Google shipped Gemma 4 this spring — and by May 06, 2026, it’s already getting faster. Not through bigger models or more parameters, but by rethinking how tokens are generated. The company quietly rolled out experimental Multi-Token Prediction (MTP) drafters for the Gemma 4 lineup, achieving up to a 3x speed boost in generation times. That’s not a theoretical edge. It’s real-world throughput, hitting local developers who are running these models on consumer hardware and hitting bottlenecks.
Key Takeaways
- Gemma 4 now supports experimental MTP drafters that predict multiple future tokens at once, speeding up output.
- The technique uses speculative decoding — a method Google says cuts latency by reducing autoregressive dependencies.
- These models are built on the same stack as Gemini but are tuned to run on local machines, including consumer GPUs.
- Google switched Gemma’s license to Apache 2.0, dropping the restrictive custom license from prior versions.
- Hardware constraints remain a barrier — which is exactly why MTP matters now.
The Speed Problem No One Talks About
Everyone focuses on model size, parameter counts, or benchmark scores. But in practice, latency kills adoption. When you’re typing a query into a locally hosted AI and waiting 8 seconds for the first word to appear, the model doesn’t matter. The experience does.
Gemma 4 was designed to run locally, meaning you don’t have to pipe your data through Google’s cloud. That’s the appeal — privacy, control, no API fees. But local hardware is limited. Even the most powerful consumer GPUs struggle with full-precision inference at speed. You can quantize the model, sure. But then you trade accuracy for performance. Google’s answer? Don’t just optimize the model. Change how it generates.
Historical Context
AI research has been focused on improving language models over the past decade, with a significant increase in attention over the past few years. In 2023, papers on speculative decoding started circulating, showcasing the potential of this technique in reducing latency. Google, in particular, has been investing heavily in improving its AI models, with Gemini being a flagship example. The company’s focus on open-sourcing and making AI more accessible to developers and researchers has been a driving force behind the adoption of AI in various industries.
The shift to local AI, enabled by models like Gemma, has been gaining traction in recent years. With the rise of cloud AI, concerns around data leaks, opaque pricing, and vendor lock-in have pushed users toward self-hosted alternatives. Gemma’s open-licensing and real-world performance gains are a significant step toward making local AI a viable option for developers and businesses.
Multi-Token Prediction Isn’t Magic — It’s Guessing
Traditional language models are autoregressive: they generate one token at a time, each depending on the last. That’s predictable. It’s stable. It’s also slow.
MTP flips the script. Instead of waiting for each token to confirm before moving on, the system uses a smaller “drafter” model to predict several tokens ahead. The main model then verifies them in bulk. If the guesses are correct, you’ve saved multiple forward passes. If not, you roll back and continue normally. It’s speculative — hence “speculative decoding” — but when it works, throughput spikes.
Google didn’t invent this idea. Papers on speculative decoding have circulated since 2023. But pulling it off in a production-grade, open model stack? That’s new. And doing it with Gemma — a model family meant for tinkering, deployment, and local use — makes it immediately useful.
How MTP Actually Works
The drafter model is lightweight — small enough to run quickly, even on modest hardware. It ingests the current context and fires off a sequence of predicted tokens. The larger Gemma 4 model then evaluates that entire sequence in one go through a process called “draft scoring.”
If the draft aligns with what Gemma would have generated, those tokens are accepted. If not, the system reverts and tries again. There’s overhead, yes. But because the drafter is fast and the verification happens in parallel, the net effect is positive — often drastically so.
According to Google’s internal testing, MTP cuts average token generation time by 60 to 70 percent across a range of prompts, from code completion to long-form text. In best-case scenarios — short, predictable sequences — the speedup hits 3x. That’s not incremental. That’s usability transformed.
Gemma’s Quiet Licensing Revolution
While MTP grabs attention, the licensing shift is just as significant. Previous versions of Gemma ran under a custom Google license — one that restricted commercial use, required attribution, and left legal teams wary. Now? It’s Apache 2.0. Full stop.
That means developers can use Gemma 4 in commercial products, modify it, redistribute it, even sell it — no strings attached. No requirement to share changes. No fear of license audits. Apache 2.0 is the gold standard for permissive open source. By adopting it, Google isn’t just open-sourcing Gemma. It’s inviting real-world adoption.
Compare that to Meta’s Llama family, which still runs under a non-commercial-friendly license. Or to Mistral, which open-weights its models but keeps training data and fine-tuning pipelines opaque. Google’s move positions Gemma as the most genuinely open frontier-scale model you can run at home.
Why Apache 2.0 Changes the Game
- Startups can embed Gemma 4 in paid apps without legal risk.
- Enterprises can fine-tune and deploy internally without licensing overhead.
- Hardware vendors can bundle Gemma as a default local AI engine.
- Researchers can publish reproducible work without permission hurdles.
That’s not theoretical freedom. That’s operational freedom. And it arrives at a time when trust in cloud AI is fraying — with data leaks, opaque pricing, and vendor lock-in pushing users toward self-hosted alternatives.
Technical Architecture
Gemma 4’s architecture is built on top of the Gemini model stack, but with a key difference: it’s designed to run on local machines, including consumer GPUs. The drafter model, responsible for predicting multiple tokens ahead, is a lightweight component that runs quickly even on modest hardware. The main model then evaluates the entire sequence in one go using “draft scoring.” This parallel processing approach reduces latency and improves throughput.
The technical specifics of MTP are crucial to its success. By using a drafter model to predict tokens ahead, Gemma 4 can save multiple forward passes and reduce the overall generation time. The speed gains are made possible by the efficient design of the drafter model, which can run quickly even on limited hardware resources.
Competitive Landscape
The AI model landscape is increasingly crowded, with major players like Meta, Microsoft, and Alibaba competing for dominance. Google’s Gemma 4, with its MTP capabilities and permissive licensing, is a significant addition to the market. While other models, like Llama and Mistral, have their strengths, Gemma 4’s focus on local AI and open-sourcing sets it apart.
In the context of the competitive landscape, Gemma 4’s speed gains and improved usability are crucial. By reducing latency and improving throughput, Gemma 4 becomes a more attractive option for developers and businesses looking for a reliable and efficient local AI solution.
Adoption Timeline
The adoption of Gemma 4 with MTP is expected to be rapid, given the speed gains and improved usability. As developers and businesses experiment with the new model, we can expect to see a significant increase in adoption over the coming months.
The adoption timeline will likely follow a predictable pattern, with early adopters being the first to implement Gemma 4 in their products and applications. As the model’s capabilities and performance become more widely known, we can expect to see a broader adoption across various industries.
What This Means For You
If you’re building local AI tools, Gemma 4 with MTP is the most compelling option on the market right now. You get a model architecture derived from Gemini — Google’s flagship AI — without the cloud dependency. The Apache 2.0 license means you can ship it in products immediately. And the 3x speed boost? That’s the kind of jump that turns prototypes into production apps.
Start experimenting with the MTP drafters now. They’re experimental, yes, but available. Test them in your pipelines. See how much latency drops in your use cases. If you’re using Llama or Mistral locally, benchmark Gemma 4 against them — especially on consumer hardware. The performance-per-watt math might surprise you.
Google is signaling that local AI isn’t just a backup plan. It’s a priority. And by combining open licensing with real performance gains, they’re making it viable.
But here’s the real question: if speculative decoding works this well on Gemma, why isn’t Google shipping it with Gemini itself?
Key Questions Remaining
The effectiveness of MTP in real-world scenarios is still being tested and refined. As developers experiment with the new model, we can expect to see a better understanding of its strengths and limitations. Key questions remaining include the following:
- How well will MTP perform on more complex tasks, such as long-form text generation or dialog systems?
- Will the speed gains and improved usability of MTP lead to increased adoption across various industries?
- How will the competitive landscape evolve as other players, like Meta and Microsoft, respond to Google’s move?
- What are the potential implications for cloud AI, given the growing trend toward local AI and open-sourcing?
These questions highlight the need for continued research and experimentation with MTP and Gemma 4. we can expect to see significant advancements in local AI and the broader AI landscape.
Sources: Ars Technica, original report
