Google released DiffusionGemma on June 8, 2026. It is a 26B-parameter Mixture-of-Experts model built on a diffusion architecture rather than standard autoregressive generation. The weights are open (Apache 2.0), and the numbers on consumer hardware are immediately actionable.
If you are running local inference for interactive coding, inline editing, or real-time chat, this changes the latency profile you have been stuck with.
The Headline Numbers
Google's published benchmarks place DiffusionGemma at over 1,000 tokens/sec on an H100 and 700+ tokens/sec on an RTX 5090. For a 26B model, that is an order of magnitude ahead of where dense autoregressive models sit on the same silicon.
Community tooling has already caught up. Unsloth released day-zero MTP GGUFs for the 12B and 26B variants. On a single high-end consumer GPU, the 12B model hits 162 tok/s with the MTP heads active, compared to a 52 tok/s baseline — a 3.1× speedup. The 26B MoE variant fits comfortably under 18 GB of VRAM when quantized, which means it runs on hardware that used to choke on 7B models.
Why Diffusion Beats Autoregressive for Chat
Standard LLMs generate text one token at a time. The GPU reads all the weights, computes a probability distribution, samples a token, and repeats. Your throughput is capped by memory bandwidth, not compute.
DiffusionGemma flips this. It borrows from image diffusion models and treats text generation as a block refinement process. The model generates entire chunks of text in parallel, then iteratively corrects them. Bidirectional attention means it can look ahead and backward simultaneously. There is no sequential decode bottleneck.
The consequence is immediate latency reduction. At batch sizes of 1 to 8 — which covers almost every interactive chat and coding scenario — the model produces output in a fraction of the wall-clock time. You stop waiting for the token stream to trickle out.
Runtime Support Is Actually Day-Zero
Usually, a new architecture requires weeks of runner adaptation. DiffusionGemma shipped with native vLLM support, which is rare for diffusion LLMs. vLLM has added continuous batching specifically for this model, allowing it to reuse KV cache concepts across parallel generation passes.
Unsloth handles the GGUF conversion, and MLX has a working loader. You can pull the weights from HuggingFace and run them through standard OpenAI-compatible endpoints. No custom inference scripts required.
Where It Breaks
Bidirectional attention is not free. The architecture cannot do prefix caching or incremental decoding the way autoregressive models do. When you chain long conversations or run high-throughput server workloads, the throughput gap narrows. At batch sizes above 32, standard autoregressive models pull ahead through KV cache reuse across concurrent requests — an advantage DiffusionGemma's architecture cannot replicate.
If your use case is batched document processing, log parsing, or high-concurrency API serving, stick to Qwen3.6 or Llama. DiffusionGemma is optimized for interactivity, not throughput density.
The Tradeoff
You are buying a different kind of speed. Autoregressive models are bandwidth-bound and linear. DiffusionGemma is parallel and iterative. The 700+ tok/s on an RTX 5090 translates to interactive coding and inline editing that feels instant, not snappy. It also means you can run a 26B model on an 18 GB VRAM card without swapping to system RAM.
The catch is that it is experimental. Google labels it as such. The tooling is mature, but the model's behavior on highly constrained or deterministic tasks still needs validation. If your agentic loop relies on strict JSON schema enforcement, test the acceptance rates explicitly.
DiffusionGemma does not replace your autoregressive stack. It sits next to it. Use the diffusion model for the interactive, low-latency tasks that dominate your daily workflow. Use the dense models for batch processing and high-concurrency serving. The gap between them was always about workload shape, not just parameter count.
Sources: