DiffusionGemma Technical Report

DiffusionGemma is an experimental open-weight language model designed to eliminate the sequential decoding bottleneck of traditional autoregressive (AR) models. By using discrete diffusion to iteratively refine blocks of 256 tokens in parallel, DiffusionGemma achieves generation speeds of approximately 1,500 output tokens per second on a single NVIDIA H100 GPU, significantly outperforming even state-of-the-art speculative decoding.

Architecture and Training Pipeline

DiffusionGemma is not trained from scratch. Instead, it is derived from the mixture-of-experts (MoE) Gemma 4 model, which features 3.8B activated parameters and 25.2B total parameters. The conversion from a decoder-only AR model to a denoiser is achieved by leveraging the logits of all tokens, a capability existing AR models possess but do not typically use during standard generation.

The training process uses a compute-efficient two-stage pipeline that requires less than 10% of the original AR model's total training token budget:

  1. Supervised Fine-Tuning (SFT): The first stage teaches the model bidirectional denoising.

  2. Reinforcement Learning (RL) and Sampler Distillation: The second stage combines RL with sampler distillation to simultaneously improve the generation quality and the overall inference efficiency.

Performance and Capabilities

DiffusionGemma achieves a new Pareto frontier for the trade-off between generation speed and model capability. On average, it generates approximately 20 tokens per forward pass, resulting in the same 1,500 tokens per second throughput on an H100 GPU.

Despite the transition to diffusion-based decoding, the model retains critical capabilities of the base Gemma 4 model, including:

  • Thinking mode: Support for internal reasoning processes.
  • Multimodal inputs: Ability to process diverse input types.
  • Longe contexts: Support for extended context windows.

Furthermore, the model remains capable of autoregressive generation with only minor performance degradation, suggesting a potential path toward hybrid diffusion-AR decoding systems.

Community Insights and Implementation

Community discussion highlights the potential for this approach to be applied to other open-weight models to create high-speed diffusion versions of existing local LLMs.

Hardware Optimization

Users implementing the model on local hardware have reported success on Apple Silicon. One developer noted that because diffusion decoding is computation-bound rather than memory-bandwidth bound, it is particularly well-suited for devices with high compute power but limited memory bandwidth.

"I've got it up to ~15tok/s on M3-class machines... DiffusionGemma with the right drafter can hit 20-30 tok/s on my machine."

Impact on Software Development

Some analysts suggest that if diffusion models reach high proficiency in coding, the extreme generation speed (1,500 tokens/sec) will shift the bottleneck of software development from model generation to CPU execution time. This could lead to a a rethink of how compilers, test suite runners, and development stacks are designed to overlap compilation and unit testing in parallel with LLM proposals.

Potential for Generalization

There is significant interest in the community regarding whether this approach can be applied to other models, such as Qwen, to dramatically increase local inference speeds for users with consumer-grade GPUs.

Sources

Related

  • Dispatch
    DiffusionGemma: 4x Faster Text Generation via Parallel Diffusion

    Google has released DiffusionGemma, an experimental 26B MoE model that uses text diffusion to generate text blocks in parallel, achieving up to 4x faster inference on dedicated GPUs compared to autoregressive models.

  • Dispatch
    DiffusionGemma: 4x Faster Text Generation via Text Diffusion

    Google DeepMind has released DiffusionGemma, an experimental 26B MoE model that uses text diffusion to generate blocks of text simultaneously, achieving up to 4x faster inference on dedicated GPUs compared to autoregressive models.

  • Dispatch
    DiffusionGemma: The First Diffusion LLM Natively Supported in vLLM

    vLLM has integrated DiffusionGemma, a 26B-parameter discrete diffusion language model, enabling high-throughput, low-latency generation by iteratively denoising token blocks instead of sequential autoregressive decoding.

  • Project
    inclusionAI/LLaDA2.X

    A series of discrete diffusion Large Language Models scaled up to 100B parameters, offering high-speed parallel decoding for text generation.

  • Dispatch
    Dreambooth Stable Diffusion fine‑tuning guide with Diffusers

    Hugging Face released detailed recommendations for training Stable Diffusion with Dreambooth using the Diffusers library, showing that low learning rates, enough steps, prior preservation for faces, and text‑encoder fine‑tuning yield the highest quality results.