Building Diffusion Language Models: Architecture, Sampling, and Scaling

Executive Summary

Diffusion language models (dLLMs) represent a paradigm shift from the standard autoregressive (AR) approach to text generation. While AR models generate tokens sequentially from left to right, diffusion models generate entire sequences in parallel, iteratively refining a rough draft into a polished output. This approach addresses three core limitations of AR models: the lack of error correction, slow sequential generation, and the restriction to causal (backward-looking) attention.

Core Mechanisms of Discrete Diffusion

Applying diffusion to discrete data like text requires redefining "noise." Unlike image diffusion, which uses Gaussian noise, language diffusion typically employs masking or state-replacement strategies.

Masked Diffusion Models (MDLM)

Masked diffusion operates as a "generative BERT." The process consists of two phases:

Forward Process: A clean sequence is gradually corrupted by replacing tokens with a [MASK] token according to a specific schedule until the sequence is fully masked.
Reverse Process: A bidirectional transformer is trained to predict the original tokens from the masked sequence. During generation, the model starts with a fully masked sequence and iteratively fills in blanks, occasionally re-masking tokens to allow for refinement.

Uniform State Diffusion (UDLM)

Instead of using a mask token, UDLMs replace tokens with random selections from the entire vocabulary. This ensures that intermediate sequences remain mask-free and allows the model to revise any token at any step, facilitating better error correction and controllability.

Advanced Architectures and Sampling

To move from theoretical models to production-ready LLMs, several architectural and algorithmic extensions are required.

Block Diffusion and Variable Length

Standard MDLMs are limited to fixed-length sequences. Block Diffusion solves this by generating text in blocks (e.g., 256 tokens) conditioned on previously generated blocks. This method supports KV caching, similar to autoregressive models, allowing for efficient variable-length generation.

Encoder-Decoder Frameworks

Modern dLLMs, such as Gemma Diffusion and Nemotron Diffusion, utilize an encoder-decoder architecture. A heavy encoder processes the clean context once, while a lightweight decoder iteratively denoises the "canvas" of tokens. This separation reduces the computational cost of the denoising steps and accelerates training.

Iterative Refinement and Error Correction

Because standard masking never re-masks a token once it is unmasked, early mistakes are permanent. Two primary solutions exist:

Remasking: A plug-in sampler that randomly re-masks a subset of predicted tokens, allowing the model to correct grammatical or logical errors as more context is filled in.
Uniform State Diffusion: By treating all tokens as potentially noisy, the model can naturally revise any position in the sequence.

Sampling Acceleration via Distillation

Diffusion models can be 5–10× faster than AR models due to parallel generation. To further increase speed, progressive distillation is used, where a student model is trained to mimic two denoising steps of a teacher model in a single step, recursively halving the required sampling steps.

Controllable Generation and Guidance

Diffusion models are inherently better at controllable generation because they refine the global sequence rather than committing to local, irreversible edits. This is managed via a Pareto trade-off between sample naturalness and property satisfaction.

Classifier-Based Guidance (CBG): Uses a separate predictor model to steer the denoising process toward a target property (e.g., binding affinity in molecules).
Classifier-Free Guidance (CFG): Trains a single model to handle both conditional and unconditional generation by randomly dropping the conditioning signal, combining the two outputs to steer the result.

Real-World Applications and Scaling

Biological and Scientific Domains

Diffusion has seen early success in biology because biological sequences lack the left-to-right bias of human language and require high controllability.

ESM3: A 100B-parameter MDLM used for protein sequence, structure, and function modeling.
Nucleotide Transformer v3 (NT-v3): A multi-billion parameter model for DNA sequence generation, utilizing CFG and remasking to design regulatory DNA sequences with specific gene-expression levels.

General Purpose Diffusion LLMs

Several frontier labs have released scaled diffusion models:

LLaDA: An 8B-parameter open-weights model that demonstrates AR-like scaling on benchmarks like GSM8K and MMLU.
Mercury 2: A commercial model achieving speeds of ~1,200 tokens/second on standard GPUs, significantly outperforming speed-optimized AR models like Claude Haiku.
Gemma Diffusion: A Google open-weights model utilizing a UDLM backbone and encoder-decoder architecture.
Nemotron Diffusion: An NVIDIA family of models (up to 35B parameters) using a joint AR-diffusion objective, offering 2–8× the throughput of comparable AR models.

Critical Analysis and Future Outlook

While diffusion offers massive gains in inference speed and hardware utilization, some technical challenges remain. Community feedback highlights a specific weakness in token coordination: when two positions both teeter between two valid states, diffusion models may fail to collapse them into a matching pair, leading to inconsistencies.

From a scaling perspective, diffusion may be to inference-time compute what the transformer was to pre-training compute. By transforming the sequential bottleneck of AR inference into a parallel process, diffusion allows for higher FLOPs per second, potentially unlocking new levels of intelligence through scaled post-training and inference-time search.