inclusionAI/LLaDA2.X

LLaDA2.0 is the diffusion language model series developed by InclusionAI team, Ant Group.

What it solves

LLaDA2.X is a series of discrete diffusion Large Language Models (dLLMs) designed to scale diffusion-based text generation to the 100-billion parameter level, overcoming the traditional limitations of autoregressive models in terms of inference speed and architecture.

How it works

The project implements a discrete diffusion process for language modeling. It utilizes a Mixture-of-Experts (MoE) architecture to scale the model size (up to 100B parameters in the LLaDA2.0-flash variant). To improve efficiency, it employs a parallel decoding mechanism and a custom inference engine based on dInfer and SGLang that supports KV-Cache reuse and block-level parallel decoding.

Who it’s for

Researchers and developers who need high-performance text generation models that can handle complex instruction-following and code generation tasks at scale, particularly those looking for alternatives to standard autoregressive LLMs.

Highlights

  • 100B Parameter Scale: The first diffusion language model to reach the 100-billion parameter mark.
  • Inference Acceleration: Achieves up to 535 tokens/s using Confidence-Aware Parallel (CAP) decoding.
  • MoE Architecture: Uses Mixture-of-Experts to maintain performance while scaling.
  • Open Source: Provides fully open-sourced model weights and training code on Hugging Face.

Related

  • Project
    ML-GSAI/LLaDA

    LLaDA is a large-scale masked diffusion language model that challenges the dominance of autoregressive architectures, achieving performance comparable to LLaMA3 8B.

  • Project
    z-lab/dflash

    A lightweight block diffusion model for speculative decoding that enables efficient, high-quality parallel drafting to speed up LLM generation.

  • Project
    pengzhangzhi/Open-dLLM

    A complete open-source framework for diffusion-based large language models, providing the entire pipeline from pretraining and datasets to evaluation and inference.

  • Dispatch
    DiffusionGemma Technical Report

    DiffusionGemma is an experimental open-weight model that uses discrete diffusion to generate text in parallel blocks of 256 tokens, achieving up to 1,500 tokens per second on a single H100 GPU.

  • Project
    ZHZisZZ/dllm

    A library that unifies the training, inference, and evaluation of diffusion language models, providing scalable pipelines and recipes for converting AR models into diffusion models.