ML-GSAI/LLaDA

Official PyTorch implementation for "Large Language Diffusion Models"

What it solves

LLaDA addresses the assumption that high-level Large Language Model (LLM) capabilities—such as in-context learning and instruction-following—are exclusively tied to autoregressive (next-token prediction) architectures. It explores the upper limits of masked diffusion models for language generation at scale.

How it works

Unlike standard LLMs that predict the next token in a sequence, LLaDA is a masked diffusion model. It uses a Transformer architecture but employs a probabilistic modeling approach where it learns to recover original tokens from a masked sequence. During training, it uses a masking ratio that varies randomly between 0 and 1, which makes its training objective an upper bound on the negative log-likelihood of the model distribution, turning it into a generative model. For inference, it iteratively refines the sequence through a remasking process.

Who it’s for

Researchers and developers interested in alternative LLM architectures, specifically those exploring diffusion-based language modeling as an alternative to autoregressive models.

Highlights

  • Scale: An 8B parameter model trained from scratch that rivals LLaMA3 8B in performance.
  • Architectural Flexibility: Uses a standard Transformer architecture without needing time $t$ as an input.
  • MoE Variant: Includes LLaDA-MoE, the first diffusion language model pretrained from scratch with a Mixture-of-Experts architecture.
  • Multimodal Extension: LLaDA-V extends the diffusion-based approach to vision-language modeling.
  • Efficiency Improvements: The iLLaDA version improves both benchmark performance and generation efficiency.

Related

  • Project
    inclusionAI/LLaDA2.X

    A series of discrete diffusion Large Language Models scaled up to 100B parameters, offering high-speed parallel decoding for text generation.

  • Project
    pengzhangzhi/Open-dLLM

    A complete open-source framework for diffusion-based large language models, providing the entire pipeline from pretraining and datasets to evaluation and inference.

  • Project
    NX-AI/xlstm

    xLSTM is a novel recurrent neural‑network architecture that extends LSTM with exponential gating and a matrix memory, enabling large‑scale language models (e.g., a 7 B‑parameter model). The repo provides PyTorch code, custom CUDA/Triton kernels (via the `mlstm_kernels` package), a single‑file 7B implementation, configuration utilities, and pretrained weights on Hugging Face. Installation is via conda + pip, and the library works on NVIDIA GPUs (fast Triton kernels) with fallbacks for AMD or Apple Silicon.

  • Project
    JetAstra/SDAR

    SDAR (Synergy of Diffusion and AutoRegression) is an open‑source LLM family that blends autoregressive training with discrete diffusion decoding. The approach yields 2‑4× faster inference while matching AR model quality, especially on science‑reasoning benchmarks. The repository provides pretrained models (1.7 B‑30 B), a simple generate script, and two high‑performance inference engines (JetEngine and LMDeploy). Training/fine‑tuning scripts use LLaMA‑Factory, and the project is MIT‑licensed and actively maintained.

  • Dispatch
    Meta Llama 3 Release Notes

    Meta has released Llama 3, an open-access LLM family featuring 8B and 70B parameter models with improved tokenization and training on 15 trillion tokens.