ZHZisZZ/dllm

dLLM: Simple Diffusion Language Modeling

What it solves

dLLM is a library designed to unify the training, inference, and evaluation of diffusion language models (dLLMs). It provides a standardized pipeline to replace fragmented development processes, making the creation and deployment of non-autoregressive text generation models more transparent and reproducible.

How it works

The library provides a modular framework consisting of core reusable modules (samplers, schedulers, and trainers) and application-specific pipelines. It integrates with industry-standard tools like the Hugging Face transformers Trainer, lm-evaluation-harness, LoRA, DeepSpeed, and FSDP for scalable training.

It supports various diffusion algorithms, including masked diffusion (MDLM) and block diffusion (BD3LM), and provides recipes to convert existing autoregressive (AR) models or BERT encoders into diffusion models (e.g., Tiny-A2D and BERT-Chat).

Who it’s for

AI researchers and developers working with diffusion-based text generation, specifically those looking to train, fine-tune, or evaluate models like LLaDA and Dream, or those wanting to convert standard LLMs into diffusion models.

Highlights

  • Unified Pipeline: Combines training, sampling, and evaluation into a single library.
  • Scalable Training: Supports LoRA, DeepSpeed, and FSDP for efficient large-scale training.
  • Model Conversion: Includes recipes to turn any autoregressive model (e.g., LLaMA, Qwen) or BERT encoder into a diffusion model.
  • Reasoning Support: Implements diffu-GRPO training for reasoning tasks like GSM8K and MATH.
  • Accelerated Inference: Integrates with Fast-dLLM for cached and confidence-threshold decoding.

Related

  • Project
    pengzhangzhi/Open-dLLM

    A complete open-source framework for diffusion-based large language models, providing the entire pipeline from pretraining and datasets to evaluation and inference.

  • Project
    yuanchenyang/smalldiffusion

    A lightweight diffusion library for training and sampling from diffusion and flow models, featuring a minimal core for easy experimentation with U-Net and DiT architectures.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is a reinforcement learning framework that provides a unified post-training loop for multimodal models, including image and video diffusion, autoregressive VLMs, and LLMs.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    huggingface/finetrainers

    A library for the accessible training and fine-tuning of diffusion models, focusing on memory-efficient text-to-video and text-to-image generation.