pengzhangzhi/Open-dLLM
Open diffusion language model for code generation — releasing pretraining, evaluation, inference, and checkpoints.
What it solves
Open-dLLM provides a fully open-source stack for creating diffusion-based large language models (dLLMs). While previous releases often only provided weights and inference scripts, this project opens the entire pipeline—including pretraining code, datasets, evaluation suites, and checkpoints—to ensure full reproducibility and accessibility for researchers.
How it works
The project uses a Masked Diffusion Model (MDM) objective, where masking ratios are sampled uniformly and reconstructed using cross-entropy loss. It supports adapting autoregressive LMs (like Qwen2.5-Coder) into diffusion LMs through continued pretraining. Additionally, it implements "representation alignment," a technique that allows for a 4x speedup in adapting these models by aligning the representations of the autoregressive and diffusion models.
Who it’s for
AI researchers and developers interested in diffusion-based language modeling, specifically those who want to complete the full cycle from raw data to trained checkpoints and evaluation.
Highlights
- Full-Stack Open Source: Includes pretraining pipelines, open datasets (FineCode), and inference scripts.
- Representation Alignment: A specialized feature to speed up the adaptation of autoregressive LMs into diffusion LMs.
- Comprehensive Evaluation: A built-in suite for standard code generation (HumanEval, MBPP) and code infilling tasks.
- Ready-to-use Weights: Provides checkpoints, such as Open-dCoder (0.5B), on Hugging Face.