JetAstra/SDAR

🚀 What is SDAR?

SDAR (Synergy of Diffusion and AutoRegression) is a family of large‑scale language models that combine two different generation paradigms:

• Autoregressive (AR) – the classic left‑to‑right token‑by‑token decoding that is cheap to train.
• Discrete diffusion – a parallel decoding technique that can generate several tokens at once.

By marrying the training efficiency of AR models with the high‑throughput decoding of diffusion models, SDAR achieves 2‑4× faster inference while keeping accuracy on par with state‑of‑the‑art open‑source AR models. The project is positioned as a generalist LLM that also shows strong specialist abilities on scientific reasoning tasks (e.g., GPQA, ChemBench).

---

📦 Available Models (as of the README)

Model	Size	Type	Hugging Face link
SDAR‑1.7B‑Chat	1.7 B	Chat	https://huggingface.co/JetLM/SDAR-1.7B-Chat
SDAR‑4B‑Chat	4 B	Chat	https://huggingface.co/JetLM/SDAR-4B-Chat
SDAR‑8B‑Chat	8 B	Chat	https://huggingface.co/JetLM/SDAR-8B-Chat
SDAR‑30B‑A3B‑Chat	30 B (MoE)	Chat	https://huggingface.co/JetLM/SDAR-30B-A3B-Chat
SDAR‑30B‑A3B‑Sci	30 B (MoE)	Science‑focused reasoning	https://huggingface.co/JetLM/SDAR-30B-A3B-Sci

All models support block sizes of 4, 8, 16, 32, or 64 (the block size controls how many tokens are generated per diffusion step).

---

⚙️ How to Use SDAR

1. Quick‑start inference (built‑in script)

python generate.py \
  --model_dir=JetLM/SDAR-1.7B-Chat \
  --trust_remote_code

The script pulls the model from Hugging Face, loads it with 🤗 Transformers (>= 4.52.4), and runs greedy generation.

2. High‑performance batch inference with JetEngine

JetEngine is a lightweight engine built on nano‑vllm that supports both dense and MoE SDAR models, tensor‑parallelism, and FlashAttention‑2. Example (Python):

from jetengine import LLM, SamplingParams
from transformers import AutoTokenizer

model_path = "/path/to/model"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

llm = LLM(
    model_path,
    enforce_eager=True,
    tensor_parallel_size=1,
    mask_token_id=151669,
    block_length=4,
)

params = SamplingParams(
    temperature=1.0,
    topk=0,
    topp=1.0,
    max_tokens=256,
    remasking_strategy="low_confidence_dynamic",
    block_length=4,
    denoising_steps=4,
    dynamic_threshold=0.9,
)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain reinforcement learning in simple terms."}],
    tokenize=False,
    add_generation_prompt=True,
)

for out in llm.generate_streaming([prompt], params):
    print(out)

Benchmark (SDAR‑4B, block‑size 4, batch 128):

• NVIDIA A800 – > 1800 tokens / s
• NVIDIA H200 – > 3700 tokens / s (FlashAttention‑2 + Triton)

3. Production‑grade serving with LMDeploy

LMDeploy (InternLM) offers a full pipeline for Tensor‑Parallel inference. The README includes a minimal example that sets dllm_block_length, dllm_denoising_steps, and a custom unmasking strategy.

---

📊 Reported Performance

Benchmark	Model (size)	Decoding	Score (higher = better)
GPQA (science QA)	SDAR‑30B‑A3B‑Sci	Greedy (block 4, 4 steps)	outperforms AR‑30B‑A3B‑Sci
ChemBench	SDAR‑30B‑A3B‑Sci	Greedy	outperforms AR counterpart
General language tasks (e.g., MMLU, HELM)	SDAR‑1.7B‑Chat	Greedy	On‑par with Qwen3‑1.7B‑AR‑SFT
Same for 4B / 8B / 30B	SDAR‑4B‑Chat, SDAR‑8B‑Chat, SDAR‑30B‑A3B‑Chat	Greedy	Comparable to Qwen3‑AR baselines

Speed – Using dynamic decoding (stop generating a block early when confidence exceeds a threshold) yields > 2× speedup over static block decoding, with negligible loss in accuracy. Speed advantage grows with model size.

---

🛠️ Development & Ecosystem

• Training framework – fine‑tuning scripts are provided under training/ and rely on the LLaMA‑Factory codebase.
• Inference engines – two first‑party options:
  • JetEngine (nano‑vllm based, open‑source on GitHub)
  • LMDeploy (InternLM’s production server)
• Roadmap – technical report already released; next steps include adding more features (unspecified) and expanding the model zoo.

---

📜 License & Contact

• License: MIT (see LICENSE).
• Contact: Shuang Cheng (Shanghai AI Lab) – chengshuang@pjlab.org.cn; Biqing Qi (corresponding author) – qibiqing@pjlab.org.cn.
• Community: A WeChat group is advertised for informal discussion.

---

TL;DR

SDAR is an open‑source family of diffusion‑augmented language models that keep the cheap training of classic autoregressive LLMs while delivering 2‑4× faster generation through parallel diffusion decoding. The repo ships pretrained weights (1.7 B – 30 B), a simple generate.py script, and two high‑performance inference back‑ends (JetEngine and LMDeploy). It is actively maintained and positioned especially for scientific reasoning tasks.