inclusionAI/LLaDA2.0-Uni

LLaDA2.0‑Uni – Unified Diffusion‑LLM for Multimodal Understanding & Generation

What it is – LLaDA2.0‑Uni is a large‑scale multimodal model that combines a diffusion‑based large language model (dLLM) with a Mixture‑of‑Experts backbone. It works with text, images, and image‑editing instructions through a single mask‑token prediction interface. The model turns visual data into discrete semantic tokens (via a SigLIP‑VQ tokenizer) and then decodes those tokens with a fast diffusion decoder (8‑step distilled inference).

Key capabilities

• Text‑to‑Image generation – high‑quality 1024×1024 images, optional “thinking” mode that shows the model’s internal reasoning steps.
• Image understanding – visual question answering and document comprehension using the same token‑based interface.
• Instruction‑guided image editing – single‑ or multi‑reference edits that preserve details while applying the requested change.
• Interleaved generation & reasoning – the unified token representation lets the model switch between generating images and reasoning about them in the same prompt.
• SPRINT acceleration – a cache‑reuse and adaptive unmasking scheme that cuts inference time (up to ~10× faster decoding with the decoder‑turbo).
• Integration hooks – ready‑made nodes for ComfyUI, Diffusers, and SGLang‑Omni for high‑throughput serving.

Model architecture

• Unified dLLM‑MoE backbone built on LLaDA 2.0.
• Discrete semantic tokenizer (SigLIP‑VQ) that converts images to a sequence of tokens.
• Efficient diffusion decoder (standard 50‑step ODE or distilled 8‑step “turbo”).

How to get started

# Clone & create environment
git clone https://github.com/inclusionAI/LLaDA2-Uni && cd LLaDA2-Uni
conda create -n llada2_uni python=3.10 -y && conda activate llada2_uni

# Install PyTorch (CUDA 12.4) and FlashAttention 2
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
pip install flash-attn --no-build-isolation

# Install the rest of the dependencies
pip install -r requirements.txt

Python quick‑start – generate an image

from transformers import AutoModelForCausalLM, AutoTokenizer
from decoder import decode_vq_tokens

model_path = "inclusionAI/LLaDA2.0-Uni"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map="cuda", torch_dtype="bfloat16", trust_remote_code=True
).eval()
model.tokenizer = tokenizer

result = model.generate_image(
    "A modern Scandinavian kitchen with white cabinetry, marble countertops, and a single orchid on the island.",
    image_h=1024, image_w=1024, steps=8, cfg_scale=2.0,
)
# Decode (8‑step turbo decoder for speed)
image = decode_vq_tokens(
    result["token_ids"], result["h"], result["w"], model_path, "cuda",
    num_steps=8, decode_mode="decoder-turbo",
)
image.save("kitchen.png")

Image understanding example

from encoder.image_tokenizer import ImageTokenizer
from decoder.smart_img_process import smart_resize_images

image_tokenizer = ImageTokenizer(model_path=model_path, device="cuda")
pil = smart_resize_images(["./assets/understanding_example.png"])[0]
info = image_tokenizer.encode_with_info(pil)
image_tokens = [t + model.config.image_token_offset for t in info["token_ids"]]
_, h, w = info["grid_thw"]

response = model.understand_image(
    image_tokens, h, w,
    question="Describe this image in detail.",
    steps=32, gen_length=2048,
)
print(response)

Image editing example

# Same tokenisation as above, then:
result = model.edit_image(
    image_tokens, h, w,
    instruction="Change the background to a beach.",
    steps=8, cfg_text_scale=4.0,
)
edited = decode_vq_tokens(
    result["token_ids"], result["h"], result["w"], model_path, "cuda",
    num_steps=8, decode_mode="decoder-turbo",
)
edited.save("beach.png")

ComfyUI integration – after cloning the repo, link the apps/comfyui folder into your ComfyUI custom_nodes directory (or run the provided install.sh). The package adds nodes for loading the model, text‑to‑image, image understanding, image editing, and token decoding.

SGLang serving – see the linked SGLang‑Omni cookbook for a ready‑to‑run server that can handle many concurrent requests with the same low‑latency inference path.

Model checkpoints – available on HuggingFace and ModelScope, both full‑precision and FP8‑quantised versions.

License – Apache 2.0.

Citation

@article{LLaDA2Uni,
  title   = {LLaDA2.0‑Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model},
  author  = {Tiwei Bie and Haoxing Chen and Tieyuan Chen and Zhenglin Cheng and Long Cui and Kai Gan and Zhicheng Huang and Zhenzhong Lan and Haoquan Li and Jianguo Li and Tao Lin and Qi Qin and Hongjun Wang and Xiaomei Wang and Haoyuan Wu and Yi Xin and Junbo Zhao},
  journal = {arXiv preprint arXiv:2604.20796},
  year    = {2026}
}

---

LLaDA2.0‑Uni brings together multimodal perception and diffusion‑based generation under a single token‑prediction model, offering a versatile toolbox for developers who need high‑quality image synthesis, visual reasoning, or instruction‑driven editing—all with fast, cache‑accelerated inference.