NVlabs/PixelDiT

PixelDiT & PixelDiT2 – Diffusion Transformers that work directly in pixel space

What it is – A research‑grade codebase from NVIDIA and the University of Rochester that implements pixel‑space diffusion transformers for image generation. The models skip the usual latent‑VAE bottleneck and generate images directly at the pixel level. Two families are provided:

Family	Main paper	Highlights
PixelDiT (CVPR 2026 Best‑Paper Finalist)	Pixel Diffusion Transformers for Image Generation	1.3 B parameter text‑to‑image model that produces 1024×1024 pictures. 2. Class‑to‑image (C2I) models for ImageNet at 256 px and 512 px.
PixelDiT2 (NeurIPS 2026)	Representation‑Grounded Pixel Diffusion Transformers	Adds a frozen representation encoder (DINO‑v3) that grounds generation, yielding faster convergence and better ImageNet FID/IS scores.

Key capabilities

Pixel‑level diffusion – No VAE encoder/decoder; the transformer directly predicts pixel values.
Two generation modes
- Class‑to‑image (C2I): generate images conditioned on ImageNet class labels.
- Text‑to‑image (T2I): a 1.3 B model that takes a prompt and outputs up to 1024 px images.
Fast sampling – PixelDiT2 uses Heun’s method with 50 steps; PixelDiT uses FlowDPMSolver with 100 steps.
Pre‑trained checkpoints – ImageNet‑trained models and the large T2I model are hosted on Hugging Face.
ComfyUI integration – Weights are repackaged for the popular ComfyUI workflow, with ready‑to‑run JSON templates.
Modular code – Core model definitions live in pixdit_core; they can be installed via pip and imported in other projects.

Typical use cases

Research on diffusion models that operate in pixel space rather than latent space.
High‑resolution text‑to‑image generation without the overhead of a VAE.
Benchmarking image synthesis (FID, IS, GenEval, DPG‑Bench) on ImageNet.
Plug‑and‑play integration into UI tools like ComfyUI for creative artists.

Quick start (Python)

# Choose the family you need and install its pinned dependencies
# PixelDiT2 (Python 3.10, specific PyTorch/CUDA)
python -m pip install -r requirements-pixeldit2.txt
# PixelDiT (C2I/T2I) – generic deps
python -m pip install -r requirements-pixeldit1.txt

Run a class‑to‑image demo

# pick a checkpoint and matching yaml config
python c2i/main.py \
    --config c2i/configs/pixeldit2_h16_in256.yaml \
    --checkpoint path/to/checkpoint.ckpt \
    --class "golden retriever" \
    --output out.png

Run a text‑to‑image demo (PixelDiT‑T2I)

python t2i/main.py \
    --config t2i/configs/PixelDiT_1024px_pixel_diffusion_stage3.yaml \
    --prompt "a futuristic city at sunset" \
    --output city.png

Install the library for reuse

pip install "git+https://github.com/NVlabs/PixelDiT"
# for PixelDiT2 grounding encoder also install timm extras
pip install "pixeldit[pixeldit2] @ git+https://github.com/NVlabs/PixelDiT"

Then import:

from pixdit_core.pixeldit_t2i import PixDiT_T2I
from pixdit_core.pixeldit_c2i import PixDiT
from pixdit_core.pixeldit2_c2i import PixelDiT2

Repository layout

├─ pixdit_core/          # Model definitions (PixelDiT, PixelDiT2, grounding encoder)
├─ c2i/                  # Class‑to‑image scripts, configs, evaluation
├─ t2i/                  # Text‑to‑image scripts, configs
├─ tools/                # Checkpoint downloader, GFLOPs calculator, evaluation helpers
├─ requirements‑*.txt   # Dependency lists for each family
└─ README.md (this file)

Further resources

Project pages – https://pixeldit.github.io/ (PixelDiT) & https://pixeldit.github.io/pixeldit2/ (PixelDiT2)
ArXiv papers – arxiv.org/abs/2511.20645 (PixelDiT) and arxiv.org/abs/2609.24919 (PixelDiT2)
Pre‑trained models – Hugging Face repos nvidia/PixelDiT-ImageNet, nvidia/PixelDiT2-ImageNet, nvidia/PixelDiT-1300M-1024px
ComfyUI integration – https://github.com/Comfy-Org/PixelDiT (weights & workflow template)
Citation – Include the two BibTeX entries from the README when using the code or models.

Who might want this?

ML researchers exploring diffusion in the raw pixel domain or representation‑grounded generation.
Engineers building high‑quality image synthesis pipelines that need to avoid latent‑space artifacts.
Artists & hobbyists who want a state‑of‑the‑art text‑to‑image model that can be run locally or via ComfyUI.

All information above is taken directly from the repository’s README; no additional features are inferred.