nv-tlabs/PiD

PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion

What it solves

PiD (Pixel Diffusion Decoder) addresses the limitations of traditional VAE (Variational Autoencoder) or RAE (Regularized Autoencoder) decoders in latent diffusion models. Instead of relying on standard decoding, it turns latent representations directly into super-resolved pixels in a single pass, unifying the decoding and upsampling processes to produce higher-resolution images with better detail.

How it works

PiD reformulates the latent-to-pixel decoding process as a conditional pixel-space diffusion model. It directly denoises in high-resolution pixel space, allowing it to function as a plug-and-play replacement for existing decoders. It can take latents from various backbones (such as FLUX, SDXL, or SD3) and decode them into resolutions ranging from 2K to 4K.

Who it’s for

This tool is designed for researchers and developers working with latent diffusion models who want to improve the final image resolution and sharpness without needing to redesign the entire generative pipeline.

Highlights

  • Plug-and-Play: Replaces standard VAE/RAE decoders across multiple popular backbones including FLUX, SDXL, SD3, and Qwen-Image.
  • Super-Resolution: Capable of decoding latents into 2K and up to 4K resolutions.
  • Single-Pass Generation: Unifies decoding and upsampling into one generative module.
  • Flexible Integration: Supports various aspect ratios and integrates with tools like ComfyUI.

Related

  • Project
    facebookresearch/pixio

    Pixio is a vision encoder optimized for dense prediction tasks like depth estimation and semantic segmentation, achieved through enhanced pixel reconstruction pre-training.

  • Dispatch
    Hugging Face Remote VAEs for Inference Endpoints

    Hugging Face has introduced an experimental feature to delegate the VAE decoding process to remote endpoints, reducing VRAM requirements for high-resolution image and video synthesis on consumer GPUs.

  • Project
    thu-ml/DiT-Extrapolation

    A plug-and-play framework for Diffusion Transformers that enables the generation of longer videos and higher-resolution images through positional embedding extrapolation.

  • Project
    rupeshs/fastsdcpu

    A high-performance implementation of Stable Diffusion optimized for CPUs, utilizing OpenVINO and Latent Consistency Models to enable near real-time image generation.

  • Project
    jasmcaus/caer

    Caer is a Python‑only, GPU‑accelerated computer‑vision library that offers fast image/video utilities (resize, colour conversion, augmentations, etc.) with a clean, type‑checked API. It can replace OpenCV for research prototypes, is installable via `pip`, and is MIT‑licensed.