hustvl/ControlAR

What it solves

ControlAR enables controllable image generation using autoregressive (AR) models, allowing users to guide the output based on spatial controls like Canny edges, depth maps, and segmentation masks. It addresses the limitation of standard AR models in following precise spatial constraints while supporting arbitrary-resolution generation without needing special tokens or resolution-aware prompts.

How it works

ControlAR implements a conditional decoding strategy that treats spatial control as a sequence perspective problem. It integrates with autoregressive models (such as LlamaGen) and uses control encoders (like DINOv2) to process spatial conditions. The system allows for adjustable control strength to balance the influence of the spatial guide against the text prompt.

Who it’s for

This project is designed for AI researchers and developers working on image synthesis, specifically those interested in moving beyond diffusion-based controllable generation to autoregressive architectures.

Highlights

• Spatial Control Support: Works with Canny, HED, and Lineart edges, depth maps, and segmentation masks.
• Arbitrary Resolution: Supports generating images of various sizes without hand-crafted tokens.
• High-Performance Inference: Includes a fast inference engine using torch.compile and CUDA graphs, achieving up to 11.4x faster single-image sampling.
• Flexible Control: Features a control strength factor to tune the level of spatial guidance.