vLLM-Omni Accelerates Inference with AutoRound Quantization

TL;DR

vLLM-Omni now fully integrates Intel's AutoRound PTQ algorithm, providing a quantize-once, serve-directly workflow for multimodal Omni, diffusion video, and multi-stage image generation pipelines. This integration delivers up to 62% checkpoint size reduction, preserves or slightly improves accuracy, and enables 1.55–1.67x faster guided generation on Intel XPU B60 via CFG Parallel execution.

Introduction

vLLM-Omni now supports AutoRound quantization, offering a simple offline quantization step followed by automatic runtime detection and serving without extra flags. AutoRound is a tuning-based post-training quantization method that jointly optimizes rounding and clipping with three learnable parameters per tensor, producing static checkpoints that incur zero inference-time quantization overhead. The integration keeps the serving API identical to loading a normal model, as vLLM-Omni reads the checkpoint metadata, detects quantization_config.quant_method = "auto-round", remaps blocks to runtime modules, and selects the matching compute backend.

Model Coverage

The validated AutoRound + vLLM-Omni ecosystem covers three primary multimodal paradigms.

Omni Multimodal Models

These models handle unified text, vision, and audio processing loops, presenting cross-modal embedding alignment challenges.

• Qwen3-Omni-30B-A3B-Instruct (Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound) – large-scale flagship multimodal model, integrated and validated in vLLM-Omni.
• Qwen2.5-Omni-7B (Intel/Qwen2.5-Omni-7B-int4-AutoRound) – lightweight, low-latency cross-modal engine, integrated and validated in vLLM-Omni.

Diffusion and Multi-Stage Image Generation

• GLM-Image (Intel/GLM-Image-int4-AutoRound) – multi-stage text-to-image pipeline, integrated and validated in vLLM-Omni.
• FLUX.1-dev (vllm-project-org/FLUX.1-dev-AutoRound-w4a16) – high-fidelity diffusion transformer (DiT), integrated and validated in vLLM-Omni.
• BAGEL-7B-MoT (Intel/BAGEL-7B-MoT-int4-AutoRound) – checkpoint available; runtime integration in progress.
• Ovis-Image-7B (Intel/Ovis-Image-7B-int4-AutoRound) – checkpoint available; runtime integration in progress.

Video Diffusion

The Wan2.2 family represents state-of-the-art spatio-temporal video generation models with AutoRound INT4 checkpoints validated in vLLM-Omni:

• I2V-A14B (Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound)
• T2V-A14B (Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound)
• TI2V-5B (Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound)

Usage

By keeping all quantization and tuning operations offline, production code remains focused on high-performance inference.

Inference with a Quantized Model

For FLUX.1-dev, the Python API mirrors a normal vLLM-Omni load; only the checkpoint path differs.

from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams

if __name__ == '__main__':
    omni = Omni(model="vllm-project-org/FLUX.1-dev-AutoRound-w4a16
    outputs = omni.generate(
        "A cat sitting on a windowsill",
        OmniDiffusionSamplingParams(num_inference_steps=28, guidance_scale=3.5),
    )
    outputs[0].images[0].save("output.png

For Wan2.2 video models, serving is a standard vLLM-Omni command; requests use the same video endpoint as the BF16 variant.

vllm serve Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound --omni --port 8091
curl -X POST "http://127.0.0.1:8091/v1/videos/sync" \
  -F 'prompt=Cherry blossoms swaying gently in the breeze, cinematic motion' \
  -F 'width=832' -F 'height=480' -F 'num_frames=48' \
  -F 'num_inference_steps=40' -F 'guidance_scale=5.0' \
  --output t2v_output.mp4

For Omni models such as Qwen2.5-Omni, the OpenAI-compatible chat interface remains unchanged.

vllm serve Intel/Qwen2.5-Omni-7B-int4-AutoRound --omni --port 8091
curl -s http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Intel/Qwen2.5-Omni-7B-int4-AutoRound",
    "messages": [{"role": "user", "content": "What is 2 + 3?"}],
    "max_tokens": 128
  }'

vLLM-Omni auto-detects quantization metadata from the checkpoint; no separate --quantization flag is needed for pre-quantized AutoRound models.

Quantizing a New Model

New checkpoints are generated offline with the AutoRound tool and then served directly by vLLM-Omni; no calibration or quantization occurs during serving.

# FLUX.1-dev
auto-round \
  --model black-forest-labs/FLUX.1-dev \
  --scheme W4A16 \
  --batch_size 1 \
  --disable_opt_rtn \
  --dataset coco2014 \
  --iters 0

# Wan2.2-T2V-A14B
auto-round \
  --model_name Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --format auto_round \
  --scheme W4A16 \
  --iters 100 \
  --nsamples 32 \
  --batch_size 1 \
  --num-inference-steps 3 \
  --guidance-scale 5.0 \
  --dataset coco2014 \
  --output_dir Wan2.2-T2V-A14B-Diffusers-int4-AutoRound

# Qwen3-Omni-30B-A3B-Instruct
auto-round \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --bits 4 \
  --group_size 128 \
  --format auto_round \
  --iters 200 \
  --lr 5e-3 \
  --output_dir tmp_qwen3_omni_w4a16 \
  --trust_remote_code

The resulting checkpoint includes quantization metadata in config.json:

{
  "quantization_config": {
    "quant_method": "auto-round",
    "bits": 4,
    "group_size": 128,
    "sym": true,
    "packing_format": "auto_round:auto_gptq"
  }
}

AutoRound and vLLM guidance suggest that 128 calibration samples and roughly 200 optimization iterations often suffice for stable convergence, though larger or more sensitive models may benefit from more tuning.

Quality Validation

For diffusion models, vLLM-Omni provides a comparison tool for same-seed regression testing between a BF16 reference and a quantized candidate.

python -m vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity \
  --task t2i \
  --reference-model black-forest-labs/FLUX.1-dev \
  --candidate-model vllm-project-org/FLUX.1-dev-AutoRound-w4a16 \
  --prompt "a cup of coffee on the table" \
  --height 512 --width 512 \
  --num-inference-steps 20 \
  --seed 142 \
  --output-json /tmp/flux_similarity/result.json

Quantitative Evaluation: Accuracy & Quality

Quantization is only useful if it preserves the model's core intelligence; extensive multi-modality regression testing was performed.

Omni Multimodal Evaluation (OmniBench)

Using evalscope across 100 highly complex multimodal tasks (incorporating image and audio modalities), the W4A16 model achieved a slightly higher aggregate OmniBench score than the BF16 baseline.

Multi-Stage Diffusion Evaluation (TIIF-Bench)

For multi-stage text-to-image systems, performance was quantified across 9 structural sub-attributes evaluating alignment, composition, and fidelity. The average accuracy degradation across all axes is ~1.3%, safely within acceptable tolerances for production deployments.

Video Generation Evaluation (Wan2.2)

Video pipelines are fragile under naive scalar quantization due to temporal consistency drift; AutoRound was evaluated using objective metrics across multiple dimensions. Under W4A16 AutoRound, the Text-to-Video variant (T2V-A14B) showed marginal improvements in structural consistency metrics, consistent with the hypothesis that clipping optimization may provide a regularization effect.

Performance, Footprint, and Serving Benchmarks

VRAM Footprint Optimization

The first-order benefit of W4A16 AutoRound is a dramatic reduction in checkpoint size and execution memory footprint. W4A16 shrinks quantized weight storage from a BF16 baseline to roughly one quarter of the original weight footprint, providing memory headroom. End-to-end speedups depend on how much of the workload was previously bottlenecked by memory capacity or bandwidth.

Trading Memory Headroom for Latency Reduction

While Section 5.1 established the first-order memory benefit of W4A16, this case study demonstrates how that memory headroom translates into architectural advantages — enabling GPU allocation strategies that deliver real throughput gains beyond what raw compute savings alone would predict. All benchmarks were conducted on Intel XPU B60.

W4A16 Reduces Minimum Hardware from 4 GPUs to only 1 GPU

The BF16 FLUX.1-dev transformer (23 GB) exceeds a single B60's 24.4 GB capacity once runtime activations are included — it requires TP=4 (all four GPUs) to serve. W4A16's 7 GB transformer fits comfortably on a single GPU with 19% headroom to spare.

W4A16 + CFG Parallel = 1.55x - 1.67x Faster Guided Generation

Classifier-Free Guidance (CFG) requires running two denoising passes per step — one with the prompt, one with a negative prompt. With BF16 occupying all 4 GPUs for tensor parallelism, these passes must run sequentially (2X latency). W4A16 fits in TP=2, freeing 2 GPUs. This enables CFG Parallel — running both guidance branches simultaneously across two GPU groups. The key insight: W4A16's value in diffusion workloads extends beyond the memory narrative; the memory headroom enables parallelism strategies that produce end-to-end speedups larger than raw dequantization overhead would predict.

Conclusion

AutoRound fits vLLM-Omni well because the integration respects operator needs: offline checkpoint generation, automatic runtime detection, predictable memory savings, and a path to verify quality before rollout. The result is a practical low-bit serving workflow spanning a meaningful slice of the vLLm-Omni ecosystem, from FLUX and Wan to GLM, BAGEL, Ovis, and Qwen Omni. For the broader community, quantization has matured into foundational infrastructure; as AutoRound broadens compatibility across model families and hardware architectures, it provides an effective path toward balancing multimodal performance, deployment cost, and output quality. Ongoing work includes broader format support and continued expansion across model families and hardware targets. Support for additional quantization formats such as MXFP4 and MXFP8 for Linear and MoE modules is being explored, along with low-bit techniques for attention layers (e.g., SageAttention). These improvements will further extend the efficiency and flexibility of multimodal serving in the near future.

Acknowledgements

Special thanks to Hongsheng Liu, Shunyang Li, and WeiQing Chen from the vLLM-Omni team, as well as Chendi Xue from Intel, for their incredible support in integrating AutoRound into vLLM-Omni. We are also deeply grateful to the vLLM-Omni community for their rapid adoption of AutoRound!