vLLM Scaling Multi-GPU Video Captioning with PyNvVideoCodec

vLLM has integrated support for NVIDIA hardware video decoding (NVDEC) via the PyNvVideoCodec library, enabling video captioning and labeling tasks to scale efficiently across multi-GPU nodes by shifting the decoding workload from the CPU to the GPU.

Removing the CPU Bottleneck in Video Captioning

Video captioning tasks—such as those used in autonomous vehicle (AV) training to classify dangerous scenarios and generate searchable metadata—often utilize lightweight Vision Language Models (VLMs) like Qwen/Qwen3-VL-8B-Instruct. Because these tasks typically produce short outputs (100-200 tokens), the time spent decoding video frames represents a significant portion of the total processing time.

Previously, vLLM relied on a CPU-based OpenCV+FFMPEG backend for video decoding. In multi-GPU configurations (where one vLLM server runs per GPU), the CPU often became a bottleneck, maxing out cores even with only 2 to 4 GPUs. By utilizing PyNvVideoCodec, vLLM moves this decoding process to the GPU's hardware decoders, allowing the system to scale throughput linearly up to 8 GPUs.

Performance Gains on Multi-GPU Nodes

Hardware-based video decoding significantly increases throughput for large-scale captioning workloads. In benchmarks using H100 GPUs with 8 vLLM replicas (each assigned a single GPU), GPU-based video decoding provided more than double the throughput compared to the CPU-based decoder.

This shift removes the CPU utilization ceiling that previously limited scaling. While CPU-based decoding would bottleneck the system before reaching 4 GPUs, the NVDEC-powered approach allows for steady-state performance across all 8 GPUs without CPU saturation.

Implementation and Configuration

Prerequisites and Installation

PyNvVideoCodec functionality is included in standard CUDA vLLM releases. Users with custom installations must include PyNvVideoCodec==2.0.4 as a PyPi dependency.

Deployment Configuration

To enable hardware video decoding, vLLM should be launched with the pynvvideocodec backend specified in the media-io-kwargs. Recommended deployment steps include:

  1. Start CUDA MPS Daemon: The nvidia-cuda-mps-control -d command is essential for maintaining performance during high-concurrency bulk VLM inference.
  2. VRAM Reservation: The --mm-ipc-gpu-memory-gb flag is used to reserve VRAM specifically for video decoding. Users should tune this value to find the minimum amount required to maintain throughput.
  3. GPU Isolation: For multi-GPU scaling, the recommended approach is to run one container per vLLM server replica, exposing a single GPU to each container (or using CUDA_VISIBLE_DEVICES). A reverse proxy is then used to distribute requests among these replicas.

Example Launch Command

vllm serve Qwen/Qwen3-VL-8B-Instruct \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --max-num-seqs 1024 \
  --max-num-batched-tokens 32768 \
  --api-server-count 4 \
  --renderer-num-workers 4 \
  --async-scheduling \
  --mm-ipc-gpu-memory-gb 2 \
  --media-io-kwargs '{"video":{"backend":"pynvvideocodec","min_frames":16,"max_frames":16,"hw_decoders":2}}' \
  --mm-processor-kwargs '{"size":{"shortest_edge":65536,"longest_edge":9437184}}'

Technical Caveats

Hardware video decoding requires a dedicated portion of VRAM. If a use case already utilizes the entire available VRAM for the KV cache, there may be a performance impact. However, vLLM reports that in actual testing, no cases were found where utilizing PyNvVideoCodec resulted in a performance downside.

Sources