Hugging Face Native-speed vLLM Transformers Modeling Backend

Hugging Face Native-speed vLLM Transformers Modeling Backend

TL;DR

Hugging Face has enhanced the transformers modeling backend for vLLM, enabling it to achieve throughput that meets or exceeds native, hand-written vLLM implementations for many LLM architectures. This allows model authors to leverage their existing transformers implementations to obtain ultra-fast vLLM inference without writing custom optimization code.

Performance Benchmarks

The transformers modeling backend was tested against native vLLM implementations across three Qwen3 model configurations. In all cases, the transformers backend met or beat native throughput:

  • Qwen3-4B (Dense): Tested on a single GPU.
  • Qwen3-32B (Dense): Tested using tensor parallelism.
  • Qwen3-235B-A22B-FP8 (MoE): Tested using data and expert parallelism on an 8‡H100 node.

To use this backend, users can apply the --model-impl transformers flag during serving. This implementation is compatible with standard parallelism options, including --tensor-parallel-size, --data-parallel-size, and --enable-expert-parallel.

Limitations

Currently, models utilizing linear attention are not supported. Additionally, custom models hosted in Hub repositories may not work if they were not written compliantly.

Technical Implementation: Dynamic Layer Fusion

Previously, the transformers vLLM backend focused primarily on attention as the main inference bottleneck. While this allowed models to run efficiently, achieving maximum performance still required custom ports to handle GPU parallelization, compilation, and fused kernels.

The updated backend now dynamically applies inference-specific layer fusions at runtime for compatible architectures. This removes the need for model authors to integrate their models twice—once for transformers and once for vLLM.

Graph Analysis and Source Manipulation

The system achieves native speeds through a two-step optimization process:

  1. Static Analysis: The backend uses torch.fx to perform static analysis on the model’s graph to identify known patterns that can be optimized.
  2. Source Rewriting: Once patterns are identified, the backend uses the Abstract Syntax Tree (AST) to manipulate the source code and rewrite operations in place.

Key Optimizations

This process enables several high-performance capabilities:

  • Fused Operations: Many-to-one mapping of operations to optimized vLLM kernels, specifically those used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models.
  • Parallel Planning: The use of MergedColumnParallelLinear and QKVParallelLinear blocks allows the system to infer parallel plans for Tensor Parallelism (TP). Pipeline Parallelism (PP) plans can also be inferred if the decoder block list is identifiable.
  • Compilation Compatibility: The manipulated models remain fully compatible with torch.compile and CUDA Graphs, mirroring the performance path of dedicated vLLM implementations.
  • Unified Codebase: Because the optimization happens at runtime, the same transformers model implementation can be used for training, evaluations, and RL rollouts, as well as high-speed inference.

Sources