vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan

vLLM × HPC-Ops: High-Performance Attention and MoE Backends from Tencent Hunyuan

Why This Matters

Production LLM serving now handles dynamic, mixed‑length batches and increasingly large MoE models, so latency is dominated by how well kernels schedule work and move data rather than raw matmul throughput. HPC‑Ops provides load‑balanced attention and fully fused FP8 MoE kernels that reduce idle SM cycles and eliminate per‑stage overhead, directly addressing these real‑world bottlenecks.

HPC-Ops: A Production Operator Library, Now in vLLM

HPC-Ops is an open‑source operator library built by the Tencent Hunyuan AI Infra team that focuses on attention, MoE, GEMM, sampling, normalization and communication‑compute fusion with native BF16 and FP8 support for Hopper GPUs. Its attention and MoE kernels have been upstreamed into vLLM as first‑class backends via PR #46020 (Attention) and PR #45924 (MoE), requiring no source changes or long‑lived fork.

Attention Backend: Dynamic Load‑Balanced Scheduling

The HPC‑Ops attention backend eliminates the idle time caused by static split‑KV schedules in mixed‑length decode by using a per‑step, load‑balanced scheduler that tiles every KV sequence into uniform 64‑token tiles, assigns tiles to CTAs based on actual work, and runs a persistent kernel grid that keeps all SMs saturated. This yields up to 2.95× speedup over a static split‑KV schedule and an average 2.25× advantage over FlashInfer and FlashAttention on H20 for mixed‑length decode.

A fused attention prologue (HpcRopeNorm) combines QK‑Norm, RoPE, KV‑cache write and, in FP8, query quantization into a single kernel, removing separate memory‑bound launches and their HBM round‑trips.

Integration with vLLM is done through HpcAttentionBackend, which inherits from vLLM’s AttentionBackend base class and is registered via the standard backend mechanism.

MoE Backend: A Fused, Low-Latency FP8 MoE Pipeline

The HPC‑Ops MoE backend removes the per‑stage overhead of conventional MoE paths by fusing routing, Gate‑Up GEMM, activation/quantization, Down GEMM and top‑k weighted reduction into a single execution pipeline. It uses a shared‑memory counting pass to build contiguous expert ranges, reads tokens directly via routing indices, keeps intermediates in registers or shared memory, and employs an occupancy‑first warp group with a persistent grid to spread uneven expert tiles evenly across CTAs. Programmatic Dependent Launch overlaps kernel launches to erase bubbles.

The pipeline runs in FP8 with per‑tensor and block‑wise scaling, matching baseline output quality. On H20 it achieves an average 1.59× speedup over Triton and CUTLASS at TP8/EP1 and 1.21× at TP1/EP8.

Integration with vLLM is via HPCExperts, which inherits from FusedMoEExpertsModular and is registered through the standard backend registration mechanism.

Using HPC-Ops Backends in vLLM

To use the backends, first install HPC‑Ops from source:

git clone https://github.com/Tencent/hpc-ops.git
cd hpc-ops
make wheel
python3 -m pip install dist/*.whl

Enable the attention backend for Hy3 (BF16) models:

vllm serve tencent/Hy3 \
    --tensor-parallel-size 8 \
    --attention-backend HPC_ATTN

For Hy3‑FP8 add KV‑cache dtype and block size:

vllm serve tencent/Hy3-FP8 \
    --tensor-parallel-size 8 \
    --attention-backend HPC_ATTN \
    --kv-cache-dtype fp8_e4m3 \
    --block-size 64

Enable the MoE backend (FP8 only):

vllm serve tencent/Hy3-FP8 \
    --tensor-parallel-size 8 \
    --moe-backend hpc

The backends are currently supported only on NVIDIA Hopper‑architecture GPUs, with best performance on H20.

Performance on H20

Fused MoE vs Triton / CUTLASS

Across batch sizes from 4 to 16384, HPC‑Ops MoE latency is lower than both Triton and CUTLASS. At TP8/EP1 the average speedup is 1.59×; at TP1/EP8 the average speedup is 1.21×, with the largest gains at small‑to‑mid batch sizes typical of low‑latency decode.

Attention Decode under Mixed‑Length Batches

Dynamic scheduling outperforms static split‑KV, FlashInfer and FlashAttention. Speedup grows with skew: from 1.00× on uniform 64×0.5K batches to 2.95× on a 1×128K + 31×4K mix. Across the tested distributions the average advantage over the best of FlashInfer/FlashAttention is 2.25×.

Attention Across Prefill, Extend and Decode Shapes

HPC‑Ops attention latency is at parity with or faster than the fastest of FlashAttention, Triton and FlashInfer for every shape benchmarked (e.g., 0.047 ms vs 0.069 ms for q512 prefill, 0.019 ms vs 0.031 ms for 8q1s1k decode).

End‑to‑End on Hy3 (8× H20)

Using both HPC‑Ops attention and MoE backends, TTFT is reduced by about 24% and TPOT by about 17% compared with the vLLM default backend. Improvements increase with batch size, reaching ~30% TTFT reduction and ~30% TPOT reduction at the largest batch sizes tested.

What's Next

The Tencent Hunyuan team will continue collaborating with the vLLM community to refine these backends, upstream further work as it matures, and welcome feedback, issues and benchmarks from users.

Acknowledgements

Thanks to the Tencent Hunyuan AI Infra team (Sethran Liu, Chase Shao, Shengy Wei, Theo Cheng, Ryann Xue, Lando Jiang, Looper Zhao, Haank Lin, Aiden Ren, Lehua Ding, Chengv Jiang, Steven Kuang, Liqi He, Kipper Gong, Reedlau Liu, Raccoon Liu, Dick Zhu), the Tencent Network Platform Department, vLLM/Inferact contributors (Kaichao You, Yongye Zhu, Yifan Qiao), NVIDIA engineers (Yuanhang Sun, Perkz Zheng, Yuxi Chi, Jiang Shao, Jun Gu, Meng Wang, River Liu, Gary Ji, Chandler Zhou), and the broader open‑source kernel community whose work this builds on and measures against.

Sources