Tencent/hpc-ops

High Performance LLM Inference Operator Library

What it solves

HPC-Ops addresses the latency and throughput bottlenecks in LLM inference. It targets the "hot paths" of serving—such as attention, Mixture-of-Experts (MoE), and sampling—where standard implementations often suffer from memory-bandwidth limits, inefficient task scheduling, or excessive kernel launch overhead.

How it works

It provides a library of production-grade CUDA kernels optimized specifically for modern NVIDIA GPUs (SM90). The project employs several advanced techniques to maximize hardware utilization:

  • Dynamic Scheduling: For decode attention, it uses a greedy bin-packing strategy to balance workloads across compute units, reducing tail latency for mixed-length requests.
  • Operator Fusion: It collapses multiple operations into single kernels. For example, it fuses AllReduce, residual addition, and RMSNorm into one NVLink-native kernel, and merges the entire sampling pipeline (penalty, temperature, top-k/p, softmax) into two kernels.
  • Precision Optimization: It implements specialized kernels for FP8 and BF16, including a "Route GEMM" that achieves FP32-level accuracy using a fused linear combination of two BF16 Tensor Core GEMMs.
  • Pipelined Execution: The Fused MoE path integrates routing, GEMM, and reduction into a single pipeline, removing standalone gather stages to reduce memory traffic.

Who it’s for

This library is designed for AI infrastructure engineers and developers building high-performance LLM inference frameworks (like vLLM or SGLang) who need to squeeze maximum performance out of NVIDIA H20 GPUs.

Highlights

  • SOTA Performance: Outperforms baselines like FlashInfer and TensorRT-LLM in various scenarios, including up to 8.5x speedup for fused sampling.
  • Rich Precision Support: Native support for BF16 and FP8 with various quantization schemes.
  • Production-Proven: Developed by the Tencent Hunyuan AI Infra team for large-scale production inference.
  • Modern CUDA Implementation: Utilizes cutting-edge features like CuTe, CUTLASS, TMA, and PDL.