Tencent/hpc-ops

What it solves

HPC-Ops is a high-performance operator library designed to reduce latency and increase throughput for Large Language Model (LLM) inference. It targets the "hot paths" of inference—the most computationally expensive operations—that often create bottlenecks in production serving environments.

How it works

The library provides a suite of deeply optimized CUDA kernels tailored for modern NVIDIA GPUs (specifically SM90/H20). It achieves performance gains through several advanced techniques:

• Fusion: It combines multiple operations into single kernels to reduce memory traffic and kernel launch overhead. Examples include fusing AllReduce, Residual Add, and RMSNorm, or collapsing the entire sampling pipeline into two kernels.
• Dynamic Scheduling: For decode attention, it uses a greedy bin-packing strategy to balance workloads across compute units, reducing tail latency for mixed-length requests.
• Precision Optimization: It supports BF16 and FP8, including a unique "Route GEMM" that simulates FP32 accuracy using a linear combination of two BF16 Tensor Core GEMMs.
• Sparsity: It implements block-sparse prefill attention to skip irrelevant KV tiles in long-context workloads.

Who it’s for

This library is intended for AI infrastructure engineers and developers building LLM inference frameworks (such as vLLM or SGLang) who need production-grade, SOTA performance on NVIDIA hardware.

Highlights

• Comprehensive Operator Catalog: Includes optimized kernels for Attention, MoE (Mixture of Experts), GEMM, Sampling, and Normalization.
• SOTA Benchmarks: Demonstrates significant speedups over baselines like FlashInfer, TensorRT-LLM, and cuBLAS (e.g., up to 8.5x for the fused sampler).
• Production-Proven: Developed by the Tencent Hunyuan AI Infra team for large-scale production use.
• Developer Resource: Serves as a practical guide for building kernels using modern CUDA tools like CuTe, CUTLASS, and TMA.