alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

What it solves

It addresses the challenge of deploying Large Language Models (LLMs) in high-traffic production environments by providing a high-performance inference acceleration engine that reduces latency and increases throughput.

How it works

RTP-LLM optimizes the inference process using several technical strategies:

  • Advanced Kernels: It uses high-performance CUDA kernels like PagedAttention, FlashAttention, and FlashDecoding.
  • Quantization: It supports WeightOnly INT8 (automatic at load time) and INT4 (via GPTQ and AWQ), as well as adaptive KVCache quantization to reduce memory footprint.
  • Memory & Scheduling: It employs a C++ rewritten scheduling and batching framework with complete GPU memory management and dynamic batching optimization.
  • Caching: It implements System Prompt Cache and Contextual Prefix Cache to speed up multi-turn dialogues.
  • Parallelism: It enables multi-machine and multi-GPU tensor parallelism for larger models.

Who it’s for

It is designed for developers and organizations needing to deploy LLMs at scale in production, particularly those using NVIDIA GPUs (with specific optimizations for V100) or ARM CPUs (Yitian).

Highlights

  • Production Proven: Used across Alibaba services like Taobao, Tmall, and Lazada.
  • Broad Compatibility: Integrates with HuggingFace models and supports SafeTensors, Pytorch, and Megatron formats.
  • Multimodal Support: Capable of handling combined image and text inputs.
  • Flexible Deployment: Supports deploying multiple LoRA services within a single model instance and supports P-tuning models.
  • Advanced Decoding: Includes support for Speculative Decoding and Prefill/Decode separation.