triton-inference-server/tensorrtllm_backend

What it solves

This project provides a backend for the Triton Inference Server that allows users to serve Large Language Models (LLMs) optimized with TensorRT-LLM. It simplifies the deployment of high-performance LLM inference, supporting advanced features like in-flight batching and paged attention to maximize throughput and efficiency on NVIDIA GPUs.

How it works

The backend integrates TensorRT-LLM into the Triton Inference Server ecosystem. It can be deployed in several ways:

• PyTorch Backend (LLM API): Allows serving HuggingFace models directly without requiring manual engine compilation.
• Deployment Modes: Supports "Leader Mode" (one Triton process per GPU, ideal for Slurm) and "Orchestrator Mode" (one orchestrator process spawning workers per GPU).
• Parallelism: Implements Tensor, Pipeline, and Expert Parallelism to distribute large models across multiple GPUs or nodes.
• Execution: Uses a C++ implementation for the core batch manager to handle in-flight batching and request scheduling.

Who it’s for

It is designed for ML engineers and DevOps professionals who need to deploy LLMs in production environments using NVIDIA hardware, specifically those requiring high throughput, multi-GPU scaling, and integration with Triton Inference Server.

Highlights

• Advanced Batching: Supports in-flight batching and paged attention.
• Flexible Deployment: Multi-node support and compatibility with Slurm-based clusters.
• Diverse Decoding: Supports Top-k, Top-p, Beam Search, and speculative decoding (Medusa, ReDrafter, Lookahead, Eagle).
• Model Support: Compatible with LLaMA, Gemma, Mistral, and various multimodal and encoder-decoder models.
• Optimization: Integration with quantization and LoRA for improved performance and memory efficiency.