NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

What it solves

TensorRT LLM is designed to optimize the inference performance of Large Language Models (LLMs) and Visual Generation models. It addresses the bottleneck of high latency and low throughput during the deployment of these massive models, enabling them to run faster and more efficiently on NVIDIA GPUs.

How it works

The project provides a pythonic framework for customizing and extending the system, combined with an efficient runtime and specialized kernels for common AI operations. It employs various optimization techniques, including:

  • Quantization and Sparsity: Reducing model precision to lower memory usage and increase speed.
  • Speculative Decoding: Using a smaller model to predict tokens, which is umapped by the larger model to boost throughput.
  • Advanced Attention Mechanisms: Implementing sparse attention, multiblock attention, and skip softmax attention to accelerate long-context inference.
  • Distributed Inference: Utilizing Distributed Weight Data Parallelism (DWDP) and expert parallelism to scale inference across multiple GPUs or racks (e.g., NVL72).

Who it’s for

It is intended for AI developers and engineers who need to deploy high-performance inference servers for LLMs and visual generation models on NVIDIA hardware, ranging from Jetson AGX Orin to H200 and Blackwell GPUs.

Highlights

  • Broad Model Support: Optimized for a wide range of models including Llama, DeepSeek, Mixtral, and diffusion models for visual generation.
  • Harnesses NVIDIA Hardware: Specifically tuned for NVIDIA Blackwell, H100, H200, and Jetson platforms.
  • Extreme Throughput: Capable of delivering massive token-per-second rates (e.g., over 40,000 tokens per second for Llama 4 on B200 GPUs).
  • Extensible Framework: Offers a pythonic interface for developers to customize the system.

Related

  • Project
    NVIDIA/TensorRT

    NVIDIA TensorRT is a high-performance deep learning inference library and SDK that accelerates AI model deployment on NVIDIA GPUs.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.