NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

TensorRT‑LLM – Fast, production‑grade inference for large language models

What it is TensorRT‑LLM is an open‑source NVIDIA library that turns a trained LLM (or visual‑generation model) into a highly‑optimized inference engine. It does this by generating custom CUDA kernels for the model’s most expensive operations (attention, GEMM, KV‑cache handling, etc.) and wrapping them in a Python‑friendly runtime that can be plugged into existing serving stacks.

Why it matters Running a 30‑B‑parameter model at interactive latency on a single GPU is hard because the raw PyTorch or ONNX implementation spends a lot of time in generic kernels and memory moves. TensorRT‑LLM replaces those with hand‑tuned kernels, supports INT8/FP8 quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token‑throughput and lower latency while keeping the same model accuracy.

Key features (as described in the README)

  • Specialized kernels for attention, GEMM, and KV‑cache reuse.
  • Support for sparse, quantized, and mixed‑precision inference (FP8, INT8, FP16).
  • Speculative / N‑gram decoding and guided decoding to boost throughput.
  • Distributed Weight Data Parallelism (DWDP) and expert‑parallelism for MoE models.
  • Integration with NVIDIA Triton Inference Server and Kubernetes autoscaling.
  • Pythonic API that lets you extend or customize the runtime.
  • Ready‑to‑run examples for GPT‑OSS‑120B, DeepSeek‑R1, Llama‑4, diffusion models, and more.
  • Documentation, performance guides, and a roadmap for future optimizations.

Typical use cases

  • Deploying chat‑style LLMs (GPT‑OSS, Llama, DeepSeek) in production services where latency and cost matter.
  • Accelerating diffusion or video‑generation models for visual AI workloads.
  • Scaling inference across multi‑node GPU clusters (HGX, Blackwell, GH200) for high‑throughput batch jobs.
  • Building AI‑augmented applications that need fast token generation (code‑completion, search, recommendation).

Ecosystem & integration

  • Works with CUDA 13.x, PyTorch 2.14, and Python 3.10/3.12.
  • Can be combined with NVIDIA Triton, BentoML, AWS EKS, Jetson AGX Orin, and other deployment tools.
  • Supports model formats exported to ONNX, then compiled into TensorRT engines.
  • Provides a set of blogs and tutorials that walk through end‑to‑end pipelines (model conversion, engine building, serving).

Getting started (quick outline)

  1. Install the package (pre‑built wheels or build from source) on a machine with a supported NVIDIA GPU and CUDA.
  2. Convert your model to ONNX (or use a provided checkpoint) and run the trtllm conversion script to generate a TensorRT engine.
  3. Run inference via the Python API or launch a Triton model repository that points to the engine.
  4. Tune performance using the provided flags (batch size, precision, KV‑cache reuse, speculative decoding) and consult the performance guide for your hardware.

Bottom line TensorRT‑LLM is a production‑focused toolkit that lets developers squeeze maximum inference performance out of modern NVIDIA GPUs for LLMs and related generative models, turning research‑grade models into cost‑effective services.

Related

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance CUDA kernel library for LLMs that optimizes GEMM and MoE operations, featuring runtime JIT compilation and extreme throughput on NVIDIA SM90/SM100 GPUs.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.

  • Project
    laugh12321/TensorRT-YOLO

    A high-performance inference deployment tool for YOLO-series models on NVIDIA devices, optimizing preprocessing and post-processing using TensorRT and CUDA.