NVIDIA/TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

What it solves

TensorRT is a high-performance inference optimizer and runtime that accelerates AI model deployment on NVIDIA GPUs. It focuses on reducing latency and increasing throughput for AI inference workflows across various modalities including LLMs, vision, audio, and multimodal models.

How it works

TensorRT optimizes models by providing specialized import paths (such as ONNX, Torch-TensorRT, and HuggingFace/Optimum) and a Network Definition API. It utilizes techniques like strongly typed networks and explicit quantization to improve performance. The open-source components of the project include the ONNX parser and TensorRT plugins, allowing developers to extend the library with custom layers.

Who it’s for

AI engineers and developers who need to deploy optimized, low-latency AI models on NVIDIA hardware, including those working with Jetson, DriveOS, and data center GPUs.

Highlights

  • Broad Model Support: Supports a wide range of models including LLMs, encoder-NLP, vision, audio, and diffusion models.
  • Extensible Architecture: Provides an open-source plugin system for custom layer implementation.
  • Multi-Platform Deployment: Supports Linux (x86_64 and aarch64), Windows, and specialized environments like DriveOS and QNX.
  • Streamlined API: Recent updates (v11.X) have streamlined the API, moving toward strongly typed networks and explicit quantization.

Related

  • Project
    onnx/onnx-tensorrt

    A TensorRT backend for ONNX that parses ONNX models for high-performance execution on NVIDIA GPUs.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    FeiYull/TensorRT-Alpha

    A high-performance C++ inference framework for YOLO models that uses custom CUDA kernels to perform end-to-end GPU processing, minimizing CPU overhead.

  • Project
    NVIDIA/TensorRT-Edge-LLM

    TensorRT Edge‑LLM is NVIDIA’s C++/Python inference runtime for running LLMs and multimodal models on Jetson, DRIVE and DGX Spark edge hardware. It converts Hugging Face checkpoints to TensorRT engines (via ONNX or a direct builder), offers quantization, in‑flight batching, speculative decoding, and a vLLM‑style Python API with an OpenAI‑compatible server. The library targets automotive, robotics, industrial IoT and consumer edge use‑cases, delivering high‑throughput, low‑latency inference on resource‑constrained GPUs.