NVIDIA/TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

What it solves

TensorRT is designed to accelerate AI inference workflows, allowing developers to optimize deep learning models for high-performance deployment on NVIDIA GPUs.

How it works

It provides a platform for importing models via various paths—including ONNX, Torch-TensorRT, HuggingFace/Optimum, and a Network Definition API—and then optimizing them for inference. The open-source components of the project include the ONNX parser, TensorRT plugins, and sample applications that demonstrate the platform's capabilities.

Who it’s for

AI developers and engineers who need to deploy high-performance deep learning models across various modalities (LLMs, vision, audio, diffusion, and multimodal) on NVIDIA hardware, including x86_64 and aarch64 (Jetson/DriveOS) platforms.

Highlights

  • Broad Model Support: Supports a wide range of model types including LLMs, encoder-NLP, vision, audio, and diffusion models.
  • Flexible Import Paths: Allows importing models from ONNX, Torch-TensorRT, and HuggingFace/Optimum.
  • Advanced Optimization: Features support for strongly typed networks and explicit quantization to improve inference efficiency.
  • Cross-Platform Deployment: Provides build support for Linux, Windows, and specialized environments like Jetson and DriveOS (including QNX safety versions).

Related

  • Project
    onnx/onnx-tensorrt

    A TensorRT backend for ONNX that parses ONNX models for high-performance execution on NVIDIA GPUs.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Project
    pytorch/TensorRT

    Torch-TensorRT integrates NVIDIA TensorRT into PyTorch to accelerate inference latency by up to 5x on NVIDIA GPUs through simple compilation or export workflows.

  • Project
    NVIDIA/TensorRT-Edge-LLM

    A high-performance C++ inference runtime for deploying text, vision, audio, and speech models on NVIDIA Jetson, DRIVE, and DGX Spark edge platforms.