pytorch/TensorRT

PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT

What it solves

Torch-TensorRT addresses the challenge of optimizing PyTorch models for high-performance inference on NVIDIA GPUs. It allows developers to reduce inference latency—potentially by up to 5x compared to standard eager execution—without needing to manually rewrite models in TensorRT.

How it works

It integrates NVIDIA's TensorRT optimization engine directly into the PyTorch ecosystem. Users can apply it in two primary ways:

  1. Just-in-Time Compilation: Using torch.compile with the tensorrt backend to optimize models on the fly.
  2. Ahead-of-Time Export: Compiling and serializing the model into a format (like TorchScript) that can be deployed in either PyTorch or a C++ environment via libtorch, removing the Python dependency for production.

Who it’s for

It is designed for machine learning engineers and developers who deploy PyTorch models on NVIDIA hardware and need to maximize throughput and minimize latency for production environments.

Highlights

  • Significant Speedups: Accelerates inference latency by up to 5x.
  • Flexible Deployment: Supports both Python-based runtime and C++ deployment.
  • Broad Model Support: Compatible with Diffusion models, LLMs from Hugging Face, and FP8 precision.
  • Seamless Integration: Can be activated with a single line of code via torch.compile.

Related

  • Project
    NVIDIA/TensorRT

    NVIDIA TensorRT is a high-performance deep learning inference library and SDK that accelerates AI model deployment on NVIDIA GPUs.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Project
    cyrusbehr/YOLOv8-TensorRT-CPP

    A C++ implementation of YOLOv8 using TensorRT for high-performance GPU inference, supporting object detection, semantic segmentation, and pose estimation.

  • Project
    triple-mu/YOLOv8-TensorRT

    A high-performance TensorRT acceleration wrapper for YOLOv8, enabling fast inference for detection, segmentation, pose, OBB, and classification in both Python and C++.

  • Project
    Ascend/pytorch

    TorchNPU is a PyTorch adapter plugin that enables the PyTorch framework to run on Huawei Ascend AI processors, providing optimized compute and distributed training capabilities.