TensorRT: an NVIDIA-powered inference accelerator for PyTorch models that reduces latency by up to 5x

What it solves

Torch-TensorRT provides a way to accelerate the inference performance of PyTorch models on NVIDIA GPUs. It addresses the problem of slow inference latency in eager execution mode, potentially reducing latency by up to 5x.

How it works

It integrates NVIDIA's TensorRT optimization engine into the PyTorch ecosystem. Users can apply optimizations via two primary methods:

1. torch.compile: A single-line integration where the backend is set to "tensorrt", allowing the model to be compiled on the first run.
2. Export workflow: An ahead-of-time optimization and serialization process that allows models to be deployed in either PyTorch or a C++ environment (via libtorch) without requiring a Python dependency.

Who it’s for

Developers and ML engineers who are deploying PyTorch models on NVIDIA hardware and need to maximize inference speed and efficiency.

Highlights

• High-performance inference acceleration (up to 5x faster than eager execution).
• Seamless integration with torch.compile for rapid prototyping.
• Support for ahead-of-time serialization for C++ deployment.
• Compatibility with Diffusion models, LLMs from Hugging Face, and FP8 precision.
• Broad platform support across Linux (AMD64, SBSA) and Windows (Dynamo only).