pytorch/TensorRT

Torch‑TensorRT — Fast PyTorch inference on NVIDIA GPUs

What it is

• An open‑source library that bridges PyTorch and NVIDIA’s TensorRT inference engine. It lets you take any PyTorch model and run it with TensorRT‑level speed without rewriting the model.

Key capabilities

• One‑line acceleration – use torch.compile(..., backend="tensorrt") or the torch_tensorrt.compile API to get up to 5× lower latency compared to regular eager PyTorch.
• Two deployment paths:
  1. Just‑in‑time (JIT) compilation via torch.compile – works entirely in Python.
  2. Ahead‑of‑time export – compile once, serialize to an ExportedProgram (.ep) or TorchScript (.ts) file, then load in Python or C++ (libtorch) for production.
• Support for modern hardware – works on Linux (x86‑64, ARM aarch64), Windows (Dynamo backend), Jetson devices (GPU/DLA), and requires CUDA 13.4 + TensorRT 11.3.
• Mixed‑precision & FP8 – tutorials show how to run models in lower‑precision formats for extra speed.

Typical workflow

import torch, torch_tensorrt
model = MyModel().eval().cuda()
inputs = [torch.randn(1,3,224,224).cuda()]
# Compile & serialize
trt_mod = torch_tensorrt.compile(model, ir="dynamo", inputs=inputs)
torch_tensorrt.save(trt_mod, "model.ts", output_format="torchscript", inputs=inputs)

Later, in Python or C++:

mod = torch.export.load("model.ep").module()
mod(*inputs)   # fast inference

auto mod = torch::jit::load("model.ts");
auto out = mod.forward({input_tensor});

Who should use it

• PyTorch developers who need production‑grade inference speed on NVIDIA GPUs.
• Teams deploying large vision or language models (e.g., Stable Diffusion, Hugging‑Face LLMs) and want a drop‑in speed boost.
• Engineers who prefer staying in the PyTorch ecosystem but need TensorRT‑level performance.

Installation

• Stable: pip install torch-tensorrt
• Nightly (CUDA 13.4): pip install --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu134
• Also available in NVIDIA NGC PyTorch containers.

Resources

• Official docs: https://nvidia.github.io/Torch-TensorRT/
• Blog posts & tutorials on accelerating diffusion models, Stable Diffusion, and Hugging‑Face LLMs.
• YouTube tech talk (GTC ’23) for deeper architecture details.

License

• BSD‑3‑Clause, permissive for commercial and research use.

---

Torch‑TensorRT is a genuine AI‑focused project that enables high‑performance inference for PyTorch models on NVIDIA hardware.