cyrusbehr/tensorrt-cpp-api

TensorRT C++ API Tutorial

What it solves

This library provides a modern, high-performance C++ wrapper around NVIDIA TensorRT, simplifying the process of running GPU-accelerated inference for CNN models. It eliminates the complexity of the raw TensorRT API, providing a leak-free, no-throw error model and removing the need for TensorRT types in public headers.

How it works

The project converts ONNX models into optimized TensorRT engines. It uses a safe engine cache system that validates the model hash, build options, and hardware UUID to prevent the use of stale caches. It supports dynamic shapes via optimization profiles and manages concurrency through an EnginePool for multi-stream inference. For Python users, it provides zero-copy bindings that allow GPU arrays (from CuPy or PyTorch) to be passed directly to the engine without host round-trips.

Who it’s for

Developers building high-performance vision applications on Linux using CUDA 12 and TensorRT 10+, specifically those targeting CNN-style vision models.

Highlights

  • Safe Engine Caching: Automatically detects and rebuilds engines if the model, options, or GPU hardware changes.
  • Zero-Copy Python Bindings: High-performance Python interface using __cuda_array_interface__ and DLPack.
  • Fused Preprocessing: A dedicated CUDA kernel for letterbox-resize, normalization, and color conversion in a single step.
  • Clean API: No-throw Status/Result error handling and PImpl pattern to hide TensorRT dependencies from public headers.