onnx/onnx-tensorrt

ONNX-TensorRT: TensorRT backend for ONNX

What it solves

It provides a way to parse and execute ONNX models using NVIDIA's TensorRT, allowing developers to optimize and run AI models for high-performance inference on NVIDIA GPUs.

How it works

The project acts as a backend for ONNX, translating ONNX model definitions into a format that TensorRT can execute. It supports full-dimensions and dynamic shape support for the latest TensorRT versions. It also provides C++ and Python APIs to load models and run inference, as well as tools like trtexec and polygraphy for testing and building engines.

Who it’s for

Developers and AI engineers who need to optimize their ONNX models for deployment on NVIDIA hardware using TensorRT.

Highlights

  • Supports the latest TensorRT 11.4 and ONNX release 1.21.0.
  • Provides both C++ and Python bindings for integration.
  • Includes tools for checking if an ONNX model can be successfully parsed and build into a TensorRT engine.
  • Offers flexible implementation options for specific operators, such as InstanceNormalization.

Related

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.

  • Project
    cyrusbehr/tensorrt-cpp-api

    A modern C++ library for high-performance GPU inference of CNN models using NVIDIA TensorRT, featuring a safe engine cache and zero-copy Python bindings.

  • Project
    onnx/onnx-tensorflow

    A backend for ONNX that converts ONNX models into TensorFlow models, enabling them to be run using the TensorFlow execution engine.

  • Project
    triple-mu/YOLOv8-TensorRT

    A high-performance TensorRT acceleration wrapper for YOLOv8, enabling fast inference for detection, segmentation, pose, OBB, and classification in both Python and C++.