onnx/onnx-tensorrt

ONNX-TensorRT: TensorRT backend for ONNX

What it solves

It provides a way to parse and execute ONNX models using NVIDIA's TensorRT, allowing developers to optimize and run AI models for high-performance inference on NVIDIA GPUs.

How it works

The project acts as a backend for ONNX, translating ONNX model definitions into a format that TensorRT can execute. It supports full-dimensions and dynamic shapes, and provides both C++ and Python APIs for integrating the parser into applications.

Who it’s for

Developers and machine learning engineers who need to optimize the execution of ONNX-formatted models on NVIDIA hardware.

Highlights

  • Support for TensorRT 11.2 and ONNX release 1.21.0.
  • Provides C++ and Python bindings for easy integration.
  • Includes tools like trtexec and polygraphy for quick model validation and benchmarking.
  • Flexible implementation options for specific operators, such as InstanceNormalization.
  • Support for dynamic shapes and full-dimensions.

Related

  • Project
    NVIDIA/TensorRT

    NVIDIA TensorRT is a high-performance deep learning inference library and SDK that accelerates AI model deployment on NVIDIA GPUs.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.

  • Project
    cyrusbehr/tensorrt-cpp-api

    A modern C++ library for high-performance GPU inference of CNN models using NVIDIA TensorRT, featuring a safe engine cache and zero-copy Python bindings.

  • Project
    microsoft/onnxruntime

    A cross-platform machine learning accelerator that optimizes inference and training for models from various frameworks like PyTorch and TensorFlow.

  • Project
    triple-mu/YOLOv8-TensorRT

    A high-performance TensorRT acceleration wrapper for YOLOv8, enabling fast inference for detection, segmentation, pose, OBB, and classification in both Python and C++.