triple-mu/YOLOv8-TensorRT

What it solves

This project provides a high-performance inference implementation for YOLOv8 models, accelerating them using NVIDIA's TensorRT. It allows users to take a trained Ultralytics YOLOv8 model and run it efficiently in either Python or C++ environments, supporting five different computer vision tasks.

How it works

The workflow follows a three-step process: exporting a PyTorch model (.pt) to ONNX format, building a TensorRT engine from that ONNX file, and then performing inference. The project includes specific export scripts for detection and segmentation to create "End2End" engines with Non-Maximum Suppression (NMS) built-in, while other tasks use raw exports.

It features a shared C++ core (libyolov8_core) that manages CUDA resources and handles pre- and post-processing. The system is designed to be version-agnostic, automatically detecting the installed versions of TensorRT and OpenCV to ensure compatibility across different hardware and software environments.

Who it’s for

This is for developers and AI engineers who need to deploy YOLOv8 models on NVIDIA GPUs with maximum throughput and minimum latency, particularly those working in C++ production environments or high-performance Python applications.

Highlights

• Multi-task support: Handles detection, segmentation, pose estimation, oriented bounding boxes (OBB), and classification.
• C++ and Python support: Provides both a lightweight C++ binary and a flexible Python entry point for inference.
• Version-agnostic build: Compatible with TensorRT versions 8 through 11 and various OpenCV versions.
• Performance tools: Includes a benchmarking script and per-layer profiling to analyze latency and throughput.
• Deployment flexibility: Includes a DeepStream bbox parser plugin and support for NVIDIA Jetson devices.