Melody-Zhou/tensorRT_Pro-Vision

What it solves

Deploying many vision models on NVIDIA GPUs usually means maintaining separate projects per task, and moving data between CPU and GPU slows things down. This framework solves that by putting 20+ tasks (detection, classification, segmentation, pose, rotated boxes, depth, OCR, lane detection, multi-object tracking) into a single C++ executable, with all preprocessing and postprocessing running on the GPU.

How it works

The project is built on TensorRT C++ API. All models share an asynchronous inference skeleton; the only differences are preprocessing and postprocessing. You export your model to ONNX (with dynamic batch), place it in the workspace folder, and run ./pro from the workspace. The framework compiles the TensorRT engine (FP32, FP16, or INT8) and runs it. The preprocessing (resize, normalize) and postprocessing (decode, NMS) are implemented as CUDA kernels, so they stay on the GPU. It also includes custom TensorRT plugins (LayerNorm, DCNv2, etc.) for models that need them. Adding a new task means implementing three pieces: an Infer interface, a preprocess() and worker() function, and a postprocess CUDA kernel, then registering the task in main.cpp.

Who it's for

C++ developers and engineers who need to deploy vision models on NVIDIA GPUs with low latency and high throughput, especially when the product includes several different vision tasks. You should be comfortable with CUDA, TensorRT, and modifying Makefile or CMake paths. It's also useful for researchers who want to quickly benchmark or ship YOLO variants, OCR, lane detection, depth estimation, and more from one codebase.

Highlights

• 20+ models, one executable: switch tasks by name; no per-task project maintenance.
• Full GPU pipeline: resize, normalize, decode, NMS, and depth resize are all CUDA kernels, avoiding CPU↔GPU round trips.
• TensorRT 8.x/10.x adaptive: conditional compilation supports both API generations without protobuf.
• Dynamic batch and three precisions: FP32, FP16, INT8, with built-in Entropy and MinMax calibrators.
• Modular extension: adding a task only requires preprocess(), worker(), and a postprocess kernel.
• Wide task coverage: detection (YOLOv3–v13, v26, DETR), classification, segmentation, pose, OBB, depth, OCR, lane lines, and ByteTrack tracking.