NVIDIA/TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

What it solves

TensorRT 旨在加速 AI 推論工作流程，優化深度學習模型在 NVIDIA GPU 上的部署效能。它提供了從各種框架匯入模型並對其進行高效執行優化的工具。

How it works

TensorRT 透過多種匯入路徑（包括 ONNX、Torch‑TensorRT、HuggingFace/Optimum 與 Network Definition API）對 AI 模型進行優化。它支援廣泛的模型類型，包括 LLM、編碼器‑NLP、視覺、音訊、擴散以及多模態模型。專案的開源元件包括 ONNX 解析器與 TensorRT 插件，開發者可以擴充平台功能。

Who it’s for

適用於需要在 NVIDIA 硬體（包括 x86_64 與 aarch64（Jetson/DriveOS）平台）上部署高效能推論的 AI 開發者與工程師。

Highlights

  • Broad Import Support: Compatible with ONNX, Torch‑TensorRT, and HuggingFace/Optimum.
  • Diverse Model Compatibility: Supports LLMs, vision, audio, and multimodal models.
  • Flexible Deployment: Provides prebuilt Python packages for easy installation and extensive build options for various OS and hardware targets.
  • Extensible: Includes open‑source plugins and an ONNX parser to customize and optimize model execution.