NVIDIA/TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

What it solves

TensorRT 旨在加速 AI 推理工作流，优化深度学习模型在 NVIDIA GPU 上的部署性能。它提供了从各种框架导入模型并对其进行高效执行优化的工具。

How it works

TensorRT 通过多种导入路径（包括 ONNX、Torch‑TensorRT、HuggingFace/Optimum 和 Network Definition API）对 AI 模型进行优化。它支持广泛的模型类型，包括 LLM、编码器‑NLP、视觉、音频、扩散以及多模态模型。项目的开源组件包括 ONNX 解析器和 TensorRT 插件，开发者可以扩展平台功能。

Who it’s for

适用于需要在 NVIDIA 硬件（包括 x86_64 和 aarch64（Jetson/DriveOS）平台）上部署高性能推理的 AI 开发者和工程师。

Highlights

  • Broad Import Support: Compatible with ONNX, Torch‑TensorRT, and HuggingFace/Optimum.
  • Diverse Model Compatibility: Supports LLMs, vision, audio, and multimodal models.
  • Flexible Deployment: Provides prebuilt Python packages for easy installation and extensive build options for various OS and hardware targets.
  • Extensible: Includes open‑source plugins and an ONNX parser to customize and optimize model execution.