NVIDIA/TensorRT-Edge-LLM

What it solves

TensorRT Edge-LLM provides a high-performance C++ inference runtime for deploying large language models (LLMs) and other foundation models (vision, audio, speech, and action) specifically on NVIDIA edge platforms like Jetson, DRIVE, and DGX Spark. It addresses the need for low-latency, memory-efficient AI execution on hardware with limited resources compared to data center GPUs.

How it works

The framework converts model checkpoints (typically from Hugging Face) into optimized TensorRT engines. This is achieved through two primary paths: a recommended pipeline that exports checkpoints to ONNX before building the engine, and an experimental direct builder that bypasses ONNX. Once the engine is built, the C++ deployment runtime handles the actual inference execution.

Who it’s for

It is designed for developers building AI applications for automotive systems (in-vehicle assistants), robotics (task planning and human-robot collaboration), industrial IoT (automated inspection), and privacy-focused edge devices requiring offline processing.

Highlights

• Broad Model Support: Compatible with a wide range of models including Gemma 4, Qwen, and Nemotron families.
• Multimodal Capabilities: Supports text, vision, audio, and speech models.
• Optimization Techniques: Implements quantization, speculative decoding (via DFlash and DSpark), and multi-turn KV-cache reuse.
• Flexible API: Includes an experimental high-level Python API and an OpenAI-compatible server.