NVIDIA/TensorRT-Edge-LLM

High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI

TensorRT Edge‑LLM – High‑Performance LLM Inference for NVIDIA Edge Devices

What it is – A C++/Python runtime built by NVIDIA for running large language models (LLMs) and multimodal models (vision, audio, speech, action) on edge platforms such as Jetson, DRIVE, and DGX Spark. The framework takes Hugging Face checkpoints, converts them to ONNX (or builds directly from the checkpoint), and compiles TensorRT engines optimized for the target GPU.

Key capabilities

  • Edge‑first inference – Designed for low‑latency, low‑memory operation on Jetson‑class devices and automotive‑grade hardware.
  • Model support – Ready‑to‑run checkpoints for many state‑of‑the‑art models (Gemma 4, Qwen 3, Nemotron‑3.5, Cosmos3‑Edge, DiffusionGemma, etc.) with optional quantization and multi‑turn KV‑cache reuse.
  • Performance tricks – In‑flight batching, speculative decoding (DSpark, DFlash, DDTree), multi‑device (TP=2) inference, and a direct engine builder that skips the ONNX export step.
  • High‑level Python API – A vLLM‑style API plus an OpenAI‑compatible HTTP server (tensorrt‑edgellm[server]) for easy integration.
  • Extensibility – Plug‑in system for custom TensorRT ops, chat‑template configuration, and a developer guide for customizing the runtime.

Typical workflow

  1. Choose a supported model from the Supported Models list.
  2. Export the checkpoint to ONNX (or use the experimental direct builder).
  3. Build a TensorRT engine with the provided C++ tools or Python helpers.
  4. Install the pre‑built Python wheel (pip install tensorrt‑edgellm==0.11.0).
  5. Run inference via the Python API or launch the OpenAI‑compatible server for REST calls.

Installation (quick)

# Requires a compatible CUDA/TensorRT stack on the target device
pip install tensorrt-edgellm==0.11.0   # base package
# optional server support
pip install "tensorrt-edgellm[server]"

For full details see the Quick Start Guide and the Support Matrix.

Performance – Benchmarks (linked in the docs) show up to 6.4× faster MLPerf Edge Agentic scores on Jetson AGX Thor versus prior solutions, with high pre‑fill and generation throughput even for 30‑B‑class models.

Use‑case highlights

  • Automotive – In‑vehicle assistants, voice‑controlled UI, driver‑assist perception.
  • Robotics – Natural‑language command handling, visual Q&A, task planning.
  • Industrial IoT – On‑device NLP for monitoring, predictive maintenance, voice‑controlled machinery.
  • Consumer edge – Offline chatbots, privacy‑preserving language processing, low‑latency inference on handheld devices.

Documentation & community – Full docs at https://nvidia.github.io/TensorRT-Edge-LLM/, issue tracker, GitHub Discussions, and NVIDIA Developer Forums for support.

License – Apache 2.0.

All information above is taken directly from the repository’s README and linked documentation.

Related

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    NVIDIA/TensorRT

    A high-performance inference optimizer and runtime from NVIDIA that accelerates AI model deployment and throughput on NVIDIA GPUs.

  • Project
    onnx/onnx-tensorrt

    A TensorRT backend for ONNX that parses ONNX models for high-performance execution on NVIDIA GPUs.

  • Project
    nndeploy/nndeploy

    A high-performance AI deployment framework that uses visual workflows to deploy AI models across desktop, mobile, and edge devices.

  • Project
    wang-xinyu/tensorrtx

    A collection of deep learning network implementations using the TensorRT network definition API for flexible, debuggable, and high-performance inference on NVIDIA GPUs.