triton-inference-server/server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

What it solves

Triton Inference Server streamlines the deployment of AI models by providing a standardized way to serve models from multiple frameworks (such as PyTorch, TensorRT, ONNX, and OpenVINO) across various hardware platforms, including NVIDIA GPUs, x86/ARM CPUs, and AWS Inferentia.

How it works

It acts as an inference serving software that manages a model repository. It uses a modular backend architecture, allowing it to execute models from different deep learning and machine learning frameworks. It supports multiple inference protocols (HTTP/REST and gRPC) and provides C and Java APIs for in-process use cases.

Who it’s for

AI teams and developers who need to deploy production-ready AI models with optimized performance across cloud, data center, edge, and embedded devices.

Highlights

  • Multi-framework support: Serves models from TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL.
  • Performance optimization: Features dynamic batching, sequence batching, and concurrent model execution to maximize throughput and only latency.
  • Flexible deployment: Supports cloud, data center, edge, and embedded devices.
  • Model pipelining: Enables complex workflows via Ensembling and Business Logic Scripting (BLS).
  • Extensibility: Provides a Backend API for creating custom backends in C/C++ or Python.

Related

  • Project
    triton-inference-server/backend

    A framework and set of utilities for creating backends that allow Triton Inference Server to execute models from various deep-learning frameworks or custom C/C++ logic.

  • Project
    triton-inference-server/python_backend

    A backend for Triton Inference Server that enables serving Python-based machine learning models without requiring C++ code.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.

  • Project
    triton-inference-server/client

    Client libraries and examples for C++, Python, and Java that simplify communication with Triton Inference Server for model inferencing and server management.

  • Project
    NVIDIA/cudnn-frontend

    NVIDIA’s cuDNN Frontend is an open‑source, header‑only C++ API plus Python bindings that simplify building high‑performance deep‑learning kernels (attention, GEMM, fused ops) on Hopper and Blackwell GPUs. It ships ready‑made, JIT‑compiled kernels for Flash‑Attention, Mixture‑of‑Experts GEMM fusions, sparse/linear attention, and more, all accessible via a unified Graph API and PyTorch‑compatible ops.