triton-inference-server/server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

What it solves

Triton Inference Server streamlines the deployment and serving of AI models. It removes the complexity of managing multiple frameworks and hardware targets, allowing teams to deploy any AI model from various deep learning and machine learning frameworks on a wide range of hardware, from the cloud to edge devices.

How it works

Triton acts as a serving layer that supports multiple backends (such as TensorRT, PyTorch, ONNX, OpenVINO, and Python). Users place their models in a model repository and configure them. Triton then provides standardized HTTP/REST and gRPC protocols for clients to send inference requests. It optimizes performance through features like concurrent model execution, dynamic batching, and sequence batching for stateful models.

Who it’s for

It is designed for AI engineers and DevOps teams who need to deploy production-grade AI models across diverse hardware (NVIDIA GPUs, x86/ARM CPUs, AWS Inferentia) and diverse frameworks.

Highlights

  • Multi-framework support: Serves models from TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL.
  • Hardware flexibility: Runs on NVIDIA GPUs, x86 and ARM CPUs, or AWS Inferentia.
  • Performance optimizations: Includes dynamic batching, concurrent model execution, and sequence batching.
  • Extensibility: Provides a Backend API for custom backends and supports model pipelines via Ensembling or Business Logic Scripting (BLS).
  • Standardized protocols: Uses HTTP/REST and gRPC based on the KServe protocol.

Related

  • Project
    triton-inference-server/backend

    A framework and set of utilities for creating backends that allow Triton Inference Server to execute models from various deep-learning frameworks or custom C/C++ logic.

  • Project
    InfiniTensor/InfiniTensor

    InfiniTensor is an open‑source C++ runtime that imports ONNX graphs, applies generic graph rewrites, and executes them on CPU or on custom accelerator back‑ends (InfiniOps/InfiniRT). It offers Python bindings, a test suite, and a modular design that lets hardware vendors plug in their own kernels.

  • Project
    openvinotoolkit/openvino

    An open-source toolkit for optimizing and deploying deep learning models across CPU, GPU, and NPU hardware to boost inference performance.

  • Project
    SeldonIO/MLServer

    An open-source inference server that provides a standardized REST and gRPC interface for serving machine learning models across various frameworks.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.