triton-inference-server/backend

What it solves

This project provides the infrastructure and documentation for creating "backends" for the Triton Inference Server. A backend is the actual implementation that executes a model, allowing Triton to support a wide variety of deep-learning frameworks (like PyTorch, TensorFlow, and ONNX Runtime) or custom C/C++ logic for operations such as image pre-processing.

How it works

Triton uses a standardized C interface (the Triton Backend API) to communicate with backends. Each backend is implemented as a shared library (named libtriton_<backend-name>.so) that Triton loads at runtime.

The API defines several key abstractions:

• Backend: The overall implementation shared across all models using that backend.
• Model: Represents a specific loaded model.
• Model Instance: An execution unit of a model; Triton can create multiple instances to handle requests.
• Request/Response: Objects used to pass input tensors to the backend and return output tensors to Triton.

Who it’s for

• ML Engineers: Who need to deploy models from different frameworks into a single production environment.
• Backend Developers: Who want to build custom C/C++ or Python logic to execute specific model operations or pre-processing steps within the Triton ecosystem.

Highlights

• Framework Agnostic: Supports a vast array of frameworks including TensorRT, ONNX Runtime, TensorFlow, PyTorch, OpenVINO, and vLLM.
• Extensible: Allows developers to create their own custom backends via a C interface.
• Flexible Deployment: Backends can be installed globally or bundled with specific model versions.
• Custom Logic: Supports the Python backend for writing model logic in Python without needing to convert to TorchScript or other formats.