triton-inference-server/server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

What it solves

Triton Inference Server 通过提供标准化的方式，使模型能够在各种硬件平台（云端、数据中心、边缘和嵌入式设备）以及多种框架上进行服务，从而简化 AI 模型的部署。它消除了在生产环境中为每种不同模型类型或框架编写自定义服务基础设施的需求。

How it works

Triton 充当服务层，支持多种后端（如 TensorRT、PyTorch、ONNX、OpenVINO 和 Python）。用户只需将模型放入模型仓库并进行配置，即可获得优化的性能。它提供 HTTP/REST 和 gRPC 协议，供客户端发送推理请求并接收响应，也可以通过 C 和 Java API 直接链接到应用程序，以用于进程内使用场景。

Who it’s for

它面向需要大规模部署生产级 AI 模型的 AI 团队和开发者，支持广泛的硬件（NVIDIA GPU、x86/ARM CPU、AWS Inferentia）以及多种深度学习框架。

Highlights

  • Multi-framework support: Serves models from TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL.
  • Optimized performance: Features dynamic batching, sequence batching, and concurrent model execution to maximize throughput and minimize latency.
  • Flexible deployment: Supports cloud, data center, edge, and embedded devices.
  • Extensible architecture: Provides a Backend API for adding custom backends and pre/post processing operations.
  • Model pipelining: Enables complex workflows using Ensembling or Business Logic Scripting (BLS).
  • Integrated metrics: Provides built-in metrics for GPU utilization, server throughput, and latency.