triton-inference-server/server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
What it solves
Triton Inference Server 透過提供一套標準化的方式，讓模型能在各種硬體平台（雲端、資料中心、邊緣與嵌入式裝置）以及多種框架上服務，簡化了 AI 模型的部署。它消除了在生產環境中為每種不同模型類型或框架撰寫自訂服務基礎設施的需求。
How it works
Triton 作為服務層，支援多種後端（如 TensorRT、PyTorch、ONNX、OpenVINO 與 Python）。使用者只需將模型放入模型倉庫並進行設定，即可獲得最佳化效能。它提供 HTTP/REST 與 gRPC 協定，讓客戶端送出推論請求並接收回應，亦可透過 C 與 Java API 直接嵌入應用程式以供內部使用。
Who it’s for
此工具設計給需要大規模部署生產等級 AI 模型的 AI 團隊與開發者，支援廣泛的硬體（NVIDIA GPU、x86/ARM CPU、AWS Inferentia）與多種深度學習框架。
Highlights
- Multi-framework support: Serves models from TensorRT, PyTorch, ONNX, OpenVINO, Python, and RAPIDS FIL.
- Optimized performance: Features dynamic batching, sequence batching, and concurrent model execution to maximize throughput and minimize latency.
- Flexible deployment: Supports cloud, data center, edge, and embedded devices.
- Extensible architecture: Provides a Backend API for adding custom backends and pre/post processing operations.
- Model pipelining: Enables complex workflows using Ensembling or Business Logic Scripting (BLS).
- Integrated metrics: Provides built-in metrics for GPU utilization, server throughput, and latency.