triton-inference-server/client

Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.

What it solves

It simplifies the process of communicating with a Triton Inference Server. Instead of manually constructing raw network requests, developers can use these standardized client libraries to perform model inferencing, check server health and status, retrieve statistics and metrics, and manage the model repository.

How it works

The project provides a set of APIs for multiple programming languages that wrap the communication protocols used by Triton. It supports two primary transport protocols:

  • HTTP/REST: A standard web-based approach for sending requests.
  • gRPC: A high-performance remote procedure call framework.

For high-performance data transfer, the libraries support using system and CUDA shared memory to pass inputs and receive outputs, reducing the overhead of copying data between the client and the server.

Who it’s for

Developers building applications that need to send data to a Triton Inference Server for AI model predictions, specifically those using C++, Python, or Java.

Highlights

  • Multi-language Support: Official libraries for C++, Python, and Java, with gRPC stubs available for many other languages via the protoc compiler.
  • High-Performance Data Transfer: Support for CUDA and system shared memory to minimize latency.
  • Secure Communication: Built-in support for SSL/TLS for both HTTP and gRPC protocols.
  • On-wire Compression: Ability to compress requests and responses to save bandwidth.
  • Extensibility: A Python Client Plugin API that allows users to add custom request headers (e.g., for authentication/authorization via gateways).
  • Async Support: Beta support for Python AsyncIO for non-blocking inference calls.

Related

  • Project
    triton-inference-server/server

    Triton Inference Server is NVIDIA’s open‑source model‑serving platform. It supports models from TensorRT, PyTorch, ONNX, OpenVINO, Python, and more, runs on GPUs, CPUs, ARM, and AWS Inferentia, and provides features like dynamic batching, sequence handling, ensembling, custom backends, HTTP/REST & gRPC APIs, and built‑in metrics. Deploy via Docker, Kubernetes, or build from source; client libraries exist for C++, Python, and Java.

  • Project
    triton-inference-server/python_backend

    A backend for Triton Inference Server that enables serving Python-based machine learning models without requiring C++ code.

  • Project
    triton-inference-server/backend

    A framework and set of utilities for creating backends that allow Triton Inference Server to execute models from various deep-learning frameworks or custom C/C++ logic.

  • Project
    hkr04/cpp-mcp

    A C++ implementation of the Model Context Protocol (MCP) that allows AI models and agents to interact with tools, resources, and services through a standardized interface.

  • Project
    SeldonIO/MLServer

    An open-source inference server that provides a standardized REST and gRPC interface for serving machine learning models across various frameworks.