smg-project/smg

What it solves

SMG (Shepherd Model Gateway) solves the complexity of managing large-scale LLM deployments. It eliminates the need for developers to manage multiple disparate endpoints for different model engines or cloud providers, while optimizing GPU utilization and providing enterprise-grade control over traffic, security, and observability.

How it works

SMG acts as a high-performance, engine-agnostic routing gateway. It provides a single unified endpoint for clients, then routes requests to a variety of backends—including self-hosted engines (like vLLM, SGLang, and TensorRT-LLM) and cloud providers (OpenAI, Anthropic, Gemini). It uses native Rust and streaming gRPC pipelines to minimize latency. To maximize efficiency, it employs cache-aware routing that tracks KV-cache states in radix trees to reuse prefixes across workers.

Who it’s for

It is designed for organizations deploying LLMs at scale who need to balance traffic across hybrid environments (cloud and self-hosted), implement strict multi-tenancy and priority scheduling, and maintain full observability over their inference infrastructure.

Highlights

• Unified API: Supports OpenAI, Anthropic, Gemini, and xAI, as well as any OpenAI-compatible endpoint.
• Cache-Aware Routing: Tracks KV-cache state to optimize GPU utilization across multiple workers.
• Enterprise Control: Includes priority admission scheduling, OIDC authentication, and pluggable chat history storage.
• C-level Performance: Built in Rust with zero-copy cache hits and prefill/decode disaggregation.
• Extensibility: Supports WebAssembly (WASM) plugins for custom logic and Model Context Protocol (MCP) integration for tool execution.
• Observability: Provides over 90 Prometheus metrics and OpenTelemetry tracing.