lightseekorg/smg

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

What it solves

Shepherd Model Gateway (SMG) simplifies the deployment of large-scale LLM infrastructures by providing a centralized routing layer. It eliminates the need to manage multiple disparate endpoints for different models and providers, while solving common production challenges like GPU underutilization, traffic spikes, and the lack of unified observability across hybrid (cloud and self-hosted) environments.

How it works

SMG acts as an engine-agnostic gateway that sits between the user and the inference workers. It routes requests to various backends—including self-hosted engines like vLLM and SGLang or cloud providers like OpenAI and Anthropic—using a single unified API. To optimize performance, it employs cache-aware routing to reuse KV cache prefixes across workers, reducing redundant computation. The system is built in Rust for high performance and uses gRPC pipelines for streaming and tool parsing, with a high-availability mesh network based on the SWIM protocol for multi-node scaling.

Who it’s for

It is designed for engineers and organizations deploying LLMs at scale who need a high-performance, enterprise-ready way to balance traffic, manage worker lifecycles, and maintain strict control over privacy and history storage.

Highlights

  • Cache-Aware Routing: Optimizes GPU utilization by understanding the KV cache state of the underlying inference engine.
  • Unified API: Provides a single endpoint for OpenAI, Anthropic, Gemini, Bedrock, and various self-hosted backends.
  • Enterprise-Grade Control: Includes multi-tenant rate limiting, OIDC authentication, and WebAssembly (WASM) plugins for custom logic.
  • High Availability: Features a mesh networking architecture with circuit breakers and automatic failover.
  • Pluggable Storage: Supports multiple backends for chat history, including PostgreSQL, Oracle, and Redis.