higress-group/higress

🤖 AI Gateway | AI Native API Gateway

What it solves

Higress is an AI-native API gateway designed to handle the specific demands of AI workloads. It solves the problem of managing multiple LLM providers through a unified protocol, while providing essential production-grade controls like token rate limiting, multi-model load balancing, and AI observability. It also simplifies the deployment of AI agents by hosting Model Context Protocol (MCP) servers, allowing agents to access tools and services more reliably.

How it works

Built on Istio and Envoy, Higress uses a Wasm (WebAssembly) plugin architecture. This allows it to extend gateway logic using plugins written in Go, Rust, or JavaScript without restarting the gateway. It supports true streaming processing for request and response bodies, which is significantly reduces memory overhead for high-bandwidth AI streaming protocols like SSE (Server-Sent Events).

Who it’s for

  • AI Developers: Those building agents or LLM-powered apps who need a unified way to connect to various model providers.
  • Platform Engineers: Teams needing a production-ready Kubernetes ingress controller or microservice gateway with low resource overhead.
  • Enterprise AI Teams: Organizations requiring fine-grained security, audit logs, and rate limiting for their AI tool calls and LLM APIs.

Highlights

  • AI Gateway Capabilities: Unified protocol for LLM providers, token-based rate limiting, and inference-aware routing on Kubernetes.
  • MCP Server Hosting: Ability to host and manage remote MCP servers, including a tool to convert OpenAPI specifications into MCP servers.
  • Production Grade: Eliminates traffic jitter during configuration changes, ensuring stable long-connections for AI streaming.
  • Wasm Plugin System: Memory-safe, sandbox-isolated plugins that support hot updates without traffic loss.
  • Cloud Native: CNCF Sandbox project compatible with Kubernetes Ingress and Gateway APIs.

Related

  • Project
    theopenco/llmgateway

    LLM Gateway is an open‑source API layer that lets you call many LLM providers through a single OpenAI‑compatible endpoint, with built‑in key management, cost tracking, and performance monitoring. It can be used as a hosted SaaS or self‑hosted via Docker.

  • Project
    agentic-community/mcp-gateway-registry

    A governed control plane and gateway for AI assets, providing centralized discovery, access control, and audit trails for MCP servers, agents, and model inference endpoints.

  • Project
    Helicone/helicone

    An AI Gateway and LLM observability platform that provides unified API access to 100+ models, request tracing, and prompt management for AI engineers.

  • Project
    higress-group/himarket

    An enterprise-grade AI open platform that provides a centralized marketplace to manage, distribute, and monitor AI resources like LLMs, Agents, and MCP Servers.

  • Project
    yym68686/uni-api

    uni‑api is an open‑source, Docker‑based gateway that lets you expose a single OpenAI‑compatible API while routing requests to many different LLM back‑ends (OpenAI, Anthropic, Gemini, Vertex, Azure, AWS, etc.). It offers YAML‑only configuration, weighted/round‑robin load balancing, automatic retries, per‑model timeouts, rate limiting, moderation, and tool/image support.