higress-group/higress

🤖 AI Gateway | AI Native API Gateway

What it solves

Higress is an AI-native API gateway designed to simplify the management of Large Language Model (LLM) APIs and Model Context Protocol (MCP) servers. It addresses the challenges of connecting to multiple model providers using a unified protocol while providing production-grade traffic management, security, and observability for AI-driven applications.

How it works

Built on Istio and Envoy, Higress uses a Wasm (WebAssembly) plugin mechanism to extend its functionality. This allows it to act as a unified entry point for LLM APIs and MCP servers, providing a layer of abstraction that handles authentication, rate limiting, and routing. It supports true streaming processing for request and response bodies, which is significantly reduces memory overhead for high-bandwidth AI services like those using Server-Sent Events (SSE).

Who it’s for

It is intended for developers and organizations building AI agents and LLM-powered applications who need a secure, scalable, and observable gateway to manage their model providers and tool-calling services.

Highlights

  • AI Gateway Capabilities: Unified protocol for all mainstream model providers, multi-model load balancing, token rate limiting, and caching.
  • MCP Server Hosting: Hosts MCP servers via plugins, allowing AI agents to call tools and services; includes a tool to convert OpenAPI specifications into remote MCP servers.
  • Wasm Plugin System: Supports plugins written in Go, Rust, or JS for memory-safe, hot-updated gateway logic without traffic loss.
  • Cloud Native: CNCF Sandbox project compatible with Kubernetes Ingress and Gateway API standards.
  • Production Grade: Eliminates traffic jitter during configuration changes, making it highly suitable for long-connection AI businesses.

Related

  • Project
    hatchet-dev/hatchet

    Hatchet is an open‑source orchestration engine that lets you run reliable background tasks, AI‑agent pipelines and durable workflows. It stores all state in PostgreSQL, provides retries, scheduling, DAGs, rate‑limiting, priority queues and a real‑time UI, and offers SDKs for Python, TypeScript, Go and Ruby. You can use it as a managed cloud service or self‑host it via Docker.

  • Dispatch
    HUGS launch: zero‑configuration, hardware‑optimized inference for open LLMs

    Hugging Face launched HUGS, a zero‑configuration, hardware‑optimized inference service for open‑source LLMs that runs on NVIDIA, AMD, and soon AWS Inferentia and Google TPUs, enabling enterprises to host models in‑house with an OpenAI‑compatible API.

  • Project
    microsoft/mcp-gateway

    MCP Gateway is an open‑source Kubernetes‑native reverse proxy and management layer for Model Context Protocol (MCP) servers. It provides a REST control‑plane for creating, updating, and monitoring MCP adapters and tools, session‑aware routing, Azure Entra ID RBAC, a built‑in React portal, and optional LLM‑driven agents. Deploy locally with Docker/K8s or with a one‑click Azure template.

  • Project
    DevYangJC/Argus

    Argus is an open‑source, enterprise‑grade RAG platform that ingests private documents, builds both vector and keyword indexes, and answers questions with a LLM that must cite real sources. It includes a Spring AI‑based ReactAgent for chat, hybrid retrieval with RRF fusion, multi‑level evidence scoring, role‑based security, and a Vue 3 front‑end.

  • Project
    yuruotong1/autoMate

    autoMate is a locally hosted service that gives AI chat clients persistent personal storage (notes, files, reminders, long‑term memory) and a toolbox of real‑world actions (shell, browser, SaaS APIs). It talks to clients via an MCP‑over‑HTTP bridge, works with OpenClaw, Claude Desktop, Cursor, Cline, etc., and can also be used through its own web chat, Chrome extension, Docker container, or binary.