eliasbiondo/antirion

🕷️ The fastest open-source AI Gateway and LLM Gateway — one binary with built-in console to route, secure, budget and observe every LLM call across OpenAI, Anthropic, Gemini, Bedrock and more.

Antirion is an open-source AI gateway: a single program that sits between your applications and AI model providers (like OpenAI, Anthropic, or Google). It gives you one common API to talk to many different providers, and adds features like smart routing, cost control, safety checks, and monitoring — all in one lightweight Rust binary that includes a built-in web console for administration.

Key features (from the README):

  • Speed: Claims sub-millisecond overhead and high throughput, with benchmarks against Bifrost and LiteLLM showing it faster and more resource-efficient.
  • Provider support: Works with OpenAI, Azure, Anthropic, Bedrock, Vertex, Gemini, Mistral, Cohere, Groq, and many others, plus any OpenAI- or Anthropic-compatible server. It translates between different API dialects (e.g., an OpenAI client can call a Claude model).
  • Routing & resilience: Fallback chains, weighted splits, least-cost/latency/busy strategies, retries, circuit breakers, and more.
  • Cost control: Rate limits, budgets in dollars or tokens, per-model pricing, a spend ledger, and alerts.
  • Safety: Guardrails for PII, secrets, keywords, JSON schemas, moderation, and prompt-injection detection, with actions like blocking or redacting.
  • Observability: Request log with full details, analytics, and caching (exact and semantic).

Getting started is simple: run a Docker container, open the console, paste an admin token, add a provider (e.g., OpenAI) with an API key, add a model, create an API key, and send your first request. No config file needed for basic use.

Deployment options: Docker (distroless image), Docker Compose (standalone or cluster), Kubernetes (Helm chart), prebuilt binaries for Linux, or building from source. It can run as a single node with an embedded store, or as a cluster using PostgreSQL and Valkey.

The project is licensed under Apache-2.0 and written in Rust. The README includes detailed performance benchmarks and reproduction steps.

Related

  • Project
    microsoft/TransformerCompression

    A post-training compression tool that makes transformer networks smaller and faster by slicing off insignificant weight matrix components using orthogonal transformations.

  • Project
    john-rocky/coreai-model-zoo

    CoreAI‑Model‑Zoo provides ready‑to‑download `.aimodel` bundles (LLMs, multimodal models) for Apple’s Core AI runtime (iOS 27/macOS 27). It includes conversion recipes, verification scripts, Swift integration via CoreAIKit, and community‑contributed ports, letting developers run sophisticated models locally on iPhone, iPad or Mac.

  • Project
    JuliaGPU/Metal.jl

    Metal.jl is a Julia package that enables GPU programming on macOS by providing interfaces to Apple's Metal framework, allowing for high-performance array operations and custom kernel execution on M-series chips.

  • Project
    Yinsongxu/LLM2Jev

    A framework that converts local LLMs and VLMs into structured decision models by extracting probabilities from logits during prefill, bypassing token-by-token decoding for lower latency.