ellipticmarketing/modelrelay

What it solves

modelrelay is an OpenAI-compatible local router designed to help developers avoid paying for API usage by routing requests to the best available free coding models across various providers. It eliminates the need to manually switch between different AI providers and their specific API formats by providing a single, unified interface.

How it works

The tool acts as a local proxy server (accessible at http://localhost:7352/v1) that benchmarks free models from over 12 providers (including NVIDIA, Groq, OpenRouter, and Google) and automatically forwards requests to the best performing model based on a Quality of Service (QoS) score. This score blends a model's intelligence percentile with its recently observed average latency.

Users can route requests using specific model IDs, capability tags (e.g., tag:coding), or the auto-fastest identifier. It also supports filtering by minimum context window size (e.g., auto-fastest+min_ctx:128k).

Who it’s for

It is primarily intended for AI coding assistants and developers who want access to a wide range of high-performance free LLMs without managing multiple API keys or paying for usage fees.

Highlights

• OpenAI-Compatible: Works as a drop-in replacement for existing tools and scripts.
• Auto-Routing: Automatically selects the fastest and most capable model available.
• Extensive Model Support: Access to 80+ models from 12+ providers.
• Context Window Filtering: Ability to filter models based on required minimum context size.
• Custom Tagging: Users can assign their own routing tags to models via a Web UI.
• Flexible Installation: Available via NPM or Docker.