mostlygeek/llama-swap

What it solves

Running multiple generative AI models locally often requires manually starting and stopping different inference servers, which is cumbersome and resource-intensive. This project automates the management of these servers, allowing you to run various models (LLMs, image generators, audio processors) and hot-swap between them on demand without manual intervention.

How it works

It acts as a lightweight, Go-based reverse proxy that sits in front of your local inference servers (such as llama.cpp, vllm, or ComfyUI). When you send an API request to a specific model, the tool detects the requested model ID, automatically loads the corresponding server if it isn't already running, and routes the request. It supports OpenAI and Anthropic API-compatible endpoints, allowing it to integrate seamlessly with existing AI tools and workflows.

Who it’s for

It is designed for developers and power users who run multiple local AI models on their own hardware and want a unified, automated interface to manage their inference stack without needing to manually juggle processes.

Highlights

• Automatic Model Switching: Dynamically loads and unloads models based on incoming API requests.
• Broad Compatibility: Supports OpenAI and Anthropic API endpoints, plus specialized support for llama.cpp, Stable Diffusion, and audio processing servers.
• Resource Management: Includes features like TTL-based automatic unloading, concurrent model matrices, and custom request filtering.
• Built-in Monitoring: Provides a web UI for testing, real-time log streaming, and detailed token/system metrics.
• Easy Deployment: Distributed as a single binary with minimal configuration, also available as a unified Docker image for various GPU backends.