mostlygeek/llama-swap

What it solves

llama-swap allows users to run multiple generative AI models on a single machine and switch between them on demand. It eliminates the need to manually start and stop different inference servers for different models, providing a unified API interface for local AI workflows.

How it works

It acts as a reverse proxy that intercepts requests to OpenAI and Anthropic compatible endpoints. When a request specifies a certain model, llama-swap extracts the model ID and automatically loads the corresponding server configuration. If the required model's server is not currently running, it swaps the active server for the correct one. For basic setups, it handles one model at a time; for advanced users, a custom DSL "matrix" allows multiple models to run concurrently.

Who it’s for

Developers and AI enthusiasts running local LLMs, image generators, and audio models who want a seamless way to manage multiple models without manually managing processes or ports.

Highlights

• Broad Compatibility: Supports OpenAI and Anthropic API endpoints, as well as specialized endpoints for llama.cpp, stable-diffusion.cpp, audio.cpp, and ComfyUI.
• Zero Dependencies: Built in Go for high performance and simplicity, requiring only a single binary and a configuration file.
• Integrated Web UI: Includes a real-time interface for testing models, monitoring token metrics, and inspecting requests/responses.
• Advanced Resource Management: Features automatic unloading of models via TTL (time-to-live) settings and support for Docker/Podman containers.
• Observability: Provides detailed log streaming (proxy, upstream, and per-model) and Prometheus metrics for system and GPU usage.