llmmanorg/llmman

What it solves

llmman is a unified interface for running AI agents and models, removing the friction of managing different inference servers, model formats, and providers. It allows users to run agents (like Claude Code or Aider) against any model, whether it is hosted locally on the user's machine or via a hosted provider, using a single command.

How it works

llmman acts as a management layer that handles model pulling, serving, and routing. It uses standard OCI (Open Container Initiative) artifacts to package and store models, allowing them to be pulled from Docker Hub, GHCR, or Hugging Face. For local inference, it leverages upstream engines like llama.cpp, vLLM, SGLang, and mlx-lm without modifying the model files. It provides a single API endpoint that is compatible with Ollama, OpenAI, and Anthropic APIs, which it then routes to the appropriate local backend or hosted provider.

Who it’s for

Developers and AI researchers who want to run various agents and models across different hardware (local or cloud) without being locked into a specific provider or registry, and those operating in air-gapped or compliance-bound environments who need secure, signed model transfers.

Highlights

• Provider and Registry Agnostic: Pull models directly from Hugging Face or any OCI registry and route requests to local or hosted providers using a single endpoint.
• One-Step Transfer: Transfer models directly between registries (e.g., Hugging Face to a private OCI registry) without storing them locally.
• Hardware Pooling (Aggregation): Pool multiple machines running llmman serve into a single logical endpoint to distribute model loads across a network.
• Unified API: A single server that speaks Ollama, OpenAI, and Anthropic APIs simultaneously.
• Cros-Model Hybrid Pairs: Automatically route requests to a hosted provider when a local model's context window is exceeded.
• Secure Model Verification: Supports cosign-format signatures to verify the authenticity of models pulled from registries.