llmmanorg/llmman

Run any agent on any model, models stored as OCI images

What is llmman?

llmman is a command‑line tool that lets you run any LLM‑style agent (Claude Code, Codex, OpenCode, GitHub Copilot, etc.) on any model, whether that model lives on your own computer or is provided by a cloud service. It does this by wrapping popular inference back‑ends (llama.cpp, vLLM, sglang, mlx‑lm, …) and exposing a single, Ollama‑compatible HTTP API. The key idea is that models are treated as OCI container images, so you can pull, push, transfer, and sign them with the same tooling you already use for Docker/Podman images.

Why it matters

Feature What it gives you
Provider‑agnostic Same launch, run, serve commands work for a locally‑hosted model or a hosted model on Groq, Baseten, OpenRouter, etc.
Registry‑agnostic Models can be stored in any OCI registry (Docker Hub, GHCR, Quay, private Harbor, etc.) or directly from Hugging Face. No proprietary model store.
Zero‑conversion Uses the original GGUF or safetensors files; no custom import step.
One‑step transfer llmman transfer hf.co/... docker.io/... moves a model between registries without ever touching the local disk – useful for air‑gapped or compliance‑heavy environments.
Aggregation Run llmman serve on several machines; they form a pool so a request is handled by whichever node has the model loaded or the most free memory.
Unified API Exposes Ollama, OpenAI and Anthropic endpoints on a single port, so any existing client (curl, LangChain, VS Code extensions, etc.) works unchanged.
Multimodal support Same run command can invoke diffusion models to generate images, short videos or audio.
Security Models can be signed with cosign; llmman verify enforces trust policies on pull.

Core concepts & workflow

  1. Models as OCI images – A model file (GGUF or safetensors) is packaged in a standard OCI Image Layout. Pulling, pushing, or transferring the model is done with ordinary container registries.
  2. Back‑ends – When a model is needed, llmman launches the appropriate inference engine:
    • llama.cpp for GGUF models (Docker, pre‑built binary, or a llama‑server on PATH)
    • vLLM, sglang or mlx‑lm for safetensors.
  3. CLI commands – The most used ones are:
    • llmman launch <agent> --model <model> – Starts an agent (e.g., Claude Code) against a model, automatically launching a local daemon if needed.
    • llmman run <model> [prompt] – Quick chat or multimodal generation.
    • llmman serve – Starts a daemon that listens on 127.0.0.1:17434 and speaks Ollama, OpenAI and Anthropic APIs.
    • llmman pull/transfer/push/verify – Registry operations.
  4. Aggregations – Configure peers with llmman config set aggregation.peers …. The daemon will forward a request to the peer that can serve it.
  5. Hybrid model pairs – Combine a cheap local model with a larger hosted model for overflow requests using --overflow-provider/--overflow-model.
  6. Sandboxes – Run an integration inside an isolated environment (Docker, macOS sandbox‑exec, etc.) via --sandbox.

Who should use it?

  • Developers who want a single tool to experiment with many agents and models without learning each provider’s CLI.
  • Ops / MLOps teams that already run OCI registries and want to treat LLMs like any other container artifact, including signing and air‑gap transfers.
  • Researchers needing to benchmark agents on different back‑ends or mix local and cloud inference.
  • Enterprises that require compliance‑friendly model handling (registry‑to‑registry transfer, cosign verification, aggregation across on‑prem hardware).

Getting started (quick)

# Install (Linux/macOS)
curl -fsSL https://llmmanorg.github.io/install.sh | sh

# Run a local model with Claude Code
llmman launch claude --model qwen3.8

# Simple chat with the same model
llmman run qwen3.8 "Explain this repository"

# Expose a compatible API for any client
llmman serve   # now use Ollama, OpenAI or Anthropic URLs

Limitations / open questions

  • Windows support relies on Docker/Podman containers or a pre‑built binary; some back‑ends may need extra toolchains (LLVM, Rust).
  • Model formats are limited to GGUF and safetensors; other formats would need a custom back‑end.
  • Performance depends on the chosen back‑end and hardware; llmman itself adds only a thin orchestration layer.
  • Security – While signing is supported, the daemon still needs to be protected (API keys, TLS) for multi‑node deployments.

TL;DR

llmman = a universal CLI & daemon that lets you run any LLM agent on any model, locally or in the cloud, using standard OCI registries and a single Ollama/OpenAI‑compatible API. It removes vendor lock‑in, simplifies model distribution, and lets multiple machines share hardware as one inference endpoint.

Related

  • Project
    ollama/ollama

    A tool to download and run open-source AI models locally on your own hardware, providing privacy, a simple CLI, a REST API, and official Python/JavaScript SDKs. It acts as a local AI server that powers coding assistants, chat interfaces, agents, and thousands of community integrations.

  • Project
    ollama/ollama-python

    A Python wrapper for the Ollama AI inference server, offering chat, generation, streaming, async calls, model management, embeddings, and a decision‑making System One API. Install via `pip install ollama` and point the client at a local or cloud Ollama instance.

  • Project
    omnimind-ai/OmniInfer

    OmniInfer is a cross‑platform inference engine that lets you run LLMs and vision‑language models locally via a simple CLI and OpenAI‑compatible HTTP server. It supports many backends (llama.cpp, MLX, MNN, etc.), works on desktop, Android and iOS, and provides hardware‑aware optimisation and a terminal UI for chatting.

  • Project
    taketwo/llm-ollama

    A Python plug‑in for the LLM CLI that discovers and talks to Ollama models (local, remote, or cloud). It adds commands for listing models, prompting, chatting, image attachments, tool use, embeddings, reasoning traces, JSON schemas, and async calls, all configurable via environment variables.

  • Dispatch
    Ollama 0.17: Simplified Setup for OpenClaw AI Assistant

    Ollama 0.17 introduces a single-command installation process for OpenClaw, a personal AI assistant capable of managing emails, calendars, and messaging apps on local hardware.