scouzi1966/maclocal-api

AFM – Local AI Infrastructure for Apple Silicon

What it is – AFM (Apple Silicon Foundation Models) is a native Swift‑based server and CLI that turns an Apple‑silicon Mac into a private, OpenAI‑compatible inference endpoint. It can run on‑device Apple Foundation Models, Hugging Face MLX models, or DwarfStar checkpoints, exposing chat, completions, embeddings, vision‑OCR, speech‑to‑text, text‑to‑speech and image‑generation APIs.

Key capabilities

• OpenAI‑compatible HTTP API – /v1/chat/completions, /v1/completions, /v1/embeddings, vision, audio, image, batch and metrics endpoints. Works with any client that expects an OpenAI server (e.g., OpenCode, Aider, Continue).
• Native Swift execution – No Python runtime; the server and CLI are compiled Swift binaries that run directly on macOS.
• Multiple runtimes –
  • MLX for community Hugging Face models (quantised, VLMs, agents).
  • Apple Foundation Models – zero‑download system model with LoRA adapters.
  • DwarfStar – Metal‑accelerated checkpoints.
  • Gateway – proxy to other local servers (Ollama, LM Studio, etc.).
• Advanced inference features – prefix‑caching, speculative decoding (MTP, Eagle3), concurrent decode queues, token‑level logprobs, structured‑output enforcement, tool‑calling formats, reasoning extraction, and detailed Prometheus metrics.
• Rich CLI / TUI – Full‑screen terminal chat with markdown, syntax‑highlighted code, LaTeX rendering, session history, transcript export, and interactive overlays.
• Web UI – A bundled browser‑based UI for quick testing.
• Packaging – Distributed via Homebrew, PyPI wheels, and a afm binary; also available as Swift packages (AFMKit*) for embedding inference directly into other macOS apps.

Typical workflow

1. Install with Homebrew (brew install scouzi1966/afm/afm) or pip (pip install macafm).
2. Start a model, e.g. afm mlx -m Qwen3-0.6B-4bit -w (Web UI) or afm --tui for terminal chat.
3. The server listens on http://127.0.0.1:9999/v1. Point any OpenAI‑compatible client at that URL with a dummy API key.
4. Use the CLI for vision, speech, or embeddings (afm vision …, afm speech transcribe …).
5. For production‑style usage, configure concurrency, caching, or speculative decoding via CLI flags.

Who it’s for

• Developers building AI agents or tools that need low‑latency, on‑device inference.
• Privacy‑conscious users who want a local LLM server without cloud keys.
• Swift/macOS developers who want to embed a model directly via the AFMKit Swift packages.

Where to learn more

• Official site: https://maclocal.ai
• Docs: https://maclocal.ai/docs (client guides, tool‑calling, performance tuning, roadmap)
• Source: https://github.com/scouzi1966/maclocal-api (includes build script, tests, and the AFMKit dependency).

---

All details are taken directly from the repository’s README.