lithos-ai/lithos-metal

lithos-metal is a fully open-source inference engine that runs large language models (like Qwen3.8-27B) efficiently on Apple silicon Macs. It generates specialized Metal GPU code ("megakernels") that fuses many small computation steps into larger, combined operations to cut overhead.

Key ideas

• Megakernels & layer-wise fusion. Instead of launching hundreds of tiny GPU kernels per token, the engine compiles an entire layer's mixer (attention or Gated DeltaNet) into a single fused GPU program. Persistent GPU workers (threadgroups) execute a sequence of tasks, reusing intermediate data and reducing memory traffic.
• DSpark speculative decoding. A small "draft" model proposes a block of up to 7 tokens; the main model verifies the whole block in one forward pass. When several tokens are accepted, one expensive model pass yields multiple output tokens.
• GPU-resident control. Token positions, accepted tokens, and state commit/rollback are tracked on the GPU itself, minimizing CPU round-trips.
• Apple-specific optimization. Uses M3+/M5 features like dynamic on-chip memory, inline tensor operations, and tuned tile geometry to maximize throughput.

Serving & integrations

It ships as a Homebrew package and a lithos-metal serve command that exposes an OpenAI-compatible API (Chat Completions, Responses, Anthropic Messages, Ollama). Companion launchers (lithos-metal opencode, claude, codex, hermes) let you point coding agents at the local server without touching their global config.

Hardware & performance

Targets M3/M4/M5 Macs; the flagship case study is a 40-core M5 Max (48 GB) running Qwen3.8-27B-NVFP4 at roughly 168 tokens/s at 128-token context, dropping to ~127 tokens/s at 32K context (projection numbers excluding draft/verification overhead).

Status: Apache 2.0, active design docs for architecture, Apple GPU execution, megakernels, speculative decoding, and extension contracts. Standalone engine — no dependence on the MPK/Mirage runtime, though it was inspired by that work.