exo-explore/exo

exo – Distributed AI inference for Apple‑silicon (and Linux) clusters

What it is – exo (by Exo Labs) is a locally‑run platform that lets you stitch together multiple Macs (or Linux machines) into a single AI “cluster”. It automatically discovers the devices, shares model shards over the network, and exposes the whole thing through familiar OpenAI‑compatible APIs and a web dashboard. The core inference engine is MLX, Apple’s Metal‑based ML library, and the system adds its own topology‑aware parallelism and RDMA over Thunderbolt 5 to cut inter‑node latency.

Key capabilities (as described in the README)

Feature	What it does
Automatic device discovery	Nodes running exo find each other on the LAN without manual config.
RDMA over Thunderbolt 5	Day‑0 support for Remote Direct Memory Access via Thunderbolt 5, claimed to reduce inter‑node latency by ~99 %.
Topology‑aware auto‑parallel	exo inspects each node’s resources and network bandwidth/latency, then decides how to split a model across the cluster.
Tensor parallelism	Shards a model across devices, giving ~1.8× speed‑up on 2 nodes and ~3.2× on 4 nodes (per the README).
MLX backend	Uses the MLX library for inference and MLX‑distributed for communication.
Multiple API compatibilities	Exposes OpenAI Chat Completions, Claude Messages, OpenAI Responses, and Ollama APIs, so existing clients work out‑of‑the‑box.
Custom model loading	Pulls any HuggingFace model (including 4‑bit/8‑bit quantised variants) and runs it on the cluster.
Dashboard	A built‑in web UI (http://localhost:52415) shows cluster status, model instances, and lets you chat with models.
Cross‑platform	Provides a macOS app (via Homebrew cask or DMG) and source builds for macOS and Linux (CPU‑only on Linux for now).

Quick start summary

Install – macOS users can brew install --cask exo or download the DMG; Linux/macOS source builds require uv, node, rust, and (on macOS) Xcode for the Metal toolchain.
Run – uv run exo starts the coordinator, dashboard, and API on port 52415. The app automatically discovers other machines running exo.
Enable RDMA (macOS 26.2+) – Reboot into Recovery, run rdma_ctl enable, then reboot.
Deploy a model – Use the dashboard or the REST endpoints (/instance/previews, /instance, /v1/chat/completions, etc.) to place a model on the cluster and start chatting.

Example API flow (from the README)

# 1. See where a model could be placed
curl "http://localhost:52415/instance/previews?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"

# 2. Create an instance (use the JSON payload from step 1)
curl -X POST http://localhost:52415/instance \
  -H 'Content-Type: application/json' -d '{"instance": {...}}'

# 3. Wait until the instance is ready
curl -N "http://localhost:52415/instance/await?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"

# 4. Chat (OpenAI‑compatible endpoint)
curl -N -X POST http://localhost:52415/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"mlx-community/Llama-3.2-1B-Instruct-4bit","messages":[{"role":"user","content":"What is Llama 3.2 1B?"}],"stream":true}'

# 5. Delete when done
curl -X DELETE http://localhost:52415/instance/<INSTANCE_ID>

Benchmarks shown in the repo

The README includes three benchmark screenshots (Qwen‑3 235B, DeepSeek v3.1 671B, and Kimi K2 Thinking) run on four M3 Ultra Mac Studios with Tensor Parallelism and RDMA, sourced from a Jeff Geerling blog post. They illustrate multi‑TB VRAM usage and multi‑fold speed‑ups, but the repo does not provide raw numbers in text.

Installation notes & platform quirks

macOS – Requires macOS 26.2 (or later) for RDMA support and a Thunderbolt 5‑capable Mac (M4 Pro/Max, M3 Ultra, etc.). The app installs a network profile and may need user permission to modify system settings.
Linux – Currently CPU‑only; GPU support is “under development”. The README points users to the --no-worker flag for nodes that should only coordinate traffic.
Model storage – Configurable via EXO_DEFAULT_MODELS_DIR, EXO_MODELS_DIRS, and EXO_MODELS_READ_ONLY_DIRS. Supports offline mode (EXO_OFFLINE=true).
Namespace isolation – Set EXO_ZENOH_NAMESPACE (or the --namespace CLI flag) to run multiple independent clusters on the same LAN.

Who might use this?

Researchers or hobbyists with several Apple‑silicon machines who want to run very large LLMs (hundreds of billions of parameters) locally.
Developers needing an OpenAI‑compatible endpoint that runs on‑premise without cloud costs.
Teams that want to experiment with low‑latency inter‑node communication via Thunderbolt 5.

Bottom line

exo is a genuine, open‑source project aimed at distributed inference of frontier‑size language models on local hardware, leveraging Apple’s MLX stack and Thunderbolt‑5 RDMA. The repository provides a full stack – from low‑level device discovery to a ready‑to‑use HTTP API and UI – and is clearly positioned as a tool for on‑premise AI workloads rather than a simple tutorial or list.