exo-explore/exo

What it solves

exo allows users to run large frontier AI models locally by clustering multiple devices together. This solves the problem of limited VRAM and compute on a single machine, enabling the execution of models that are too large to fit on one device while increasing speed as more hardware is added.

How it works

exo connects multiple devices into a unified AI cluster using automatic device discovery. It uses MLX as an inference backend and MLX distributed for communication. To optimize performance, it employs topology-aware auto-parallelism to split models across devices based on real-time network latency and bandwidth. It also supports tensor parallelism for speedups and leverages RDMA over Thunderbolt 5 to drastically reduce latency between nodes.

Who it’s for

It is designed for users with multiple Apple Silicon Macs or Linux machines who want to run massive LLMs (like DeepSeek v3.1 or Qwen3) locally without relying on cloud providers.

Highlights

• RDMA over Thunderbolt 5: Reduces inter-device latency by up to 99%.
• Automatic Discovery: Devices find each other on the network without manual configuration.
• Topology-Aware Parallelism: Automatically determines the most efficient way to shard models based on hardware resources and network links.
• Broad API Compatibility: Supports OpenAI, Claude, and Ollama APIs, allowing it to work with existing AI tools.
• Custom Model Support: Ability to load any model from the HuggingFace hub.