Mesh-LLM/mesh-llm

What it solves

Mesh LLM addresses the hardware limitations of running large language models by allowing multiple machines to work together. It enables users to run models that are too large for a single device by distributing the workload across a network of nodes, providing a unified OpenAI-compatible API for the entire cluster.

How it works

The system uses a mesh architecture where each node exposes the same API. When a request arrives, the mesh routes it to the appropriate peer based on the model requested. If a model is too large for one machine, the system uses "Skippy stage splits" to distribute model layers across different nodes. It also features an experimental "Mixture-of-Agents" mode that can fan out a single prompt to multiple models in the mesh to arbitrate and return a single response.

Who it’s for

• Individual users wanting to run massive models by connecting multiple existing computers.
• Developers needing an OpenAI-compatible interface for distributed model inference.
• Operators looking to build private or public meshes of hardware for shared AI compute.

Highlights"

• Distributed Inference: Pools GPU and memory across multiple machines.
• OpenAI Compatibility: Exposes a standard /v1 API for easy integration with existing tools.
• Stage Splitting: Uses the Skippy runtime to split large models into manageable layers across nodes.
• Mesh Discovery: Supports public discovery via Nostr or private meshes via invite tokens.
• Multi-Model Arbitration: Experimental Mixture-of-Agents (MoA) feature to aggregate responses from multiple models.