Mesh LLM: Distributed AI Computing via Iroh
Mesh LLM: Distributed AI Computing via Iroh
Mesh LLM is a distributed computing framework that pools available GPU and memory resources across multiple machines to run large language models (LLMs). By leveraging the Iroh networking library, it allows users to run models that are too large for any single machine's VRAM by splitting the workload across a mesh of peers, exposing the entire cluster as a single OpenAI-compatible API at localhost:9337/v1.
Distributed Model Inference Architecture
Mesh LLM enables three primary methods of serving model requests to ensure flexibility and resource optimization:
- Local Execution: The request is processed on the local machine's GPU.
- Peer Routing: The request is routed to a peer in the mesh that already has the required model loaded.
- Pipeline Splitting (Skippy): For models too large for any single node, Mesh LLM uses a "split mode" called "Skippy." This partitions a model by layer ranges into stages (e.g., layers 0-15 on one node, 16-31 on the next). Activations flow sequentially from one stage to the next across the network.
The system supports a catalog of over 40 models, ranging from small models fitting on laptops to mixture-of-experts (MoE) giants up to 235B parameters.
Networking and Transport via Iroh
Mesh LLM utilizes Iroh to handle the complexities of peer-to-peer networking, including NAT traversal, hole-punching, and relay fallbacks. Each node is identified by a public key, and all communications occur over authenticated QUIC connections.
Protocol Specification
The system uses QUIC's ALPN negotiation to separate traffic into three distinct channels:
mesh-llm/1: Handles the main mesh operations, including gossip, routing, HTTP tunnels, and plugin channels.mesh-llm-control/1: Manages the owner control plane for configuration synchronization and ownership attestation.skippy-stage/2: A dedicated channel for latency-sensitive activation transport used in split models.
Within the mesh-llm/1 connection, bidirectional QUIC streams are demultiplexed using a leading byte to identify the stream type:
| Byte | Stream Type | Description |
|---|---|---|
| 0x01 | GOSSIP | Peer announcements (models, GPU, RTT, capabilities) |
| 0x04 | TUNNEL_HTTP | Inference requests proxied to a peer |
| 0x05 | ROUTE_REQUEST | Queries regarding hosted models |
| 0x06 | PEER_DOWN | Dead-peer notifications |
| 0x07 | PEER_LEAVING | Graceful shutdown |
| 0x08 | PLUGIN_CHANNEL | Plugin RPC |
| 0x0e | DIRECT_PATH_REQUEST | Direct address sharing for NAT traversal |
Deployment and Accessibility
Mesh LLM is distributed as lightweight software (approximately 18 MB) that can be configured for either public mesh participation or private deployments. To ensure global connectivity, the project maintains two Iroh relays in different regions to provide fallback paths for nodes that cannot establish direct connections.
Future developments include a mobile application built on Iroh's Swift SDK and integration with ACP (the emerging agent standard) to allow more diverse clients to join the mesh.
Community Insights and Performance
Community discussions highlight both the ease of deployment and the inherent challenges of distributed inference over consumer networks.
Setup and Usability
Users have reported a seamless onboarding experience, noting that the mesh-llm --auto command simplifies the process of joining a swarm, downloading models from peers, and serving content.
Performance Trade-offs
While network latency is a primary concern for distributed LLMs, some performance data exists for larger models. For example, the Qwen 235B MoE model has been proven to run at 16 tokens per second across two nodes. However, some users remain skeptical about the viability of interactive use over standard consumer internet connections compared to local VRAM or system RAM.
Security and Privacy
Concerns have been raised regarding data privacy in a distributed mesh, specifically whether the host node processing a request can see the user's data. While Iroh provides secure transport, the data processed by the node is accessible to the host of that node.