Mesh LLM: Peer‑to‑Peer Distributed LLM Inference on iroh

Mesh LLM lets you turn a collection of personal GPUs into a single OpenAI‑compatible inference service

Mesh LLM pools the GPUs and RAM you already own—laptops, desktop rigs, mini‑PCs, servers, or cloud instances—into a mesh that presents a single localhost:9337/v1 endpoint. The mesh decides whether a request runs locally, is forwarded to a peer that already has the model loaded, or is split across several nodes for models that exceed any single device’s memory.

Why a mesh matters: control, cost and privacy

Control – You keep the model version, hardware, and data in‑house. No external API can silently upgrade the model or change its privacy policy.
Cost – By sharing idle GPU memory across a team, you can run larger models without buying a single monstrous GPU.
Privacy – All traffic travels over iroh’s authenticated QUIC connections, so the mesh does not expose a public API endpoint that third parties can scrape.

Architecture at a glance

iroh endpoint per node – Each node’s identity is a public‑key pair. iroh handles NAT traversal, hole‑punching and fallback relays, providing a direct, authenticated QUIC link between any two peers.
Three ALPN protocols –
- mesh-llm/1 – Main mesh (gossip, routing, HTTP tunnels, plugin channels)
- mesh-llm-control/1 – Owner control plane (config sync, ownership attestation)
- skippy-stage/2 – Low‑latency activation transport for split models
Stream multiplexing – Inside mesh-llm/1, every QUIC stream begins with a single byte that identifies its purpose (gossip, HTTP tunnel, route query, peer lifecycle, plugin RPC, etc.).
Plugin manifest – Plugins declare the models they serve and the resources they expose. The runtime loads plugins, routes calls, and advertises capabilities via the gossip layer.
Model catalog – Over 40 models ship with the runtime, ranging from 0.5 B‑parameter models that fit on a laptop to 235 B‑parameter mixture‑of‑experts (MoE) models.

Splitting gigantic models with "Skippy"

When a model cannot fit on any single GPU, Mesh LLM uses a pipeline parallelism engine called Skippy. The model is partitioned by layer ranges (e.g., layers 0‑15 on node A, 16‑31 on node B, …). Activations flow from one stage to the next over a dedicated skippy-stage/2 QUIC stream. This enables inference on models such as the 235 B‑parameter MoE, which the project reports can generate 16 tokens per second across two nodes—fast enough for many non‑interactive workloads.

Real‑world experience from early adopters

"I ran mesh-llm --auto on my MacBook Pro, and it worked on the first try. The UI let me pick a model, download it from peers, and I could see the GPU working in real time." – maccam912

"The lack of performance data is concerning; consumer networks are orders of magnitude slower than local RAM. The project lists 16 tok/s for a 235 B MoE across two nodes, which is respectable but not interactive." – SwellJoe

"I tried to get Mesh LLM running on an older GPU, but the bundled llama.cpp builds failed. It seems the system can proxy an external llama.cpp service, but the integration still has rough edges." – kennywinker

These comments highlight both the ease of onboarding (a single --auto command) and the current limitations: performance depends heavily on network bandwidth and hardware compatibility.

Security and privacy considerations

Encryption – All QUIC streams are encrypted and authenticated via the node’s public key, preventing passive eavesdropping.
Access control – The mesh’s gossip layer enforces admission policies; only trusted peers (identified by public keys) can join a private mesh.
Data exposure – While traffic is encrypted, the host node that processes a request can still see the prompt and response. Mesh LLM does not provide end‑to‑end encryption of user data beyond the transport layer.

Comparison to other distributed inference projects

AI Horde – A large‑scale volunteer inference network that uses KoboldCPP and focuses on text generation. Mesh LLM differentiates itself by offering a standard OpenAI‑compatible API, built‑in model catalog, and QUIC‑based mesh rather than HTTP‑only workers.
Exo, query.mt, and other libp2p‑based meshes – These projects also explore peer‑to‑peer inference, but Mesh LLM’s tight integration with iroh’s NAT‑traversing QUIC and its plugin architecture provide a more turnkey experience.

Getting started

Install the lightweight binary (≈ 18 MB).
Run mesh-llm --auto to join the public mesh or configure a private mesh via a manifest.
Point any OpenAI client to http://localhost:9337/v1.
Optionally, explore the mobile Swift SDK for iOS devices.

The source code lives at https://github.com/Mesh-LLM/mesh-llm, and the underlying iroh library is documented at https://iroh.computer/docs.

Outlook

Mesh LLM demonstrates that peer‑to‑peer QUIC meshes can make distributed LLM inference practical for teams that already own GPU hardware. While network latency remains a bottleneck for interactive use, the ability to run multi‑hundred‑billion‑parameter models on modest hardware without a central provider is a compelling step toward more open, cost‑effective AI.