mesh-llm: a distributed inference system that pools GPU and memory resources across machines to run massive LLMs

What it solves

Mesh LLM allows users to pool GPU and memory resources across multiple machines to run large language models (LLMs) that would otherwise be too big for a single computer. It provides a unified, OpenAI-compatible API, making it easy to integrate with existing tools while distributing the computational load across a network of peers.

How it works

• Resource Pooling: It connects multiple nodes into a "mesh," where each node can either host a full model or act as a part of a larger distributed system.
• Intelligent Routing: Requests are routed to the peer best suited to serve the model. If a model fits on one machine, it runs locally; otherwise, it is routed to a peer.
• Skippy Stage Splits: For models too large for any single machine, the system splits the model into contiguous layer ranges (stages). Nodes fetch only the necessary GGUF fragments (layer packages) to serve their assigned stage.
• Discovery: Users can join public meshes via Nostr discovery or create private meshes using invite tokens.
• Mixture-of-Agents (MoA): An experimental feature that fans out a single prompt to every available model in the mesh, arbitrating the responses to return a single consolidated reply.

Who it’s for

• Hardware-constrained users: People who want to run massive models on consumer hardware by pooling resources with others.
• Developers: Those needing an OpenAI-compatible API for distributed LLM inference.
• Self-hosters: Users who want to deploy private or public distributed inference clusters.

Highlights

• OpenAI-compatible API: Seamlessly integrates with existing LLM applications.
• Huge Model Support: Supports a wide range of model families (Qwen, Llama, Gemma, Mistral, DeepSeek, etc.) and multimodal models.
• Flexible Deployment: Supports various backends including CUDA, ROCm, Vulkan, and Metal.
• Distributed Execution: Implements layer-based splitting for massive models across multiple nodes.
• Experimental MoA Gateway: Parallel execution across multiple models with deterministic arbitration.