Mesh-LLM/mesh-llm

Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.

What it solves

Mesh LLM allows users to pool GPU and memory resources across multiple machines to run large language models (LLMs) that would otherwise be too large for a single computer. It provides a unified, OpenAI-compatible API, making it easy to integrate with existing AI tools while distributing the computational load across a network of peers.

How it works

  • Distributed Routing: Every node in the mesh network exposes the same API. Requests are routed to the peer that can host the specific model requested.
  • Skippy Stage Splits: For models that exceed the capacity of any single machine, the system uses "stage splits," where the model is divided into contiguous layer ranges and distributed across multiple nodes. Each node processes its part of the model and passes the activations to the next stage.
  • Encrypted Transport: Traffic between nodes, including inference requests and model activations, is end-to-end encrypted using QUIC.
  • Discovery: Nodes can join public meshes via Nostr discovery or private meshes using invite tokens.
  • Mixture-of-Agents (MoA): An experimental feature that fans out a single prompt to every model in the mesh, arbitrates the responses, and returns a single consolidated reply.

Who it’s for

  • Hardware-constrained users: People who want to run massive models on consumer-grade hardware by pooling resources.
  • Developers: Those needing an OpenAI-compatible endpoint for distributed LLM serving.
  • System Operators: Users who want to create private or public distributed inference clusters.

Highlights

  • OpenAI-compatible API: Seamless integration with existing LLM applications.
  • Broad Model Support: Supports a wide range of GGUF-based model families (Qwen, Llama, Mistral, DeepSeek, etc.).
  • Cross-Platform: Available for macOS, Linux, and Windows (with CUDA, ROCm, Vulkan, and Metal support).
  • Dynamic Scaling: Start with one node and add more nodes to the mesh as needed.

Related

  • Dispatch
    Mesh LLM: Peer‑to‑Peer Distributed LLM Inference on iroh

    Mesh LLM lets you pool GPUs across laptops, workstations and servers into a single OpenAI‑compatible inference API using iroh’s authenticated QUIC mesh, eliminating reliance on centralized AI providers.

  • Project
    b4rtaz/distributed-llama

    A distributed inference engine that clusters home devices over Ethernet to share RAM and accelerate LLM performance using tensor parallelism.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.

  • Project
    hpcaitech/ColossalAI

    A distributed deep learning framework that makes training and inference for large AI models faster and cheaper through advanced parallelism and memory management.

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.