Mesh-LLM/mesh-llm

Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.

What it solves

Mesh LLM allows users to pool GPU and memory resources across multiple machines to run large language models (LLMs) that would otherwise be too large for a single computer. It provides a unified, OpenAI-compatible API, making it easy to integrate with existing tools while distributing the computational load across a network of peers.

How it works

The system consists of two main components: Skippy, which handles model management, native inference, and split execution, and Mesh, which manages peer discovery, routing, and the user interface.

Key mechanisms include:

  • Mesh Routing: Requests are routed to the peer capable of serving the requested model.
  • Encrypted Transport: Uses QUIC for end-to-end encrypted communication between nodes.
  • Skippy Stage Splits: For models too large for one machine, the system splits the model into contiguous layer ranges (stages) across multiple nodes.
  • Layer Packages: Peers download only the specific GGUF fragments needed for their assigned stage.
  • Mixture-of-Agents (MoA): An experimental feature that fans out a single prompt to every model in the mesh and arbitrates the responses into one reply.

Who it’s for

  • Hardware-constrained users: People who want to run massive models on a collection of smaller GPUs across different machines.
  • Developers: Those needing an OpenAI-compatible endpoint for distributed LLM inference.
  • Self-hosters: Users who want to create private or public meshes for shared AI resources.

Highlights

  • OpenAI-compatible API: Seamlessly integrates with existing LLM clients.
  • Distributed Inference: Supports splitting large models across multiple machines via stage splits.
  • Broad Model Support: Compatible with a wide range of GGUF model families (Qwen, Llama, Mistral, DeepSeek, etc.).
  • Cross-Platform: Available for macOS, Linux, and Windows (with CUDA, ROCm, Vulkan, and Metal support).
  • Experimental MoA: Ability to query multiple models in parallel and merge results.

Related

  • Dispatch
    Mesh LLM: Peer‑to‑Peer Distributed LLM Inference on iroh

    Mesh LLM lets you pool GPUs across laptops, workstations and servers into a single OpenAI‑compatible inference API using iroh’s authenticated QUIC mesh, eliminating reliance on centralized AI providers.

  • Project
    ServerlessLLM/ServerlessLLM

    A high-performance system for deploying multiple AI models on shared GPUs, featuring ultra-fast checkpoint loading and GPU multiplexing to enable efficient serverless inference.

  • Project
    Nehanth/pooled

    Pooled is an open‑source web app that lets multiple browsers pool their GPUs via WebGPU and WebRTC to run large LLMs (up to 35 B parameters) entirely in the browser. Devices split the model’s layers, exchange tiny hidden‑state packets per token, and the host assembles the final output. It supports speculative decoding, a built‑in coding‑assistant (“Code mode”), and an OpenAI‑compatible Serve API, all with zero‑install, no‑account usage.

  • Project
    iimeta/fastapi

    An enterprise-grade LLM API integration system that provides a unified, OpenAI-compatible interface to access multiple AI model providers.

  • Project
    mlc-ai/mlc-llm

    MLC LLM is a universal deployment engine and ML compiler that enables high-performance native execution of large language models across diverse hardware and platforms.