leyten/shard

Pipeline-parallel LLM inference across GPUs on separate machines.

What it solves

Shard solves the problem of running massive AI models that are too large for any single GPU's VRAM. It enables a decentralized, permissionless network of consumer-grade GPUs across the open internet (WAN) to pool their memory and compute, allowing them to serve frontier-size models (like GLM-5.2 744B) at interactive speeds.

How it works

Shard splits a transformer model's layers into contiguous blocks, assigning one block per GPU (a "swarm"). A request is processed by streaming activations through these shards in order. To overcome the high latency of the public internet, Shard employs several key optimizations:

  • Speculative Decoding: A small, fast "draft" model proposes tokens, which the large distributed model then verifies in a single pipeline traversal.
  • Async Pipelining: Multiple verification chunks are kept in flight simultaneously, shifting the bottleneck from network latency to system throughput.
  • Ring Direct-Return: The final node in the chain returns the result directly to the coordinator, reducing the number of network hops.
  • CUDA-Graphed Drafts: The draft model is captured as a CUDA graph to minimize launch overhead.

Who it’s for

Developers and GPU owners who want to build or use a decentralized AI compute network that is uncensored, decentralized, and private, avoiding reliance on centralized datacenters.

Highlights

  • Frontier-Scale Performance: Proven to run a 744B parameter model at ~30 tok/s across GPUs in six different US states.
  • Permissionless Joining: Designed for one-command joining and dynamic layer allocation across heterogeneous GPUs.
  • Lossless Sampling: Supports temperature and top-p/top-k sampling that is bit-identical to a single-machine setup.
  • Secure Transport: Uses authenticated and encrypted (ChaCha20-Poly1305) pickle-free framing to prevent code execution and eavesdropping.
  • Fault Tolerance: Demonstrated mid-request healing where a request resumes on a spare node if a node fails mid-generation.

Related

  • Project
    Mesh-LLM/mesh-llm

    A distributed LLM serving system that pools GPUs and memory across multiple machines to run large models via an OpenAI-compatible API.

  • Project
    cosmosgl/graph

    A high-performance WebGL rendering engine and force graph algorithm that enables real-time simulation of massive network graphs on the GPU.

  • Project
    b4rtaz/distributed-llama

    A distributed inference engine that clusters home devices over Ethernet to share RAM and accelerate LLM performance using tensor parallelism.

  • Project
    hyperspaceai/hyperspace-node

    A decentralized P2P AI inference network that allows users to contribute GPU, CPU, and storage resources to run AI models and earn rewards.

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.