leyten/shard

Pipeline-parallel LLM inference across GPUs on separate machines.

What it solves

Shard solves the problem of running massive AI models that are too large for any single GPU's memory (VRAM). Instead of relying on centralized data centers, it allows a decentralized network of scattered consumer GPUs to pool their resources and serve these models collectively over the open internet.

How it works

Shard treats a transformer model as a stack of layers and splits them into contiguous blocks. Each GPU in a "swarm" holds only a portion of the model's layers. To generate a token, activations are streamed through these shards in order across the network.

To overcome the high latency of the public internet (WAN), Shard employs several optimizations:

  • Speculative Decoding: A small, fast "draft" model proposes multiple tokens, which the large distributed model then verifies in a single pass, amortizing the network round-trip cost.
  • Async Pipelining: Multiple verification chunks are kept in flight simultaneously, shifting the bottleneck from network latency to system throughput.
  • CUDA-graphed Draft: The draft model is captured as a CUDA graph to reduce launch overhead, significantly speeding up the proposal phase.
  • Direct Return: The final stage in the pipeline returns the result directly to the coordinator, reducing the number of network hops.

Who it’s for

It is designed for users and providers who want a permissionless, decentralized compute network for running frontier-scale AI models without needing a single massive machine or high-end enterprise hardware.

Highlights

  • Frontier-Scale Performance: Proven to serve a 744B parameter model (GLM-5.2) at ~30 tokens per second over the WAN.
  • Decentralized Architecture: No single node holds the entire model; anyone can join the network with a GPU.
  • Permissionless Join: Supports one-command joining and dynamic layer allocation across heterogeneous GPUs.
  • Fault Tolerance: Demonstrated the ability to actually resume a request mid-generation if a node fails.
  • Secure Transport: Uses authenticated encryption (ChaCha20-Poly1305) and pickle-free framing to prevent code execution and eavesdropping.