Lumabri: P2P Swarm Inference for Mixture-of-Experts Models

Lumabri enables the execution of massive Mixture-of-Experts (MoE) models by distributing both the model weights and the actual computation across a peer-to-peer (P2P) swarm. Unlike traditional distributed inference that splits transformer layers across devices, Lumabri splits at the expert granularity, allowing nodes with limited resources—including those without GPUs—to contribute to the inference process.

Distributed Expert Execution

Lumabri optimizes for MoE sparsity by keeping only dense weights, the router, and the KV cache on the local "chatter" machine. When an expert is required for a token, the chatter sends a small 4 KB activation to a peer holding that specific expert. The peer executes the expert and returns the result.

This architecture ensures that expert weights never need to reach the chatter, significantly reducing the bandwidth required compared to weight-streaming approaches. Because both the chatter and the peers are built from the same engine source, the output is byte-for-byte identical to a local run. To prevent silent errors caused by different hardware instructions (e.g., -march=native), Lumabri requires peers to advertise their exact build, including source hash, ISA, and compiler; a chatter will refuse any peer whose build differs.

P2P Model Distribution and Lazy Loading

Lumabri implements a lazy-loading mechanism for model bytes via an LD_PRELOAD shim (liblumabri.so). This shim interposes standard libc calls (open, fopen, opendir, pread) to make remote model files appear as sparse local mirrors.

• On-demand fetching: Bytes are fetched from peers only when the inference engine actually touches them.
• Local Caching: Once fetched, bytes are stored in a local mirror and a content-addressed store (CAS) shared across checkpoints. Subsequent requests for the same bytes are served from the local disk at full speed.
• Integrity: Every MiB of the model is verified via sha256. The origin can sign the model root with an ed25519 key, allowing the chatter to verify every block against a trusted public key.

Trust and Verification in Untrusted Swarms

Because peers are not trusted, Lumabri employs several mechanisms to ensure the correctness of remote compute:

• Replica Verification: The LUMABRI_VERIFY=N setting allows the chatter to rerun a percentage of expert calls on a second replica. If the two honest peers disagree, the run stops as it is proof of a lie.
• Hedged Requests: To mitigate latency from slow peers, LUMABRI_HEDGE_MS=N can send duplicate requests to a second replica after a specified timeout, using the first valid deterministic result received.
• Encrypted Transport: When LUMABRI_ENCRYPT=1 is enabled, tokens, model blocks, and activations are protected using an X25519/Ed25519 handshake and ChaCha20-Poly1305 frames.

Supported Engines and Models

Lumabri integrates with the Colibri engine, supporting several MoE architectures. Each engine requires a specific expert-node binary built from the engine's own source to ensure deterministic output.

Engine	Supported Model	Expert Node Provenance
olmoe	OLMoE	phase2_test.sh
colibri	GLM	phase2_glm_test.sh
inkling	Inkling	phase2_inkling_test.sh
kimi_k3	Kimi K3	phase2_kimi_test.sh
deepseek	DeepSeek V4	phase2_deepseek_test.sh

Comparison to Other P2P Inference

Lumabri differs from projects like Petals or llama.cpp RPC in its point of partitioning. While those projects split consecutive transformer layers across devices—which typically requires high-speed GPU interconnects to be practical—Lumabri splits by expert. This allows a swarm to function effectively even on a network of CPUs and SSDs, as only the 4 KB activations travel over the network rather than entire layer activations.

Community Insights and Use Cases

Discussion among users highlights the potential for using Lumabri in local area networks (LANs) to pool the resources of low-power devices or RAM-constrained GPUs.

"The biggest benefit I see is to enable RAM constrained GPUs to perform inference of large parameter models with surprisingly high throughput... the memory to compute ratio over the network is limited only by the activations, not the weights."

Other users noted that this architecture serves as an "insurance policy" against centralized censorship or monitoring by cloud providers, returning LLM inference to a more decentralized, community-driven culture.