Nehanth/pooled

What is Pooled?

Pooled is a web‑based system that lets a group of ordinary devices – a laptop, a desktop, a phone, even a friend’s computer – combine their GPUs to run a single large language model (LLM) that none of them could hold on its own. The devices join a room by opening a URL; each device downloads only the slice of the model it will compute (a set of layers) and runs those layers with custom WebGPU (WGSL) kernels. For every generated token the host sends a tiny hidden‑state packet (a few KB) over a direct WebRTC data channel to the workers, collects the final logits, samples the next word, and repeats. The whole pipeline stays in the browser – no native install, no cloud server does the inference, and the only thing that leaves the browser is the model weight files (fetched from Hugging Face or a peer).

---

Key features (as described in the README)

Feature	What it means	Why it matters
Peer‑to‑peer LLM inference in the browser	Uses WebGPU for on‑device GPU compute and WebRTC for low‑latency peer connections.	No server‑side GPU costs; anyone with a modern browser can contribute compute.
Layer‑wise model sharding	Each participant holds a different contiguous block of the model’s layers (e.g., layers 0‑21 on a laptop, 22‑42 on a desktop, 43‑63 on a phone).	Allows very large models (27‑35 B parameters) to be run with total memory spread across devices.
Supported models	Qwen 3.8 27B (Q4_0, ~17 GB total), Qwen 3.6 35B MoE (Q4_0, ~22.5 GB total), Qwen 3 1.7B (Q8_0, ~5.6 GB).	Demonstrates that both dense and mixture‑of‑experts (MoE) architectures work with the system.
Speculative decoding	The model includes a built‑in draft layer that proposes tokens; the host validates them, dramatically speeding up generation.	Brings token‑per‑second rates close to native llama.cpp on a single GPU while keeping correctness.
Code mode	An agent loop that can write files, serve a sandboxed preview (:5173), read console errors and iteratively fix them.	Turns the shared LLM into an in‑browser coding assistant that can build small apps (e.g., a Tetris game) that all participants can see.
OpenAI‑compatible Serve API	A tiny CLI (npx @pooled/cli serve …) exposes the room as an OpenAI‑style /v1/chat/completions endpoint (also Anthropic messages).	Lets existing tools (Codex CLI, Claude Code, Continue, LiteLLM, curl, etc.) talk to the pooled model as if it were a hosted API.
Zero‑install, no‑account	Users only need to open a link; the host can optionally protect the room with a token.	Low friction for ad‑hoc collaboration or demos.
Relay fallback	If a corporate network blocks UDP, the system can fall back to a TURN relay (TCP/TLS on 443).	Makes the system usable behind firewalls while still keeping model weights out of the relay.
Testing & benchmarking	Unit tests (CPU‑only), end‑to‑end tests without GPU, GPU golden tests on a 0.6 B model, and detailed benchmark logs.	Shows the project is engineered with reproducibility in mind.
Open source & MIT‑licensed	All code, kernels, and documentation are publicly available under the MIT license.	Encourages community contributions and reuse.

---

Who might use it?

Use case	How Pooled helps
Live coding demos / workshops	Participants can bring laptops and phones; the model runs collectively, letting the presenter showcase a 30‑B‑parameter LLM without any cloud credits.
Privacy‑conscious teams	Because no central server processes the prompts, a team that trusts only its own devices can still run a powerful model locally.
Edge‑device research	Researchers exploring layer‑wise sharding, MoE off‑loading, or WebGPU kernels can experiment directly in the browser.
Rapid prototyping of AI‑augmented web apps	The built‑in Code mode gives a ready‑made agent that writes and serves code, useful for quick proof‑of‑concepts.

---

Limitations & open challenges (as acknowledged by the project)

• Network latency matters – token‑per‑second drops as round‑trip latency grows; speculative decoding mitigates but does not eliminate the effect.
• Prefill speed – the first token of a long prompt is still slower than native llama.cpp; the roadmap lists faster pre‑fill as a priority.
• Browser support – only Chrome on macOS is fully vetted; Safari on iPhone works as a worker, but Safari on Mac may reload under memory pressure. Firefox/Linux need WebGPU manually enabled.
• Privacy of hidden state – the hidden‑state packets are not cryptographically protected; anyone in the room could reconstruct the prompt. Rooms are intended for trusted participants.
• Device memory requirements – even when split, the total RAM/VRAM needed across the room is ~17‑22 GB for the 27‑35 B models, so a room of low‑end phones alone cannot host the biggest models.

---

Quick start (from the README)

# Clone and serve the static site
git clone https://github.com/Nehanth/pooled && cd pooled
npx -y serve -l 8080 .   # opens http://localhost:8080/room

Open the displayed URL on each participant’s browser, click “Start a room”, and watch the devices negotiate layer assignments.

---

Bottom line

Pooled is a genuine, open‑source project that pushes the frontier of collaborative, in‑browser LLM inference. It combines WebGPU‑accelerated kernels, WebRTC‑based peer networking, and a full OpenAI‑compatible API surface, all without requiring any native installation. The README provides detailed architecture docs, performance numbers, and a clear roadmap, making it a solid reference for anyone interested in distributed inference, browser‑based AI, or building coding agents that run entirely client‑side.