Nehanth/swarmllm

What it solves

SwarmLLM allows multiple devices (laptops, phones, PCs) to collectively run large language models (LLMs) that would be too large for any single device's memory. It eliminates the need for specialized installations, accounts, or central servers by running the model across browser tabs using WebGPU and WebRTC.

How it works

The project splits a model's layers across the participating devices in a "room." Each device holds and processes a only a slice of the model's weights on its own GPU. During inference, a small activation vector (approximately 10 KB) is passed between devices over direct peer-to-peer WebRTC connections. The system uses a custom WebGPU engine with WGSL kernels to optimize memory bandwidth and employs speculative decoding to reduce the impact of network latency.

Who it’s for

It is designed for users who want to run powerful LLMs locally and privately without installing software or simply opening a URL. It is also for developers interested in high-performance WebGPU-based inference engines.

Highlights

• Zero Install: Runs entirely in browser tabs via a URL.
• Cross-Device Compatibility: Works across different operating systems and hardware (e.g., a MacBook and an iPhone together).
• Cross-Network Support: Rooms can span different networks via WebRTC.
• High Performance: Features a custom WGSL engine that achieves native-competitive decode speeds on some hardware.
• Privacy-Centric: No server processes the data; conversations stay within the peer-to-peer room.
• Speculative Decoding: Uses multi-token-prediction to maintain speed even over slower network links.