ngxson/wllama
WebAssembly binding for llama.cpp - Enabling on-browser LLM inference
What it solves
wllama provides a way to run Large Language Models (LLMs) directly in the web browser without requiring a backend server or a dedicated GPU. It removes the dependency on external infrastructure by bringing the inference engine to the client side.
How it works
It acts as a WebAssembly (Wasm) binding for llama.cpp, allowing the C++ inference engine to run in a browser environment. The project leverages WebAssembly SIMD for CPU-based inference and WebGPU for hardware-accelerated GPU inference. To handle browser memory limits (specifically the 2GB ArrayBuffer limit), it supports splitting large GGUF model files into smaller chunks that can be downloaded in parallel.
Who it’s for
Web developers who want to integrate local LLM capabilities into their applications using an OpenAI-compatible API, and users who prefer privacy-focused or offline-capable AI experiences where data never leaves the browser.
Highlights
- Hardware Acceleration: Supports both WebGPU for GPU offloading and WebAssembly SIMD for CPU inference.
- Multimodal Capabilities: Supports image and audio file inputs in addition to text.
- Developer Friendly: Provides a fully-typed OpenAI-compatible API and a pre-built npm package.
- Performance Optimizations: Includes the ability to split models into smaller files for faster parallel downloads and to avoid memory restrictions.
- Non-blocking UI: Runs inference inside a web worker to ensure the browser UI remains responsive.