ngxson/wllama

WebAssembly binding for llama.cpp - Enabling on-browser LLM inference

What it solves

wllama provides a way to run Large Language Models (LLMs) directly in the web browser without requiring a backend server or a dedicated GPU. It solves the problem of deploying AI models to the web while maintaining privacy, reducing server costs, and enabling offline capabilities by leveraging WebAssembly (Wasm) and WebGPU.

How it works

It acts as a WebAssembly binding for llama.cpp, allowing the browser to execute GGUF model files. The project uses WebAssembly SIMD for CPU-based inference and WebGPU for hardware-accelerated GPU inference. To handle browser memory limits and improve download speeds, it supports splitting large model files into smaller chunks that can be loaded in parallel.

Who it’s for

Web developers who want to integrate LLM capabilities (like chat, embeddings, or multimodal inputs) into their frontend applications using a familiar OpenAI-compatible API.

Highlights

  • Browser-Native Inference: Runs models locally in the browser using Wasm and WebGPU, ensuring no data leaves the user's device.
  • OpenAI-Compatible API: Provides a fully-typed built-in API for easy integration.
  • Multimodal & Tool Support: Supports image and audio file inputs as well as tool calling.
  • Efficient Loading: Allows splitting models into smaller files to bypass the 2GB ArrayBuffer limit and speed up downloads via parallel loading.
  • Non-Blocking UI: Executes inference inside a web worker to prevent the browser UI from freezing.

Related

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    abetlen/llama-cpp-python

    Python bindings for llama.cpp that enable local LLM inference with an OpenAI-compatible API and support for multiple hardware acceleration backends.

  • Project
    SciSharp/LLamaSharp

    A cross-platform C# library based on llama.cpp that allows developers to run LLaMA and other LLMs locally on CPU and GPU with high-level APIs.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.

  • Project
    go-skynet/go-llama.cpp

    Go bindings for llama.cpp that enable high-performance local LLM inference within Go applications using the GGUF format.