ngxson/wllama

WebAssembly binding for llama.cpp - Enabling on-browser LLM inference

What it solves

wllama provides a way to run Large Language Models (LLMs) directly in the web browser without requiring a backend server or a dedicated GPU. It bridges the gap between the powerful C++ implementation of llama.cpp and web applications by providing WebAssembly (Wasm) bindings.

How it works

The project compiles llama.cpp into WebAssembly, allowing the browser to execute the model's inference logic. It supports multiple execution paths:

  • WebGPU: Offloads model layers to the GPU for significantly faster performance.
  • Wasm SIMD: Runs inference on the CPU using SIMD instructions for efficiency when a GPU is unavailable.
  • Web Workers: Executes inference inside a worker thread to ensure the browser's user interface remains responsive and does not freeze during generation.

To handle large model files, wllama can load models split into smaller chunks, which bypasses browser memory limits (ArrayBuffer restrictions) and enables parallel downloading.

Who it’s for

Web developers who want to integrate local, private, and serverless LLM capabilities into their frontend applications using TypeScript or JavaScript.

Highlights

  • OpenAI-compatible API: Simplifies integration for those already familiar with OpenAI's request/response format.
  • Hardware Acceleration: Supports both WebGPU and Wasm SIMD for optimized performance.
  • Multimodal Capabilities: Supports image and audio file inputs for multimodal models.
  • Tool Calling: Enables models to interact with external tools.
  • Parallel Model Loading: Ability to split and load large GGUF models in parallel to improve speed and bypass the 2GB file size limit.

Related

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without server-side processing.

  • Project
    abetlen/llama-cpp-python

    Python bindings for llama.cpp that enable local LLM inference with an OpenAI-compatible API and support for multiple hardware acceleration backends.

  • Project
    SciSharp/LLamaSharp

    LLamaSharp is a cross-platform .NET library that enables efficient local inference of Large Language Models by wrapping the llama.cpp engine.

  • Project
    hybridgroup/yzma

    yzma is a Go library and CLI that lets developers run llama.cpp language and vision models locally with hardware acceleration, supporting native platforms and WebAssembly without needing CGo.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.