mlc-ai/web-llm

High-performance In-browser LLM Inference Engine

What it solves

WebLLM brings high-performance large language model (LLM) inference directly into the web browser. It eliminates the need for server-side processing, which enhances user privacy and removes the dependency on external APIs for model execution.

How it works

WebLLM leverages WebGPU for hardware acceleration and WebAssembly (WASM) for optimal performance. It is designed as a modular npm package that can be integrated into web applications, Chrome extensions, or used via CDN. The engine supports various cache backends (such as the Cache API, IndexedDB, and OPFS) to manage model weights efficiently within the browser.

Who it’s for

Web developers building AI assistants, chatbots, or browser extensions who want to run LLMs locally on the client side with GPU acceleration.

Highlights

  • OpenAI API Compatibility: Fully compatible with the OpenAI API, supporting streaming, JSON-mode, and seeding.
  • Broad Model Support: Natively supports Llama 3, Phi 3, Gemma, Mistral, and Qwen families.
  • Client-Side Execution: Runs entirely in-browser with no server support required.
  • Performance Optimization: Supports offloading computations to Web Workers or Service Workers to keep the UI responsive.
  • Custom Model Integration: Allows users to deploy custom models in MLC format.

Related

  • Project
    ngxson/wllama

    A WebAssembly binding for llama.cpp that enables running LLMs, multimodal models, and tool calling directly in the browser via WebGPU or Wasm SIMD.

  • Project
    Ryan-yang125/ChatLLM-Web

    A private, browser-native AI studio and agent workspace that runs LLMs locally on your device using WebGPU, requiring no API keys or backend servers.

  • Project
    sauravpanda/BrowserAI

    A JavaScript SDK that enables running production-ready LLMs and multimodal AI models directly in the browser using WebGPU acceleration for privacy and zero server costs.

  • Project
    mlc-ai/mlc-llm

    MLC LLM is a universal deployment engine and ML compiler that enables high-performance native execution of large language models across diverse hardware and platforms.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.