mlc-ai/web-llm

High-performance In-browser LLM Inference Engine

What it solves

WebLLM brings high-performance Large Language Model (LLM) inference directly into the web browser. It eliminates the need for server-side processing, allowing AI applications to run locally on the user's hardware, which enhances privacy and reduces server costs.

How it works

WebLLM leverages WebGPU for hardware acceleration and WebAssembly (WASM) for optimal performance. It is designed as a modular npm package that can be integrated into web applications. The engine supports various cache backends (such as the Cache API, IndexedDB, and OPFS) to store model weights locally, and can be offloaded to Web Workers or Service Workers to prevent UI blocking.

Who it’s for

Web developers building AI assistants, chatbots, or browser extensions who want to provide local, GPU-accelerated LLM capabilities without managing a backend infrastructure.

Highlights

  • Full OpenAI API Compatibility: Use the same API for streaming, JSON-mode, and seeding across various open-source models.
  • In-Browser Execution: Runs entirely on the client side using WebGPU, ensuring no data leaves the browser.
  • Extensive Model Support: Natively supports Llama 3, Phi 3, Gemma, Mistral, and Qwen.
  • Structured Generation: Implements high-performance JSON mode for guaranteed structured output.
  • Flexible Deployment: Supports integration via NPM, Yarn, or CDN, and works within Chrome Extensions.

Related

  • Project
    ngxson/wllama

    A WebAssembly binding for llama.cpp that enables running LLMs, multimodal models, and tool calling directly in the web browser using WebGPU and Wasm.

  • Project
    Ryan-yang125/ChatLLM-Web

    A private, browser-native AI studio and agent workspace that runs LLMs locally on your device using WebGPU, requiring no API keys or backend servers.

  • Project
    sauravpanda/BrowserAI

    A JavaScript SDK that enables running production-ready LLMs and multimodal AI models directly in the browser using WebGPU acceleration for privacy and zero server costs.

  • Project
    mlc-ai/mlc-llm

    MLC LLM is a universal deployment engine and ML compiler that enables high-performance native execution of large language models across diverse hardware and platforms.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.