ngxson/wllama

What it solves

wllama 提供了一種直接在網頁瀏覽器中執行大型語言模型 (LLM) 的方式，而不需要後端伺服器或專用的 GPU。它透過將推論引擎帶到用戶端，消除了對外部基礎設施的依賴。

How it works

wllama 作為 llama.cpp 的 WebAssembly (Wasm) 綁定，允許 C++ 推論引擎在瀏覽器環境中執行。該專案利用 WebAssembly SIMD 進行基於 CPU 的推論，並利用 WebGPU 進行硬體加速的 GPU 推論。為了處理瀏覽器記憶體限制（特別是 2GB ArrayBuffer 限制），它支援將大型 GGUF 模型檔案拆分為較小的區塊，以便進行並行下載。

Who it’s for

想要透過使用 OpenAI 相容的 API 將本地 LLM 能力整合到應用程式中的網頁開發者，以及偏好隱私導向或離線可用的 AI 體驗，且數據從不離開瀏覽器的使用者。

Highlights

Hardware Acceleration: Supports both WebGPU for GPU offloading and WebAssembly SIMD for CPU inference.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
Developer Friendly: Provides a fully-typed OpenAI-compatible API and a pre-built npm package.
Developer Friendly: Provides a fully-typed OpenAI-compatible API and a pre-built npm package.
Performance Optimizations: Includes the ability to split models into smaller files for faster parallel downloads and to avoid memory restrictions.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
Multimodal Capabilities: Supports image and audio file inputs in addition to text.
imodal Capabilities: Supports image and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and and00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000