Picovoice/picollm

On-device LLM Inference Powered by X-Bit Quantization

What it solves

picoLLM is a cross-platform inference engine designed to run compressed large language models (LLMs) locally on a wide variety of hardware, including mobile devices, web browsers, and single-board computers like the Raspberry Pi. It addresses the challenge of running high-accuracy LLMs on resource-constrained devices while maintaining privacy by keeping all inference 100% local.

How it works

The engine uses a proprietary quantization algorithm called picoLLM Compression. Unlike standard techniques that use fixed bit allocation, this algorithm automatically learns the optimal bit allocation strategy across and within the model's weights based on a task-specific cost function. This approach significantly reduces the accuracy degradation typically seen in quantization methods like GPTQ.

Who it’s for

Developers building local AI applications across different platforms, including those targeting Android, iOS, Web (Chrome, Safari, Edge, Firefox), Linux, macOS, Windows, and Raspberry Pi. It is specifically intended for those who need to deploy open-weight models on CPU or GPU without relying on cloud services.

Highlights

  • Broad Hardware Support: Runs on CPU and GPU across desktop, mobile, and web browsers.
  • ** wysokie Accuracy**: Recovers a significant portion of MMLU score degradation compared to GPTQ at 2, 3, and 4-bit settings.
  • 100% Local Inference: Ensures data privacy by processing all prompts locally.
  • Extensive Model Support: Compatible with a wide range of open-weight models including Llama-3, Gemma, Mistral, Mixtral, Phi-3.5, and specialized models for OCR and vision.
  • Multi-Language SDKs: Provides official SDKs for Python, .NET, Node.js, Android, Java, Swift, and C.

Related

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without a server.

  • Project
    ggml-org/llama.cpp

    A plain C/C++ implementation for high-performance LLM and VLM inference across a wide range of hardware with minimal setup.

  • Project
    ModelCloud/GPTQModel

    A comprehensive LLM quantization toolkit that compresses models using methods like GPTQ and AWQ with hardware acceleration for NVIDIA, AMD, Intel, and Huawei NPUs.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Dispatch
    ChonkLM: Bringing Tiny Language Models to the Browser via WebGPU

    An exploration of ChonkLM, a browser-based inference runtime that allows users to run small language models entirely offline using WebGPU.