Picovoice/picollm
On-device LLM Inference Powered by X-Bit Quantization
What it solves
It provides a highly accurate, cross-platform SDK for running compressed large language models (LLMs) locally. This eliminates the need for cloud-based inference, ensuring data privacy and allowing AI to run on a wide range of hardware, from mobile devices to single-board computers.
How it works
The engine uses a proprietary quantization algorithm called picoLLM Compression. Unlike standard fixed-bit allocation schemes, this algorithm automatically learns the optimal bit allocation strategy for weights based on a task-specific cost function, which significantly reduces accuracy loss compared to methods like GPTQ.
Who it’s for
Developers building local AI applications for Android, iOS, Web (Chrome, Safari, Edge, Firefox), Linux, macOS, Windows, and Raspberry Pi.
Highlights
- Local Inference: Runs 100% locally on CPU and GPU for maximum privacy.
- Broad Model Support: Compatible with open-weight models including Llama-3, Gemma, Mistral, Mixtral, Phi-3.5, and Qwen3-VL.
- High Accuracy: Recovers a significant portion of MMLU score degradation seen in standard GPTQ quantization.
- Multi-Platform SDKs: Native support for Python, .NET, Node.js, Android, iOS, Web, and C.