microsoft/BitNet

Official inference framework for 1-bit LLMs

What it solves

bitnet.cpp is an inference framework designed to run 1-bit (ternary) Large Language Models (LLMs) with high efficiency. It addresses the high computational and energy costs of full-precision models, enabling large models (up to 100B parameters) to run on a single CPU with speeds comparable to human reading.

How it works

The framework provides optimized kernels for CPU (x86 and ARM) and GPU, utilizing ternary weights (1.58-bit) to perform fast and lossless inference. It supports various quantization types (such as I2_S and TL1) and can convert .safetensors checkpoints into a GGUF format for execution.

Who it’s for

It is intended for developers and researchers looking to deploy LLMs on edge devices or resource-constrained environments where CPU-only inference or low energy consumption is critical.

Highlights

  • Significant Speedups: Achieves up to 6.17x speedup on x86 CPUs and 5.07x on ARM CPUs compared to full-precision models.
  • Energy Efficiency: Reduces energy consumption by up to 82.2% on x86 and 70.0% on ARM.
  • Broad Hardware Support: Optimized for x86 and ARM CPUs, with official GPU kernels available.
  • 1-bit Embedding Models: Includes support for 1-bit embedding models (0.6B and 270M) that maintain competitive quality with faster CPU inference.

Related

  • Project
    NVIDIA/TensorRT-LLM

    An open-source library from NVIDIA that optimizes inference for LLMs and visual generation models using specialized kernels and an efficient runtime to maximize throughput and latency on NVIDIA GPUs.

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    xLLM-AI/xllm

    xLLM is an efficient LLM inference framework optimized for Chinese AI accelerators, designed for high-throughput, low-latency enterprise-grade deployment.

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.