Falcon-Edge: Powerful, Universal, and Fine-Tunable 1.58-bit LLMs

Falcon-Edge is a series of ternary language models based on the BitNet architecture, designed for extreme efficiency on edge devices. By utilizing ternary weights ({-1, 0, 1}), these models enable a "matmul-free" design that significantly reduces memory usage and increases inference speed compared to traditional floating-point models.

Key Model Variants and Availability

Falcon-Edge is available in two sizes—1 Billion and 3 Billion parameters—with each size offered in both base and instruction-tuned versions. To support diverse deployment and development needs, the team introduced a pre-training paradigm that generates three distinct variants from a single training process:

  • Native BitNet Model: The standard ternary format model optimized for ultra-efficient inference.
  • bfloat16 Variant: A non-quantized version that approximates the BitNet model by injecting weight scales, allowing it to be loaded via standard Hugging Face transformers.
  • Pre-quantized Variant: Weights specifically engineered for effortless fine-tuning and continuous pre-training.

Technical Architecture and Performance

Falcon-Edge models were pre-trained on an internal data mixture of approximately 1.5 Tera Tokens using a WSD learning rate scheduler.

In evaluations using the Hugging Face leaderboard v2 benchmark, Falcon-Edge demonstrated performance on par with or better than other models of comparable size. The models also showed competitiveness against Microsoft's BitNet models on leaderboard v1 tasks, proving that BitNet architecture can be trained to be powerful across various domains.

The Universality Approximation

The team discovered that by injecting the weight scale after quantizing weights, they could create a bfloat16 counterpart of the BitNet model. This approximation allows the model to function as a non-BitNet model while maintaining high performance, as confirmed by end-to-end evaluations of the 1B and 3B base models.

Fine-Tuning with onebitllms

To move beyond inference-only releases, the Falcon-LLM team released the onebitllms Python package. This toolkit provides the necessary utilities to fine-tune pre-quantized BitNet models, which would otherwise produce gibberish output if run through standard nn.Linear layers.

onebitllms provides the following core functionalities:

  • Checkpoint Conversion: Converts pre-quantized model checkpoints into BitNet training format for use with fine-tuning frameworks like Hugging Face's trl library.
  • Quantization Utilities: Tools to quantize trained checkpoints into both BitNet and bfloat16 formats.
  • Low-level Components: Bare BitnetLinear layers and Triton kernels for integration into custom pre-training frameworks.

Currently, the package supports full fine-tuning; Parameter-Efficient Fine-tuning (PEFT) for BitNet models remains an open research question.

Future Research Directions

The release of Falcon-Edge and onebitllms identifies several key areas for future development in ternary LLMs:

  • GPU Inference Kernels: Developing specialized kernels to make BitNet models faster than native floating-point models on GPUs, similar to the goals of bitnet.cpp.
  • PEFT Support: Exploring the possibility of applying parameter-efficient fine-tuning to 1-bit models.
  • Universality Optimization: Further reducing the performance gap between BitNet checkpoints and their bfloat16 counterparts.
  • Multi-modal Expansion: Using these foundational models to create the first multi-modal BitNet Vision Language Models (VLMs).
  • Training Efficiency: Optimizing BitNet training kernels to reduce the current ~20% overhead observed during pre-training compared to non-BitNet models.

Sources