Falcon 3 release notes / what's new

TL;DR

Technology Innovation Institute (TII) has released Falcon 3, a family of decoder-only large language models under 10 billion parameters. These models prioritize training efficiency and high performance in science, mathematics, and coding, with the Falcon3-10B-Base model achieving state-of-the-art performance for models under 13B parameters.

Model Family and Variants

The Falcon 3 family consists of five base models, available in various formats including Instruct, GGUF, GPTQ-Int4, GPTQ-Int8, AWQ, and 1.58-bit variants:

  • Falcon3-1B-Base: A compact model developed via pruning and knowledge distillation.
  • Falcon3-3B-Base: A compact model developed via pruning and knowledge distillation.
  • Falcon3-7B-Base: The primary transformer-based model trained on 14 trillion tokens.
  • Falcon3-10B-Base: An upscaled version of the 7B model, created by duplicating redundant layers and continuing pre-training.
  • Falcon3-Mamba-7B-Base: An enhanced State Space Language Model (SSLM) trained on an additional 1.5 trillion tokens.

Technical Innovations and Training Methodology

Falcon 3 employs several strategies to optimize performance while reducing training costs:

Transformer-Based Training

The 7B model was trained using 1,024 H100 GPU chips on 14 trillion tokens comprising web, code, STEM, and multilingual data. To create the 10B model, TII used depth up-scaling by duplicating redundant layers of the 7B model and continuing pre-training with 2 trillion tokens of high-quality data.

Efficiency in Small Models

The 1B and 3B models were developed using knowledge distillation and pruning techniques, utilizing less than 100GT of curated data.

State Space Model (SSM) Integration

Falcon3-Mamba-7B-Base is a pure SSM that matches or surpasses leading transformer-based LLMs at the 7B scale. It supports a 32K context length and maintains the same architecture as the original Falcon Mamba 7B for seamless integration.

Performance Benchmarks

General and Specialized Capabilities

Falcon 3 models demonstrate competitive performance across several domains:

  • Mathematics: Falcon3-10B-Base scores 22.9 on MATH-Lvl5 and 83.0 on GSM8K.
  • Coding: Falcon3-10B-Base achieves 73.8 on MBPP, and Falcon3-10B-Instruct scores 45.8 on Multipl-E.
  • Reasoning: Falcon3-7B-Base and Falcon3-10B-Base score 51.0 and 59.7 on BBH, respectively.
  • Scientific Knowledge: Falcon3-10B-Base achieves 73.1 on MMLU and 42.5 on MMLU-PRO.

Comparative Performance

  • Falcon3-1B-Base is on par with gemma-2-2b and surpasses SmolLM2-1.7B.
  • Falcon3-3B-Base outperforms Llama-3.1-8B and Minitron-4B-Base.
  • Falcon3-7B-Base is on par with Qwen2.5-7B.
  • Falcon3-10B-Base is state-of-the-art for the under-13B category.
  • Instruct Models: Falcon3-7B-Instruct and Falcon3-10B-Instruct outperform all instruct models under the 13B scale on the open leaderboard.

Architectural Specifications

  • Context Length: Most models support up to 32k tokens, except for the 1B model, which supports up to 8k.
  • Architecture: Transformer-based models are compatible with the Llama architecture. The transformer models range from 18 to 40 layers, while the Mamba model has 64 layers.
  • Activation and Vocabulary: All models use the SwiGLU activation function. The vocabulary size is 131K tokens for transformer models and 65K for Mamba-7B.
  • Optimization: Falcon3-7B-Base uses a head dimension of 256, optimizing it for high throughput with FlashAttention-3.

Future Roadmap and Licensing

All Falcon 3 models are released under the Falcon LLM license. TII has announced that in January 2025, they will release additional Falcon 3 family models featuring multi-modal capabilities (image, video, and audio support) and a full technical report.

Sources