Falcon 2 11B Release Notes
TII has launched Falcon 2, a new generation of open-source models featuring an 11B parameter Large Language Model (LLM) and a corresponding Vision-Language Model (VLM). These models are designed to provide the community with smaller, more efficient models that reduce inference costs while maintaining high performance across multiple languages and modalities.
Falcon2-11B LLM Technical Specifications
Falcon2-11B is a pretrained language model trained on over 5,000 billion tokens of RefinedWeb data, supplemented by curated corpora including technical data, code, and conversational data.
Training Strategy and Data
The model was trained using a four-stage strategy to incrementally increase context length and refine performance:
| Stage | Context Length | Tokens (GT) |
|---|---|---|
| Stage 1 | 2048 | 4500 |
| Stage 2 | 4096 | 250 |
| Stage 3 | 8192 | 250 |
| Stage 4 | 8192 | 500 |
Data sources include RefinedWeb-English and RefinedWeb-Europe, covering ten languages: Czech (cs), German (de), Spanish (es), French (fr), Italian (it), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), and Swedish (sv).
Model Architecture and Training Procedure
Falcon2-11B utilizes a Transformer-based architecture with the following specifications:
- Transformer Blocks: 60
- Query Heads: 32
- Key/Value Heads: 8
- Head Dimensions: 128
- Parallel Attention: Yes
- MLP Upscale Factor: 4
Training was conducted on 1,024 A100 40GB GPUs using a 3D parallelism strategy (TP=8, PP=1, DP=128), ZeRO, and Flash-Attention 2. The model used bfloat16 precision and the AdamW optimizer with a cosine decay learning rate schedule for the first stage.
Falcon2-11B LLM Performance Evaluation
Falcon2-11B demonstrates competitive performance against larger models and other 7B-11B class models.
English and General Benchmarks
On Open LLM Leaderboard tasks, Falcon2-11B achieved an average score of 64.28, placing it on par with Gemma-7B (64.29) and outperforming Llama3-8B (62.38) and Mistral-7B (60.97). Notably, zero-shot evaluations indicate that Falcon2-11B provides performance similar to the much larger Falcon-40B despite being four times smaller.
Multilingual and Coding Capabilities
Falcon2-11B outperforms Falcon-40B and several other multilingual models across six tested languages (German, Spanish, French, Italian, Dutch, and Romanian). In code generation, the model achieved a pass@1 of 29.59% on the HumanEval benchmark for Python via the BigCode Leaderboard.
Falcon2-11B Vision-Language Model (VLM)
Falcon2-11B VLM extends the LLM's capabilities to include image understanding, allowing users to engage in text-based chats about visual content.
Architecture and Training
The VLM integrates a pretrained CLIP ViT-L/14 vision encoder with a chat-finetuned Falcon2-11B model. To improve the perception of small objects and fine-grained details, it employs a dynamic high-resolution encoding mechanism similar to LLaVA-Next.
Training occurred in two stages:
- Pretraining: The LLM and visual encoder remained frozen while the multimodal projector was trained on 558K image-caption pairs to map visual embeddings to text space.
- Finetuning: Both the projector and LLM weights were trained on 1.2M image-text instruction data, including multi-round conversations.
VLM Evaluation
Falcon2-11B VLM shows strong performance across several vision benchmarks, achieving an average score of 74.4, surpassing LLaVA-1.6 (Vicuna-7B) at 72.1 and LLaVA-1.6 (Mistral-7B) at 73.3.
| Model | MME | GQA | SQA | POPE | VQAv2 | TextVQA | MM-Bench | SEED-IMG | Average |
|---|---|---|---|---|---|---|---|---|---|
| Falcon2-11B VLM | 1589/343 | 64.5 | 74.9 | 88.4 | 82.1 | 66.7 | 72.0 | 72.3 | 74.4 |
| LLaVA-1.6 (Vicuna-7B) | 1519/332 | 64.2 | 70.1 | 86.5 | 81.8 | 64.9 | 67.4 | 70.2 | 72.1 |
| LLaVA-1.6 (Vicuna-13B) | 1575/326 | 65.4 | 73.6 | 86.2 | 82.8 | 67.1 | 70.0 | 71.9 | 73.8 |
Licensing
Falcon 2 models are released under the TII Falcon 2 License, a permissive Apache 2.0-based license that includes an acceptable use policy to ensure responsible AI deployment.