Falcon models release and Hugging Face ecosystem integration

The Falcon models

Falcon is a new family of state‑of‑the‑art language models created by the Technology Innovation Institute in Abu Dhabi and released under the Apache 2.0 license. The family consists of Falcon‑40B and its smaller sibling Falcon‑7B. At the time of release Falcon‑40B topped the Open LLM Leaderboard, while Falcon‑7B was the best model in its weight class. Falcon‑7B was trained on 1.5 trillion tokens and Falcon‑40B on 1 trillion tokens, with over 80 % of the data coming from the RefinedWeb web corpus. Both models use multiquery attention, which greatly reduces the size of the key‑value cache during autoregressive decoding. Falcon‑40B needs about 90 GB of GPU memory for full‑precision inference, whereas Falcon‑7B needs only about 15 GB. Instruct variants Falcon‑7B‑Instruct and Falcon‑40B‑Instruct are also available and are recommended for quick experimentation.

Demo

The Falcon‑40B model can be tried immediately in a Hugging Face Space or through an embedded playground. The Space uses Hugging Face’s Text Generation Inference backend, the same technology that powers HuggingChat. A Core ML version of the Falcon‑7B‑Instruct model has been built and demonstrated running on an M1 MacBook Pro, with a video showing the lightweight app and a link to download the Core ML weights for further exploration.

Inference

Falcon models can be run with the standard transformers library, but the bfloat16 datatype must be used and remote code execution must be enabled because the modeling code resides in the model repository. For the 7B instruct model a simple pipeline can be constructed with AutoTokenizer, transformers.pipeline, torch_dtype=torch.bfloat16, trust_remote_code=True, and device_map="auto". Running the 40B model requires more memory; loading in 8‑bit mode reduces the footprint to about 45 GB, which fits on an A6000 (48 GB) but not on a 40 GB A100. With multiple GPUs and accelerate, device_map="auto" can distribute layers across cards or offload to CPU. Using the latest bitsandbytes, transformers, and accelerate enables 4‑bit loading, bringing the 40B model’s memory requirement down to roughly 27 GB.

Evaluation

Falcon‑40B base and instruct models ranked first and second on the Open LLM Leaderboard, with average scores of 60.4 and 63.2 respectively. Falcon‑7B base achieved a score of 48.8, outperforming LLaMA‑7B (47.6) and MPT‑7B (48.6). The leaderboard evaluates reasoning and truthfulness across AI2 Reasoning Challenge, HellaSwag, MMLU, and TruthfulQA. Notably, Falcon‑40B reached this performance with only 2 800 PF‑days of pretraining compute, about half that required for LLaMA‑65B (6 300 PF‑days), suggesting that further efficiency gains in LLM pretraining remain possible.

Fine-tuning with PEFT

Fine‑tuning large Falcon models is made feasible with parameter‑efficient methods. Using the PEFT library and QLoRA, the Falcon‑7B model was fine‑tuned on the Guanaco dataset on a single NVIDIA T4 GPU (16 GB), and the Falcon‑40B model was fine‑tuned on a single NVIDIA A100 GPU (80 GB). Training with 4‑bit quantized base models and the SFTTrainer from the TRL library yields adapter files that are only a few megabytes in size—for example, the fine‑tuned Falcon‑7B adapter is about 65 MB compared to the original 15 GB half‑precision model. The hidden states from the frozen model are augmented by low‑rank adapters applied to the query and key projection layers, and the resulting model can be used for inference without storing the full weights.

Conclusion

Falcon models are permissively licensed for commercial use, straightforward to run with Hugging Face’s inference tools, and easy to adapt via PEFT‑based fine‑tuning on modest hardware. The release invites the community to build applications, experiments, and further improvements on top of these openly available large language models.