Falcon 180B Release Notes

TII has released Falcon 180B, a 180-billion parameter open-access language model trained on 3.5 trillion tokens. It establishes a new state-of-the-art for open models, rivaling the performance of proprietary systems such as Google's PaLM-2 Large.

Model Architecture and Training

Falcon 180B is a scaled-up version of Falcon 40B, incorporating innovations like multiquery attention to enhance scalability. The model was trained using Amazon SageMaker on up to 4,096 GPUs simultaneously, totaling approximately 7 million GPU hours.

Key training specifications include:

  • Dataset: Predominantly web data from RefinedWeb (85%), supplemented by curated conversations, technical papers, and a small fraction of code (3%).
  • Scale: 3.5 trillion tokens, which constitutes less than a single epoch of the pretraining dataset.
  • Compute: 2.5 times larger than Llama 2 and trained with four times more compute.

Performance and Benchmarks

At the time of its release, Falcon 180B was the highest-scoring openly released pre-trained LLM on the Hugging Face Leaderboard with a score of 67.85 (later updated to 67.85, placing it on par with Llama 2 70B according to updated methodology).

Technical performance highlights include:

  • Competitive Standing: It outperforms Llama 2 70B and OpenAI's GPT-3.5 on MMLU.
  • Proprietary Parity: It is on par with PaLM 2-Large across several benchmarks, including HellaSwag, LAMBADA, WebQuestions, Winogrande, PIQA, ARC, BoolQ, CB, COPA, RTE, WiC, WSC, and ReCoRD.
  • Quantization Stability: Evaluation metrics remain similar across torch.float16, 8bit, and 4bit versions, though 8-bit inference is noted as being significantly faster than 4-bit.

Hardware Requirements for Deployment

Running Falcon 180B requires significant computational resources depending on the task. The following memory requirements are identified for specific configurations:

Use Case Method Memory Required Example Hardware
Training Full fine-tuning 5120GB 8x 8x A100 80GB
Training LoRA with ZeRO-3 1280GB 2x 8x A100 80GB
Training QLoRA 160GB 2x A100 80GB
Inference BF16/FP16 640GB 8x A100 80GB
Inference GPTQ/int4 320GB 8x A100 40GB

Implementation and Usage

Falcon 180B is integrated into the Hugging Face ecosystem via Transformers version 4.33. Users must accept the model's license, which allows commercial use under restrictive conditions that exclude "hosting use."

Base vs. Chat Model

  • Base Model: A pre-trained platform intended for further fine-tuning; it does not follow a specific prompt format and is not designed for conversational responses out of the box.
  • Chat Model: Fine-tuned on large-scale conversational and instruction datasets. It uses a specific prompt structure:
    • System: [optional system prompt]
    • User: [user input]
    • Falcon: [model response]

Technical Integration

Integration is achieved through the AutoModelForCausalLM class in the transformers library. The model supports bfloat16 for high precision and bitsandbytes for 8-bit and 4-bit quantization to reduce hardware overhead.

Sources