Llama 4 Maverick & Scout Release Notes

Meta has announced the release of Llama 4 Maverick and Llama 4 Scout, natively multimodal large language models based on a Mixture-of-Experts (MoE) architecture. These models introduce significant advancements in context window capacity, multimodal processing, and architectural efficiency, with both models utilizing 17B active parameters.

Model Variants and Specifications

Llama 4 introduces two primary model variants tailored for different deployment scales:

Llama 4 Maverick: A high-capacity model with approximately 400B total parameters and 128 experts, utilizing 17B active parameters.
Llama 4 Scout: An efficient model with approximately 109B total parameters and 16 experts, utilizing 17B active parameters.

Both models are trained on up to 40 trillion tokens across 200 languages, with specialized fine-tuning for 12 languages, including Arabic, Spanish, German, and Hindi. They leverage early fusion for native multimodality, allowing them to process text and image inputs simultaneously.

Architectural Innovations

Llama 4 employs several novel architectural choices to support extreme context lengths and improve computational efficiency:

iRoPE Architecture and NoPE Layers

To manage long contexts, Llama 4 uses the iRoPE architecture, which interleaves traditional Rotary Positional Embeddings (RoPE) with NoPE (No Positional Encoding) layers. NoPE layers occur every four layers and use a full causal mask over the context, which is critical for maintaining performance over long sequences.

Chunked Attention

In the RoPE layers (three out of every four), Llama 4 implements chunked attention. With a chunked attention length of 8,192, these layers only track context in 8K blocks, reducing memory and compute requirements compared to standard attention.

Attention Temperature Tuning

To prevent attention probability scores from fading toward zero in very long sequences—a common issue with the softmax function—Llama 4 uses temperature tuning (scaled softmax) in the NoPE layers to improve generalization across arbitrary context lengths.

Additional Technical Enhancements

QK Normalization: Llama 4 Scout uses RMS normalization without learnable parameters for Query and Key states in RoPE layers after embeddings are applied.
MoE Interleaving: While Scout is a full MoE, Maverick interleaves MoE and dense layers, applying experts in only half of the layers.
Co-distillation: Maverick was co-distilled from a larger model, Llama Behemoth, using a dynamic loss function to weight student and teacher logits.
MetaP: The models utilize MetaP, a methodology for optimally tuning hyperparameters across training budgets and model sizes.

Context Window and Performance

Llama 4 models are pre-trained with a 256K context length. The instruction-tuned variants expand this capacity significantly:

Model	Instruct	Context Length
Scout (16E)	Yes	10M
Maverick (128E)	Yes	1M
Scout (16E)	No	256K
Maverick (128E)	No	256K

Evaluation and Benchmarks

Instruction-tuned Llama 4 models outperform predecessors like Llama 3.1 405B in reasoning, knowledge, and multimodal tasks:

Reasoning & Knowledge: Maverick achieves 80.5% on MMLU Pro and 69.8% on GPQA Diamond.
Coding: Maverick scores 43.4% on LiveCodeBench (pass@1).
Multimodal Reasoning: Maverick achieves 73.4% on MMMU and 90.0% on ChartQA.
Image Understanding: Scout scores 91.6% on DocVQA (pre-trained) and 94.4% on DocVQA test (instruction-tuned).

Deployment and Integration

Llama 4 is integrated into the Hugging Face ecosystem via transformers v4.51.0, Text Generation Inference (TGI), and the Xet storage backend.

Quantization: Llama 4 Scout supports on-the-fly 4-bit or 8-bit quantization to fit on a single server-grade GPU. Llama 4 Maverick is available in BF16 and FP8 formats.
Storage: The use of Xet storage achieves approximately 25% deduplication for base models and up to 40% for derivative models, accelerating uploads and downloads.
Licensing: Models are released under the custom Llama 4 Community License Agreement.