Meta Llama 3 Release Notes

Meta has released Llama 3, a new generation of open-access large language models (LLMs) available in 8B and 70B parameter sizes. This release significantly advances open-source AI capabilities through a massive increase in training data and a redesigned tokenizer.

Model Variants and Specifications

Llama 3 is available in four primary configurations, each with a context length of 8K tokens and compatibility with consumer-grade hardware:

  • Meta-Llama-3-8B: The base pre-trained model designed for efficient deployment.
  • Meta-Llama-3-8B-Instruct: The instruction-tuned version of the 8B model.
  • Meta-Llama-3-70B: The base pre-trained model for large-scale applications.
  • Meta-Llama-3-70B-Instruct: The instruction-tuned version of the 70B model.

Additionally, Meta released Llama Guard 2, a safety-specific model fine-tuned on Llama 3 8B. It is designed for production environments to classify both LLM prompts and responses to detect unsafe content based on a risk taxonomy.

Technical Improvements over Llama 2

Llama 3 introduces several architectural and data-driven enhancements to improve performance and efficiency:

Expanded Tokenizer and Vocabulary

Llama 3 utilizes a new tokenizer that increases the vocabulary size to 128,256 tokens, up from 32,000 in Llama 2. This expansion allows for more efficient text encoding and potentially stronger multilingual capabilities. This change contributed to the increase in the small model's parameter count from 7B to 8B due to larger embedding input and output matrices.

Training Scale and Data Curation

The models were trained on over 15 trillion tokens—approximately 8x more data than previous iterations—using a new mix of publicly available online data. Training was conducted on two clusters utilizing 24,000 GPUs.

Architectural Optimizations

The 8B model now implements Grouped-Query Attention (GQA), which improves efficiency when handling longer contexts.

Instruction Tuning Process

Llama 3 Instruct models were optimized for dialogue using over 10 million human-annotated data samples. The tuning process combined supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct policy optimization (DPO).

Performance Evaluation

According to the Open LLM Leaderboard, Llama 3 shows significant improvements over its predecessor. Llama 3 8B achieved a score of 13.41 compared to Llama 2 7B's 8.72. The Llama 3 70B model achieved a score of 26.37, outperforming Llama 2 70B's 18.25.

Deployment and Integration

Llama 3 is fully integrated into the Hugging Face ecosystem, providing several paths for deployment and inference:

  • 🤗 Transformers: Compatible with release 4.40, supporting safetensors, 4-bit quantization via bitsandbytes, and Flash Attention 2. Llama 3 models are compatible with torch.compile() with CUDA graphs, which can provide up to a 4x speedup at inference time.
  • Inference Endpoints: Deployable via Text Generation Inference (TGI) for production-ready features like continuous batching and token streaming.
  • Cloud Providers: One-click deployment is available through Google Cloud (Vertex AI and GKE) and Amazon SageMaker (AWS Jumpstart).

Fine-Tuning and Prompting

Prompt Format

While base models have no specific format, the Instruct versions require a strict conversation structure to function effectively:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_msg_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ model_answer_1 }}<|eot_id|>

Efficient Fine-Tuning

Llama 3 can be fine-tuned on consumer GPUs using the 🤗 TRL (Transformer Reinforcement Learning) library. By utilizing 4-bit quantization and QLoRA, the 8B model can be trained on a single A10G GPU in approximately four hours.

Licensing

Llama 3 uses a permissive license that allows for redistribution, fine-tuning, and derivative works. A new requirement for Llama 3 is explicit attribution: derivative models must include "Llama 3" at the beginning of their name and mention "Built with Meta Llama 3" in derivative works or services.

Sources