Llama 3.1 Release Notes: Multilinguality, Long Context, and 405B Model
Meta has released Llama 3.1, a new iteration of the Llama family that introduces a massive 405B parameter dense model alongside 8B and 70B versions. This release is significant because it expands context windows to 128K tokens, adds native multilingual support, and updates licensing to allow the use of model outputs to improve other LLMs.
Model Variants and Core Capabilities
Llama 3.1 consists of eight open-weight models, including three base models and five fine-tuned variants. All models support a context length of 128K tokens and are capable of processing eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
Generative Models
- Llama 3.1 8B: Optimized for efficient deployment and development on consumer-grade GPUs.
- Llama 3.1 70B: Designed for large-scale AI-native applications.
- Llama 3.1 405B: A very large dense model intended for synthetic data generation, distillation, and serving as an "LLM as a Judge."
Each of these sizes is available in both base (pre-trained) and instruct-tuned versions.
Safety and Classification Models
- Llama Guard 3: A safeguard model fine-tuned on Llama 3.1 8B to classify LLM inputs and responses based on a risk taxonomy. It supports the 128K context length and is multilingual.
- Prompt Guard: A 279M parameter BERT-based classifier designed to detect prompt injections and jailbreaking attempts.
Technical Specifications and Training
Llama 3.1 models were trained on over 15 trillion tokens using a custom GPU cluster totaling 39.3M GPU hours. The training breakdown is as follows:
- 8B: 1.46M GPU hours
- 70B: 7.0M GPU hours
- 405B: 30.84M GPU hours
Llama 3.1 Instruct models were optimized for instruction following using publicly available datasets and over 25M synthetically generated examples via Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF).
Memory and Hardware Requirements
Running Llama 3.1 requires significant VRAM, which varies by model size and precision.
Inference Memory (Checkpoint Loading)
| Model Size | FP16 | FP8 | INT4 |
|---|---|---|---|
| 8B | 16 GB | 8 GB | 4 GB |
| 70B | 140 GB | 70 GB | 35 GB |
| 405B | 810 GB | 405 GB | 203 GB |
KV Cache Memory (FP16)
As context length increases, the KV cache becomes a critical memory factor. For 128K tokens, the memory requirements are:
- 8B: 15.62 GB
- 70B: 39.06 GB
- 405B: 123.05 GB
Training Memory (Estimated)
| Model Size | Full Fine-tuning | LoRA | Q-LoRA |
|---|---|---|---|
| 8B | 60 GB | 16 GB | 6 GB |
| 70B | 500 GB | 160 GB | 48 GB |
| 405B | 3.25 TB | 950 GB | 250 GB |
Tool Use and Prompting
Llama 3.1 Instruct models are fine-tuned for agentic use cases and support tool calling.
Built-in Tool Calling
By including Environment: ipython in the system prompt, the model enables code interpreter mode. It can generate Python code using the <|python_tag|> and <|eom_id|> delimiters. Built-in tools include brave_search, wolfram_alpha, and code_interpreter.
Custom Tool Calling
The models support custom JSON function calling. When a tool is called, the model outputs the function name and parameters in JSON format. The response from the tool is then passed back to the model using the <|python_tag|> delimiter to generate the final answer.
Licensing and Synthetic Data
Llama 3.1 features a more permissive license than its predecessors. A key change is that it explicitly enables the use of model outputs to improve other LLMs. This allows for synthetic data generation and distillation, making the 405B model particularly valuable as a teacher model for smaller, specialized LLMs.
Ecosystem Integration
Llama 3.1 is integrated into the Hugging Face ecosystem via transformers >= 4.43.2.
- Quantization: Official FP8 quantized weights are available for the 405B model to allow it to run on 8xH100 GPUs. Additional AWQ and GPTQ INT4 variants are provided by Hugging Face for further memory reduction.
- Deployment: Support is available through Hugging Face Inference API (for PRO users), Inference Endpoints, and partners including AWS, Google Cloud, Microsoft Azure, and DELL.