Llama 3.2 Release Notes: Multimodal Vision and On-Device Small Language Models
Meta has released Llama 3.2, a collection of ten open-weight models that introduce multimodal vision capabilities and lightweight, on-device text models. This release expands the Llama ecosystem to include visual reasoning and efficient local execution for mobile and edge devices.
Llama 3.2 Vision: Multimodal Capabilities
Llama 3.2 Vision provides visual understanding and reasoning, enabling tasks such as document question answering, image-text retrieval, and visual grounding. The models are available in two sizes: 11B for consumer-grade GPU deployment and 90B for large-scale applications. Both sizes offer base and instruction-tuned variants.
Technical Architecture
Llama 3.2 Vision models are built by combining Llama 3.1 LLMs with a vision tower and an image adapter. Specifically, the 11B Vision model utilizes Llama 3.1 8B, and the 90B Vision model utilizes Llama 3.1 70B. To maintain text-only performance, the text models were frozen during the vision training process.
Key Specifications and Performance
- Context Length: 128k tokens, supporting multi-turn conversations with images.
- Training Data: Trained on 6 billion image-text pairs.
- Image Processing: The 11B base model uses tile sizes of 448, while the 90B and all instruction-tuned versions use tile sizes of 560.
- Language Support: In text-only mode, the models support English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
Vision Benchmarks
| Model | MMMU (val) | VQAv2 | DocVQA | AI2D |
|---|---|---|---|---|
| 11B | 41.7 | 66.8 (val) | 62.3 (val) | 62.4 |
| 11B (Instruct) | 50.7 (CoT) | 75.2 (test) | 88.4 (test) | 91.1 |
| 90B | 49.3 (zero-shot) | 73.6 (val) | 70.7 (val) | 75.3 |
| 90B (Instruct) | 60.3 (CoT) | 78.1 (test) | 90.1 (test) | 92.3 |
Llama 3.2 1B and 3B: On-Device Text Models
Llama 3.2 introduces 1B and 3B text-only models designed for local execution on devices. These models are intended for use cases such as prompt rewriting, summarization, tool usage, and locally running assistants.
Performance and Training
These models follow the Llama 3.1 architecture and were trained on up to 9 trillion tokens. They maintain a 128k token context length and support the same eight languages as the vision models.
Notably, the 3B Instruct model performs as strongly as the 8B model on the IFEval benchmark, making it highly effective for agentic applications that require strict instruction following.
Memory Requirements for Inference
| Model Size | BF16/FP16 | FP8 | INT4 |
|---|---|---|---|
| 3B | 6.5 GB | 3.2 GB | 1.75 GB |
| 1B | 2.5 GB | 1.25 GB | 0.75 GB |
Llama Guard 3: Safety and Safeguards
Meta has released Llama Guard 3 to classify model inputs and generations for safety. This includes:
- Multimodal Support: The main Llama Guard 3 can detect harmful multimodal prompts or assistant responses.
- Llama Guard 3 1B: A small version designed to be deployed alongside the 1B and 3B text models to evaluate responses in multi-turn conversations. This version allows developers to customize or exclude pre-defined safety categories.
Deployment and Ecosystem Integration
Llama 3.2 is integrated across several frameworks to facilitate immediate deployment:
- Hugging Face Transformers: Requires version 4.45.0 or later for Vision models.
- On-Device Frameworks:
- Llama.cpp & Llama-cpp-python: Supports cross-platform CPU/GPU inference with 4-bit and 8-bit quantized weights.
- Transformers.js: Enables execution in browsers or JavaScript runtimes (Node.js, Deno, Bun) via ONNX.
- MLC.ai Web-LLM: Provides hardware-accelerated in-browser inference using WebGPU.
- Fine-tuning: Supported via TRL (Transformer Reinforcement Learning) for both text and vision models, including LoRA fine-tuning via PEFT.
Licensing Restrictions
Llama 3.2 follows a license similar to Llama 3.1, with one critical exception: individuals or companies domiciled in the European Union are not granted license rights to use the multimodal models included in Llama 3.2. This restriction does not apply to end users of products that incorporate these models.