SmolVLM release notes / what's new
Hugging Face has released SmolVLM, a 2B parameter Vision Language Model (VLM) designed for efficiency, speed, and local deployment. The model is fully open-source, with all checkpoints, datasets, training recipes, and tools released under the Apache 2.0 license.
Model Variants and Ecosystem
SmolVLM is available in three distinct versions to support different use cases:
- SmolVLM-Base: Designed for downstream fine-tuning.
- SmolVLM-Synthetic: A variant fine-tuned on synthetic data.
- SmolVLM-Instruct: An instruction-tuned version ready for interactive end-user applications.
The ecosystem includes integration with the transformers library, a supervised fine-tuning script, and a public demo. The models were trained using open-source datasets, specifically The Cauldron and Docmatix.
Technical Architecture
SmolVLM's architecture is based on Idefics3 but introduces several optimizations to reduce memory and compute requirements:
- Language Backbone: Replaces Llama 3.1 8B with SmolLM2 1.7B.
- Visual Compression: Employs a pixel shuffle strategy to compress patched visual information by 9x (compared to 4x in Idefics3).
- Image Patching: Uses 384x384 patches (divisible by 3 for the pixel shuffle strategy) instead of 364x364.
- Vision Backbone: Utilizes a shape-optimized SigLIP with 384x384 patches and 14x14 inner patches.
Performance and Efficiency
Benchmarks
SmolVLM demonstrates competitive performance across several multimodal benchmarks while maintaining a significantly lower memory footprint than other 2B-class models:
| Model | MMMU (val) | MathVista (testmini) | MMStar (val) | DocVQA (test) | TextVQA (val) | Min GPU RAM (GB) |
|---|---|---|---|---|---|---|
| SmolVLM | 38.8 | 44.6 | 42.1 | 81.6 | 72.7 | 5.02 |
| Qwen2-VL 2B | 41.1 | 47.8 | 47.5 | 90.1 | 79.7 | 13.70 |
| InternVL2 2B | 34.3 | 46.3 | 49.8 | 86.9 | 73.4 | 10.52 |
| PaliGemma 3B | 34.9 | 28.7 | 48.3 | 32.2 | 56.0 | 6.72 |
| moondream2 | 32.4 | 24.3 | 40.3 | 70.5 | 65.2 | 3.87 |
Memory and Throughput
SmolVLM is highly efficient in image encoding, converting each 384x384 image patch into only 81 tokens. For a single image and prompt, SmolVLM uses approximately 1.2k tokens, whereas Qwen2-VL uses 16k tokens. This efficiency leads to:
- Reduced RAM: The lowest memory usage among the existing suite of VLMs in
transformers. - Higher Throughput: Prefill throughput is 3.3 to 4.5 times faster, and generation throughput is 7.5 to 16 times faster than Qwen2-VL.
Video Analysis
SmolVLM can be used for basic video analysis by extracting up to 50 evenly sampled frames. In tests on the CinePile benchmark, it achieved a score of 27.14%, placing its performance between InternVL2 (2B) and Video LlaVa (7B).
Training and Fine-Tuning
Context Extension
To support multiple images and long sequences, the pre-training context window of SmolLM2 was extended to 16k tokens. This was achieved by increasing the RoPE base value from 10k to 273k and fine-tuning on a mixture of long-context (books from Dolma and code from The Stack) and short-context (FineWeb-Edu, DCLM, and a math dataset) data.
Checkpoint Selection
Optimal checkpoints were selected using a weighted metric across MMMU, MMStar, DocVQA, TextVQA, MathVista, and AI2D. The team noted that while general performance typically improved with more training, performance on DocVQA tended to decrease over time.
Customization
SmolVLM can be fine-tuned using the transformers and TRL libraries. A provided notebook demonstrates fine-tuning on the VQAv2 dataset using LoRA or QLoRA. With 8-bit loading and gradient checkpointing, the model can be fine-tuned on a consumer GPU (e.g., L4) using approximately 16 GB of VRAM.