SmolVLM 256M and 500M Release Notes

Hugging Face has released SmolVLM-256M and SmolVLM-500M, two new lightweight Vision Language Models (VLMs) designed for constrained devices, consumer laptops, and browser-based inference. The 256M model is currently the smallest VLM in the world, yet it outperforms the Idefics 80B model released 17 months prior.

Model Variants and Capabilities

Two model sizes are available, each provided as a base model and an instruction fine-tuned version:

  • SmolVLM-256M: The smallest VLM available, capable of image and short video captioning, document Q&A (PDFs and scanned text), and basic visual reasoning involving charts and diagrams.
  • SmolVLM-500M: A half-billion-parameter model that provides higher performance headroom and greater robustness to prompting, making it more suitable for out-of-the-box production use. It pushes performance on MMMU and DocVQA closer to larger models.

Technical Improvements over SmolVLM 2B

Hugging Face implemented several design trade-offs and optimizations to reduce the footprint while preserving multimodal performance:

Vision Encoder Optimization

Instead of the SigLIP 400M SO encoder used in SmolVLM 2B, these models use the SigLIP base patch-16/512 (93M parameters). Testing showed that the larger encoder offered only marginal gains. The smaller encoder also processes images at a larger resolution, which improves visual understanding with minimal overhead.

Data Mixture Updates

Training data was sourced from The Cauldron and Docmatix, with the addition of MathWriting. The data proportions were adjusted to emphasize:

  • Document Understanding: 41%
  • Image Captioning: 14%

Tokenization Enhancements

To increase efficiency, the models now encode images at a rate of 4096 pixels per token, up from 1820 pixels per token in the 2B version. Additionally, special tokens were introduced to replace multi-token strings for sub-image separators (e.g., mapping <row_1_col_1> through <row_6_col_6> to single tokens), which improved training stability and result quality.

Ecosystem and Integration

Multimodal Retrieval with ColSmolVLM

Using the ColBERT-like retrieval architecture, the team developed ColSmolVLM 256M and 500M. These models deliver state-of-the-art multimodal retrieval speeds with performance rivaling models 10x their size, reducing the cost and time required to build searchable databases.

Document Processing with SmolDocling

In partnership with IBM, Hugging Face is integrating these models into Docling. Early results indicate that the 256M model is highly effective for document layout and table understanding.

Deployment and Framework Support

SmolVLM-256M and 500M are compatible with existing SmolVLM code and support the following frameworks:

  • Transformers: Direct loading for inference and fine-tuning.
  • MLX: Optimized for Apple Silicon.
  • ONNX: Checkpoints are provided for WebGPU demos and browser-based inference.
  • TRL: Supported for alignment and Direct Preference Optimization (DPO).

Sources