SmolVLM2: Bringing Video Understanding to Every Device

Hugging Face has introduced SmolVLM2, a series of efficient multimodal models designed to bring video and image understanding to local devices. By releasing models in three sizes—2.2B, 500M, and 256M parameters—SmolVLM2 enables high-performance visual analysis on hardware ranging from mobile phones to servers without requiring massive computing resources.

Model Variants and Performance

SmolVLM2 provides three distinct model sizes to balance performance and memory consumption, with the 500M and 256M versions representing the smallest video language models released to date.

SmolVLM2 2.2B

The 2.2B model serves as the primary choice for general vision and video tasks. It demonstrates significant improvements over the previous SmolVLM family in several key areas:

  • Mathematical Reasoning: Enhanced ability to solve math problems using images.
  • OCR and Diagrams: Improved text reading in photos and understanding of complex diagrams.
  • Scientific QA: Better performance on scientific visual questions.

On the Video-MME benchmark, the 2.2B model outperforms all existing models in the 2B parameter range. Its memory efficiency allows it to run on free tiers of Google Colab.

SmolVLM2 500M and 256M

The 500M model provides video understanding capabilities very close to the 2.2B model while utilizing less than a quarter of the parameters. The 256M model is an experimental release intended to push the boundaries of how small a video-capable model can be, serving as a foundation for specialized fine-tuning and creative applications.

Practical Applications and Demos

To demonstrate the utility of small-scale video models, Hugging Face has developed three specific applications:

  • iPhone Video Understanding: A local application using the 500M model that analyzes video content directly on the device without cloud connectivity.
  • VLC Media Player Integration: A collaboration with VLC to enable semantic search and navigation within videos, allowing users to jump to specific sections based on natural language descriptions.
  • Video Highlight Generator: A tool capable of processing long-form videos (over one hour), such as soccer matches, to automatically extract the most significant moments.

Technical Implementation and Deployment

SmolVLM2 is integrated with the transformers library and MLX for Apple Silicon optimization from launch.

Transformers Integration

Inference is handled via a conversational API using chat templates. The models support:

  • Video Inference: Processing videos by passing a path via {"type": "video", "path": "path_to_video.mp4"}.
  • Multi-Image Inference: Conducting conversations involving multiple images provided via URLs, file paths, or PIL objects.

MLX and Apple Silicon

For deployment on Apple Silicon, SmolVLM2 supports the mlx-vlm library in Python and the Swift language via mlx-swift-examples. The use of system prompts is highlighted as a critical factor in controlling the level of detail and behavior of the model during video analysis.

Fine-Tuning

SmolVLM2 can be fine-tuned on video-caption pairs using the transformers library. For the 500M variant, full fine-tuning is recommended over QLoRA or LoRA due to its small size, while QLoRA remains an option for the 2.2B variant.

Sources