merveenoyan/smol-vision

What it solves

Smol Vision provides a collection of recipes for shrinking, optimizing, and customizing cutting-edge vision and multimodal AI models. It addresses the challenge of making large vision-language models (VLMs) and foundation models more efficient for deployment on smaller hardware or specific tasks.

How it works

The project consists of a series of notebooks and scripts that demonstrate various optimization and fine-tuning techniques:

• Optimization: Uses tools like 🤗 Optimum, ONNX Runtime, and Quanto to perform quantization and graph optimizations to reduce model size and reduce latency.
• Fine-tuning: Provides recipes for QLoRA and full fine-tuning of models like PaliGemma, Florence-2, IDEFICS3, and SmolVLM on specific datasets (e.g., VQAv2, DocVQA).
• Multimodal RAG: Implements retrieval-augmented generation using ColPali and Qwen2-VL to retrieve documents without heavy processing.
• Any-to-Any: Demonstrates fine-tuning and RAG across multiple modalities including audio, text, image, and video using models like Gemma-3n and OmniEmbed.

Who it’s for

AI developers and researchers who want to optimize vision and multimodal models for performance, size, and memory efficiency, or customize them for specialized vision tasks.

Highlights

• Comprehensive Optimization: Covers quantization, knowledge distillation, and torch.compile for speed-ups.
• VLM Customization: Recipes for grounded fine-tuning and specialized VLM fine-tuning for OCR and document QA.
• Multimodal RAG: Workflows for multimodal document retrieval and generation.
• Any-to-Any Capabilities: Support for fine-tuning and RAG across audio, text, image, and video modalities.