jingyaogong/minimind-v

👀 Train a 65M-parameter VLM from scratch in just 2h!

What it solves

MiniMind-V provides a lightweight, accessible way to build and train a Vision Language Model (VLM). It solves the high barrier to entry for VLM development by offering a minimal implementation that can be trained on a single consumer GPU (like an NVIDIA 3090) in a very short time and at a very low cost.

How it works

MiniMind-V extends a small language model (MiniMind) by adding a Visual Encoder and a projection module.

  • Visual Encoder: It uses the SigLIP2 model to extract image features, converting an image into a set of visual tokens.
  • Projection Module: An MLP (Multi-Layer Perceptron) projector translates these visual features into the semantic space of the language model, effectively acting as a "dictionary" that translates images into a language the LLM understands.
  • Training: The process involves two stages: an optional Pretrain stage to align visual tokens with language tokens (training only the projector) and a mandatory SFT (Supervised Fine-Tuning) stage to refine the model's ability to follow instructions and describe images (training the projector and the first/last layers of the LLM).

Who it’s for

  • AI Learners: Those wanting a clear, jargon-light tutorial on how VLMs are structured and trained.
  • Individual Developers: People with limited hardware who want to experiment with multimodal models on personal GPUs.
  • Researchers: Those looking for a minimal, baseline implementation of the VLM paradigm.

Highlights

  • Extremely Lightweight: The smallest version has only 65M parameters, making it significantly smaller than GPT-3.
  • Low Cost: Can be trained for approximately 3 RMB (roughly $0.40) in 2 hours on a single RTX 3090.
  • Comprehensive Pipeline: Includes full code for dataset cleaning, pretraining, and SFT.
  • Flexible Architecture: Supports both dense and Mixture-of-Experts (MoE) modes.
  • Efficient Data Handling: Uses Parquet format for integrated image-text storage to speed up loading.

Related

  • Project
    jingyaogong/minimind

    A lightweight framework for training ultra-small LLMs from scratch using native PyTorch, designed as a practical tutorial for understanding the full LLM lifecycle.

  • Dispatch
    nanoVLM: A Minimalist PyTorch Toolkit for Training Vision Language Models

    Hugging Face has released nanoVLM, a lightweight, pure PyTorch toolkit designed to simplify the training and understanding of Vision Language Models (VLMs) for beginners and researchers.

  • Dispatch
    Vision Language Models Explained

    Hugging Face provides a comprehensive guide to Vision Language Models (VLMs), detailing their architecture, open-source options, evaluation benchmarks, and new experimental support for fine-tuning via the TRL library.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Project
    zai-org/GLM-V

    A series of vision-language models (GLM-4.6V, 4.5V, 4.1V) that enable complex multimodal reasoning, native function calling, and precise visual grounding for agents.