TinyLLaVA/TinyLLaVA_Factory

A Framework of Small-scale Large Multimodal Models

What it solves

TinyLLaVA Factory provides a modularized codebase for creating and training small-scale Large Multimodal Models (LMMs). It simplifies the process of customizing multimodal models by reducing coding effort and errors, while ensuring training results are reproducible.

How it works

The framework allows users to mix and match different components to build a multimodal model. It integrates several key building blocks:

  • LLMs: Supports models like OpenELM, TinyLlama, StableLM, Qwen, Gemma, and Phi.
  • Vision Towers: Supports CLIP, SigLIP, Dino, and combinations of CLIP and Dino.
  • Vision-Language Connectors: Supports MLP, Qformer, and Resampler.
  • Training Recipes: Includes options for Frozen, Fully, and Partially tuning, as well as LoRA and QLoRA.

Who it’s for

It is designed for researchers and developers who want to build, fine-tune, or evaluate small-scale multimodal models without starting from scratch.

Highlights

  • Modular Architecture: Easy to swap LLMs, vision towers, and connectors.
  • High Performance: The TinyLLaVA-Phi-2-SigLIP-3.1B model outperforms some existing 7B models like LLaVA-1.5.
  • Extensible: Provides clear guides and registration patterns for adding new LLMs, vision towers, and connectors.
  • Comprehensive Tooling: Includes scripts for pretraining, finetuning, and evaluation, as well as a Gradio web demo and CLI inference tools.

Related

  • Project
    hiyouga/EasyR1

    An efficient and scalable reinforcement learning training framework that extends support to vision language models and multiple RL algorithms like GRPO.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.

  • Project
    hiyouga/LlamaFactory

    LlamaFactory is an open‑source toolkit that lets you fine‑tune, evaluate, and serve over 100 large language models with a simple YAML config, offering both a command‑line interface and a Gradio‑based web UI. It supports many modern training methods (LoRA, QLoRA, RLHF, PPO, DPO, etc.), a wide range of models (LLaMA‑3/4, Qwen 3, Mixtral‑MoE, Gemma 3, etc.), and runs on NVIDIA, AMD, Ascend NPU, or cloud services. Integrated monitoring (TensorBoard, WandB, SwanLab) and deployment options (OpenAI‑style API, vLLM/SGLang) make it suitable for researchers, product teams, and educators.

  • Project
    jingyaogong/minimind-v

    A minimal implementation of a Vision Language Model (VLM) that allows users to train a 65M parameter multimodal model on a single consumer GPU in just 2 hours.