mbzuai-oryx/LLaVA-pp

What it solves

LLaVA++ extends the visual capabilities of the LLaVA 1.5 model by integrating it with more modern and powerful Large Language Models (LLMs) like LLaMA-3 and Phi-3. This allows for improved performance on instruction-following and academic-task-oriented visual tasks.

How it works

The project provides a framework to integrate LLaVA's visual architecture with Phi-3 Mini Instruct (3.8B) and LLaMA-3 Instruct (8B). It supports multiple training stages, including pre-training on the LCS-558K dataset and fine-tuning using LoRA or full fine-tuning (including S2 fine-tuning) on the LLaVA-Instruct-665K dataset.

Who it’s for

Developers and researchers in the field of multimodal AI who want to use or train multimodal models based on the latest LLM backbones

Highlights

• Integration with LLaMA-3 Instruct 8B and Phi-3 Mini Instruct 3.8B.
• Support for LoRA, full fine-tuning, and S2 fine-tuning methods.
• Pre-trained and fine-tuned model weights available in the Model-Zoo.
• Provided scripts for pre-training and fine-tuning the models.