hiyouga/LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
What it solves
LLaMA Factory provides a unified, zero-code interface for fine-tuning a vast array of large language models (LLMs). It removes the technical complexity of managing different training scripts and hyperparameters for over 100 different models, allowing users to adapt models to specific tasks without writing extensive code.
How it works
The project offers a comprehensive toolkit that integrates various training methods and optimization algorithms. Users can interact with the system via a Command Line Interface (CLI) or a web-based GUI called LlamaBoard. It supports a wide range of fine-tuning techniques, including full-tuning, freeze-tuning, and parameter-efficient methods like LoRA and QLoRA (across multiple bit-depths), as well as advanced optimizers and preference learning algorithms (PPO, DPO, KTO, ORPO).
Who it’s for
It is designed for developers and AI researchers who need to fine-tune LLMs for specific domains or tasks—such as multi-turn dialogue, tool use, or multimodal understanding—without needing to build their own training pipelines from scratch.
Highlights
- Extensive Model Support: Compatible with 100+ models including LLaMA, Mistral, Qwen, DeepSeek, and Gemma.
- Zero-Code Interface: Offers both a CLI and a Gradio-powered Web UI (LlamaBoard) for easy configuration.
- Scalable Tuning: Supports everything from 16-bit full-tuning to highly compressed QLoRA (2-bit to 8-bit).
- Multimodal Capabilities: Supports fine-tuning for image understanding, visual grounding, video recognition, and audio understanding.
- High-Performance Inference: Integrates with vLLM and SGLang workers for accelerated model deployment via OpenAI-style APIs.