OptimalScale/LMFlow

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.

LMFlow – A toolbox for fine‑tuning large language models

What it is – LMFlow is an open‑source Python library that helps researchers and developers adapt (fine‑tune) large language models (LLMs) such as Llama‑3, Phi‑3, Llama‑2, GPT‑2, etc. It bundles scripts, utilities, and optional integrations that make the whole pipeline – data preparation, training, inference, and deployment – faster and easier.

Key capabilities

Area What LMFlow provides
Training • Full‑parameter fine‑tuning (updates every weight)
• LoRA (low‑rank adaptation) – a lightweight, parameter‑efficient method
• LISA – a new memory‑efficient algorithm that freezes most layers and only updates a few, enabling 7 B‑scale models on a single 24 GB GPU
• QLoRA (8‑bit/4‑bit quantized LoRA) for even lower memory use
• Gradient checkpointing, FlashAttention‑2, DeepSpeed ZeRO‑3, and other tricks to squeeze out speed and reduce GPU RAM.
Inference • Standard Hugging‑Face‑style inference
• vLLM and SGLang back‑ends for high‑throughput serving
• CPU‑only 4‑bit inference via llama.cpp conversion scripts
Deployment • Gradio UI for a quick chatbot demo
• Flask integration for custom web services
Extras • Optional packages (vllm, sglang, trl, deepspeed, flash_attn, ray, multimodal, gradio, flask) that can be installed on demand.

How you use it

  1. Install – pip install -e . (or add the needed extras).
  2. Prepare data – follow the docs for supported dataset formats (e.g., Alpaca).
  3. Run a training script – pick the method you want (run_finetune.sh for full, run_finetune_with_lora.sh, run_finetune_with_lisa.sh).
  4. Chat – after training, run_chatbot.sh launches a simple terminal chatbot; for a web UI, use the Gradio example.
  5. Deploy – launch a Flask or Gradio service, optionally with DeepSpeed for multi‑GPU scaling.

Hardware guidance – The README includes a handy table showing approximate GPU memory needed for different model sizes and training modes (e.g., a 7 B model needs ~120 GB for full‑precision fine‑tuning, but only ~16 GB with LoRA, ~10 GB with 8‑bit QLoRA, etc.).

Community & support – The project maintains a website, Discord, Slack, and WeChat groups, and it publishes regular updates (new model support, new algorithms like speculative decoding, conversation‑template presets, etc.).

Why it matters – Fine‑tuning LLMs is resource‑intensive; LMFlow bundles many of the latest memory‑saving tricks (LISA, LoRA, QLoRA, FlashAttention‑2, DeepSpeed) under a single, easy‑to‑install package, lowering the barrier for labs and hobbyists to adapt large models for specific tasks or domains.

Quick start snippet

# clone a stable tag
git clone -b v1.0.0 https://github.com/OptimalScale/LMFlow.git
cd LMFlow
conda create -n lmflow python=3.9 -y && conda activate lmflow
conda install mpi4py
pip install -e .

# fine‑tune GPT‑2 on Alpaca data
bash ./scripts/run_finetune.sh \
  --model_name_or_path gpt2 \
  --dataset_path data/alpaca/train_conversation \
  --output_model_path output_models/finetuned_gpt2

# chat with the result
bash ./scripts/run_chatbot.sh output_models/finetuned_gpt2

Related

  • Project
    zai-org/GLM-5

    GLM‑5 is an open‑source series of very large LLMs (320 B‑744 B total parameters) optimized for multimodal input, 1 M‑token contexts, and long‑horizon agentic tasks. It introduces hybrid sparse/linear attention, IndexShare, and a custom RL engine (slime) to achieve frontier‑level performance on coding and planning benchmarks while offering cheap FP8 inference. Models are downloadable from Hugging Face/ModelScope and can be served with SGLang, vLLM, KTransformers, etc., with optional ‘reasoning_effort’ control.

  • Project
    modelscope/ms-swift

    A large model and multimodal fine-tuning and deployment framework that supports over 1,000 models and provides a full pipeline from training to deployment.

  • Project
    meta-pytorch/torchtune

    torchtune is a PyTorch‑based library that provides ready‑to‑run recipes and configs for fine‑tuning, RL‑HF, distillation, quantization‑aware training, and inference of many modern LLMs (Llama, Gemma, Qwen, Phi, etc.). It abstracts distributed training, memory‑saving tricks, and device support behind a simple CLI (`tune`) and YAML files, integrating with Hugging Face, torch‑ao, FSDP2, and logging tools. Though development stopped in 2025, it remains a practical, open‑source toolkit for LLM experimentation.

  • Project
    sambanova/bloomchat

    BLOOMChat is a 176‑billion‑parameter multilingual chat LLM fine‑tuned from the open‑source BLOOM model. The repo provides data‑prep, tokenisation, and training scripts (the latter for SambaNova’s RDU hardware) plus detailed GPU inference instructions using Hugging Face’s Bloom inference code. It’s a genuine, large‑scale LLM project aimed at researchers and developers who want to reproduce or run the model.

  • Dispatch
    2023 Year of Open LLMs Review

    Hugging Face’s 2023 recap shows a surge of open‑source LLM releases, smaller high‑performing models, and new fine‑tuning techniques that dramatically broaden access and community participation.