baidu-baige/LoongForge

A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs. Supports Pi0.5, GR00T-N1.7, FastWAM, Cosmos3, DreamZero, DeepSeek-V4, GLM-5.3, Kimi-K3 and more.

LoongForge – High‑performance training framework for LLMs, VLMs, diffusion and embodied models

What it is – LoongForge is an open‑source training stack from Baidu AI Cloud’s Baige team. It wraps a patched Megatron‑LM core (for language, vision‑language and diffusion models) and a separate torch‑native subsystem (for embodied agents) and adds a large collection of speed‑up tricks: MoE expert‑placement, FP8/INT4 support, long‑sequence pipelines, heterogeneous parallelism, custom fused kernels, etc.

Why it matters – Training today’s foundation models can cost thousands of GPU‑hours. LoongForge claims 1.4×‑5× higher throughput than the vanilla Megatron‑LM or other open‑source baselines while keeping loss curves identical, and it works on both NVIDIA GPUs and Baidu’s Kunlun XPUs.

Quick start (GPU)

# Pull the official Docker image (includes CUDA, Megatron patches, etc.)
docker pull loongforge/loongforge:latest
mkdir -p workspace
docker run --gpus all --ipc=host -it --rm \
  -v "$(pwd)/workspace:/workspace/data" \
  -w /workspace/LoongForge \
  loongforge/loongforge:latest bash

Inside the container you can launch any of the ready‑made scripts under examples/. For instance, a 5‑B DreamZero LoRA fine‑tune on 8 GPUs:

cd /workspace/LoongForge
export WAN22_CKPT_DIR=/workspace/data/dreamzero/checkpoints/Wan2.2-TI2V-5B
export WAN21_CKPT_DIR=/workspace/data/dreamzero/checkpoints/Wan2.1-I2V-14B-480P
export TOKENIZER_PATH="$WAN22_CKPT_DIR/google/umt5-xxl"
export DATA_PATH=/workspace/data/dreamzero/data/droid_lerobot

bash examples/embodied/dreamzero/run_dreamzero_wan22_5b_lora_fsdp_finetune.sh \
  GPUS_PER_NODE=8 TRAIN_ITERS=20 SAVE_INTERVAL=20 \
  OUTPUT_DIR=/workspace/data/dreamzero/outputs/lora

The script runs a short 20‑step test and writes checkpoints to OUTPUT_DIR.

Key capabilities

Area Highlights
LLM / VLM / Diffusion Patched Megatron‑LM with MoE expert‑parallel (EP) overlap, topology‑aware dynamic expert replication (TAOT paper), adaptive FP8/INT4 training, context‑parallel long‑sequence support, composable ViT+LLM models (e.g., GLM‑5.2 + MoonViT).
Embodied agents Separate torch‑native stack offering DDP / ZeRO‑1 / FSDP / HSDP, per‑model I/O and kernel optimizations, FP8 communication, unified evaluation on LIBERO, CALVIN, SimplerEnv, RoboTwin, and data conversion from first‑person video to LeRobot format.
Hardware Runs on NVIDIA GPUs (CUDA) and Baidu Kunlun XPUs via a lightweight plugin; Docker images provide a ready‑to‑run environment.
Usability Ready‑to‑run config files (configs/models/), launch scripts (examples/), and a “Mcore Bridge” that converts between Megatron checkpoints and HuggingFace format both offline and on‑the‑fly.

Model coverage (selected)

  • LLMs: LLaMA 2/3/3.1, Qwen 1‑5/2/2.5/3/3‑Next, DeepSeek‑V2/V3/V4, MiniMax‑M2.1/2.5/2.7, GLM‑5/5.2/5.3‑flash, Kimi‑K2.5/2.6/K3, MiniCPM‑V‑4.6, etc.
  • VLMs: Qwen2.5‑VL, Qwen3‑VL, Qwen3.5/3.6/3.8, Kimi‑K2.5/2.6/K3, MiniCPM‑V‑4.6, GLM‑5.2 + MoonViT, ERNIE4.5‑VL, LLaVA‑OneVision 1.5/2.0, InternVL 2.5/3.5, custom composable models.
  • Diffusion: Wan2.1, Wan2.2, Qwen‑Image‑Edit‑2511.
  • Embodied: Pi0.5, GR00T‑N1.6/N1.7, xVLA, Wall‑OSS‑0.5, FastWAM, LingBot‑VA, Cosmos3, DreamZero.

Reported performance

The README shows benchmark speed‑ups (training throughput) over the corresponding open‑source baselines on identical hardware:

  • 1.45× on Qwen3‑VL
  • up to 5.04× on DeepSeek‑V3.2 Lite (DSA‑optimized variant)
  • 4.38× on DreamZero (embodied) compared with the baseline
  • 2.3× on GR00T‑N1.6, 1.79×‑4.38× across various embodied models All gains are claimed to keep loss curves aligned with the un‑optimized baselines.

Community & resources

  • Docs: https://loongforge.readthedocs.io
  • Blog & release notes: linked throughout the README (e.g., TAOT paper, embodied launch)
  • Docker Hub: loongforge/loongforge
  • Discord / WeChat / RedNote groups for discussion
  • GitHub: open‑source under Apache‑2.0, welcomes PRs, issue tracker for bugs and feature requests.

Bottom line

LoongForge is a production‑grade, multi‑backend training framework that targets the full spectrum of frontier models—large language, multimodal vision‑language, diffusion, and robot‑control agents. Its main selling point is a suite of engineering optimizations that promise several‑fold speed‑ups on both GPU and Baidu’s XPU hardware, while providing a fairly complete set of ready‑to‑run examples and a Docker‑based entry point for newcomers.

Related

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.

  • Project
    ByteDance-Seed/VeOmni

    A versatile framework for scaling the pre-training and post-training of single- and multi-modal models across various hardware accelerators.

  • Project
    kvcache-ai/ktransformers

    KTransformers is a research framework for efficient LLM inference and fine-tuning via CPU-GPU heterogeneous computing, enabling large MoE models like DeepSeek-R1 to run on a single consumer GPU (24GB VRAM) with 3-28x speedups, and offering fine-tuning with 6-12x speedup over ZeRO-Offload.

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.