baidu-baige/LoongForge

A unified, high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs.

What it solves

LoongForge is a high-performance training framework designed to unify the training of Large Language Models (LLMs), Vision-Language Models (VLMs), diffusion models, and embodied AI models (VLA/WAM). It addresses the inefficiency and fragmentation of training different model types across various hardware, providing a single stack that supports pre-training, continued pre-training, and supervised fine-tuning (SFT).

How it works

LoongForge utilizes multiple distributed backends depending on the model type: LLMs, VLMs, and diffusion models run on a patched version of Megatron-LM, while embodied models use a torch-native DDP/FSDP stack. The framework is supports both NVIDIA GPUs and Kunlun XPUs. It employs several optimization techniques to increase throughput:

  • Heterogeneous Parallelism: Allows independent tensor parallelism (TP), data parallelism (DP), and recomputation for different model components (e.g., separating ViT and LLM).
  • MoE Optimization: Includes overlapped All2All, activation offload, and a topology-aware expert load balancing algorithm.
  • Adaptive FP8 Training: Uses blockwise FP8 with an optional adaptive mode that selects precision based on operator efficiency.
  • DP Load Balancing: Redistributes data to mitigate sequence-packing imbalance.
  • Custom Fused Operators: Implements high-performance kernels, such as FusedDSA, to accelerate specific model architectures.

Who it’s for

LoongForge is intended for researchers and engineers working on large-scale model training, particularly those needing to train multimodal or embodied AI models across heterogeneous hardware (NVIDIA and Kunlun XPU).

Highlights

  • Unified Framework: Supports LLMs, VLMs, Diffusion, and Embodied models in one place.
  • Broad Model Support: Compatible with a wide range of SOTA models including DeepSeek, Qwen, LLaMA, and GR00T.
  • Hardware Agnostic: Native support for both NVIDIA GPUs and Kunlun XPUs.
  • Significant Speedups: Demonstrates substantial training acceleration over baselines (e.g., 5.04x for DeepSeek-V3.2 Lite).
  • Flexible Composition: Configuration-driven assembly of VLMs from interchangeable components.
  • Seamless Checkpointing: Bidirectional conversion between Megatron and HuggingFace formats.