baidu-baige/LoongForge

A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs.

What it solves

LoongForge is a high-performance training framework designed to accelerate the training of Large Language Models (LLMs), Vision-Language Models (VLMs), diffusion models, and embodied AI models. It aims to reduce training costs and time by optimizing parallelism strategies, memory usage, and kernel efficiency while maintaining the same training loss curves as baseline implementations.

How it works

LoongForge uses a multi-backend architecture to apply the best training strategy for different model types:

  • Megatron Stack: A patched version of Megatron-LM used for LLMs, VLMs, and diffusion models. It includes optimizations for Mixture-of-Experts (MoE) parallelism, long-sequence training via Context Parallel (CP), and adaptive FP8 training.
  • Torch-Native Stack: A dedicated subsystem for embodied models (VLA and WAM) using DDP, ZeRO-1, FSDP, and HSDP strategies, decoupled from the Megatron core to maximize throughput for robotics-focused models.

Who it’s for

It is intended for researchers and engineers training large-scale foundation models across multiple modalities, specifically those using NVIDIA GPUs or Kunlun XPUs, and those needing to scale training runs to thousands of accelerators.

Highlights

  • MoE Optimizations: Features topology-aware dynamic expert replica placement (TAOT) to balance workloads and reduce communication overhead.
  • Flexible Composition: Allows assembling VLMs from interchangeable ViT and LLM components via configuration without writing custom code.
  • Heterogeneous Parallelism: Supports independent parallelism settings (TP/DP/recompute) for different model components (e.g., ViT vs. LLM).
  • Embodied AI Support: Specialized training and evaluation modules for VLA and world-action models (e.g., Pi0.5, GR00T).
  • Hardware Compatibility: Native support for both NVIDIA GPUs and Kunlun XPUs.

Related

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    PaddlePaddle/PaddleFormers

    A high-performance Transformers library for PaddlePaddle that enables efficient pre-training and post-training of LLMs and VLMs with extensive support for distributed strategies and domestic AI hardware.

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning that uses CPU-GPU heterogeneous computing to run ultra-large MoE models on consumer hardware.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.