ByteDance-Seed/VeOmni

VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo

What it solves

VeOmni is designed to simplify and scale the training of single- and multi-modal models. It addresses the complexity of scaling models across different hardware accelerators and the rigidity of traditional trainer classes, providing a flexible, modular framework for both pre-training and post-training.

How it works

VeOmni leverages a modular design that decouples components, allowing users to replace them with custom implementations. It uses a "trainer-free" approach with linear training scripts to give users full transparency and control over training logic. The framework is built on PyTorch native functions and supports various distributed training techniques including FSDP2, Sequence Parallelism (via Deepspeed Ulysses), and Experts Parallelism for large MoE models.

Who it’s for

It is intended for AI researchers and engineers who need to scale large-scale multi-modal or omni-models across diverse hardware (NVIDIA GPUs, AMD ROCm, and Ascend NPUs) and require high levels of control over the training process.

Highlights

  • Multi-Accelerator Support: Compatible with NVIDIA GPU, AMD ROCm, and Ascend NPU.
  • MoE Optimization: Includes Experts Parallelism and efficient GroupGemm kernels (Liger-Kernel) for Mixture-of-Experts models.
  • Broad Model Support: Compatible with HuggingFace Transformers models like Qwen, Llama, and DeepSeek.
  • Distributed Training: Implements FSDP2, Sequence Parallelism, and Torch Distributed Checkpoint.
  • Flexible Training Logic: Supports linear scripts to avoid rigid trainer classes.

Related

  • Project
    verl-project/verl-omni

    VeRL‑Omni is an open‑source RL training framework for multimodal generative models (diffusion, unified, and omni‑modality). Built on verl and vLLM‑Omni, it provides fast rollout, asynchronous multi‑reward serving, distributed back‑ends, and many RL algorithms (FlowGRPO, DiffusionOPD, DPO, etc.). Supports GPUs and Ascend NPUs, with extensive docs, examples, and Apache‑2.0 licensing.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Dispatch
    VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

    vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.

  • Project
    baidu-baige/LoongForge

    LoongForge is Baidu’s open‑source training framework for LLMs, VLMs, diffusion and embodied models. It combines a patched Megatron‑LM stack with a torch‑native subsystem, adds MoE expert‑placement, FP8/INT4 support, long‑sequence pipelines and other speed‑ups, and ships ready‑made Docker images and launch scripts for dozens of popular models.