ByteDance-Seed/VeOmni

VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo

What it solves

VeOmni addresses the complexity of scaling the training of single- and multi-modal models across different hardware accelerators. It provides a flexible, modular framework that removes the rigidity of traditional trainer classes, allowing developers to have more direct control over training logic while scaling large models (including MoE and omni-modality models) efficiently.

How it works

VeOmni leverages a "model-centric distributed recipe zoo" to manage training configurations. It utilizes PyTorch native functions and several advanced distributed training techniques to optimize performance:

  • FSDP2: Used as the primary backend for training.
  • Sequence Parallelism: Implemented via Deepspeed Ulysses (supporting both async and non-async modes).
  • Experts Parallelism: Enables the training of large Mixture-of-Experts (MoE) models.
  • Hardware Compatibility: Supports NVIDIA GPUs, AMD ROCm, and Ascend NPUs.
  • Modular Design: Decouples components so users can replace them with custom implementations or use linear training scripts instead of structured trainer classes.

Who it’s for

It is designed for AI researchers and engineers who need to pre-train or post-train large-scale multimodal or text-only models and require high transparency, hardware flexibility, and the ability to scale across diverse accelerators.

Highlights

  • Trainer-free approach: Supports linear training scripts for maximum transparency and control over training logic.
  • Broad Model Support: Compatible with HuggingFace Transformers and supports models like DeepSeek, Llama 3, Qwen series (including VL and MoE), and Wan.
  • Multi-Accelerator Support: Works across NVIDIA, AMD, and Ascend hardware.
  • Advanced Parallelism: Integrates FSDP2, Sequence Parallelism, and Experts Parallelism for massive model scaling.

Related

  • Project
    verl-project/verl-omni

    A general RL training framework for multimodal generative models, providing fast rollouts and stable post-training for diffusion and omni-modality models.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.

  • Dispatch
    VeRL-Omni: RL Training Framework for Diffusion and Omni-Modality Models

    vLLM has announced the pre-release of VeRL-Omni, a general reinforcement learning post-training framework designed specifically for multimodal generative models, including diffusion and omni-modality architectures.

  • Project
    PrimeIntellect-ai/prime-rl

    A framework for large-scale asynchronous reinforcement learning designed to train frontier models with 1T+ parameters on up to 1000+ GPUs.