EvolvingLMMs-Lab/lmms-engine

A simple, unified multimodal models training engine. Lean, flexible, and built for hacking at scale.

What it solves

LMMs-Engine provides a unified, lean framework for training multimodal models at scale. It eliminates the need for fragmented training scripts by offering a single engine capable of handling vision-language models (VLMs), diffusion models, and standard large language models (LLMs), while integrating high-performance distributed training and memory optimizations.

How it works

The engine uses a builder pattern to compose training pipelines, allowing users to initialize models from pretrained weights, apply custom data processors, and select specific trainers (such as hf_trainer for general VLMs or wan_trainer for video generation). It employs a factory pattern for datasets and processors to ensure extensibility.

To achieve scale and efficiency, it integrates several advanced techniques:

  • Distributed Training: Uses FSDP2 (PyTorch DTensor-based sharding) and Ulysses Sequence Parallelism to handle ultra-long contexts.
  • Kernel Optimizations: Implements Liger kernels (Triton-fused operations) and Flash Attention to reduce memory usage and eliminate padding computation.
  • Advanced Optimization: Includes the Muon optimizer, which uses Newton-Schulz orthogonalization for faster convergence than AdamW.
  • Flexible Patching: A monkey patching system allows the injection of model-specific optimizations at runtime without modifying the original source code.

Who it’s for

It is designed for AI researchers and engineers who need to pretrain or fine-tune large-scale multimodal models, particularly those dealing with high-resolution images, long videos, or complex multimodal (image, audio, text) inputs.

Highlights

  • Broad Model Support: Supports 20+ architectures including Qwen2.5-VL, Qwen3-VL MoE, WanVideo, and LLaVA-OneVision.
  • High Efficiency: Features sequence packing (first-fit bin packing) and Native Sparse Attention (NSA) for efficient long-context processing.
  • Unified Modalities: Capable of training models that unify vision, audio, and text in a single framework.
  • Production-Grade Tooling: Provides Docker images and native torchrun support for seamless cluster-scale deployment.

Related

  • Project
    baidu-baige/LoongForge

    A unified, high-performance training framework for LLMs, VLMs, diffusion, and embodied models that supports NVIDIA GPUs and Kunlun XPUs.

  • Project
    InternLM/xtuner

    A next-generation LLM training engine optimized for ultra-large-scale MoE models up to 1T parameters, supporting multimodal training and advanced reinforcement learning.

  • Project
    UbiquitousLearning/mllm

    A fast and lightweight multimodal LLM inference engine for mobile and edge devices that optimizes model execution across CPUs, GPUs, and NPUs.

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.