HITsz-TMG/Uni-MoE

Uni-MoE: Lychee's Large Multimodal Model Family.

What it solves

Uni-MoE addresses the challenge of creating a single, unified model capable of understanding and generating across multiple modalities (omnimodality) without sacrificing performance in any specific domain. It aims to overcome data imbalance and task conflicts that typically occur when training models to handle diverse inputs like text, images, speech, and music simultaneously.

How it works

The project utilizes a Mixture-of-Experts (MoE) architecture to scale capabilities across different versions:

  • Uni-MoE 2.0: Built on a Qwen2.5-7B dense architecture, it uses a dynamic-capacity MoE design, a progressive training strategy with iterative reinforcement, and curated multimodal data matching to enable cross- and tri-modality understanding and generation of images, text, and speech.
  • Uni-MoE-Audio: Employs a dynamic-capacity routing mechanism based on Top-P sampling for adaptive expert allocation and a hybrid expert design that separates shared universal representations from domain-specific dynamic experts. It uses a three-stage training curriculum to handle tasks ranging from voice cloning and TTS to text-to-music and video-to-music.
  • Uni-MoE 1.0: Uses a three-stage process: first building connectors to map modalities to a unified language space, then developing modality-specific experts, and finally integrating these into an LLM and refining them via LoRA on mixed multimodal data.

Who it’s for

Researchers and developers working on omnimodal AI, universal audio generation, and scalable multimodal LLMs who need a model that can reason and generate across text, vision, and audio.

Highlights

  • Omnimodal Capabilities: Supports understanding and generation across ten-plus modalities, including text, images, and speech.
  • Dynamic MoE: Uses adaptive expert allocation to optimize computation based on the input modality.
  • Unified Audio Generation: The Audio version is the first to unify speech and music generation in one model.
  • Scalable Training: Supports distributed MoE modules for parallel processing at both expert and modality levels.

Related

  • Project
    inclusionAI/Ming

    Ming-flash-omni 2.0 is an open-source omni-MLLM that unifies multimodal perception and generation across text, image, audio, and video in a single MoE-based architecture.

  • Project
    PaddlePaddle/ERNIE

    A family of large-scale multimodal MoE models and development toolkits providing state-of-the-art performance in text and vision-language understanding.

  • Dispatch
    Mixture of Experts (MoE) Explained

    Mixture of Experts (MoE) allows transformer models to scale parameters while maintaining efficient pretraining and faster inference by activating only a subset of neural network experts per token.

  • Project
    microsoft/Tutel

    An optimized Mixture-of-Experts (MoE) implementation that accelerates training and inference for massive LLMs on NVIDIA and AMD GPUs using advanced parallelism and quantization.

  • Project
    OpenSenseNova/SenseNova-U1

    SenseNova-U1 is a native multimodal model series that unifies understanding, reasoning, and generation within a single architecture, eliminating the need for modality adapters.