PaddlePaddle/ERNIE

What it solves

ERNIE 4.5 is a family of large-scale multimodal models designed to provide state-of-the-art performance in text understanding, image/video understanding, and cross-modal reasoning. It addresses the challenge of training multimodal models where one modality might hinder the learning of another, while providing a scalable infrastructure for efficient training and high-performance inference across various hardware platforms.

How it works

The project utilizes a heterogeneous Mixture-of-Experts (MoE) architecture that allows for parameter sharing across modalities while maintaining dedicated parameters for each. This is supported by modality-isolated routing and specific loss functions (router orthogonal and multimodal token-balanced loss) to ensure mutual reinforcement between text and visual data. The models are built on the PaddlePaddle deep learning framework and employ hybrid parallelism, FP8 mixed-precision training, and advanced quantization (4-bit/2-bit) for efficient deployment.

Who it’s for

This project is for AI researchers and developers who need high-performance LLMs or Vision-Language Models (VLMs) for tasks such as instruction following, world knowledge memorization, and visual reasoning. It also provides tools for those looking to fine-tune models using SFT, DPO, or LoRA.

Highlights

• Multimodal Heterogeneous MoE: A novel structure that enhances multimodal understanding without compromising text performance.
• Scaling-Efficient Infrastructure: High pre-training throughput via intra-node expert parallelism and memory-efficient pipeline scheduling.
• Thinking and Non-Thinking Modes: VLMs support both modes to balance perception and complex reasoning.
• ERNIEKit & FastDeploy: Industrial-grade toolkits for the full lifecycle of training, compression, and production-ready inference.