Baidu ERNIE 5.5 Flash Base Release Notes

Baidu has released ERNIE 5.5 Flash Base, a sparse Mixture-of-Experts (MoE) language model featuring 280B total parameters with 14B activated per token. This release provides a raw pre-training checkpoint, explicitly excluding mid- or post-training, to serve as a foundational baseline for developers interested in domain adaptation, instruction tuning, and reinforcement learning research.

Model Architecture and Efficiency

ERNIE 5.5 Flash is built with a focus on balancing model capacity with computational efficiency. The architecture utilizes a 43-layer backbone with several specialized components:

  • Hybrid Sparse Attention: The model employs a mix of Multi-head Latent Attention (MLA) and Heavily Compressed Attention (HCA). Most layers use HCA to compress context, while a subset of layers uses MLA to maintain token-level access to the full context, optimizing the KV-cache budget.
  • Multi-Router Latent MoE: Replacing traditional single-router MoE designs, this model uses multiple independently initialized sigmoid routers. This approach improves expert selection accuracy and reduces sensitivity to noisy features.
  • ECHO (Efficient Contextual Head Optimization): Replacing standard Multi-Token Prediction (MTP) modules, ECHO reuses the backbone’s final layer to predict future tokens, encouraging the model to anticipate sequence progression without the overhead of auxiliary modules.
  • Manifold-Constrained Hyper-Connections (mHC): Standard residual connections are replaced with mHC, which uses a constrained mixing matrix across four parallel residual streams to improve information flow.

Training and Data

The model was trained from scratch on 33.5T tokens using the Muon optimizer and FP8 mixed precision. The training pipeline included a context length of 128K, with support for up to 1M tokens during inference. The pre-training corpus emphasizes high-quality web data, repository-scale code, and long-context sequences designed to facilitate reasoning and agentic workflows.

Baidu also introduced two infrastructure components for this release: a distributed version of the Muon optimizer designed for large-scale GPU clusters, and FlashMask-4, an attention kernel optimized for large head dimensions on SM90 and SM100 architectures.

Community Reception and Licensing

The release has sparked discussion regarding the role of base models in the current AI landscape. Supporters of the release argue that providing unaligned base models is essential for decentralized alignment, as it allows researchers to pursue diverse training methodologies rather than relying on a single, centralized approach.

However, some community members have raised concerns regarding the license agreement, which contains geographic restrictions on usage. Additionally, while some developers see the model as a powerful alternative to existing open-source base models like DeepSeek-V4-Flash, others have noted that the "ERNIE" branding may feel dated compared to newer, more rapidly evolving model families.

Technical Specifications

  • Total Parameters: 280B
  • Activated Parameters: 14B
  • Context Window: 128K (pre-training), 1M (inference)
  • Vocabulary: ~200K (BPE with right-to-left number tokenization)
  • Availability: The model is available on Hugging Face in both FP8 and BF16 formats.

Sources