Aya Expanse Release: Advancing Multilingual LLM Performance

Hugging Face and Cohere For AI have released the Aya Expanse family, featuring 8B and 32B parameter models designed to close the performance gap between multilingual and monolingual AI. These models set a new state-of-the-art for multilingual performance, with the 32B model outperforming Llama 3.1 70B, Gemma 2 27B, and Mistral 8x22B in pairwise comparisons on the m-ArenaHard dataset.

Performance Benchmarks

Aya Expanse models demonstrate superior performance compared to other leading open-weights models in their respective size classes:

  • Aya Expanse 32B: Outperforms Llama 3.1 70B (a model more than twice its size), Gemma 2 27B, and Mistral 8x22B.
  • Aya Expanse 8B: Outperforms Gemma 2 9B, Llama 3.1 8B, and Ministral 8B, with win rates ranging from 60.4% to 70.6%.

Evaluation was conducted using the Arena-Hard-Auto dataset translated into 23 languages (m-ArenaHard).

Avoiding Model Collapse via Multilingual Arbitrage

To combat model collapse associated with over-reliance on synthetic data, Cohere For AI implemented "data arbitrage." This technique strategically samples from a pool of teacher models rather than relying on a single teacher model, which is critical for low-resource languages where no single model excels across all languages.

The Arbitrage Process:

  1. Model Pool: A pool of models is trained for specific language groups.
  2. Arbiter: An internal reward model (RM) acts as an arbiter to score generations from all models in the pool for a given prompt.
  3. Reward-Based Routing: The completion with the highest score is selected as the final training data.

This approach resulted in a 9.1% improvement in win-rate for the 8B model against Gemma 2 9B during the SFT stage compared to the previous Aya 23 model.

Multilingual Preference Optimization

Alignment to human preferences in multilingual settings is challenging due to the scarcity of high-quality non-English preference datasets. Aya Expanse utilizes a hybrid offline and online preference training pipeline:

Synthetic Preference Data Generation

The team generated high-quality preference pairs by contrasting in-language completions from a high-performance multilingual LLM against lower-quality completions translated from English (generated by a weaker model). This steers the model away from translation artifacts and poor multilingual completions.

Hybrid DPO Pipeline

  • Offline Stage: The model is first trained on data curated from the highest and lowest reward responses from the Arbitrage stage.
  • Online Iterative DPO: The model undergoes three iterations of online DPO. In each iteration, multiple generations are sampled from the current model, ranked by a Reward Model, and used for further training.

For the 8B model, this combined offline and online preference training provided an additional 7.1% gain in win rate against Gemma 2 9B.

Performance Maximization through Model Merging

To reduce the computational cost of hyperparameter tuning for data mixtures, the team used model merging. This involves training separate models on different language families to maximize diversity between checkpoints while leveraging cross-lingual transfer.

Key Merging Strategies:

  • Diversity via Language Families: Models are trained on different language clusters to ensure differentiation, while shared languages (English, Spanish, and French) are included in each cluster to maintain robustness.
  • Weighted Averaging: While SLERP, TIES-merging, and DARE-TIES were tested, weighted linear averaging was found to be the most consistent method.
  • Scaling Effects: Model merging provided significantly larger gains at the 32B scale compared to the 8B scale, with gains up to 3x larger.

Summary of the Post-Training Pipeline

The final Aya Expanse models are the result of a multi-stage pipeline combining Multilingual Arbitrage for synthetic data quality, a hybrid offline/online DPO process for preference alignment, and strategic model merging to optimize multi-tasking across diverse language families.

Sources