Apertus 1.5 release notes

Overview

Apertus 1.5 is a significant update to the Apertus 1.0 model series, expanding the 8B and 70B parameter models with native multimodal understanding, a 262,144-token context window, and an optional thinking mode for improved reasoning. Released under an Apache 2.0 license, the update includes open weights, training data, and comprehensive research documentation.

Key Enhancements

  • Multimodal Understanding: The models now natively process images and audio alongside text using an early-fusion approach. Images are processed via the Emu3.5 tokenizer, and audio is handled by WavTokenizer.
  • Thinking Mode: An optional mode allows the model to generate reasoning traces before providing a final answer, significantly improving performance on complex reasoning tasks.
  • Extended Context: The context window has been increased to 262,144 tokens, a 4x improvement over the previous generation.
  • Instruction and Tool Use: Refinements in the training pipeline have led to better instruction adherence and more reliable integration with external tools and APIs.
  • Transparency: The models maintain adherence to the Apertus Charter, with all training data documented for transparency and compliance.

Training Pipeline

The Apertus 1.5 training process involves continued pretraining followed by a multi-stage post-training pipeline. The pretraining phase adds 4 trillion tokens to the 8B model and 2 trillion tokens to the 70B model, utilizing a mix of text, image, and audio data.

Post-training consists of five distinct stages:

  1. Supervised Fine-Tuning (SFT): Training on 3.7 million text samples plus multimodal data.
  2. Long-context SFT: Optimization at the full 262k token window.
  3. Reinforcement Learning with Verifiable Rewards (RLVR): Targeted training on math, code, and reasoning prompts.
  4. Direct Preference Optimization (DPO): Improving chat quality and safety using judge-scored preference pairs.
  5. Charter Alignment (SDPO): Self-distillation to align model outputs with the Apertus Charter.

Evaluation and Reproducibility

Apertus 1.5 utilizes the Every Eval Ever (EEE) schema to provide transparent, machine-readable evaluation records. This approach aims to meet AI governance requirements, such as those outlined in the EU AI Act, by providing detailed metadata alongside performance scores. The models show competitive performance in coding, instruction following, and multilingual tasks compared to other open-weight models of similar scale.

Detailed recipes, ablations, and intermediate checkpoints are available through the official Apertus Hugging Face collection, allowing researchers to reproduce the training results and investigate the impact of each stage on model behavior.

Sources