Pixtral 12B Release Notes

Mistral AI has announced Pixtral 12B, a natively multimodal model designed to process interleaved image and text data. Pixtral 12B is positioned as a drop-in replacement for Mistral Nemo 12B, providing state-of-the-art multimodal reasoning while maintaining high performance on text-only benchmarks, including coding, math, and instruction following.

Technical Architecture

Pixtral 12B utilizes a two-component architecture consisting of a vision encoder and a multimodal transformer decoder.

Vision Encoder and Variable Image Support

Pixtral features a new 400M parameter vision encoder trained from scratch. A key technical advancement is its native support for variable image sizes and aspect ratios. Images are passed through the encoder at their native resolution, converted into image tokens for every 16x16 patch. To maintain spatial awareness, the model uses [IMG BREAK] tokens to distinguish between rows and [IMG END] tokens to mark the end of an image. This allows the model to handle high-resolution complex diagrams and charts efficiently while maintaining fast inference for smaller images.

Multimodal Decoder and Context Window

The model employs a 12B parameter multimodal decoder based on Mistral Nemo. It is trained on interleaved image and text data to predict the next text token. This architecture supports the processing of multiple images within a long context window of 128K tokens.

Performance and Benchmarks

Pixtral 12B is designed to deliver best-in-class multimodal reasoning without degrading the text capabilities found in its base text model.

Multimodal Reasoning

Pixtral achieves a score of 52.5% on the MMMU reasoning benchmark, surpassing several larger models. It demonstrates strong capabilities in document question answering, chart and figure understanding, and multimodal reasoning.

Instruction Following

Pixtral shows a significant advantage in instruction following compared to other open multimodal models. Mistral AI reports a 20% relative improvement in text IF-Eval and MT-Bench over the nearest open-source (OSS) model. To validate these claims, Mistral AI developed multimodal versions of these benchmarks, MM-IF-Eval and MM-MT-Bench, where Pixtral also outperforms open-source alternatives.

Comparative Evaluation

Using a consistent evaluation harness and prompts, Pixtral was compared against both open and closed models. The results indicate that Pixtral substantially outperforms open models of similar scale and matches or exceeds the performance of much larger models, such as LLaVa OneVision 72B, on multimodal benchmarks. It also outperforms closed models like Claude 3 Haiku in several instances.

Capabilities and Use Cases

Pixtral 12B is capable of handling a variety of complex visual tasks:

  • Complex Figure Reasoning: Extracting and structuring data from visual diagrams (e.g., identifying top GDP countries from a chart).
  • Chart Analysis: Interpreting training loss curves and identifying specific points of failure in data trends.
  • Multi-image Processing: Combining information from multiple image-based tables into a single structured Markdown table.
  • Image-to-Code Generation: Converting visual website mockups into functional HTML and CSS code.
  • Natural Scene Understanding: Interpreting optical illusions and spatial relationships in natural photographs.

Availability and Licensing

Pixtral 12B is released under the Apache 2.0 license. It can be accessed via:

  • Le Chat: Mistral AI's conversational chat interface.
  • La Plateforme: Available via API for integration into applications.
  • Local Deployment: Supported through mistral-inference and the vLLM library for higher serving throughput.

Sources

Related

  • Dispatch
    Pixtral Large Release Notes

    Mistral AI has released Pixtral Large, a 124B open-weights multimodal model that achieves state-of-the-art performance on MathVista, DocVQA, and VQAv2 while maintaining the text capabilities of Mistral Large 2.

  • Dispatch
    Mistral NeMo Release Notes

    Mistral AI and NVIDIA have released Mistral NeMo, a 12B parameter model featuring a 128k context window and a new, highly efficient multilingual tokenizer called Tekken.

  • Dispatch
    Mistral Small 4 release notes / what's new

    Mistral AI has released Mistral Small 4, a unified open-source model under Apache 2.0 that integrates reasoning, multimodal, and agentic coding capabilities into a single versatile architecture.

  • Dispatch
    Mistral AI Shieldstral 1.0 3B Release

    Mistral AI has released Shieldstral, a 3B open-weights multimodal safety classifier that uses a binary question-answering framework to enable policy-adaptive moderation without retraining.

  • Dispatch
    Mistral Large 2 release notes / what's new

    Mistral AI has released Mistral Large 2, a 123B parameter model designed for high throughput single-node inference with competitive performance in coding, reasoning, and multilingual capabilities.