Mistral 7B Release Notes

Mistral AI has announced the release of Mistral 7B, a 7.3 billion parameter language model designed to provide high performance relative to its size. The model is released under the Apache 2.0 license, allowing for unrestricted use and deployment.

Performance Benchmarks

Mistral 7B outperforms Llama 2 13B on all evaluated benchmarks and is comparable to Llama 1 34B on many. In terms of code and reasoning, it is described as vastly superior to these models and approaches the performance of CodeLlama 7B while maintaining proficiency in English tasks.

According to Mistral AI, the model performs equivalently to a Llama 2 model more than three times its size in areas of reasoning, comprehension, and STEM reasoning (MMLU). This efficiency results in significant memory savings and increased throughput.

Evaluation Categories

The performance was measured across several themes:

• Commonsense Reasoning: 0-shot average of Hellaswag, Winogrande, PIQA, SIQA, OpenbookQA, ARC-Easy, ARC-Challenge, and CommonsenseQA.
• World Knowledge: 5-shot average of NaturalQuestions and TriviaQA.
• Reading Comprehension: 0-shot average of BoolQ and QuAC.
• Math: Average of 8-shot GSM8K (maj@8) and 4-shot MATH (maj@4).
• Code: Average of 0-shot Humaneval and 3-shot MBPP.
• Aggregated Results: 5-shot MMLU, 3-shot BBH, and 3-5-shot AGI Eval (English multiple-choice questions only).

Technical Architecture

Mistral 7B implements two primary architectural features to optimize inference speed and sequence handling:

Grouped-query attention (GQA)

Mistral 7B uses Grouped-query attention (GQA) to enable faster inference speeds.

Sliding Window Attention (SWA)

Sliding Window Attention (SWA) allows the model to handle longer sequences at a lower computational cost. In this mechanism, each layer attends to the previous 4,096 hidden states, resulting in a linear compute cost of O(sliding_window.seq_len).

SWA exploits the stacked layers of the transformer to allow tokens to access information further in the past than the window size suggests. For example, a token at layer k attends to tokens in the window at layer k-1, which in turn attended to tokens further back. This effectively extends the reach of the model's attention.

In practice, combined with changes to FlashAttention and xFormers, SWA provides a 2x speed improvement for sequences of 16k length with a 4k window. Additionally, a fixed attention span allows the model to limit its cache to the size of the sliding window using rotating buffers, reducing cache memory for inference on sequence lengths of 8,192 by half without impacting quality.

Fine-tuning and Instruction Following

To demonstrate the generalization capabilities of the base model, Mistral AI released Mistral 7B Instruct, a version fine-tuned on publicly available instruction datasets from HuggingFace.

Mistral 7B Instruct outperforms all other 7B models on the MT-Bench benchmark and is comparable to 13B chat models. The model was fine-tuned without the use of proprietary data or specific "tricks."

Mistral AI notes that the Instruct model does not currently have a moderation mechanism and is seeking community engagement to develop guardrails for deployment in environments requiring moderated outputs.