Mixtral 8x7B Release Notes

Mistral AI has released Mixtral 8x7B, a high-quality sparse mixture-of-experts (SMoE) model with open weights under the Apache 2.0 license. This model is designed to optimize the trade-off between performance and cost, matching or outperforming GPT-3.5 and Llama 2 70B on most standard benchmarks while maintaining significantly lower inference latency.

Sparse Mixture-of-Experts (SMoE) Architecture

Mixtral 8x7B utilizes a sparse mixture-of-experts network, which allows the model to increase its total parameter count without a proportional increase in computational cost per token.

Technical Implementation

As a decoder-only model, Mixtral 8x7B employs a feedforward block that selects from eight distinct groups of parameters. For every token at every layer, a router network chooses two of these "experts" to process the token, combining their outputs additively.

Parameter Efficiency

This architecture enables a high total parameter count while controlling latency. Mixtral 8x7B has 46.7B total parameters, but only uses 12.9B parameters per token. Consequently, the model generates output at the same speed and cost as a 12.9B parameter model.

Performance and Benchmarks

Mixtral 8x7B demonstrates superior efficiency and quality compared to the Llama 2 family and GPT-3.5 base models.

General Benchmarks

Mixtral matches or outperforms Llama 2 70B and GPT-3.5 on most standard benchmarks. Compared to Llama 2 70B, Mixtral provides 6x faster inference.

Multilingual Capabilities

The model is proficient in five languages: English, French, Italian, German, and Spanish.

Specialized Capabilities

• Context Window: The model handles a context of 32k tokens.
• Code Generation: Mixtral shows strong performance in generating code.
• Bias and Hallucination: On the BBQ benchmark, Mixtral presents less bias than Llama 2. On the BOLD benchmark, it displays more positive sentiments than Llama 2 with similar variances across dimensions.

Mixtral 8x7B Instruct

Alongside the base model, Mistral AI released Mixtral 8x7B Instruct. This version is optimized for instruction following through supervised fine-tuning and direct preference optimization (DPO).

Instruction Following Performance

On MT-Bench, Mixtral 8x7B Instruct achieves a score of 8.30, making it the strongest open-source model in its class with performance comparable to GPT-3.5.

Moderation and Guardrailing

While the model can be prompted to ban specific outputs for applications requiring moderation, the base behavior of the model is to follow the instructions provided without built-in restrictions unless specifically prompted or preference-tuned.

Deployment and Ecosystem

To support the open-source community, Mistral AI has integrated Megablocks CUDA kernels into the vLLM project for efficient inference. Deployment of vLLM endpoints can be managed via Skypilot on any cloud instance.

Mixtral 8x7B is also available as the mistral-small endpoint in beta on the Mistral AI platform.