Mixtral 8x7B Release Notes

Mixtral 8x7B is a high-performance large language model utilizing a Mixture of Experts (MoE) architecture to achieve state-of-the-art results for open-access models. It outperforms Llama 2 70B and matches or beats GPT-3.5 across many benchmarks, providing a commercially permissive alternative under the Apache 2.0 license.

Mixture of Experts (MoE) Architecture

Mixtral 8x7B employs a sparse Mixture of Experts (MoE) technique, replacing standard Feed-Forward layers with a MoE layer containing a router network. This router selects two experts for each timestep to process tokens most efficiently.

While the name suggests 56B parameters, the model actually contains approximately 45B parameters. This is because only the feed-forward blocks are replicated; other parameters remain the same as in a 7B model. This architecture allows Mixtral to decode at the speed of a 12B parameter-dense model despite its larger effective parameter count.

Key Capabilities and Performance

Mixtral 8x7B is available in both base and Instruct versions with the following specifications:

  • Context Window: Supports a context length of 32k tokens.
  • Multilingual Support: Proficient in English, French, German, Spanish, and Italian.
  • Coding Proficiency: Achieves 40.2% on the HumanEval benchmark.
  • Benchmark Performance:
    • The base model scores 68.42 on the Open LLM Leaderboard, surpassing Llama-2-70b (67.87).
    • Mixtral Instruct scores 8.30 on MT-Bench, outperforming all other open-access models and achieving performance comparable to GPT-3.5-turbo-0613 (8.32).

Inference and Hardware Requirements

Inference can be performed via the Hugging Face pipeline() function or through Text Generation Inference (TGI) for production-ready deployment. Due to the model size, VRAM requirements vary by precision:

Precision Required VRAM
float16 >90 GB
8-bit >45 GB
4-bit >23 GB

For users with limited hardware, 4-bit quantization via bitsandbytes or GPTQ can be used. GPTQ quantization specifically avoids quantizing expert gating layers to maintain performance, resulting in a perplexity of 4.40 compared to 4.25 for the half-precision model.

Fine-Tuning and Optimization

Mixtral can be fine-tuned on a single A100 GPU using the Hugging Face TRL library. To optimize memory, 4-bit quantization and QLoRA are recommended. Technical guidelines specify that only linear layers in the attention blocks (q_proj, k_proj, v_proj, o_proj) should be targeted; MLP layers should not be targeted as they are sparse and do not interact well with Parameter-Efficient Fine-Tuning (PEFT).

Implementation Details

Prompt Format

The base model has no specific prompt format. The Instruct model requires a specific conversation structure for effective use:

<s> [INST] User Instruction 1 [/INST] Model answer 1</s> [INST] User instruction 2[/INST]

Known Limitations

There are currently no public details regarding the pretraining dataset size, composition, or preprocessing methods. Additionally, details concerning the fine-tuning datasets and hyperparameters for Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) for the Instruct model have not been shared.

Sources