Hugging Face Training Efficiency: Packing with Flash Attention 2

Hugging Face has introduced boundary-aware packing for instruction tuning examples, which is now compatible with Flash Attention 2. This update allows for the concatenation of sequences without padding, providing up to a 2x increase in training throughput and a 20% reduction in peak memory usage without impacting convergence quality.

Increased Training Throughput and Memory Efficiency

Packing examples without padding significantly reduces the computational overhead associated with irrelevant padding tokens. The degree of improvement depends on the variance of sequence lengths within the training dataset:

  • High Variance Datasets (e.g., FLAN): Training on the FLAN dataset showed a 2x throughput increase for models including Llama 2-7B, Mistral-7B, and Granite-8B-code. Peak memory usage was reduced by 20%.
  • Low Variance Datasets (e.g., OrcaMath): Training on the OrcaMath dataset, which has longer examples and lower variance, resulted in a 1.4x throughput increase and a 6% reduction in peak memory.

Because the new implementation retains minibatches and maintains the same number of optimization steps as padded training, validation loss remains identical, ensuring no degradation in training convergence.

Technical Implementation: Boundary Awareness

Previous packing implementations often ignored example boundaries when using Flash Attention 2, leading to undesired cross-example attention that harmed model quality. Hugging Face addresses this by maintaining boundary awareness during packing.

This is achieved by providing position_ids to Flash Attention 2 and utilizing the flash_attn_varlen_func, which calculates the cumulative sequence lengths (cu_seqlens) for each mini-batch. This approach allows the model to concatenate sequences into a single tensor while ensuring that attention is restricted to the correct sequence boundaries.

Supported Models

The solution requires models to expose position_ids. Currently, 14 models are supported, including:

  • Llama 2 and 3
  • Mistral and Mixtral
  • Granite
  • DBRX
  • Falcon
  • Gemma
  • OLMo
  • Phi 1, 2, and 3 (including phi3)
  • Qwen 2 and 2 MoE
  • StableLM
  • StarCoder 2

Integration and Usage

Users can implement packing with Flash Attention 2 through two primary paths depending on the library used:

Using Hugging Face Trainer

To use this feature with the Trainer class from the Transformers library, users must:

  1. Instantiate the model with attn_implementation="flash_attention_2".
  2. Use the DataCollatorWithFlattening collator.

Using TRL SFTTrainer

For users of the SFTTrainer from the TRL library using DataCollatorForCompletionOnlyLM, the requirements are:

  1. Instantiate the model with Flash Attention 2.
  2. Set the padding_free=True flag when calling DataCollatorForCompletionOnlyLM.

Conclusion

By eliminating padding tokens and implementing boundary-aware attention, Hugging Face has significantly improved the efficiency of instruction tuning. The most substantial gains in throughput and memory are realized when training on datasets with high variance in example lengths.

Sources