Fine-Tuning Gemma Models in Hugging Face

Hugging Face has introduced a detailed workflow for fine-tuning Google DeepMind's Gemma models using Parameter-Efficient Fine-Tuning (PEFT). This approach allows developers to adapt the 2 billion and 7 billion parameter Gemma models to specific datasets with significantly lower memory and compute requirements than full-weight training.

Low-Rank Adaptation (LoRA) for Gemma

Low-Rank Adaptation (LoRA) is the primary PEFT technique highlighted for Gemma models. Instead of updating all model parameters, LoRA freezes the original weights and trains only a small set of adapter layers composed of low-rank matrices. This drastically reduces the computational overhead of fine-tuning.

To implement LoRA with Gemma, the Hugging Face PEFT library allows users to target specific linear layers. In the provided example, the following modules are targeted for adaptation:

  • q_proj
  • o_proj
  • k_proj
  • v_proj
  • gate_proj
  • up_proj
  • down_proj

Memory Optimization via QLoRA

To further reduce memory usage, Hugging Face recommends using QLoRA. This method quantizes the base model to 4-bit precision using the bitsandbytes library. By passing a BitsAndBytesConfig to the from_pretrained method, users can load Gemma in a memory-efficient format (e.g., using nf4 quantization type and bfloat16 compute dtype), making fine-tuning accessible on consumer-grade GPUs or free platforms like Google Colab.

Implementation Workflow: Learning to Quote

Using the SFTTrainer from the trl library, Hugging Face demonstrates a practical application where Gemma-2b is fine-tuned to generate quotes in a specific format (Quote followed by Author).

The process involves:

  1. Model Loading: Loading the model with 4-bit quantization via BitsAndBytesConfig.
  2. Dataset Preparation: Utilizing the Abirate/english_quotes dataset and tokenizing it for the model.
  3. Configuration: Applying a LoraConfig with a rank (r) of 8.
  4. Training: Running the trainer with a learning rate of 2e-4 and the paged_adamw_8bit optimizer.

Hardware Acceleration: PyTorch/XLA and FSDP on TPU

Gemma models in the Hugging Face transformers library are optimized for both PyTorch and PyTorch/XLA, enabling deployment on both GPUs and Cloud TPUs.

For TPU users, Hugging Face has improved the Fully Sharded Data Parallel (FSDP) experience via SPMD (Single Program, Multiple Data). By adding an fsdp_config to the transformers.Trainer, users can wrap the GemmaDecoderLayer and enable xla_fsdp_v2. This integration significantly accelerates fine-tuning speeds on TPU hardware.

TPU Configuration Summary

Feature Setting
Layer to Wrap GemmaDecoderLayer
FSDP Mode full_shard
XLA FSDP v2 Enabled (True)
XLA FSDP Grad Checkpointing Enabled (True)

Sources