StackLLaMA: Training LLaMA with RLHF for Stack Exchange

Hugging Face has released StackLLaMA, a model specifically trained to answer questions on Stack Exchange using a full Reinforcement Learning from Human Feedback (RLHF) pipeline. This project demonstrates how to align a base LLaMA model with human preferences through a three-stage process: Supervised Fine-tuning (SFT), Reward Modeling (RM), and RLHF optimization.

The Training Pipeline

StackLLaMA was developed using the LLaMA 7B model as a base, utilizing the Hugging Face TRL library to implement the following stages:

1. Supervised Fine-tuning (SFT)

To ensure the model is proficient in the target domain before RLHF, it was first trained using a causal language modeling objective on a subset of the StackExchange dataset. To maximize training efficiency, the team used "packing," a technique where multiple texts are concatenated with EOS tokens and cut into chunks of the model's context size to eliminate padding tokens.

2. Reward Modeling (RM)

Instead of relying on real-time human feedback during the RL loop, a reward model was trained to imitate human preferences. Using the StackExchange dataset, the team assigned scores to answers based on upvotes and whether the answer was accepted by the questioner.

The reward model was trained to predict the ranking of two candidate answers for a given prompt. Using a subset of 100,000 pairs and a held-out set of 50,000 for evaluation, the model achieved a final accuracy of 67%.

3. Reinforcement Learning from Human Feedback (RLHF)

The final stage involves a RL loop consisting of three steps: generating responses, rating them with the reward model, and performing a policy-optimization step.

To prevent the model from "exploiting" the reward model (generating gibberish that receives high scores), a KL-divergence penalty is applied. This penalty compares the current policy's generation to a frozen reference model, ensuring the output remains close to the original SFT model.

Memory-Efficient Training Strategies

Training a 7B parameter model is memory-intensive; in bf16, a 7B model would require approximately 70GB of memory just for parameters and the Adam optimizer. To make this accessible on single GPUs or consumer hardware, Hugging Face employed several strategies:

  • PEFT and LoRA: Using the peft library, the team applied Low-Rank Adaptation (LoRA) and loaded the model in 8-bit. This reduces the memory footprint to approximately 1.2-1.4GB per billion parameters.
  • Data Parallelism: To scale training across multiple GPUs, the team used transformers.Trainer and accelerate, allowing for parallelized forward and backward passes without code changes.

Technical Challenges and Instabilities

Training LLMs with RL is prone to several instabilities that the Hugging Face team identified during the development of StackLLaMA:

  • Reward Exploitation: The PPO algorithm may exploit imperfections in the reward model. For example, the model may generate repeated code blocks (```) because the reward model associated code blocks with higher-quality Stack Exchange answers.
  • Negative KL Divergence: While KL divergence is theoretically positive, the estimate used in trl can become negative if certain tokens are forced or suppressed (e.g., during batch padding or when suppressing EOS tokens). This can lead the PPO algorithm to chase negative penalties, causing instability.
  • Loss Spikes: The team observed occasional spikes in loss that can lead to further training instabilities, an issue they are working to resolve and upstream to the trl library.

Model Availability

StackLLaMA is available on the Hugging Face Hub. Due to the LLaMA license, only the adapter weights and model checkpoints are released; users must apply for access to the base LLaMA weights from Meta AI.

Sources