lucidrains/PaLM-rlhf-pytorch

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

What it solves

This project provides a PyTorch implementation of Reinforcement Learning from Human Feedback (RLHF) applied to the PaLM architecture, enabling developers to align a large language model with human preferences.

How it works

The workflow consists of three primary stages:

  1. Base Model Training: An autoregressive PaLM transformer is trained on a large dataset.
  2. Reward Model Training: A RewardModel is trained using curated human feedback to score model responses. The implementation supports both standard fine-tuning and Low-Rank Adaptation (LoRA) to mitigate overfitting.
  3. RLHF Training: The RLHFTrainer takes the pretrained PaLM model and the trained reward model to optimize the language model's output based on the reward signals.

Who it’s for

Researchers and developers interested in replicating the RLHF process used in models like ChatGPT using an open-source framework.

Highlights

  • PaLM Architecture: Built on the PaLM transformer design.
  • LoRA Support: Includes Low-Rank Adaptation for efficient reward model fine-tuning.
  • Hugging Face Integration: Utilizes the accelerate library for distributed training.
  • Flash Attention: Integrated for improved memory efficiency and speed.

Related

  • Dispatch
    Illustrating Reinforcement Learning from Human Feedback (RLHF)

    Hugging Face explains Reinforcement Learning from Human Feedback (RLHF), a three-step process used to align large language models with complex human values by optimizing them using a reward model based on human preferences.

  • Project
    FareedKhan-dev/train-llm-from-scratch

    A complete PyTorch implementation of LLMs from scratch, covering the entire pipeline from raw text tokenization and pretraining to advanced RL alignment (PPO, DPO, GRPO).

  • Project
    OpenRLHF/OpenRLHF

    A high-performance RLHF framework that combines Ray and vLLM to scale reinforcement learning from human feedback for LLMs and VLMs.

  • Dispatch
    Anthropic: Training a Helpful and Harmless Assistant with RLHF

    Anthropic demonstrates that Reinforcement Learning from Human Feedback (RLHF) improves language model performance across most NLP evaluations while ensuring the assistant remains helpful and harmless.