OpenAI Research: Improving Language Model Behavior via Curated Dataset Fine-Tuning

OpenAI has demonstrated that language model behavior can be improved with respect to specific behavioral values by fine-tuning on a small, curated dataset of fewer than 100 examples. This approach allows model operators to constrain the universal set of possible model behaviors to a specific set of values tailored to a given social context or application.

Behavioral Alignment via Small-Scale Fine-Tuning

Fine-tuning on a highly curated, small-scale dataset is an effective method for aligning model outputs with desired behavioral values. OpenAI found that this process becomes more effective as the model size increases, suggesting that larger language models require fewer samples to adapt their behavior to specific values. This method provides a more scalable alternative to retraining a model from scratch, which is necessary when attempting to align models for large, diverse groups of people to avoid marginalizing minority voices.

The Three-Step Alignment Process

OpenAI developed a process to improve model behavior in specific social contexts using the following three steps:

1. Defining Sensitive Topic Categories

The researchers identified categories with a direct impact on human wellbeing, basing desired behaviors on international human rights law and Western social movements for human equality. The prioritized categories included:

  • Abuse, Violence, and Threat: Opposing violence and threats, and encouraging help from authorities.
  • Health, Physical and Mental: Opposing non-conventional medicines as scientific alternatives and refusing to diagnose conditions or prescribe treatment.
  • Human Characteristics and Behavior: Supporting the subjectivity of goodness and likeability and opposing unhealthy beauty standards.
  • Injustice and Inequality: Opposing human injustices, inequalities, and harmful stereotypes, particularly against social groups defined by international law.
  • Political Opinion and Destabilization: Maintaining nonpartisanship unless human rights or law are undermined, and opposing interference in democratic processes.
  • Relationships: Opposing non-consensual actions or violations of trust.
  • Sexual Activity: Opposing illegal and non-consensual sexual activity.
  • Terrorism: Opposing terrorist activity or threats.

2. Dataset Crafting and Fine-Tuning

The researchers crafted a values-targeted dataset consisting of 80 text samples in a question-answer format, with each sample ranging between 40 and 340 words. This dataset totaled approximately 120KB, which represents about 0.000000211% of the total GPT-3 training data (570GB).

GPT-3 models ranging from 125M to 175B parameters were then fine-tuned on this dataset using standard fine-tuning tools.

3. Evaluation Metrics

To measure the effectiveness of the alignment, OpenAI used a combination of quantitative and qualitative metrics:

  • Human Evaluations: Three different humans rated samples from 1 to 5 based on how well the text matched the specified sentiment position.
  • Toxicity Scoring: Using the Perspective API, though the researchers noted that toxicity scores can contain demographic and racial biases (e.g., flagging identity terms or African American English as toxic).
  • Co-occurrence Metrics: Used to examine gender, race, and religion.

Evaluation compared base GPT-3 models, values-targeted GPT-3 models, and control GPT-3 models (fine-tuned on a dataset of similar size and style but without the targeted values).

Key Findings and Implications

Qualitative probes and human evaluations indicate that values-targeted models adhered more closely to desirable behavior than base or control models. This improvement in behavior occurred without compromising performance on downstream tasks.

OpenAI notes that this technique is nascent and raises several open questions regarding the robustness of the methodology against real-world prompt distributions, the accountability for outputs not aligned with user values, and the application of this research to non-English languages and other generative modalities like image, video, or audio.

Sources