What Makes a Dialog Agent Useful? Technical Analysis
TL;DR
Modern dialog agents are created by applying a series of fine-tuning techniques to a pre-trained base language model to move it from simple text prediction to instruction following. The primary drivers of utility are Instruction Fine-Tuning (IFT) for task versatility, Supervised Fine-Tuning (SFT) for safety and helpfulness, Reinforcement Learning from Human Feedback (RLHF) for preference alignment, and Chain-of-Thought (CoT) for complex reasoning.
Comparative Landscape of Conversational Agents
While ChatGPT popularized these techniques, several organizations have developed similar dialog agents using varying combinations of training data and alignment methods.
| Model | Organization | Size | Pre-trained Base | Web Access | RLHF |
|---|---|---|---|---|---|
| LaMDA | 137B | Unknown | Yes | No | |
| BlenderBot 3 | Meta | 175B | OPT | Yes | No |
| Sparrow | DeepMind | 70B | Chinchilla | Yes | Yes |
| ChatGPT/InstructGPT | OpenAI | 175B | GPT-3.5 | No | Yes |
| Assistant | Anthropic | 52B | Unknown | No | Yes |
Common across these models is the goal of instruction following, which allows the agent to execute user-specified tasks, such as writing a poem or summarizing text.
Instruction Fine-Tuning (IFT)
Instruction Fine-Tuning transforms a base model's objective from predicting the next token to following specific directions. This process involves fine-tuning the model on demonstrations consisting of three components: an instruction, an optional input, and an output.
Data Sourcing for IFT
IFT datasets are created through a spectrum of human and model contributions:
- Purely Model-Generated: Datasets like Unnatural Instructions are generated entirely by LMs.
- Bootstrapped: Self-instruct uses a small set of high-quality seed data to generate new instructions and outputs.
- Hand-Crafted: Super-natural instructions rely on large-scale community efforts for human-written data.
- Template-Based: Projects like T0, FLAN LM, and Natural Instructions cast existing NLP datasets into a unified instruction schema.
Safety and Alignment Techniques
To prevent models from being evasive or generating unsafe content, developers employ specific alignment strategies.
Supervised Fine-Tuning (SFT)
SFT involves fine-tuning the base model on high-quality, human-annotated data specifically focused on helpfulness and harmlessness. While IFT is a subset of SFT, the SFT phase is frequently used specifically for safety topics after the initial instruction tuning.
Reinforcement Learning from Human Feedback (RLHF)
Used by InstructGPT, Sparrow, and Anthropic's Constitutional AI, RLHF aligns models with human preferences through a three-step process:
- Human annotators rank multiple model responses.
- A preference model is trained on these rankings to provide a scalar reward.
- The dialog agent is trained via reinforcement learning to maximize that reward.
Improving Reasoning with Chain-of-Thought (CoT)
Chain-of-Thought prompting and fine-tuning elicit step-by-step reasoning from the agent. By training on datasets that include human annotations of the reasoning process, models show significant performance gains in:
- Commonsense reasoning
- Arithmetic
- Symbolic reasoning
Furthermore, CoT fine-tuning can improve harmlessness more effectively than RLHF in some cases, reducing the frequency of evasive responses (e.g., "I cannot respond to this question") when dealing with sensitive prompts.
Key Technical Takeaways
- Data Efficiency: Instruction fine-tuning requires only a tiny fraction of the data used in pre-training (often in the order of a few hundred samples).
- Safety via SFT: Human annotations in SFT are critical for ensuring model outputs are safe and helpful.
- Reasoning via CoT: CoT fine-tuning is essential for complex thinking tasks and reduces model evasiveness on sensitive topics.
Open Questions in Dialog Agent Development
Several unresolved challenges remain for the field:
- RL Necessity: It is unclear if the performance of RLHF can be achieved using only higher-quality IFT or SFT data.
- SFT vs. RLHF: The relative effectiveness of SFT-only safety (as in LaMDA) versus SFT+RLHF (as in Sparrow) requires further comparison.
- Pre-training Trade-offs: The optimal amount of pre-training required when combined with advanced fine-tuning techniques is not yet defined.
- Red-Teaming Reproducibility: There is currently no systematic way to record and reproduce the effects of red-teaming (searching for failure modes to influence future training).