OpenAI Introducing ChatGPT

OpenAI has introduced ChatGPT, a conversational AI model designed to interact in a dialogue format. This format allows the model to answer follow-up questions, admit mistakes, challenge incorrect premises, and reject inappropriate requests.

Technical Methodology: RLHF and GPT-3.5

ChatGPT is a fine-tuned version of a model from the GPT-3.5 series, which completed training in early 2022. The model was developed using Azure AI supercomputing infrastructure and trained using Reinforcement Learning from Human Feedback (RLHF).

The training process involved several key stages:

1. Supervised Fine-Tuning: Human AI trainers provided conversations where they played both the user and the AI assistant. These trainers used model-written suggestions to help compose responses. This dialogue dataset was then mixed with the InstructGPT dataset, which was converted into a dialogue format.
2. Reward Model Creation: To facilitate reinforcement learning, OpenAI collected comparison data by having AI trainers rank multiple model-generated completions for a randomly selected message within a conversation.
3. Proximal Policy Optimization (PPO): Using the reward models, the model was further fine-tuned using PPO over several iterations.

Key Capabilities and Dialogue Format

Unlike previous models, ChatGPT's dialogue format enables more natural and iterative interactions. For example, when presented with a coding error, the model can ask for more context or suggest potential fixes based on the user's subsequent clarifications.

Additionally, the model demonstrates an improved ability to handle incorrect premises. In a comparison with InstructGPT, when asked about Christopher Columbus arriving in the US in 2015, InstructGPT provided a factually incorrect response. ChatGPT, however, identified the historical inaccuracy and challenged the premise while still engaging with the prompt in a hypothetical scenario.

Known Limitations

OpenAI identifies several critical limitations in the current version of ChatGPT:

• Factuality: The model sometimes produces "plausible-sounding but incorrect or nonsensical answers." This is attributed to the lack of a source of truth during RL training, the risk of the model becoming too cautious and declining correct answers, and the fact that supervised training can mislead the model based on what a human demonstrator knows rather than what the model knows.
• Sensitivity: The model is sensitive to input phrasing and may provide different answers to the same prompt if it is slightly rephrased.
• Verbosity: Due to biases in training data (where trainers prefer longer, comprehensive-looking answers) and over-optimization, the model is often excessively verbose and overuses specific phrases.
• Ambiguity: Instead of asking clarifying questions for ambiguous queries, the model typically guesses the user's intent.
• Safety: Despite efforts to refuse inappropriate requests, the model may still respond to harmful instructions or exhibit biased behavior. OpenAI uses a Moderation API to block or warn against unsafe content, though false positives and negatives occur.

Iterative Deployment and Feedback

ChatGPT is released as a research preview to gather user feedback and learn about the model's strengths and weaknesses. This release follows OpenAI's strategy of iterative deployment, utilizing lessons from GPT-3 and Codex to reduce harmful and untruthful outputs via RLHF.

OpenAI is actively seeking feedback on problematic outputs, particularly regarding harmful outputs in real-world, non-adversarial conditions, as well as novel risks and possible mitigations. To encourage this, they have introduced a ChatGPT Feedback Contest to provide API credits to users who identify critical issues.