Finding GPT-4’s mistakes with GPT-4

OpenAI has developed CriticGPT, a specialized model based on GPT-4 that identifies inaccuracies in ChatGPT's code output. This tool is designed to assist human AI trainers during Reinforcement Learning from Human Feedback (RLHF), enabling them to spot subtle mistakes that would otherwise be difficult for humans to detect as models become more advanced.

Enhancing RLHF with AI-Assisted Review

CriticGPT addresses a fundamental limitation of RLHF: as AI models become more knowledgeable and their errors more subtle, it becomes increasingly difficult for human trainers to accurately rate responses. This gap in human ability to provide high-quality feedback can hinder the alignment of advanced AI systems.

By integrating CriticGPT into the labeling pipeline, OpenAI provides trainers with explicit AI assistance. In experimental settings, human trainers assisted by CriticGPT outperformed unassisted trainers 60% of the time. The combination of human oversight and AI critique results in feedback that is more comprehensive than human-only reviews and contains fewer hallucinated bugs than model-only reviews.

Training Methodology and Performance

CriticGPT was trained using RLHF, similar to ChatGPT, but with a specific focus on error detection. The training process involved:

  • Manual Error Insertion: AI trainers manually inserted mistakes into ChatGPT-generated code.
  • Example Feedback: Trainers wrote example critiques as if they had discovered the inserted bugs.
  • Comparative Rating: Trainers compared multiple critiques to identify which ones successfully caught the inserted bugs.

In tests involving "naturally occurring" ChatGPT bugs, trainers preferred CriticGPT's critiques over standard ChatGPT critiques in 63% of cases. This preference is attributed to CriticGPT producing fewer "nitpicks" (unhelpful minor complaints) and fewer hallucinations.

To further improve quality, OpenAI utilized additional test-time search against the critique reward model. This allows for a configurable precision-recall trade-off, balancing the aggressiveness of bug hunting against the risk of hallucinations to generate the most helpful critiques for RLHF.

Current Limitations

Despite its effectiveness, CriticGPT has several technical limitations:

  • Input Length: The model was trained on relatively short ChatGPT answers; supervising future agents will require methods for handling long and complex tasks.
  • ** CURLOPT Hallucinations:** The model still hallucinates, which can occasionally lead human trainers to make labeling mistakes.
  • Error Distribution: The current focus is on errors that can be identified in a single location, whereas real-world mistakes are often dispersed across multiple parts of a response.
  • Complexity Ceiling: For extremely complex tasks, neither the expert human nor the AI assistant may be able to correctly evaluate the response.

Future Implications for AI Alignment

OpenAI intends to scale the use of CriticGPT-like models within their RLHF pipelines. The research demonstrates that applying RLHF to GPT-4 can help humans produce higher-quality RLHF data for GPT-4, creating a cycle of improvement that is essential for aligning increasingly complex AI systems.

Sources