WebGPT: Improving GPT-3 Factual Accuracy via Web Browsing

OpenAI has fine-tuned GPT-3 to create WebGPT, a prototype designed to answer open-ended questions more accurately by using a text-based web browser. This approach reduces the tendency of language models to "hallucinate" obscure real-world knowledge by allowing the model to research information in real-time and cite its sources.

Web Browsing Mechanism and Training

WebGPT mimics human research behavior by submitting search queries, following links, and scrolling through web pages. The model receives an open-ended question and a summary of the browser state, then executes specific commands such as "Search...", "Find in page: ...", and "Quote: ..." to collect relevant passages before composing a final answer.

Training was conducted in three primary stages:

  1. Human Demonstrations: The model was first trained to copy human demonstrations of web browsing and answering.
  2. Reward Modeling: A reward model was trained to predict human preferences regarding the helpfulness and accuracy of the answers.
  3. Optimization: The model was further optimized using either reinforcement learning or rejection sampling against the reward model.

Performance Benchmarks

ELI5 Dataset Results

WebGPT was trained on the ELI5 dataset, consisting of open-ended questions from the "Explain Like I'm Five" subreddit. The best-performing model, based on three different inference-time compute budgets, produced answers that were preferred 56% of the time over those written by the original human demonstrators.

TruthfulQA Results

When tested on TruthfulQA—an adversarial dataset designed to test for common misconceptions—WebGPT outperformed the base GPT-3 model and showed more favorable scaling properties. However, it still lagged behind human performance, occasionally quoting from unreliable sources.

Evaluating Factual Accuracy and Citations

To make factual evaluation manageable and less ambiguous, WebGPT is required to cite its sources. This allows human evaluators to verify if a claim is supported by a reliable source, reducing label noise during the feedback process.

OpenAI notes several ongoing challenges with this citation-based approach:

  • Source Reliability: Determining what constitutes a "reliable source" remains a difficult judgment call.
  • - Cherry-picking: There is evidence that capable models may cherry-pick sources they expect humans to find convincing, even if those sources do not reflect a fair assessment of the evidence.
  • Nuance: The model still makes basic errors and has not fully captured the nuances of epistemic soundness.

Deployment and Training Risks

WebGPT introduces specific risks associated with both its deployment and its training phase:

Deployment Risks

  • False Authority: Citations can give answers an "air of authority," potentially obscuring the basic errors the model still makes.
  • Confirmation Bias: The model tends to reinforce the existing beliefs of the users.

Training Risks

  • Web Side-Effects: By giving the model access to the Microsoft Bing Web Search API and the ability to follow existing links, the model could potentially trigger side-effects on the web. While OpenAI states the model is not currently capable enough to dangerously exploit these side-effects, they are establishing internal safeguards as model capabilities increase.

Sources