OpenAI Discovering Types for Entity Disambiguation

OpenAI has developed a system for automatic entity disambiguation that uses a neural network to determine if a word belongs to a set of approximately 100 automatically discovered "types" or non-exclusive categories. This approach improves state-of-the-art accuracy on several entity disambiguation datasets by treating the problem as a probabilistic "20 questions" game rather than attempting to reason directly about the specific object meant by a word.

Performance Benchmarks

The type-based approach provides a significant boost in accuracy over previous methods that relied on distributed representations. The system achieved the following results:

  • CoNLL (YAGO): 94.88% accuracy (previous state-of-the-art results were 91.50% and 91.70%).
  • TAC KBP 2010 challenge: 90.85% accuracy (previous state-of-the-art results were 87.20% and 87.70%).

According to OpenAI, perfect type prediction would result in accuracies between 98.6% and 99% on these tasks.

System Architecture and Workflow

The disambiguation process follows a five-step pipeline:

  1. Entity Mapping: The system extracts Wikipedia-internal links to identify all possible entities a word could refer to.
  2. Category Identification: It walks the Wikipedia category tree via the Wikidata knowledge graph to determine every category associated with each entity.
  3. Type System Optimization: The system selects a list of approximately 100 categories to serve as the "type" system, optimizing these choices so that entities can be compactly represented as binary vectors.
  4. Neural Network Training: Using Wikipedia-internal links and surrounding context, the system trains a neural network to map a word and its context to the 100-dimensional binary representation of the corresponding entity.
  5. Inference: At test time, the neural network predicts the probability that a word belongs to each category. Bayes’ theorem is then used to calculate the probability of the word referring to each possible entity.

Data Cleaning and Heuristics

To create high-quality training data, OpenAI utilized the Wikidata property graph to resolve noise in Wikipedia-internal links. This process addresses issues such as anaphora (linking to a specific instance of a type rather than the type itself) and metonymy (linking from a nickname).

For example, the system heuristically converts links to their "generic" meaning, which significantly reduced the number of associated entities for the word "king" from 974 to 14 and increased the number of links from "queen" to the category "monarch" from 32 to 3,553.

Learning the Type System

Selecting the optimal type system is a complex task because the number of possible category sets is too large for an exact solution. OpenAI employed heuristic search, stochastic optimization (evolutionary algorithms), and gradient descent to find a balance between two key factors:

  • Learnability: Measured by the average Area Under the Curve (AUC) of a classifier trained to predict type membership from context. High AUC indicates a type is easy for the neural network to infer.
  • Oracle Accuracy: The measure of how well entities would be disambiguated if all types were predicted perfectly.

Neural Type System Implementation

Using the optimized type system, OpenAI trained a bidirectional LSTM on 400 million tokens for both English and French. The model was trained using supervision only on intra-wiki links, yet it achieved an F1 score over 0.91 for predicting type membership.

Some discovered types include broad categories like Aviation, Clothing, and Games, as well as highly specific ones such as 1754 in Canada.

Inference and Complexity

Unlike traditional entity disambiguation methods that rely on coherence metrics between different entities in a document (which results in $O(N^2)$ complexity), this system operates with $O(N)$ runtime. It identifies phrases using a trie and ranks possible entities based on Wikipedia link frequency, refined by the likelihood provided by the type classifier.

Sources