OpenAI Text and Code Embeddings by Contrastive Pre-training

TL;DR

OpenAI has introduced a method for generating high-quality vector representations of text and code through contrastive pre-training on unsupervised data at scale. This approach allows a single unsupervised model to achieve state-of-the-art results across multiple tasks, including linear-probe classification and large-scale semantic search, often outperforming previously fine-tuned supervised models.

Contrastive Pre-training for General Purpose Embeddings

Contrastive pre-training on unsupervised data at scale enables the creation of vector representations (embeddings) that capture deep semantic meaning. Unlike previous approaches that required models to be customized for specific use cases—varying in architecture, training objectives, and dataset choices—this method produces general-purpose embeddings that are useful for a wide range of applications such as semantic search and text similarity.

Performance in Text Embedding Classification

Unsupervised text embeddings produced by this method achieve new state-of-the-art results in linear-probe classification. On an average across seven different tasks, the best unsupervised model demonstrated a relative improvement of 4% over the previous best unsupervised model and 1.8% over the previous best supervised text embedding model.

Advancements in Semantic Search

The contrastive pre-training approach significantly improves the performance of large-scale semantic search. When evaluated against standard benchmarks, the best unsupervised model achieved the following relative improvements over previous best unsupervised methods:

• MSMARCO: 23.4% relative improvement
• Natural Questions: 14.7% relative improvement
• TriviaQA: 10.6% relative improvement

Code Embedding Capabilities

By training code embedding models on (text, code) pairs, OpenAI has demonstrated that the same contrastive pre-training logic can be extended to the same effect for programming languages. This process resulted in a 20.8% relative improvement over prior best work in the domain of code search.

Summary of Technical Impact

This research demonstrates that scale and contrastive pre-training on unsupervised data are sufficient to produce embeddings that are-first, answer-first, and highly competitive with supervised, fine-tuned models across both natural language and programming code.