OpenAI Text and Code Embeddings Release
OpenAI has introduced a new set of embedding models designed to map text and code into high-dimensional vector representations. These models outperform top existing models across three standard benchmarks, including a 20% relative improvement in code search.
Overview of Embedding Technology
Embeddings convert concepts into numerical sequences (vectors), allowing computers to analyze the semantic relationships between different pieces of data. In this high-dimensional space, vectors that are numerically similar are also semantically similar. OpenAI's embeddings are generated using neural network models derived from GPT-3, where each dimension of the resulting vector captures a specific aspect of the input text or code.
Available Embedding Model Families
OpenAI provides three distinct families of embedding models, each optimized for specific use cases:
| Model Family | Use Cases | Specific Models |
|---|---|---|
| Text Similarity | Clustering, regression, anomaly detection, and visualization | text-similarity-{ada, babbage, curie, davinci}-001 |
| Text Search | Semantic information retrieval and context relevance | text-search-{ada, babbage, curie, davinci}-{query, doc}-001 |
| Code Search | Finding relevant code using natural language queries | code-search-{ada, babbage}-{code, text}-001 |
Text Similarity and Linear Probing
Text similarity models allow users to compare two pieces of text using the dot product of their embeddings to generate a similarity score (cosine similarity) between -1 and 1. Higher scores indicate greater semantic similarity.
These models are particularly effective as features for machine learning classification tasks, known as "linear probes." OpenAI's text similarity models have achieved state-of-the-art results on linear probe classification within the SentEval benchmark.
Semantic Text Search
Text search models facilitate large-scale information retrieval by producing separate embeddings for queries and documents. By comparing the cosine similarity between a query vector and document vectors, the system can identify the most relevant content.
Unlike classical keyword search, which relies on word overlap, embedding-based search captures semantic meaning and is less sensitive to exact phrasing. OpenAI's text search models demonstrated superior performance on the BEIR search evaluation suite compared to previous methods.
Real-World Applications and Performance
Industry implementations demonstrate significant gains in accuracy and efficiency over previous embedding techniques:
- Educational Content Mapping: FineTune Learning used the
text-search-curiemodel to map textbook content to learning objectives, achieving a top-5 accuracy of 89.1%. This outperformed Sentence-BERT, which achieved 64.5%, reducing the labeling time for textbooks from hours to seconds. - Customer Insight Analysis: Fabius utilized embeddings to tag customer call transcripts for feature requests. By moving away from fuzzy keyword search to OpenAI embeddings, they identified 2x more general examples and 6x–10x more examples for abstract use cases that lacked clear keywords.