Inside OpenAI's In-House Data Agent
OpenAI has developed a bespoke internal-only AI data agent designed to streamline data exploration and analysis across the company's massive internal data platform. By combining natural language interfaces with a sophisticated multi-layered context system, the agent allows employees across Engineering, Data Science, Finance, and Research to move from complex questions to actionable insights in minutes rather than days.
End-to-End Autonomous Data Analysis
The OpenAI data agent handles the entire analytics workflow—from initial data discovery and SQL generation to the publication of notebooks and reports. Unlike traditional tools that follow fixed scripts, the agent employs a closed-loop, self-learning process where it evaluates its own progress in real-time. If an intermediate result is incorrect (such as a query returning zero rows due to a join error), the agent investigates the failure, adjusts its approach, and retries the analysis while retaining full context.
A Six-Layer Context Architecture
To ensure accuracy and prevent common failures like misestimating user counts or misinterpreting internal terminology, the agent is grounded in six distinct layers of context:
1. Table Usage
The agent utilizes schema metadata (column names and data types) and table lineage to understand relationships between datasets. It also ingests historical queries to infer how tables are typically joined and queried.
2. Human Annotations
Domain experts provide curated descriptions of tables and columns to capture business meaning, intent, and caveats that cannot be inferred from schemas alone.
3. Codex Enrichment
By deriving code-level definitions of tables, the agent understands how data is constructed via pipeline logic. This allows it to distinguish between similar-looking tables (e.g., identifying if a table only includes first-party ChatGPT traffic) and understand data freshness and granularity.
4. Institutional Knowledge
The agent integrates with Slack, Google Docs, and Notion to retrieve context on product launches, reliability incidents, internal codenames, and canonical metric definitions. This information is managed via a retrieval service with strict access control.
5. Memory
The agent implements a memory system to retain non-obvious corrections, filters, and constraints. When a user corrects the agent or a nuance is discovered, the agent saves this learning to ensure future queries start from a more accurate baseline.
6. Runtime Context
When prior context is stale or missing, the agent issues live queries to the data warehouse and communicates with other Data Platform systems (such as Airflow and Spark) to validate schemas and understand data in real-time.
Technical Implementation and Scaling
OpenAI manages the scale of over 70,000 datasets and 600 petabytes of data using a daily offline pipeline. This pipeline aggregates usage, annotations, and Codex enrichment into a normalized representation, which is then converted into embeddings via the OpenAI Embeddings API. At query time, the agent uses Retrieval-Augmented Generation (RAG) to pull only the most relevant context, ensuring low runtime latency.
Evaluation and Quality Control
To prevent quality drift, OpenAI uses a systematic evaluation framework based on curated question-answer pairs. Each pair consists of a natural language question and a "golden" SQL query. The agent's generated SQL and the resulting data are compared against the golden set using an Evals grader, which accounts for syntactic variations in SQL that still produce correct results.
Security and Permissions
The agent operates as a pass-through interface layer. It inherits and enforces existing OpenAI security and access-control models, meaning users can only query tables for which they already have explicit permission.
Key Engineering Lessons
OpenAI identified three primary lessons during the development of the internal agent:
- Tool Consolidation: Providing too many overlapping tools confused the agent; restricting and consolidating tool calls improved reliability.
- High-Level Guidance: Rigid, prescriptive prompting degraded results. Shifting to high-level goals and relying on the reasoning capabilities of GPT-5 produced more robust outcomes.
- Code-Centric Meaning: While schemas describe shape, the true meaning of data lives in the pipeline code. Using Codex to crawl the codebase provided the most accurate understanding of what a table contains and when to use it.
Tooling Stack
The agent was built using the same tools available to external developers, including:
- GPT-5 flagship model for core reasoning.
- Codex for code-level table enrichment.
- Evals API for systematic regression testing.
- Embeddings API for context retrieval.