OpenAI Research Agenda for Economic Impacts of Code Generation Models
OpenAI is establishing a research program to evaluate the economic impacts of code generation models, specifically using Codex as a primary case study. This research is intended to provide evidence-based insights to inform deployment policies, AI system design, and broader public policy regarding the integration of large language models (LLMs) into the workforce.
Economic Impact Assessment via Codex
Codex, an LLM fine-tuned from GPT-3 on billions of lines of public GitHub code, serves as the foundation for this research agenda. Because Codex has demonstrated the ability to generate functionally correct code 28.8% of the time on a sample of evaluation problems, OpenAI posits that its deployment may significantly alter the economics of software development and the industries that rely on it.
OpenAI views the use of Codex as a critical starting point for establishing methodologies that will be applicable to future, more capable LLMs. By studying the current impacts of a specific tool, researchers can generate the evidence needed to guide decision-making in three key areas:
- Deployment Policy: Determining how and when to release these tools to the workforce.
- AI System Design: Improving the architecture and functionality of models to mitigate negative economic externalities.
- Public Policy: Creating regulations and social safety nets to address shifts in the economic landscape.
Priority Outcome Areas for Research
OpenAI has identified six priority areas where the economic impacts of code generation models should be measured:
Productivity and Employment
Research will focus on whether code generation models increase the overall productivity of developers and how this shift affects the total demand for software engineers. The goal is to understand if these tools act as complements to human labor or as substitutes for specific tasks.
Skill Development
The research agenda seeks to determine how the use of AI-assisted coding affects the learning curve for new developers. There is a focus on whether these tools accelerate skill acquisition or potentially hinder the development of fundamental coding skills.
Inter-firm Competition
OpenAI intends to study how the accessibility of code generation tools affects competition between firms. This includes analyzing whether these tools lower the barrier to entry for new software companies or consolidate power among existing dominant players.
Consumer Prices
The research will examine if the productivity gains from AI-assisted coding are passed down to consumers in the form of lower prices for software products and services.
Economic Inequality
The research agenda addresses the potential for LLMs to increase or decrease economic inequality. This includes analyzing the distribution of productivity gains across different skill levels of developers and who captures the majority of the economic value created by these models.
Collaboration with External Researchers
To accelerate these findings, OpenAI has issued a Call for Expressions of Interest, inviting academic and policy researchers to collaborate with OpenAI researchers and customers. This collaboration aims to leverage external expertise to better measure the economic impacts of code generation models and other LLMs on individuals, firms, and society.