Evaluating Large Language Models Trained on Code: OpenAI Codex

OpenAI has introduced Codex, a GPT language model fine-tuned on publicly available code from GitHub. Codex is designed to synthesize Python code from natural language docstrings, significantly outperforming general-purpose models like GPT-3 and GPT-J on functional correctness.

Performance on HumanEval

Codex was evaluated using HumanEval, a new evaluation set released by OpenAI to measure the functional correctness of programs synthesized from docstrings. On this benchmark, Codex solves 28.8% of the problems on a single attempt, compared to 0% for GPT-3 and 11.4% for GPT-J.

Repeated Sampling Strategy

Repeated sampling from the model is an effective strategy for increasing the success rate of producing working solutions to difficult prompts. By generating 100 samples per problem, the success rate for Codex increases to 70.2%.

Model Limitations

Despite its capabilities, Codex exhibits specific technical limitations in its ability to generate correct code. The model struggles with:

  • Long chains of operations: Codex has difficulty when docstrings describe a sequence of long, complex operations.
  • Variable binding: The model occasionally fails to correctly bind operations to variables.

Broader Impacts

OpenAI discusses the potential broader impacts of deploying powerful code generation technologies, focusing on three primary areas:

  • Safety: The risks associated with generating potentially insecure or buggy code.
  • Security: The vulnerabilities that could be introduced by AI-generated code.
  • Economics: The impact of automated code generation on the software engineering profession and the economy.

Sources