OpenAI Advances Coding Capabilities in Low-Resource Programming Languages
Our latest model advancements provide stronger capabilities for building and migrating software written in low-resource programming languages, such as COBOL, Fortran, and Visual Basic. These languages support critical global infrastructure—for example, COBOL supports over 40% of online banking systems—but they have less available code and documentation for models to learn from compared to languages like Python or JavaScript.
This work addresses the difficulty developers face when maintaining and modernizing software in these languages, where documentation may be incomplete and business rules often reside only within the code. The models were guided by feedback from developers and enterprise customers who noted that existing coding models struggle with these specific languages.
To measure progress, OpenAI developed two complementary evaluations: MultiLangBench, which tests the ability to build new software, and a COBOL migration evaluation, which tests the ability to preserve existing software behavior.
MultiLangBench: Building Software Across Languages
MultiLangBench requires a model to build a SQLite-compatible database engine from scratch in a specified programming language. The evaluation assesses the model's ability to organize a codebase, manage state, handle errors, and add new features while preserving existing functionality.
To ensure consistent testing across languages, the evaluation uses database files as a shared interface. The candidate program receives an initial database and a SQL query, and then produces a modified database, which is checked against the reference SQLite implementation.
Performance of GPT-6 Astra
At maximum reasoning effort, GPT-6 Astra achieved an average test pass rate of 87.4% across 20 programming languages, marking a substantial increase from 26.8% achieved by GPT-5.6 Sol. Improvements were observed in every language tested, including COBOL (rising from 10.9% to 72.1%), OCaml (from 25.6% to 85.8%), Fortran (from 21.4% to 83.5%), and Visual Basic (from 27.5% to 89.4%). Astra scored above 83% in 19 of the 20 languages, with COBOL remaining the most challenging at 72%.
Cost-Efficient Option: GPT-6 Luna
GPT-6 Luna provides a lower-cost option that also outperforms GPT-5.6 Sol in every language tested. At maximum reasoning effort, it reached an average test pass rate of 40.3%. These gains were achieved at an estimated average API cost of $0.77 per sample, which is approximately 78% lower than the $3.48 cost of GPT-5.6 Sol.
COBOL Migration Evaluation: Preserving Business Logic
The second evaluation challenges models to migrate existing COBOL programs to modern languages, including C++, Java, Kotlin, Python, or Rust, while strictly preserving the program's original functionality. Successful migration requires the model to understand both the program's business logic and COBOL's specific language behaviors.
For example, COBOL programs often handle financial transactions where details like rounding rules are critical. COBOL’s fixed-point arithmetic, for instance, discards excess decimal digits unless rounding is explicitly requested. The evaluation tests whether the model preserves these behaviors using hidden tests that require byte-for-byte parity on observable outputs.
Migration Accuracy and Cost
On this migration evaluation, GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared to only 5.4% for GPT-5.6 Sol at the same setting, while costing 61% less. When the reasoning effort was increased, accuracy reached 97.9%, bringing the evaluation close to saturation. These results demonstrate significant progress toward more accurate and affordable code migration for businesses modernizing their legacy systems.
Implications for Enterprise Software Development
These evaluations, combined with direct feedback from enterprise customers, demonstrate that the models are becoming significantly better at applying software engineering skills across a wider range of languages. The overall goal is to enable models to handle tasks such as understanding unfamiliar repositories, planning changes, utilizing build tools, running tests, and debugging failures.
Developers can access the MultiLangBench evaluation setup and test suite on GitHub to inspect the evaluation criteria and review performance in their specific languages. Users are encouraged to provide feedback on model performance to help shape future training and evaluation efforts.