OpenAI's COBOL migration tests reach 98%, but the gap remains

By Tom Whitfield

OpenAI said GPT-6.1 Sol achieved 87% behavior-preservation accuracy at low reasoning effort in its COBOL migration tests, rising to 98% at high reasoning effort. Reasoning effort here measures how many computational steps a model takes before answering. These were vendor-run code-translation evaluations, not trials showing that a critical production system can be replaced without extensive integration and testing.

What did OpenAI’s tests show?

OpenAI reported substantial improvements in its latest models’ ability to move aging software from older languages into modern code, with an eye to helping organizations maintain and modernize critical systems. In a Monday blog post, the company described COBOL migrations into C++, Java, Kotlin, Python, and Rust while preserving the original program’s behavior.

OpenAI considered a migration successful only if every test passes, and described the results as progress toward more accurate, affordable code migration for teams modernizing business software. It also reported better ability to build database software in several programming languages, including COBOL.

At low reasoning effort, GPT-6.1 Sol’s 87% result exceeded GPT-6 Astra at about 86%, GPT-6 Sol at 21%, and GPT-5.6 Sol at 5%. The score difference is substantial, but the benchmark does not establish that an entire legacy system can be replaced safely.

Why is replacing COBOL harder than rewriting it?

Introduced in 1959 as common business-oriented language, COBOL remains embedded in critical infrastructure. According to IBM, the language forms the foundation for more than 40% of online banking systems, supports 80% of in-person credit-card transactions, and handles 95% of ATM transactions. Government software includes Social Security, the IRS, and services run by most states.

One rough estimate placed the annual cost of COBOL inefficiencies at roughly $105 billion, measured against U.S. GDP. Fortran also remains in use for scientific computing, weather forecasting, atmospheric and ocean studies, and software considered critical to nuclear security.

Replacing either language is difficult because the systems carry too much operational risk to break, and many specialist programmers have retired or died. AI coding tools have historically struggled with these systems, and Zaytsev said they are not yet playing a major role in replacing or migrating legacy COBOL systems. A replacement must also preserve undocumented behavior, data formats, technical links to other systems, and the expectations of the organizations that use them.

Vadim Zaytsev, an associate professor of software evolution at the University of Twente, said the largest COBOL system he had encountered contained 250 million lines, which he stressed was not the world’s largest. “Rewriting code is not the same as replacing a system,” he said. The constraint is therefore system complexity and scarce human knowledge, not a shortage of generated code.

What did Anthropic add?

In February, Anthropic published a step-by-step code modernization playbook and promoted Claude Code as a tool for moving COBOL into Java or Python. The company said the economics of COBOL modernization had shifted. It supplied no performance benchmarks showing how well Claude Code performs those migrations.

IBM shares nosedived following Anthropic’s announcement; IBM derives part of its revenue from helping companies maintain legacy systems. OpenAI spokesperson Oscar Haines said legacy code remains far from solved, adding that the company is listening to feedback from developers and enterprises and exploring better tests and stronger model understanding of coding languages.

Anthropic did not provide comment in time for publication. Experts said AI coding tools have made progress on legacy code, but view OpenAI’s advance as a long way from safe replacement of critical systems. The two model companies have identified a commercial opening, while neither has demonstrated autonomous replacement of critical workloads.

How can migration teams verify behavior?

Arie van Deursen, a software engineering professor at Delft University of Technology, called OpenAI’s blog superficial because it gave few details about the migration task. He said the language benchmark covered a relatively simple, well-defined problem resembling systems encountered during model training, rather than a full modernization project.

Van Deursen said maintaining aging software is much easier than modernizing it, and described the benchmark as light-years from real modernization. The evaluations also did not test the key step of breaking a complex legacy system into smaller units that teams can replace individually.

Alastair Donaldson, a programming languages professor at Imperial College London, said the main challenge is not simply understanding how a system works, because that understanding is often missing from legacy code and few people can still program in COBOL. He also said no organization would choose COBOL for a new project. For live systems, he said, even minor behavioral differences before and after migration are unacceptable.

One validation method would run the same workflows through old and new systems to check for identical behavior. AI could also document each code translation so specialists familiar with the legacy language can review it. Formal verification could use mathematical techniques to check behavior automatically.

Jacopo Mauro, a mathematics and computer science professor at the University of Southern Denmark, said formal verification would require a huge amount of work. Such methods already verify some AI-generated mathematical results, and he said automated checks would be better than relying on a human to confirm every translation. Human verification nevertheless remains essential, he said.

Zaytsev used a simple test: “Suppose you have a magic AI wand that handles 99 percent of your code. What do you do with the remaining 1 percent? That 1 percent can be a million lines of code, spread all over the place and subtly failing in more ways you can imagine within your lifetime.” His point was that residual code can conceal failures too numerous for human teams to inspect manually.

AI can save time and money, but partial automation does not remove the hard cases. Until automated verification proves equivalent behavior across the whole system, OpenAI and Anthropic will need to pair faster code generation with dependable verification; until then, coding agents will support specialist migration teams rather than replace them.