Alibaba Benchmark: Best Coding Agent Fully Solves Only 26.4% of Real Requests

The best of ten AI coding agents fully completed only 26.4% of real user requests on Alibaba's new Co-Design Bench, according to a paper posted to arXiv on Oct. 26. OpenAI's GPT-6 Astra topped the field. It passed 69.9% of each task's hidden acceptance criteria on average, but it satisfied every criterion on just 26.4% of the 273 tasks. Nearly three in four outputs would still need a person to finish the job, as The Decoder put it.

The benchmark is built from failure. Alibaba researchers drew on 37,463 deduplicated requests from 26,314 real coding sessions. They ranked candidates partly on signs of past failure: a user rephrasing a request two or more times, repeated follow-ups, or an explicit rollback. Experts then refined each request and wrote hidden grading rubrics — 2,331 criteria in total.

Tasks split into two tracks. Localized multimodal edits let the agent receive images, audio, video or documents. Repository edits require coordinated changes across a codebase. An agentic judge tests each finished app in a browser and checks the source code. A second verifier challenges every pass or partial verdict to catch mock-ups and hardcoded results.

How did the ten models score?

The main comparison ran every model through Claude Code as the harness. Anthropic's Claude Opus 5 and Alibaba's Qwen3.8-Max followed GPT-6 Astra. xAI's Grok 4.6 came last.

• GPT-6 Astra: 69.9% criteria met, 26.4% fully solved, 41.0% multimodal, 17.9% repository
• Claude Opus 5: 66.1%, 18.3%, 30.0%, 11.6%
• Qwen3.8-Max: 65.3%, 16.5%, 31.0%, 8.1%
• Kimi K3: 62.9%, 15.8%, 26.0%, 9.8%
• Qwen3.8-27B: 64.4%, 15.0%, 23.0%, 10.4%
• GPT-5.6 Sol: 61.1%, 14.3%, 20.0%, 11.0%
• GLM-5.3 Flash: 57.3%, 10.6%, 18.0%, 6.4%
• Claude Opus 4.8: 58.1%, 10.3%, 16.0%, 6.9%
• GLM-5.2: 56.7%, 9.5%, 16.0%, 5.8%
• Grok 4.6: 51.1%, 6.6%, 6.0%, 6.9%

Multi-file repository edits were the hardest task type. GPT-6 Astra fully solved 17.9% of them, against 41.0% of localized multimodal edits. Every model scored lower on repository edits spanning three or four kinds of change than on those spanning one or two.

Higher scores did not require more compute. On multimodal edits, GPT-6 Astra used a median of about 1.24 million tokens per task. GLM-5.2 used about 7.29 million tokens for a lower score — roughly six times the compute for a weaker result.

Why does the harness matter?

When the authors reran two models through other harnesses, the ranking reshuffled. Qwen3.8-27B scored 0.749 on repository edits in opencode and 0.560 in qwen-code. The gap between it and Kimi K3 shifted by up to 16.2 points depending on the tool. The authors want model and harness reported together, and The Decoder reported their conclusion that leaderboards listing only the model tell part of the story.

The paper also argues older edit benchmarks such as WebCompass are near saturation, with models above 99% — though that comparison used a different task pool.

What are the benchmark's limits?

The authors state the constraints plainly. Each request is fixed, so agents cannot ask follow-up questions or receive mid-task corrections. AI judges can still make errors. The results therefore measure autonomous completion under a strict rubric, not interactive development with a human in the loop.

The findings add to a pattern of agents underperforming on real work. Earlier this month, MLQ covered CWE-bench v2, where top agents still missed one in five security fixes. If the harness effect holds across other evaluations, model-only leaderboards may soon look incomplete.