Co-Design Bench: the best coding agent meets 70% of the spec but fully finishes only 26% of real app edits

Published November 10, 20264 min read

The paper, Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models, went up on arXiv on October 26. The Decoder covered it on November 8. The project site publishes the leaderboard, and the code and data are public.

We haven't rerun it. All numbers below are the authors', and the grading is done by AI judges, which the authors checked by hand on a sample.

What the benchmark tests

Most coding benchmarks hand an agent a synthetic issue. Co-Design Bench starts from 37,463 deduplicated requests across 26,314 real user coding sessions. The authors favored requests where users had to rephrase or roll back, so the set leans toward the requests agents have struggled with.

• 273 tasks, two tracks. Multimodal Editing has 100 localized edits, many with an attached image, audio clip, video, or document. Repository Editing has 173 coordinated changes across a web app, and 166 of them need two or more kinds of edit.
• 2,331 hidden acceptance criteria, written by experts. That's about 5.5 per task on the multimodal track and 10.3 on the repository track.
• One request, no follow-ups. The agent gets a single fixed request. Clarifying questions and mid-task corrections aren't tested.
• One harness for every model. The paper ran all ten endpoints in Claude Code with shared prompts, tools, and containers, capped at 3 hours and 120 turns.

Grading uses what the authors call judge-then-refute. Each app is built and run in a sandbox, a judge collects source, screenshots, and browser actions for every criterion, and a second model challenges every pass, looking for stubs, hardcoded results, and features nobody can reach. Two frontend experts reviewed 200 scored outputs, though the paper reports no agreement figure.

The results

“Strict” is the share of criteria that fully pass. “Full pass” is the share of tasks where every criterion passes. From the paper's main tables:

The newest models from several labs, including Claude Opus 5.5, aren't in the table. On repository edits, no model fully finishes even one task in five.

Compare that with older edit benchmarks. On 150 WebCompass-edit tasks, the project site reports Opus 4.8 reaching 99.4% graded feature completion in agentic mode. The same model fully finishes 10.3% of Co-Design Bench tasks.

Most failures are one detail short

The gap between 69.9% of criteria and 26.4% of tasks is the headline. Agents rarely fail outright. The Decoder sums up the pattern: the agent does most of the work but “leaves at least one detail unfinished.”

• Broader edits are worse. Repository tasks that mix three or four kinds of change, such as layout, styling, interaction, and content, score 59.1% strict against 65.4% for one or two kinds. Every one of the ten models drops.
• Attachments hide traps. Tasks with a document attached had the highest pooled criteria score, yet only 18.2% of model-task pairs fully passed, against 37.0% for text-only tasks.
• Working features are hardest. Pooled full-pass rates are 20.2% for understanding the input, 13.6% for UI changes, and 10.3% for functional changes.

That matches what most of us see in review. The diff looks complete, the main path works, and the third requirement in the ticket quietly didn't happen.

More tokens didn't help, and the harness moved the ranking

On the multimodal track, GPT-6 Astra scored 75.4% strict using a median of 32 turns and 1.24 million tokens. GLM-5.2 used 69 turns and 7.29 million tokens and scored 57.2%. Spending more effort didn't buy more credit.

A separate experiment reran two models in Codex, OpenCode, and Qwen Code. On repository edits, Qwen3.8-27B scored 63.8%, 74.9%, and 56.0% strict depending on the harness, a spread of almost 19 points. On the multimodal track, which of the two models came out ahead reversed depending on the harness. These numbers don't line up with the main table, and the paper doesn't say why.

The Decoder's summary puts it well: leaderboards that list only the model “only tell part of the story.” Our benchmark guide covers how the older leaderboards were built.

What to do with this

1. Write acceptance criteria into the prompt. The benchmark grades against hidden criteria. Your agent does better when the criteria aren't hidden. A numbered list of what “done” means gives the agent, and you, something to check off.
2. Review against the list, not the diff. A clean diff is exactly where the missing 30% hides. Walk each requirement in the running app before you merge.
3. Split broad requests. If a ticket mixes layout, behavior, and content changes, run them as separate tasks. The paper's drop on mixed edits applies to every model it tested.
4. Test your model in your harness. A 19-point swing from the harness alone means a vendor leaderboard tells you little about your setup. Run five of your own past tickets through each candidate.
5. Add a refute step. The benchmark's second judge looks for stubs and hardcoded results. A reviewer subagent with the same brief is cheap to add to your workflow.

Sources

• Wei et al., Co-Design Bench (arXiv 2610.35138, October 26, 2026): task construction, harness, main results, breakdowns, and harness sensitivity.
• Co-Design Bench project site: leaderboard, track sizes, metrics, token figures, and the WebCompass-edit comparison.
• The Decoder (November 8, 2026): summary and quotes.