The Best AI Coding Agent Ships a Mergeable PR 13% of the Time on Hard Tasks. Stop Reading SWE-bench — This Benchmark Measures What You Actually Pay For.
The Best AI Coding Agent Ships a Mergeable PR 13% of the Time on Hard Tasks. Stop Reading SWE-bench: This Benchmark Measures What You Actually Pay For.
Cognition shipped a benchmark on June 8 called FrontierCode, and the headline number is uncomfortable in a useful way. On its Diamond subset (the 50 hardest tasks), the leading model, Claude Opus 4.8, scores 13.4%. GPT-5.5 gets 6.3%. Gemini 3.1 Pro lands at 4.7%.
Those numbers look broken if you're used to SWE-bench scores in the 60s and 70s. They're not. FrontierCode is measuring something different, and the something different is the thing you actually care about: not "did the code pass a test" but "could I merge this pull request without rewriting it."
What FrontierCode measures that the others don't
Most coding benchmarks you've seen score correctness. The agent gets a task, produces a patch, and the patch either passes the hidden tests or it doesn't. Pass rate goes up, leaderboard moves, headline gets written. The problem is that "passes the tests" was always a low bar for production code. A patch can pass tests and still be unmergeable: wrong scope, no new tests, a regression in a path the existing tests don't cover, or just sloppy enough that a reviewer sends it back.
FrontierCode scores mergeability instead. Cognition built it with open-source maintainers, each task taking 40-plus hours to construct, and it grades on behavioral correctness, regression safety, mechanical cleanliness, test correctness, scope control, and overall code quality: the dimensions a senior engineer actually checks in review. It's nested into three subsets: Diamond (the 50 hardest), Main (the 100 hardest), and Extended (150 total). The 13.4% is the Diamond number.
So the gap between a 70% SWE-bench score and a 13% FrontierCode Diamond score isn't a contradiction. It's the distance between "technically correct" and "I'd merge this." That distance has a name in your workflow, and it's the part of the bill nobody quotes you.
How to read 13.4% without overreacting
Two ways to misread this number, and I want to head off both because I've watched people do each.
The doomer read: "agents are useless, 13% is a disaster." That's wrong. Diamond is the 50 hardest tasks Cognition could build with veteran maintainers: gnarly, multi-file, regression-prone changes in real open-source projects. That is not your average Tuesday ticket. Most of what you hand an agent is closer to "add a field, wire it through, update the form," and agents are genuinely strong at that. The Diamond score is the ceiling-of-difficulty measurement, not the expected value of your day.
The hype read: "13% on the hardest tasks, so on normal tasks it's basically solved." Also wrong, and more dangerous, because it leads you to merge without looking. The same dimensions FrontierCode grades (scope creep, missing tests, quiet regressions) are exactly the failures that slip through on easy tasks too, just less often. The agent that writes a clean-looking patch for a simple ticket can still expand the scope, skip the test, and introduce a regression your existing suite doesn't catch. The benchmark didn't invent those failure modes on hard tasks. It just made them impossible to ignore.
The honest read is in between: agents produce a lot of value and a lot of drafts, and the best available models are nowhere near "merge it unread," even at the top of the leaderboard. That's not a knock. It's an operating instruction.
The verification tax, finally with a number on it
I've written before that you should treat agent output as a draft. FrontierCode is the first benchmark I've seen that puts a defensible number on why. The gap between correctness and mergeability is the verification tax (the time you spend reading, testing, and cleaning what the agent produced before it's safe to ship), and Cognition just published a measurement of how wide that gap is on hard problems.
Here's what that should change in how you work, concretely.
Budget the review, not just the generation. When you estimate "the agent will do this feature in an hour," you're pricing the generation. The FrontierCode gap is the part you forgot: the review-and-repair pass that turns a passing patch into a mergeable one. On anything non-trivial, that pass is not optional and it is not fast. Put it in the estimate or you'll blow every deadline you set with an agent.
Make the agent do the boring safety work up front. The dimensions FrontierCode penalizes (missing tests, scope creep, regressions) are the ones you can push left. Tell the agent to write the test first, to touch only the files in scope, to explain what existing behavior its change could break. You won't close the gap to 100%, but you'll convert "passing patch" into "reviewable patch," and reviewable is the whole game.
Stop quoting the leaderboard you've been quoting. If your read on which model to use is built on SWE-bench pass rates, you're optimizing for correctness when the thing that costs you money is mergeability. The model that's 2 points higher on SWE-bench but produces sloppier, scope-creeping diffs is the more expensive model once you count your review time. FrontierCode is a better proxy for your actual bill, even early and imperfect as it is.
What I'd actually do
Pick your default coding model based on the benchmark that maps to your cost, not the one that maps to the press release. Right now that's a mergeability benchmark, and right now Opus 4.8 leads it, but the specific winner matters less than the lens. Then run your own one-week test: take ten real tickets you've shipped, hand them to your agent, and honestly log how long the review-and-repair took versus the generation. That ratio is your personal FrontierCode gap, and it's the number that should drive whether you're actually saving time.
The honest counter-take: FrontierCode is brand new, built by a company that sells a coding agent, and a single vendor's benchmark deserves skepticism until it's been independently reproduced. The Diamond set is also deliberately brutal, so reading the 13.4% as "what agents can do" instead of "what agents can do on the 50 hardest curated tasks" would be a real mistake, one I'd push back on if someone quoted it at me out of context. And benchmarks drift; the next model release will move these numbers and someone will write the "agents solved it" post regardless of whether they did.
But the underlying point survives all of that. The metric that decides whether an agent saves you time isn't whether the test passed. It's whether you can merge the diff without rebuilding it. FrontierCode is the first benchmark pointed at that question, and the answer (even at the top) is "read it first." Which is what you should have been doing anyway.
Author
Lukas
@lukcombinator