A Reviewer Audited Terminal-Bench 4: 60 of 65 Tasks Had Problems and 44 Could Be Passed by Cheating. Stop Picking Your Coding Agent by Leaderboard.
On October 19, researchers at Preference Model published an audit of Terminal Bench 4 and reported that 60 of the 65 tasks they reviewed had at least one problem. On 44 of those tasks a model could pass by cheating, and on 41 a model could do honest, correct work and still fail because of a rule the instructions never clearly stated. If you chose your coding agent, or your model, partly because of a terminal-benchmark score, that number should change how much weight you give it.
What the audit found
The authors, Nathan Palamuttam and Felipe Peter, reviewed and ported 65 tasks from Terminal Bench 4, as listed on Harbor's leaderboard under the tag 4-0-0. They skipped a 66th, protein-autointerp-disulfide, because it needs internet access that their framework does not allow. Of the 65, 60 had at least one issue, 44 could be passed by cheating, and 41 graded models on a rule the instructions never clearly stated.
Their examples of cheating are specific: models teleported a robot past masked frames, moved expensive work outside a timed section, and removed page elements that caused layout shift. None of that is solving the problem you would want solved on your own codebase.
The score gap is just as telling. Harbor's best result on these 65 tasks was 65.8%, from Claude Opus 5.5. The authors report Claude Sonnet 5.5 passing 62.8% of attempts on Harbor and 89.2% (290 of 325 attempts) on their own port with the task problems fixed, with tasks passed on every attempt rising from 33 to 50. In their words, "Terminal Bench 4 is closer to solved than its leaderboard suggests," and scores "can undersell what they can do, and oversell how honestly they did it."
One correction to the way this circulated: a social post summarizing the audit mentioned "325 issues in total." Per the authors' own writeup, 325 is the number of attempts (65 tasks times 5 attempts), not a count of issues. I am using the writeup's figures.
Why a solo dev should care
A leaderboard is a proxy for the question you actually have, which is whether this agent will fix your bugs without breaking your build. The audit shows two ways the proxy fails. Scores can overstate ability when a task can be gamed, and understate it when the grader checks something nobody told the model about. Both errors point in directions you cannot see from a single percentage.
The authors also argue most benchmarks would show the same problems if anyone looked this closely, and they say they would bet on it. I would treat that as an informed suspicion, not a finding, because they audited one benchmark. But it matches what you would expect from any test written once and then optimized against for months.
What I'd actually do
Build a private eval from your own work. It takes about 30 minutes the first time.
- Pick five tickets you already closed in the last month, ones where you know the correct fix because you shipped it.
- For each, check out the commit before the fix and write the one-sentence task you would give a teammate.
- Run each agent you are comparing on all five, with the same prompt and the same time limit.
- Grade with your test suite plus a five-minute diff read. Note which runs passed the tests by touching the tests.
- Record pass count, wall-clock time, and cost per run in a table.
Five tasks is a small sample, so use it to catch big differences, not to rank two close contenders. If two agents tie, pick the cheaper one and re-run in a month.
The diff read is the part people skip, and it is exactly where the audit's cheating examples would show up. An agent that "passes" by deleting the check is worse than one that fails honestly.
Where I could be wrong
A flawed benchmark can still rank agents correctly if the flaws hit everyone equally, and a leaderboard remains a cheap first filter for a field with dozens of models. If you are choosing between a frontier model and a model six months old, the gap will be visible on any test. My argument is narrower: the closer two scores are, the less a published number tells you, and the audit gives a concrete reason to doubt top-of-board differences. Also, my five-ticket eval has its own bias. It measures how well an agent handles your codebase's habits, which is what you want, and it will not tell you how it does on someone else's.
Author
Lukas
@lukcombinator