A New Benchmark Gave Agents 50 Real Tax Assignments. Claude Sonnet 5.5 Passed 10.8% of Them. Here Is What I Would Hand an Agent in My Books and What I Would Not.
On October 27, Surge AI published Dayjob: Tax, a benchmark that gives AI agents 50 professional tax assignments with real workspace files and grades them against a rubric. Claude Sonnet 5.5 passed 10.8% of attempts, Claude Fable 5.1 passed 9.6% and Claude Opus 5.5 passed 9.2%, using a bar of 95% on the rubric. The authors' conclusion is that agents "haven't yet passed the audit." If you run a one-person business and you have been wondering whether you can hand your books to an agent, this is a number worth keeping.
What the benchmark actually measured
The test has 50 environments across eight tax practice areas, with 1,789 workspace files in total, a median of 30 per task and up to 123 in the largest. This is closer to a real client folder than to an exam question. The agent has to find the right documents, work out what is being asked and what is not, and carry a workflow through to a finished result.
Scores are pass@1 at a 95% rubric threshold, which is strict by design. Tax work is the kind where a 90% answer can still be an expensive mistake. Sonnet 5.5 and Opus 5.5 sit at the high-performance end of the page's cost-performance Pareto frontier, which means that among the models tested, paying more did not buy a better result. I could only see the three Claude scores in the write-up I read. The full leaderboard lives on the benchmark's own page, so I will not claim these three are the top of the field.
Two failure modes that matter to you
The authors describe where agents fail, and both patterns are familiar to anyone who has delegated finances to a tired assistant.
The first is missing a consequential issue nobody asked about. In one case, Astra never flagged a deadline after which a condo sale could lose a tax exclusion, while Opus flagged it in all five attempts. The second is acting on an unverified assumption. Opus assumed a separating couple was unmarried in all five attempts, despite records suggesting a possible valid marriage elsewhere, while Astra questioned that assumption in four of five attempts.
Notice that neither failure is arithmetic. Models are fine at adding up columns. They fail on the questions a human advisor asks without being prompted, such as "is there a deadline here," "are you sure about the filing status" and "does this transaction trigger anything else." For a solo operator, those are the questions that matter. You usually have no one to ask them, which is why you reached for an agent in the first place.
What I'd hand an agent, and what I wouldn't
This benchmark scores whole assignments, so it does not say agents are useless at pieces of the job. The authors themselves say agents can do individual tax steps. I would split the work along that line.
Safe enough to delegate with a check: categorizing a month of transactions against a list of categories I wrote, drafting a summary of what came in and went out, pulling invoices into a folder structure, and flagging receipts with missing dates or amounts. These have a right answer I can spot-check in ten minutes, and mistakes show up in the totals.
Not safe to delegate: anything involving filing status, deadlines, residency, entity choice, depreciation or what counts as deductible. These are judgment calls with penalties attached, and the benchmark's failure modes sit exactly there. If a decision comes with a date, a form number or a "does this qualify" attached to it, a person who is licensed or accountable should look at it.
Where I could be wrong
A pass rate of 10.8% against a 95% rubric does not mean the agent was wrong 89% of the time. The bar is demanding, and partial credit is not counted as a pass. A model scoring 85% on the rubric would still count as a failure. Tax is also unusually hard, with jurisdiction details and edge cases, so a bookkeeping workflow for one country and one simple business might do much better than this suite. These models also improve fast. A score taken today may be stale by spring.
I could also be underestimating how much a good prompt and a checklist change the result. The benchmark tests agents given an assignment, not agents wrapped in a process built by someone who knows their own books.
What I'd actually do
I would use an agent for the clerical layer and keep a short written list of questions it must answer before I accept any output: what deadlines does this touch, what did you assume that I did not tell you, and what is the one thing I did not ask that I should have. That list turns the two failure modes above into a routine rather than a hope. For anything that changes how I file, I would pay an accountant for an hour. One hour of professional time is cheap next to a missed exclusion deadline, and no model that exists today has given me a reason to skip it.
Author
Lukas
@lukcombinator