A New Paper Says the Highest Reasoning Effort Costs Up to 42x the Lowest on Tasks Every Level Solves. Start Low and Escalate Only on Verified Failure.
A paper submitted to arXiv on October 25, titled "What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents," reports that the highest reasoning effort setting costs up to 42 times as much as the lowest on tasks that every level solves. The authors, led by Kaiser Sun with Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou, and Federico Bianchi, tested four benchmarks covering single-call reasoning and long-horizon agent work.
If you have ever left a coding agent on "high" because it felt safer, this is about your invoice.
The credence good problem
The authors describe reasoning effort as a credence good. You buy it without being able to tell whether you needed it. If a task succeeds at high effort, you can't know whether low effort would have worked too. If it fails, you can't be sure more effort wouldn't have fixed it.
So they measured the counterfactual: the same tasks, across proprietary and open-weight models, at each effort level. Three results matter for anyone paying per token.
First, higher effort raises accuracy in aggregate, but it often fails on tasks that a lower level solved. More effort is not a strict upgrade for a given task. Second, on easy tasks that all levels solve, higher effort is usually more expensive, which is where the 42x ceiling comes from. Third, higher effort changes how agents behave: they verify more and search more, which lengthens their trajectories without necessarily improving outcomes.
The policy they propose
The recommendation is simple. Start at low effort, and escalate only after a verified failure. Across all models and benchmarks in the study, this matched or improved on running fixed top effort, and for most proprietary models it did so at lower cost.
"Verified failure" is the important phrase. Escalation is triggered by a check, not by a feeling that the answer looks thin. The paper's setup relies on tasks where success can be checked. For an agent that means something like a failing test, a type error, or a lint rule.
How I'd wire this into a solo setup
Most solo builders I know set one effort level in a config file and never touch it again. That is the 42x scenario waiting to happen on every small edit.
My plan is a two-step loop. Run the task at the lowest setting. Then run the project's existing checks: tests, type check, linter, a build. If they pass, stop. If they fail, retry once at the next level up, and repeat until you reach the top or the checks pass.
Where this works well: bug fixes with a reproducing test, refactors with a type checker, content tasks with a schema or link validator. Where it works badly: open-ended design work or writing, where nothing returns a clean pass or fail. For those I'd still start low and read the output myself before paying for more.
What to measure
I would add two numbers to whatever I already track: the share of tasks that pass at the first (lowest) level, and the average cost per task by level. If the first-level pass rate on your real work is high, you were overpaying. If it is low, the loop will cost you extra retries, and you should look at which task types fail so you can start those one level higher.
I can't promise your numbers match the paper's, because the study used its own benchmarks, and I have not rerun it on my workloads. The 42x figure is also a ceiling on tasks all levels solve, not an average saving. Expect a smaller gain.
Where this could be wrong
Escalation has its own cost. A task that fails at low effort and succeeds at high effort costs you both runs, and for slow tasks, the delay matters as much as the money. If your work is mostly hard problems, starting low just adds latency.
There is also a risk in "verified." A weak test suite will pass a wrong answer at low effort, and you'll never escalate. The policy is only as good as the check behind it.
What I'd actually do
This week, I'd lower the default effort in every agent config I control, keep a one-line note of which tasks needed escalation, and add a failing-check trigger where I can. After two weeks, I'd compare spend per merged change with the previous two weeks. If the number drops and nothing breaks, I'd keep it. If quality slips on a specific task type, I'd pin that type to a higher level and leave everything else low.
Author
Lukas
@lukcombinator