METR Says Anthropic's Capability Growth Is About 1.9x Its Old Pace and Mostly AI-Driven. Build Like the Next Model Ships Sooner Than You Planned.
METR published an assessment on November 5 estimating that Anthropic's capability growth on benchmark tasks is running at about 1.9x its pre-October-2025 rate. The 95% confidence interval is 1.5x to 2.8x, which is wide, and Anthropic redacted most of the evidence METR found most informative. I don't think the exact number matters to you. The direction does, because it changes how long your workarounds will be worth keeping.
What METR actually reported
The evaluation was run by about 2.5 full-time-equivalent METR staff with high access, between August 12 and October 29, 2026. The headline is that growth in Anthropic's internal capability index, AECI, is 1.9x the earlier rate. METR's robustness checks gave a range of 1.1x to 2.9x. On attribution, METR's subjective estimate is that a majority, perhaps around 80%, of the October 2026 acceleration is due to AI itself. It says that is highly uncertain.
Two numbers deserve suspicion. Anthropic staff self-reported productivity uplift of about 12x in an internal Slack poll, which METR calls "very loose evidence." And METR says it did not identify any clear bottlenecks for the next 12 months, while noting its own view changed several times during the work.
The caveats are the story
METR had no query access to unreleased models. A majority of the most informative evidence was redacted by Anthropic. Benchmarks are saturating, so the index may be measuring the tail of what the tests can see. And METR warns the index may not track real-world impact at all. Claude was also involved in most of the data collection and analysis, which is either a nice demonstration or a conflict of interest depending on your mood.
So the honest summary is: an independent group with partial access thinks progress sped up, and cannot rule out that it's less than 2x or that it doesn't matter for your product.
What it means for a solo builder
If the cadence is faster than you planned for, three habits get more expensive. Prompt scaffolding written to cover a model's weakness, such as retry loops, output validators and giant few-shot blocks, has a shorter shelf life. Fine-tuned or prepaid commitments to one model age quickly. And a hand-built benchmark you ran once last spring is stale.
I made the pricing version of this point in the post about eight LLM price moves in nine weeks: build the swap layer. This is the capability version. Every workaround is a bet that the model will stay weak at that task for a while, and that bet is getting shorter.
What I'd actually do
Schedule a quarterly deletion review. On each new model release, pick one piece of scaffolding, remove it, and run your private eval set. I keep a list of the twenty requests my product handles worst, and that list is the eval. If quality holds, the code stays deleted. If it drops, you've confirmed the workaround still earns its place and you know the date to check again.
Keep model names and prompts in config, never hard-coded across files, so a swap is a one-line change. And don't promise customers a specific model by name in contracts unless you must.
Where I could be wrong
Benchmark progress is not product usefulness, as METR itself says, and the confidence interval includes 1.5x, which would feel almost like the old pace. My advice costs little either way, since deleting dead scaffolding is good hygiene at any speed. The danger in the other direction is over-reading a 1.9x as a promise that your specific task gets better soon. It may not. A model can improve on a coding benchmark and stay unchanged at your invoice parser.
Author
Lukas
@lukcombinator