Amazon Told Its Engineers That Counting Tokens Is "Deeply Flawed." Here Is the Metric I Would Track Instead.
On October 27, Amazon published revised guidelines for its software engineers, and the line that matters is blunt: counting tokens and measuring lines of code are "deeply flawed" ways to judge productivity, according to Fortune's report on the memo. The author, a VP and distinguished engineer in Amazon's stores division, went further and said that turning those numbers into targets "is at odds with frugality."
That is a company with one of the largest cloud bills on earth telling its own staff that spending more on AI is not the same as doing better work. I think solo operators should read it as a warning aimed at them, because nobody else is watching their usage.
What the memo actually says
The guidance describes what it calls the "Renaissance Developer," someone who combines AI with their own expertise and judgment. Two instructions stand out. First, engineers should tell management when AI is not useful for their work and explain why, and the memo calls that outcome "a finding, not a failing." Second, managers should compare the 10th and 90th percentile of AI users on their teams: ask the light users why they use it less, and check whether the heavy users' output actually rises with their usage.
Amazon did not drop AI adoption metrics entirely. The memo says they still "correlate well with the outputs that matter," and that the company plans to find better metrics over time. So this is a correction, not a retreat.
The backdrop, per Fortune citing the Financial Times, is an unofficial internal leaderboard called Kirorank. Employees reportedly ran agents on meaningless tasks to inflate their token counts and climb it. Amazon ended it in May over concerns about wasted compute. Fortune also relays an Axios report that an unnamed company accidentally spent $500 million on Claude in one month after failing to limit employee usage. I can't verify that figure beyond the report, so treat it as a story about missing guardrails rather than a benchmark.
Why this lands harder on a one-person shop
Amazon's fix is a manager comparing percentiles. I don't have a team, so I have no percentile to compare myself against. I am simultaneously the heavy user, the light user, and the person paying the invoice.
That creates a specific trap. Token spend feels like effort. A long agent session that burns through a large context window feels productive because it was expensive, and a dashboard that shows rising usage feels like progress. Nothing in a solo setup pushes back on that feeling, which is exactly what the Kirorank story shows happening at scale when the incentive is a leaderboard instead of a bill.
The metric I would track instead
Tokens in and tokens out are inputs. The output a solo operator cares about is a change that shipped and stayed shipped. So the number I would track is cost per shipped change: the spend across your AI tools for the week, divided by the number of merged changes, published posts, or closed customer tickets that did not come back.
You don't need tooling for this. Most providers show weekly spend in a billing page, and your git log or CMS already counts what shipped. Put both numbers in a note every Friday. It takes ten minutes and gives you a trend line that the percentile comparison would have given you if you had a team.
Then run Amazon's two-percentile test on yourself, using weeks instead of people. Look at your lowest-spend week and ask what you did differently. Look at your highest-spend week and check whether the shipped count rose with it. If the expensive weeks ship the same as the cheap ones, you've found waste, and it is probably a default you never questioned, like a high reasoning setting or an agent left running on a task a script could do.
Where this could be wrong
Cost per shipped change is a crude number. A single large migration can make one week look terrible, and a week of tiny edits can make it look great. It also ignores quality: a cheap week of sloppy merges is not a win.
So I would pair it with one honest check. Once a month, pick the three most expensive tasks and ask whether a person or a plain script would have done them faster. Amazon's "a finding, not a failing" framing is useful here. If AI does not help with a task, writing that down is the point, not an embarrassment.
What I'd actually do
This week, I'd do three things. Pull last month's AI spend from every provider and tool into one note. Count what shipped over the same period. Divide, and write the number at the top of the note.
Then set a soft cap, not a goal. A spend ceiling you review monthly protects you from the $500 million story at your own scale, and it avoids the Kirorank problem, because nobody is climbing a leaderboard when the only person watching is you. If the ratio improves over three months, keep the setup. If it doesn't, cut the most expensive tool first and see whether anything you can measure gets worse.
Author
Lukas
@lukcombinator