Amazon Engineers Replaced Sonnet 4.6 With a Decision Model and Cut Ticket Labeling Cost 98%. The Post Gives No Accuracy Number. Here Is the Test I'd Run First.
A post on the Strands Agents blog, dated November 9, says two engineers on Amazon's internal ticketing system replaced Claude Sonnet 4.6 with a small open model called Strands Decider and cut the cost of labeling tickets from about $22,200 per million to about $500. That is the 98% in the headline. The post reached the Hacker News front page on November 10. If you pay a large model to pick one label from a fixed list, the numbers are worth an afternoon. They are also missing the one figure I would want before copying the idea.
What the post measured
The job is classification. Each ticket gets one of 20 category labels. Under the old setup, Sonnet 4.6 received a long system prompt, reasoned over the label set and returned JSON. The post's table gives 6,554 input tokens and 170 output tokens per ticket, at an average of 4,215 ms per call.
The replacement is the E4B build of Strands Decider, which sits on Google's Gemma 4. It reads 2,009 input tokens, emits a single output token and averages 897 ms. The team had to rewrite the labels as one or two sentence criteria, because the original label descriptions alone did not fit in the 4,096-token window they were working with.
The cost side uses list prices. Sonnet 4.6 on Bedrock at $3.00 per million input tokens and $15.00 per million output comes to $0.0222 per ticket, and I get the same figure from the token counts. Decider runs on a SageMaker ml.g5.xlarge at $1.408 per hour. At full utilization that works out to roughly $250 per million tickets, and the authors say it is about $500 in practice once you leave headroom for bursts. They call that 44 times cheaper.
What Strands Decider is
Decider is a different kind of model from a chat LLM, which is where the saving comes from. According to the model card and the repo, it is a LoRA adapter on a Gemma 4 base with a small scoring head. You give it a state and a question, and it scores the options in one forward pass without generating text. It supports yes or no questions, a choice among N options and a score on an ordered scale, and each answer comes with a confidence value.
The weights and code are Apache 2.0 and live on Hugging Face and GitHub. There are four Gemma 4 sizes, from E2B at about 9 GB of weights to a 26B mixture-of-experts build at about 51 GB. There is no hosted API and no per-token price. You run it yourself, and the model card says no inference provider serves it.
The number that is missing
The post gives no accuracy figure for either model. It has no precision or recall, no agreement rate between the two, and no test set size. It says Decider "matched Sonnet 4.6's performance" and that the E4B build was "beating production Sonnet" on a separate set of tickets, and leaves it there. It does not say how ground truth was established, what confidence threshold the team chose, or whether the public weights were used unchanged.
I would also keep in mind who published it. The Strands team makes the model, the blog is theirs, and two of the three bylines belong to the engineers who did the migration. That does not make the numbers false. It does mean the comparison was framed by people who wanted it to come out well, and several Hacker News commenters asked why a cheaper Claude tier or a classic classifier was not in the table.
The cost comparison also leaves out the Lambda, the queue, the engineering time and the period when both models ran side by side.
The break-even for a solo product
Amazon's saving depends on volume, and the post never states the volume. It says only that the change will save "millions of dollars" a year. From its own per-ticket figures, each million dollars saved corresponds to about 46 million tickets a year.
Now run it at my scale. One ml.g5.xlarge left on all month costs about $1,014 at that hourly rate. At Sonnet's $0.0222 per call, the GPU pays for itself at roughly 45,700 calls a month, or about 1,500 a day. Below that, the API is cheaper and you carry no pager.
The comparison gets harder against a small hosted model. Using the $0.10 and $0.50 per million list price for Claude Haiku 5.5 that I quoted in yesterday's post, the same 6,554 input and 170 output tokens cost about $0.0007 per call. The always-on GPU does not break even against that until about 1.4 million calls a month. For most solo products, routing the task to a cheaper hosted model captures most of the saving with a one-line change.
What I'd actually do
- Pick the one task in your product that is a closed choice: a label from a list, a yes or no, a routing decision. If the model has to explain itself or the list changes every week, stop here. The authors say the same.
- Pull 200 real examples and label them by hand. This is the step the Amazon post skips in public, and it is the only way you get the accuracy number they did not print.
- Try the cheapest hosted model first with a shortened prompt. The Amazon team cut input tokens by about 70% just by rewriting their labels as short criteria. Half the lesson of the post is prompt size.
- Only if you are above roughly 1,500 calls a day on an expensive model, or you already own a GPU, try Decider. The E4B card lists about 15 GB of weights, so a 24 GB card is the practical floor.
- Run both in parallel on live traffic and read every disagreement yourself. The post says this taught the team more than any offline eval, and it is the best idea in the piece.
- Sort your JSON keys. The team found that key order alone moved Decider's output probabilities by up to 0.19.
Where this could be wrong
My break-even assumes an always-on instance. If you batch classification into one hour a night, or you have a spare GPU under your desk, the fixed cost drops and Decider looks good at far lower volume. The latency gain is also real, and I gave it no weight. Going from about 4.2 seconds to under one matters if a person is waiting on the label.
I may also be too suspicious of the missing accuracy number. A team that ran both models on production traffic and then switched the old one off presumably liked what it saw. I would still rather measure my own 200 examples than take that on trust, because my labels are not their labels.
Author
Lukas
@lukcombinator