A 9B Coding Model That Runs on a Phone Just Scored 70.6 on SWE-Bench Verified. The License Is MIT.
Ornith released Ornith-1.5 on August 19 in three sizes: a 397B mixture-of-experts model, a 35B MoE, and a 9B dense model. The headline number is the flagship scoring 86.1 on Terminal-Bench 2.1 against Claude Opus 4.8's 85.0. That is the number every aggregator picked up, and it is the least interesting thing in the release.
The 9B is the story. It scores 70.6 on SWE-bench Verified and 46.2 on Terminal-Bench 2.1. It ships with a quantized variant the team says deploys on iPhone and Android. And the weights are MIT.
What 70.6 on a 9B actually means
Put the 9B next to models several times its size. Gemma-4-31B scores 52.0 on SWE-bench Verified. Qwen3.5-9B, the closest size comparison, scores 53.2. Qwen3.6-35B-A3B scores 73.4, which is still ahead, but that is a 35B MoE against a 9B dense model.
On Terminal-Bench 2.1 the gap narrows differently: the 9B gets 46.2 with the Terminus-2 harness, against Gemma-4-31B's 42.1. A model you can hold in your hand is now beating a model that needs a workstation.
I want to be precise about what this does and does not mean. SWE-bench Verified measures whether a model can produce a patch that passes tests on real GitHub issues. It does not measure whether the patch is good, whether the model understood the codebase, or whether you would merge it without reading it. A 70.6 means roughly seven in ten of those specific tasks got a passing patch. That is useful. It is not "this model can maintain your repo."
The license is the part nobody leads with
Ornith-1.5 is on Hugging Face under MIT. Not a custom community license with a monthly-active-user threshold. Not an Apache variant with a use restriction attached. MIT.
For most people reading a model announcement, license text is the paragraph you skip. If you are building something you intend to charge for, it is the only paragraph that determines whether you can ship at all. I have walked away from two otherwise-good open-weights models in the last year over exactly this, because the license had a revenue clause I would have had to re-read with a lawyer before invoicing anyone.
MIT means you embed the weights, ship the app, sell it, and owe nobody a conversation. Combined with a build that runs on a phone, that is a genuinely different product surface: an app that does real code work with no API key, no per-token cost, and no network dependency at all.
Where the flagship actually lands
The 397B is competitive rather than dominant, and the release page is honest enough to show it. Terminal-Bench 2.1 (Terminus-2): Ornith-1.5 at 86.1, Opus 4.8 at 85.0, Kimi K3 at 88.3. DeepSWE: Ornith at 56.0, Opus 4.8 at 59.0, Kimi K3 at 67.5. Frontier-Bench v0.1: Ornith at 13.5, Opus 4.8 at 21.1, Kimi K3 at 23.0.
So the flagship beats Opus on some agentic coding rows, loses on others, and trails Kimi K3 on most of the hard ones. "Matches Claude Opus 4.8" is defensible for two specific benchmarks and misleading as a summary.
What "self-improvement" is and is not
The paper's actual contribution is the training loop. Ornith-1.5 generates its own training tasks, builds a scaffold for each one, produces solution rollouts, and optimizes all three stages with GRPO against a reward that multiplies validity, frontier difficulty, and novelty. Difficulty targets a 20% success rate, so as the model gets better at a task the reward for generating that task drops and the generator is pushed toward harder problems.
That is a real engineering result and it is genuinely interesting. It is also entirely a training-time property. The model you download does not improve itself on your machine. It is a static set of weights like any other. If you saw "self-improving LLM" in a headline and pictured something that gets better as you use it, that is not what shipped.
What I would actually do
If you have a product idea that needs code understanding on-device, this is the first release I would spend a weekend on rather than bookmark. Pull the 9B, run it against your own repo on ten issues you already know the answers to, and see whether the patches are reviewable. Ten real issues from your codebase will tell you more than the entire benchmark table, because benchmark suites are curated and your repo is not.
If you are choosing a hosted frontier model for an agent, this release does not change your decision. The 397B is not meaningfully better than what you are already paying for, and you would be trading a managed endpoint for infrastructure work.
And treat every number here as provisional. They are self-reported by the lab that trained the model, averaged over five runs, with harness configurations documented in a footnote. That is more transparency than most releases offer, which is worth acknowledging. It is still the vendor grading its own exam. The independent reproductions will land within a couple of weeks, and if the 9B holds anywhere near 70.6 in someone else's harness, that is when I would build on it.
Author
Lukas
@lukcombinator