· 9 min read

Innodata Tested 10 AI Models Against Real CVEs. The Best One Patched 23% — Even With the Bug Circled.

AI Lessons

Innodata Tested 10 AI Models Against Real CVEs. The Best One Patched 23% — Even With the Bug Circled.

Innodata tested ten leading AI models and agents against real, verified CVEs and asked them to fix the bug. Not find it, not explain it — fix it, with a live exploit sitting in a sandbox ready to prove whether the patch actually worked. The best open-weight model repaired at most 23% of the flaws, and that's the generous number: the one where the model was told exactly where the vulnerability lived. I write and ship AI-generated code every week with nobody else reading the diff, and this is the first hard number I've seen that puts a percentage on the thing I already suspected.

What Innodata actually built

On August 4, 2026, Innodata (Nasdaq: INOD) released the first stage of what it's calling the AI Cyber Training Suite — twelve datasets and evaluation systems built from thousands of real security flaws pulled from the last decade of CVE history. The languages covered are Python, TypeScript, JavaScript, Rust, C, and Go, and the platforms span Linux, macOS, Android, Windows, AWS, and GCP. This isn't a synthetic benchmark somebody generated with an LLM and called a dataset. According to the company's press release, Innodata's cybersecurity engineers hand-rebuilt each flaw, attack included, inside a sealed offline copy of the vulnerable software.

That last detail is the part that makes the benchmark worth trusting. A "fix" only counts if the original exploit fails against the patched code and the software still does its job. That's a meaningfully higher bar than most AI coding benchmarks, which tend to check whether output compiles or whether a unit test passes. Here, an attacker script tries to actually break in after the model is done, and if it gets in, the patch failed regardless of how clean the diff looked.

The number that should worry you

Ten leading AI models and agents were tested on their ability to repair these verified flaws. The best open-weight model topped out at 23% — and that's with the vulnerability location handed to it. No hunting required, no ambiguity about which function or which file. Just: here's the bug, fix it. It still failed on more than three out of every four.

I want to sit with that for a second, because it cuts against how coding agents get marketed. The pitch is usually "point it at the problem and it solves the problem." Innodata's test is close to the best-case version of that pitch — full context, exact location, single well-defined task — and the result is still a coin flip stacked against you. If that's the ceiling on a security fix with the answer key in hand, imagine the number for a vulnerability nobody flagged, buried three files deep in a refactor your agent did unsupervised at 11pm.

Fine-tuning helped a lot — and that's not something you get for free

The more interesting number in the release is what happened after Innodata fine-tuned an open-weight model on a slice of its own suite data: the model's ability to patch vulnerabilities without any guidance at all — no location hint — rose from 18.4% to 41.2%. More than doubled, in one round of fine-tuning.

That's a genuinely encouraging result for model builders, and it tells you where this is headed: security-specific training data measurably improves security-specific performance, which sounds obvious until you remember that almost no foundation model has been trained on this kind of adversarially-verified data at scale. But notice what that improvement required — curated CVE data, a fine-tuning run, and an evaluation harness with a live exploit sandbox. That's Innodata's job, not yours. You are not going to fine-tune Claude or GPT-5 or Llama on a private vulnerability dataset before your Tuesday deploy. The 41.2% number is a preview of where frontier labs might take base model security over the next year or two. The 23% number, or something like it, is what you're working with today, out of the box, on whatever model your coding agent defaults to.

Why this lands harder on solo operators than on enterprises

A company with a security team treats AI-generated code as input to a review process: SAST scanning, a pen test cadence, a person whose whole job is finding exactly this kind of hole before it ships. Solo operators don't have that. If you're running Claude Code, Cursor, or Codex to ship features on your own SaaS, the agent's output goes straight from "looks right to me" to production, and "looks right to me" is doing a lot of unverified work.

Jack Abuhoff, Innodata's CEO, put it plainly in the release: without a professional security team reviewing everything an agent produces, businesses can't fully depend on AI-generated code — and he's framing that as a blocker to enterprise adoption. Flip that around for a one-person shop and it's worse, not better. You don't have the fallback he's describing. You are the security team, and until now you didn't have a number telling you how much that team is actually catching.

What I'd actually do

Here's my actual response to this, not a hand-wave about "reviewing your code more carefully."

First, treat any AI-written patch to a security-relevant path — auth, session handling, file uploads, anything touching user input or a database query — as a draft, not a fix, until you've independently confirmed the exploit it was supposed to close. If you reported a bug via a scanner or a bug bounty note, keep the original repro steps and rerun them against the patch yourself. Don't trust the agent's self-report that the issue is resolved; ask it to write the exploit test and watch it fail before the patch and pass after.

Second, run a dependency and static analysis scanner as a hard gate, not a suggestion. Semgrep's free tier, Trivy, or GitHub's built-in CodeQL scanning catch a different class of bug than a manual review does, and they cost you nothing but CI minutes. Wire one into your pipeline this week if you haven't.

Third, be specific with your agent prompts on security-sensitive changes — ask it to enumerate the attack surface it's touching before it writes code, not just to "fix the bug." Innodata's own finding is that giving the model the exact location roughly doubles its success rate versus finding it blind; giving it the threat model probably helps the same way, even if nobody's benchmarked that yet.

The honest counter-take here: this benchmark measures models patching known, previously-catalogued CVEs in isolated sandboxes, which is a narrower and arguably harder task than the everyday work of an agent avoiding a fresh mistake while writing new code from scratch. A 23% repair rate on hand-picked historical exploits doesn't necessarily mean your agent introduces new vulnerabilities 77% of the time — those are different tasks measuring different failure modes, and Innodata's own suite includes prevention data alongside repair data that this release doesn't fully break out. I'd also flag that Innodata sells cybersecurity data and evaluation services, so they have a commercial interest in a headline number that makes the problem look large. That doesn't make the finding wrong — the methodology (live exploit, sealed sandbox, ten independently tested models) is rigorous and the press release is specific about how the test worked — but it's a reason to treat "23%" as a floor worth taking seriously rather than a universal law of AI-written code.

Either way, the number that matters for a solo operator isn't the exact percentage. It's that the percentage isn't 100, isn't close to 100, and nobody but you is checking.

Author

Lukas

Sources

More AI articles →

Stay in the Loop

Get new posts delivered to your inbox. No spam, unsubscribe anytime.

Newsletter coming soon. Set PUBLIC_CONVERTKIT_FORM_ID in .env to activate.

Related Posts