Artificial Analysis Built Most of Its Intelligence Index on Private Datasets. Build a Private Eval of Your Own for the Same Reason.
When Artificial Analysis announced version 5 of its Intelligence Index on October 27, its changelog said that for the first time the majority of the index is built on private datasets. A week later, on November 3, it shipped v5.0.1 with grading fixes. If you pick models by looking at that number, the number just became harder to audit. I think that is the right call for the benchmark and a reason for you to stop treating it as the decision.
What changed in v5
Based on the changelog and methodology pages, v5.0 was announced on October 27. The November 3 update incorporated grading robustness improvements from the authors of CritPt and regraded all models on CritPt v2. The index now lists nine evaluations, including Terminal-Bench 4.0, FrontierCode 2.0, HLE-Diamond, AA-Omniscience v1.1 and a private 40-question hard subset of CritPt v2. Three older pieces were removed: GDPval-AA v2, SciCode and AA-LCR.
On the homepage as I read it today, Gemini 4 Carbon at its High setting sits at 60 on the index. On November 5 the changelog also describes a model called Inkling 2 Small at 31 as the highest-scoring open-weights model outside China, pending its weights release. Those are snapshots and will move.
I tried to check a news report that the gap between open-weight and closed models had widened on a revised index. The article was blocked for me and I could not find the claim on Artificial Analysis's own pages, so I am not repeating the numbers.
Why private data is the right move and still a problem
Public test sets get memorized. Once questions leak into training data, scores stop measuring ability. Keeping most of the data private is the standard fix, and it is the same reason schools keep exam questions secret.
The cost is that you cannot reproduce the result, argue with a specific question, or tell whether the test resembles your work. A single blended score also hides that a model can be excellent at hard science questions and mediocre at the boring extraction task you actually run.
The private eval I'd build
You need a small set of your own tasks that the model has never seen. Mine would look like this:
- Collect 20 real tasks from your own work: support replies you wrote, a function you refactored, an invoice you needed parsed, a summary you needed from a messy thread.
- Write what a good answer looks like for each, in a sentence or two, before you run anything.
- Keep the set out of any public repository and out of any prompt you share.
- Run each candidate model on all 20 with the same prompt, and score pass, partial or fail by hand.
- Record cost and latency per task alongside the score.
Twenty tasks will not give you statistical certainty. They will tell you which of three shortlisted models fails the thing you care about, which a leaderboard cannot.
Where this breaks down
A private eval is small and subjective, and your own scoring can drift. If you are choosing for a task that really is general-purpose reasoning, the aggregate index may be a decent first filter. And the index maintainers have information you do not: far more questions and consistent grading. I would still use their number only to decide what goes on my shortlist.
What I'd actually do
Keep a folder called evals with your 20 tasks and your scoring notes, and re-run it whenever a model you use ships a new version or changes price. When Artificial Analysis publishes a new top score, check the index to find candidates, then check your folder to decide.
Author
Lukas
@lukcombinator