Every model launch arrives with a bar chart. The new model is taller than the old model on four benchmarks, sometimes by two or three percentage points, and the chart is presented as though those points settle something.
They rarely do. Benchmarks are genuinely useful — they are the only broadly comparable evidence we have — but reading them naively produces worse decisions than ignoring them, because a number feels like proof in a way that a vague impression does not.
Here is how to read them properly: what the main ones actually measure, the four ways they mislead, and how to weight them against your own testing.
What the major benchmarks measure
Four appear in most comparisons. They measure genuinely different things, and averaging them without understanding that is how you get a meaningless composite.
LiveCodeBench — coding. Programming problems drawn from recent competitive-programming sources, refreshed over time specifically to reduce the chance that problems appeared in training data. It measures algorithmic problem-solving in isolated, well-specified tasks. It does not measure working in a large existing codebase, understanding unfamiliar conventions, or debugging someone else’s mess — which is most professional programming.
AIME — mathematics. Competition maths at pre-university level. Multi-step symbolic reasoning with exact answers. Correlates reasonably with structured logical reasoning generally. Does not tell you much about applied numerical work or handling ambiguous quantitative questions.
GPQA Diamond — graduate-level reasoning. Science questions written to be resistant to search: a skilled non-expert with a web browser still scores poorly. This is the closest thing to a genuine reasoning test in common use, and it is correspondingly the hardest to game.
LMArena Elo — human preference. Blind pairwise comparisons where people pick the better of two responses. It measures what people prefer, which is not the same as what is correct. Response formatting, length, tone, and confidence all move this score. A model that hedges appropriately can lose to one that asserts confidently and wrongly.
Understanding this last distinction matters enormously. Three of these measure capability; one measures likeability. They are all worth knowing and they should not be weighted equally for every decision.
The four ways benchmarks mislead
1. Contamination
Benchmark questions leak into training data. When a model has effectively seen the test, its score reflects recall rather than reasoning. Benchmark maintainers work hard against this — refreshing question sets, holding out private splits — but the problem is structural and never fully solved.
Signal to watch: a model that scores dramatically better on an older, widely-published benchmark than on a newer or private one. That divergence is a contamination fingerprint.
2. Selective reporting
Vendors publish the benchmarks they win. This is not fraud, it is marketing, and it is universal. The absence of a benchmark from a launch post is information.
Signal to watch: a launch announcement covering three benchmarks when the previous one covered six. Ask what happened to the other three.
3. Configuration differences
The same model can score very differently depending on prompting strategy, sampling temperature, number of attempts allowed, and whether tools were available. A score obtained with elaborate scaffolding and multiple attempts is not comparable to a single zero-shot pass — but both get published as a percentage with no asterisk.
Signal to watch: any comparison that does not state the evaluation conditions. If the methodology is not published, the number is a claim rather than a measurement.
4. Staleness
Model releases now outpace independent re-evaluation. A leaderboard that has not been rebuilt in six months is a historical document. This is the most common failure in practice, because outdated leaderboards keep circulating and keep getting cited.
Signal to watch: the date. If a ranking does not carry one, do not use it.
What a good ranking looks like
A ranking worth reading states four things:
- Which benchmarks feed it, by name
- How they are combined — the actual formula, including normalisation
- When it was last updated
- Which models are missing scores, and why
Composite indices are useful precisely because they resist single-benchmark gaming: a model tuned to excel at competition maths does not automatically rise if reasoning and human preference are weighted alongside it. But a composite is only as trustworthy as its disclosure. A composite AI model leaderboard that publishes its formula — averaging normalised coding, maths, reasoning, and human-preference scores, and stating how many of the four each model was scored on — can be checked. A bare “intelligence score” cannot.
Treat even a well-documented composite as a snapshot with a date attached, not a standing verdict. Read the date first, the formula second, and the ordering third.
The gap between benchmarks and your work
Here is the uncomfortable truth about all of this: benchmark performance and usefulness for your specific task correlate, but loosely.
A model can lead on coding benchmarks and be frustrating in a real codebase because it does not follow your conventions or handle your framework version. A model can trail on maths and be excellent at your work because your work is not competition maths. A model can win on human preference and irritate your team because the style it optimises for is not the style you want.
The strongest predictor of whether a model suits you is not a benchmark. It is a test on your own material, and it takes about twenty minutes.
Build a small evaluation set. Ten to fifteen tasks drawn from real work you have already done, where you know what good output looks like. Include your hardest recurring case and your most ambiguous one.
Run it across candidate models unchanged. Same prompt, same inputs, no per-model tuning — you are comparing models, not your prompt-writing.
Score on what you actually need. Accuracy, format compliance, whether it declines when it should, how it handles missing information.
Re-run quarterly. Models update, and your set is now a regression test for the tools you already depend on, which is arguably more valuable than the original comparison.
Running the same set across several models used to mean several subscriptions, which is why most teams never did it. Multi-model platforms such as Perspective AI collapse that into one account with access to models from multiple labs, which makes a fifteen-task comparison an afternoon rather than a procurement exercise.
How to weight benchmarks by decision type
Not every decision needs the same evidence.
| Decision | Weight benchmarks | Weight own testing |
|---|---|---|
| Which model family to explore first | High | Low |
| Which model for a specific production workflow | Low | High |
| Whether to switch after a new release | Medium | High |
| Explaining a choice to a stakeholder | High | Medium |
Benchmarks are a good filter and a poor decider. Use them to narrow eighteen candidates to three. Use your own evaluation set to pick among the three.
Frequently asked questions
Are AI benchmarks reliable? Reliable for what they measure, under the conditions they were measured. Unreliable as a general statement of which model is “better”, because that depends entirely on the task.
What is benchmark contamination? Test questions appearing in a model’s training data, so the score reflects memorisation rather than capability. Suspect it when a model performs far better on old public benchmarks than on new or private ones.
Why do models score differently on different leaderboards? Different question sets, prompting strategies, sampling settings, attempt allowances, and dates. Two leaderboards disagreeing is normal and does not mean one is wrong.
Should I choose a model based on the leaderboard? Use it to shortlist. Choose on a test set built from your own work. The leaderboard tells you which models are worth twenty minutes; only your own testing tells you which one to use.
What does LMArena Elo actually measure? Blind human preference between two responses. It captures helpfulness and presentation quality well, and correctness poorly — a confident wrong answer frequently beats a hedged right one.
The habit worth building
When you next see a benchmark chart, ask four questions before reading the bars: what does this benchmark measure, when was it run, under what conditions, and what is missing from the chart.
Most published comparisons fail at least two of those. The ones that survive all four are worth your attention — and even then, they only earn you a shortlist. The decision still belongs to fifteen tasks from your own work.