How to read an AI benchmark without being misled

A car's fuel figure comes from a set test on a set route. It is useful for comparing cars, but your own mileage depends on how and where you drive. AI benchmarks work in a similar way.

Model announcements often lead with benchmark scores. Those numbers can help, but they can also mislead a buyer who reads them as a guarantee. This piece explains what a benchmark measures, where it falls short, and what to ask before relying on one.

What a benchmark is

A benchmark is a fixed set of tasks with known answers. A model attempts the tasks, and its answers are scored. The result is usually a single number or a percentage.

Some benchmarks test knowledge, such as answering exam-style questions. Others test reasoning, coding, maths, following instructions or reading long documents. A few measure speed or cost rather than quality.

Because the tasks are fixed, everyone can run the same test. That is the main value: it allows a like-for-like comparison under stated conditions.

What a score can tell you

Used carefully, a benchmark gives a few useful signals.

A rough ranking on a narrow skill. If one model scores much higher on a coding test, it is probably stronger at that kind of coding.

Progress over time. Scores on the same test, across versions, show whether a model family is improving at that task.

A shortlist. Scores can help narrow many options to a few worth testing yourself.

What a score cannot tell you

The limits matter more for a buyer than the strengths.

Your task is not the test. A model that does well on exam questions may still struggle with your documents, your language or your format.

Small gaps may be noise. A difference of a point or two can come from how the test was run, not from a real difference in ability.

Tests can leak. If test questions appeared in a model's training data, its score may reflect memory rather than skill. This is known as contamination.

Settings change results. The wording of the prompt, the number of attempts allowed and how answers are graded can all move a score.

Quality is not cost or speed. A high score says nothing about price per token or how long users wait. We look at those trade-offs in AI inference cost: why one task gets different prices.

Who ran the test, and how

A score is only as reliable as the way it was produced. Results reported by the model's own maker may be accurate, but they are chosen and run by an interested party.

Independent results are useful as a check. Even then, look for a clear description of the method. Reputable results state the test version, the prompt format, the number of attempts and the date.

The date matters because models are updated. A score from an earlier version may not describe the model you can buy today. We make the same point about prices in A price without a date is a rumour: why we publish dated prices.

Questions to ask before you trust a score

When a benchmark result is put in front of you, it helps to ask:

1. What exactly does this test measure, and how close is it to my task?
2. Who ran it, and is the method published?
3. Which model version and settings were used, and on what date?
4. Were all models compared under the same conditions?
5. Is the gap between models large enough to matter, or within normal variation?
6. Is there any sign the test questions were in the training data?
7. What do cost and speed look like for the same models?

If the answers are unclear, treat the score as a hint rather than evidence.

Test on your own work

The most reliable benchmark is a small one built from your own tasks. Take a sample of real requests, decide what a good answer looks like, and run a few shortlisted models against it.

Score the results the same way each time. Record the cost and waiting time alongside quality. Repeat the test when models or prices change.

This takes some effort, but it answers the question public benchmarks cannot: how well does this model do the job you actually need? For a dated view of model prices and speeds to set beside your own results, see Model Watch.

What we are building

A model choice is only part of the decision. Where the model runs, and on what terms, shapes cost, speed and data handling too.

We're building standardized compute close to power, connectivity and demand, connected through one programmable network for AI inference. Workloads are designed to be placed by policy, on latency, data residency, model and load, and the network is designed so every workload can generate an auditable record of commitment, placement and delivery.

We would rather show a record of what happened than ask anyone to trust a headline number.

If you are choosing models for an inference workload, use Request capacity below.