Every claim about a model’s capability rests on a measurement, and measurements can be gamed. As artificial intelligence has become commercially important, the business of measuring it has grown from academic exercise into a competitive industry with its own incentives, and its own failure modes.
The result is a field under pressure from three directions at once. Companies want favourable numbers. Regulators want numbers they can trust. Researchers want numbers that mean something. Those goals overlap less than the vocabulary suggests.
Why benchmarks decay
A benchmark is a fixed set of questions with known answers. Once it is published, it becomes a target. Training data scraped from the web may include the questions, and even without deliberate contamination, repeated exposure during development shifts results upward. A benchmark that was informative in one year can be saturated the next.
This is a normal lifecycle rather than misconduct. The difficulty is that decisions are still made on stale instruments, because replacing a familiar benchmark requires agreement on a successor and a period of uncertainty in between.
Contamination and how to detect it
Detecting contamination is possible but not easy. Techniques include comparing performance on paraphrased questions, examining whether a model can reproduce distinctive strings from a test set, and holding back a portion of the data entirely. Each has limits, and none is conclusive on its own.
Organisations that publish methodology, such as the team behind the AI Index at Stanford, have argued for transparency about how data was collected and whether overlap with training corpora was checked. Without that disclosure, a score is difficult to interpret.
The move toward private and dynamic sets
One response is to keep the test set private and rotate it. Another is to generate questions dynamically so that memorisation is useless. Both make comparison harder, because a private test cannot be re-run by an independent party.
The compromise that has emerged is a public leaderboard fed by a private set, with periodic refresh and published confidence intervals. It is better than a static public set, and it still depends on trusting the organisation holding the data.
Evaluation beyond accuracy
Accuracy is the easiest thing to measure and often the least important. A system that is accurate but slow, expensive, unstable or unsafe is not useful in production. Serious evaluation therefore includes latency, cost per request, refusal behaviour, susceptibility to prompt injection and performance on inputs that differ from the training distribution.
Safety evaluation in particular has matured, with dedicated groups probing for harmful outputs, jailbreaks and unintended tool use. The NIST AI risk framework provides a vocabulary that has been widely adopted for organising this work.
Human evaluation and its costs
Many of the properties that matter most, such as whether an answer is helpful, are judged by people. Human evaluation is expensive, slow and vulnerable to inconsistency between raters. Good practice involves written rubrics, multiple raters per item and measurement of agreement.
It is also where the largest disagreements appear. Two competent reviewers can reach different conclusions about the same response, and the aggregate depends on who was hired and what instructions they received. Publishing the rubric matters as much as publishing the score.
Benchmarks for agents and multi-step tasks
Single response benchmarks are a poor fit for systems that take sequences of actions. Evaluating an agent means judging the whole trajectory, including whether it used tools appropriately, whether it stopped when it should have and whether it achieved the goal without damaging side effects.
Designing these evaluations is harder than it looks. The environment must be simulated so that mistakes are safe, the success criteria must be specified precisely, and the results must be reproducible across runs, which conflicts with the randomness inherent in many systems.
Several groups are working on standardised environments for exactly this purpose, and the early results are sobering. Performance that looks strong on a single turn often degrades sharply over a longer horizon, which is the kind of finding a static benchmark cannot produce, and it is the reason agent evaluation has become an active research area rather than a solved one.
What to ask when you see a number
Ask which version of the model was tested, on which data, with what prompt format, at what temperature, and by whom. Ask whether the test set was public and whether contamination was checked. Ask for a confidence interval, and be sceptical if none is offered.
Then ask the practical question: what does this measure predict about the task I actually have? A benchmark is a proxy, and proxies fail in specific ways. The teams that get useful answers are the ones that build their own small, well documented evaluation for their own workload, and treat published benchmarks as context rather than proof. That evaluation should be versioned alongside the model, because a test that is not maintained stops measuring anything useful within a few months.
Image: Photo.iep · CC BY-SA 3.0 · via Wikimedia Commons.