# Ceilings, uncertainty and rank claims

Lesson 1 of [Audit AI benchmark claims before you buy](https://aibenchmarkindex.com/academy/audit-ai-benchmark-claims.md), 18 minutes.

A vendor claim often says 'first on benchmark X'. Your job is to find out if the benchmark can support that ranking.

## Five signals of saturation

1. **Top scores cluster.** Compare the gap between neighbouring systems with the run-to-run variation or the reported interval. If small changes reorder the table, the table shows ties, not a ranking.
2. **Easy items dominate.** Per-task results can show that nearly every model passes the same large block of tasks. A small tail then decides the rank. The nominal task count overstates the real evidence.
3. **The benchmark is old and public.** Long exposure gives task formats time to appear in training data, tutorials and fine-tuning sets.
4. **Prompt choices move the rank.** Change the format, the examples, the answer extraction or the seed. If a small, reasonable change moves the winner, the benchmark supports a sensitivity report, not one ranking.
5. **Real failures remain common.** If systems pass the public tasks but still fail your workflow, the construct is too narrow or the tasks are too easy.

## Which population does the score describe?

NIST AI 800-3 warns against collapsing different performance quantities and sources of uncertainty into one unexplained number. Ask what the score describes: these exact tasks, a larger population of tasks, or future real work. A benchmark can support the first claim and still give weak evidence for the third.

## What to ask the vendor

- The scores of comparable systems under the same settings.
- Per-task results, not only the mean.
- Intervals or repeated runs with different seeds.
- The prompt variants they tried, and the results of each.

A reproduced 92 from a deterministic runner can be more useful for a decision than an unverified 99 from a different prompt with unpublished settings.


## Sources

- AI Benchmark Index guide, What a saturated benchmark is and how to spot one
- NIST AI 800-3, Expanding the AI Evaluation Toolbox with Statistical Models

Also on the [web page](https://aibenchmarkindex.com/academy/audit-ai-benchmark-claims/ceilings-and-uncertainty).