Define the decision and the failures that matter

A useful benchmark measures a capability that matters to your decision. It measures it in conditions close enough to your use that the result changes what you do.

Write the decision

Write one sentence that names the workflow, the cost of failure and a behaviour you can observe. For example: 'Choose a support model that follows our refund policy while it uses tools.'

The NIST draft practices for automated benchmark evaluations start with three items. Define the evaluation objective. Define the construct to measure. Define the intended use of the result. Then select the benchmark.

Make two lists

• Must catch: failures that block deployment. Examples are unauthorized purchases, fabricated citations, broken code and unsafe tool calls.
• Nice to know: smaller differences, such as style or a small gain in latency.

Choose benchmarks for the first list before you spend budget on the second.

Write the claim and the test that can disprove it

Write the claim that you hope to support. Then write the result that would disprove it. If no result can disprove the claim, the claim is too vague. Make it narrower.

One benchmark is usually not enough

Pair a broad capability check with a task-specific evaluation. Add a small set of your own cases from the real workflow. The private cases catch the gap between a clean public task and the system that you plan to ship.

Sources

• AI Benchmark Index guide, How to choose an AI benchmark
• AI Benchmark Index guide, How to run an AI benchmark yourself for under $20
• NIST AI 800-2 (initial public draft), Practices for Automated Benchmark Evaluations of Language Models