Audit AI benchmark claims before you buy
Check a vendor's benchmark claim like an evaluator: ceilings and uncertainty, contamination, grader and judge bias, score provenance and the licence of each asset.
Start the courseAbout this course
- Level
- Specialist
- Access
- Membership, $499.00 a year
- Time
- 1 h 15 min
- Lessons
- 4
- Pass mark
- 80%
- Exam attempts
- 3
- Certificate valid for
- 1 year
What you will learn
- Find the five signals of a saturated benchmark and decide when a rank claim is not supported.
- Explain how contamination can raise scores and which questions to ask about test freshness.
- Name the biases of LLM judges in the MT-Bench study and the controls that reduce them.
- Ask for complete run metadata and reproduce a claim on a sample.
- Record the licence of each benchmark asset with SPDX identifiers.
Lessons
Questions
Does this course make me a qualified auditor?
No. It is a certificate of completion from this site. It records that you passed this course's exam. It is not a licence, an accreditation or a professional qualification.
Does the course give legal advice on data licences?
No. It teaches you to read and record licence terms. When ownership of a valuable dataset is unclear, ask counsel or the publisher.
Does the course say which vendors make false claims?
No. It names no vendor. It teaches checks that you apply to any benchmark claim.