Skip to content

AI benchmark list

There are 264 source-checked AI benchmarks in this index.

Results

264 benchmarks

  • CommerceOnline

    WebShop

    Evaluates agents that search, customize, and purchase products in a simulated e-commerce site from natural-language instructions.

    Task count
    12,087

  • RetrievalOnline

    WebWalkerQA

    Measures long-horizon information seeking through multi-hop traversal of structured websites.

  • ReasoningOnline

    WikiText

    Evaluates language modeling on more than 100 million tokens drawn from verified Wikipedia articles.

  • ReasoningOnline

    WildBench

    Tests chat models on challenging real-user requests using task-specific checklists and model-based judges.

    Task count
    1,024

  • SafetyOnline

    WinoBias

    Measures gender bias in coreference resolution by changing pronoun gender while preserving sentence structure.

  • ReasoningOnline

    WinoGrande

    Tests commonsense reasoning by resolving pronouns in adversarially collected sentence pairs.

    Task count
    44,000

  • BrowserOnline

    WorkArena

    Measures whether browser agents can complete knowledge-work tasks in a ServiceNow environment.

    Task count
    33

  • ReasoningOnline

    XNLI

    Evaluates cross-lingual natural-language inference and language transfer across 15 languages.

    Task count
    112,500

  • ReasoningOnline

    XQuAD

    Evaluates cross-lingual extractive question answering on the same question-answer pairs translated across 11 languages.

    Task count
    1,190

  • SafetyOnline

    XSTest

    Tests whether safety-tuned language models refuse harmless prompts that superficially resemble unsafe requests.

    Task count
    450

  • MemoryOnline

    ZeroSCROLLS

    Evaluates zero-shot understanding of long texts across summarization, question answering, aggregation, and reordering tasks.

    Task count
    10

  • MultimodalOnline

    ActivityNet-QA

    Tests question answering over long videos with questions that require spatial and temporal reasoning.

    Task count
    58,000

  • ScienceOnline

    AI2 Reasoning Challenge

    Tests grade-school science reasoning with natural multiple-choice questions split into easy and challenge sets.

    Task count
    7,787

  • MultimodalOnline

    AI2D

    Measures parsing and question answering over grade-school science diagrams with annotated constituents and relationships.

  • AgentsOnline

    ARC-AGI-3

    Tests exploration, goal discovery, world-model building, and planning in novel interactive environments.

  • MultimodalOnline

    AVA

    Evaluates spatiotemporally localized recognition of 80 atomic actions in densely annotated movie clips.

    Task count
    430

  • AgentsOnline

    BabyAI

    Measures grounded language understanding by training agents to follow generated instructions in grid worlds.

  • CommerceOnline

    Benchmark Apology Pizza

    Tests whether an agent can correct a wrong pizza order without exceeding a refund limit.

  • SafetyOnline

    Benchmark MCP Safety

    Tests resistance to malicious tool descriptions, tool shadowing, and excess privilege.

  • SafetyOnline

    Benchmark Sycophancy

    Tests whether a model resists endorsing a user's increasingly harmful mistaken plan.

  • CodingOnline

    Benchmark Test Weakening

    Tests whether a coding agent makes tests pass by weakening the tests instead of fixing the code.

  • MultimodalOnline

    CIFAR-10

    Evaluates image classification on 60,000 small color images evenly divided among ten classes.

    Task count
    60,000

Categories in Online

Questions

What kinds of benchmarks are in Online?

Categories in Online: Multimodal (51 benchmarks), Reasoning (50 benchmarks), Agents (31 benchmarks), Safety (25 benchmarks), and Science (24 benchmarks).

How many AI benchmarks are in the index?

AI Benchmark Index currently lists 264 benchmarks that link to a primary source.

Can I submit an AI benchmark?

Yes. Submit a public primary source, licence, release date, category, and a one-sentence description of what it measures.

Published by . How we check each benchmark.