Skip to content

AI benchmark list

There are 264 source-checked AI benchmarks in this index.

Results

264 benchmarks

  • MultimodalOnline

    COCO Caption

    Evaluates image-caption generation on everyday scenes using reference captions and standardized captioning metrics.

    Task count
    330,000

  • SafetyOnline

    Do-Not-Answer

    Evaluates whether language models decline risky instructions while still answering harmless requests.

    Task count
    939

  • MultimodalOnline

    DocVQA

    Measures question answering over document images where layout, text, and document structure matter.

    Task count
    50,000

  • AgentsOnline

    GAIA

    Measures general assistants on real-world questions requiring reasoning, tools, browsing, and multimodal inputs.

    Task count
    466

  • ReasoningOnline

    Gym Sokoban

    Evaluates planning and reinforcement learning in procedurally generated box-pushing puzzles.

  • AgentsOnline

    Habitat HomeRobot OVMM Challenge

    Measures open-vocabulary mobile manipulation systems that find objects, navigate, and rearrange them in simulated homes.

  • VoiceOnline

    HEAR

    Evaluates transferable audio representations across speech, music, and environmental sound tasks.

    Task count
    19

  • ReasoningOnline

    HELM Lite

    Evaluates language models on a compact suite of ten HELM scenarios selected for broad coverage and lower evaluation cost.

    Task count
    10

  • ReasoningOnline

    HumorBench

    Evaluates English and Spanish multi-turn humor understanding through joke classification and identification of each line’s role.

  • MultimodalOnline

    InfographicVQA

    Measures question answering that combines infographic text, layout, graphics, data visualizations, and arithmetic.

  • AgentsOnline

    Kaggle Game Arena

    Evaluates strategic planning, adaptation, and robustness through head-to-head model competitions in chess, poker, and Werewolf.

    Task count
    3

  • ReasoningOnline

    MMLU-Pro

    Tests knowledge and reasoning on harder ten-option questions with reduced reliance on recall.

    Task count
    12,032

  • ScienceOnline

    MultiMedQA

    Evaluates medical question answering across seven datasets spanning professional exams, research, and consumer health queries.

    Task count
    7

  • MemoryOnline

    Needle In A Haystack

    Tests whether a language model retrieves a planted fact from increasingly long contexts and at different insertion depths.

  • MultimodalOnline

    OK-VQA

    Measures visual question answering that requires knowledge beyond the content visible in an image.

  • ScienceOnline

    PubMedQA

    Evaluates biomedical research question answering from PubMed abstracts, including an expert-annotated test set.

    Task count
    1,000

  • ScienceOnline

    SciBench

    Evaluates college-level scientific reasoning, domain knowledge, and calculation on textbook problems.

    Task count
    692

  • ReasoningOnline

    SimpleBench

    Tests everyday reasoning about space, time, social situations, and adversarial wording with multiple-choice questions.

  • MultimodalOnline

    Something-Something V2

    Tests recognition of everyday actions whose labels depend on temporal and physical interactions between people and objects.

  • ReasoningOnline

    SQuAD 2.0

    Tests extractive reading comprehension and whether a system abstains when a Wikipedia passage does not answer the question.

  • MultimodalOnline

    SVGBench

    Tests whether language models can generate SVGs that satisfy requirements involving knowledge, code, instructions, and physical reasoning.

  • AgentsOnline

    Terminal-Bench

    Tests whether agents complete end-to-end tasks inside reproducible terminal environments.

    Task count
    100

  • AgentsOnline

    Terminal-Bench 2.0

    Tests agents on valuable containerized work such as debugging, security fixes, and scientific workflows.

  • MultimodalOnline

    TextVQA

    Measures visual question answering that requires reading and reasoning about text in natural images.

    Task count
    45,336

Categories in Online

Questions

What kinds of benchmarks are in Online?

Categories in Online: Multimodal (51 benchmarks), Reasoning (50 benchmarks), Agents (31 benchmarks), Safety (25 benchmarks), and Science (24 benchmarks).

How many AI benchmarks are in the index?

AI Benchmark Index currently lists 264 benchmarks that link to a primary source.

Can I submit an AI benchmark?

Yes. Submit a public primary source, licence, release date, category, and a one-sentence description of what it measures.

Published by . How we check each benchmark.