Skip to content

AI benchmark list

There are 264 source-checked AI benchmarks in this index.

Results

264 benchmarks

  • ToolsOnline

    Spider

    Tests cross-domain text-to-SQL generalization to unseen database schemas and complex queries.

    Task count
    10,181

  • AgentsOnline

    Spider 2.0

    Tests agents on realistic enterprise text-to-SQL workflows with long contexts and cloud or local databases.

    Task count
    632

  • SafetyOnline

    StereoSet

    Measures stereotypical bias and language-modeling ability in sentence-level and discourse-level contexts.

  • SafetyOnline

    StrongREJECT

    Evaluates jailbreak responses with a forbidden-prompt dataset and an automated judge designed to penalize empty refusals and ineffective attacks.

    Task count
    313

  • VoiceOnline

    SUPERB

    Evaluates general-purpose speech representations across recognition, speaker, semantic, and paralinguistic downstream tasks.

    Task count
    10

  • CodingOnline

    SWE-bench

    Tests whether coding systems can resolve real GitHub issues in complete software repositories.

    Task count
    2,294

  • CodingOnline

    SWE-bench Verified

    Tests repository-level issue resolution on a human-validated subset of 500 solvable SWE-bench tasks.

    Task count
    500

  • MultimodalOnline

    TasteBench

    Studies whether aesthetic-preference benchmarks reward expert-aligned craft judgment rather than shortcuts; its item pool remains pending licence clearance.

  • ReasoningOnline

    TAT-QA

    Evaluates numerical question answering over combined tables and prose from financial reports.

    Task count
    16,552

  • ToolsOnline

    tau-bench

    Measures customer-service agents through policy-constrained conversations and database tool use.

  • ToolsOnline

    tau2-bench

    Measures conversational agents when both the user and agent can change shared task state.

  • SafetyOnline

    ToxiGen

    Tests hate-speech classifiers on machine-generated implicit toxic and benign statements about 13 minority groups.

  • ReasoningOnline

    TriviaQA

    Tests reading comprehension and open-domain question answering with trivia questions paired to independently gathered evidence.

  • ReasoningOnline

    TruthfulQA

    Tests whether language models reproduce common human misconceptions and falsehoods when answering questions.

    Task count
    817

  • ReasoningOnline

    TyDi QA

    Evaluates information-seeking question answering across 11 typologically diverse languages.

    Task count
    204,000

  • ScienceOnline

    Vendi Score

    Measures diversity in a collection from pairwise similarities using the effective rank of a similarity matrix.

  • MultimodalOnline

    Video-Bench

    Evaluates video language models on perception, prior-knowledge use, and video-based decision making.

  • MultimodalOnline

    Video-MME

    Evaluates multimodal models on short, medium, and long videos across visual domains, durations, and subtitle settings.

    Task count
    2,700

  • MultimodalOnline

    Video-MMMU

    Measures whether multimodal models acquire and apply new knowledge from professional lecture videos.

    Task count
    900

  • AgentsOnline

    VirtualHome

    Evaluates generation and execution of household activity programs in simulated 3D homes.

  • BrowserOnline

    VisualWebArena

    Tests multimodal web agents on visual tasks in realistic, reproducible browser environments.

    Task count
    910

  • MultimodalOnline

    We-Math

    Tests whether multimodal models solve visual math problems through coherent one-step and multi-step reasoning.

    Task count
    6,500

  • BrowserOnline

    WebArena

    Tests autonomous web agents on realistic tasks in reproducible websites with execution-based evaluation.

    Task count
    812

  • CommerceOnline

    WebShop

    Measures agents that search, customize, and purchase products in a simulated e-commerce website from natural-language instructions.

    Task count
    12,087

Categories in Online

Questions

What kinds of benchmarks are in Online?

Categories in Online: Multimodal (51 benchmarks), Reasoning (50 benchmarks), Agents (31 benchmarks), Safety (25 benchmarks), and Science (24 benchmarks).

How many AI benchmarks are in the index?

AI Benchmark Index currently lists 264 benchmarks that link to a primary source.

Can I submit an AI benchmark?

Yes. Submit a public primary source, licence, release date, category, and a one-sentence description of what it measures.

Published by . How we check each benchmark.