Skip to content

AI benchmark list

There are 264 source-checked AI benchmarks in this index.

Results

264 benchmarks

  • CodingOnline

    BigCodeBench

    Tests function-level code generation on software-engineering tasks requiring diverse library calls.

    Task count
    1,140

  • SafetyOnline

    BIPIA

    Evaluates model and defense robustness against indirect prompt injection across five application tasks.

    Task count
    5

  • ToolsOnline

    BIRD-SQL

    Tests text-to-SQL generation against large database contents spanning many professional domains.

    Task count
    12,751

  • MultimodalOnline

    BLINK

    Measures core visual perception abilities through multiple-choice tasks designed to resist verbal shortcuts.

    Task count
    3,807

  • SafetyOnline

    BOLD

    Measures social biases in open-ended English generation from prompts spanning profession, gender, race, religion, and politics.

    Task count
    23,679

  • ReasoningOnline

    BoolQ

    Tests yes-or-no question answering from a supporting passage using naturally occurring information-seeking questions.

    Task count
    15,942

  • BrowserOnline

    BrowserGym

    Provides a common environment for evaluating browser agents across several web-task benchmarks.

  • ReasoningOnline

    C-Eval

    Evaluates Chinese foundation models with multiple-choice questions spanning 52 disciplines and four difficulty levels.

    Task count
    13,948

  • Long-horizonOnline

    CALVIN

    Measures long-horizon, language-conditioned robot manipulation through sequences of continuous-control tasks.

  • MultimodalOnline

    ChartQA

    Measures visual and logical reasoning over charts with human-authored and generated questions.

  • AgentsOnline

    Cicero Diplomacy

    Evaluates strategic planning, negotiation, and natural-language coordination by agents playing the board game Diplomacy.

  • MultimodalOnline

    CLIPScore

    Measures image-caption compatibility without requiring reference captions by comparing CLIP embeddings.

  • MultimodalOnline

    ClockBench

    Evaluates whether vision-language models can read analog clocks across varied faces and times.

    Task count
    720

  • ReasoningOnline

    CMMLU

    Evaluates Chinese language and cultural knowledge across 67 multiple-choice subjects.

  • CodingOnline

    CodeContests

    Tests competitive-programming solution generation with problems, descriptions, tests, and human submissions.

    Task count
    13,328

  • CodingOnline

    CodeXGLUE

    Tests code understanding and generation across ten tasks including search, repair, translation, and summarization.

    Task count
    10

  • MultimodalOnline

    CogVideo

    Evaluates text-to-video generation; the repository provides model code and references for the original CogVideo and later CogVideoX systems.

  • AgentsOnline

    Crafter

    Measures reinforcement-learning agents across a survival game designed to expose a broad spectrum of capabilities.

  • AgentsOnline

    Crafter

    Evaluates reinforcement-learning agents on survival, exploration, resource collection, and technology-tree achievements.

    Task count
    22

  • AgentsOnline

    CRMArena

    Measures agents on professional customer-relationship-management tasks in realistic business scenarios.

  • CodingOnline

    CrossCodeEval

    Tests cross-file code completion and context retrieval in Python, Java, TypeScript, and C# repositories.

  • SafetyOnline

    CrowS-Pairs

    Measures stereotypical preferences in masked language models using paired sentences across nine types of social bias.

    Task count
    1,508

  • RetrievalOnline

    CRUD-RAG

    Evaluates retrieval-augmented generation across create, read, update, and delete application scenarios.

    Task count
    4

  • CodingOnline

    CRUXEval

    Tests code reasoning by asking models to predict inputs or outputs for short Python functions.

    Task count
    800

Categories in Online

Questions

What kinds of benchmarks are in Online?

Categories in Online: Multimodal (51 benchmarks), Reasoning (50 benchmarks), Agents (31 benchmarks), Safety (25 benchmarks), and Science (24 benchmarks).

How many AI benchmarks are in the index?

AI Benchmark Index currently lists 264 benchmarks that link to a primary source.

Can I submit an AI benchmark?

Yes. Submit a public primary source, licence, release date, category, and a one-sentence description of what it measures.

Published by . How we check each benchmark.