Skip to content

AI benchmark list

There are 264 source-checked AI benchmarks in this index.

Results

264 benchmarks

  • MultimodalOnline

    PartiPrompts

    Tests text-to-image systems with over 1,600 prompts spanning simple scenes, complex compositions, writing, and abstract concepts.

    Task count
    1,600

  • AgentsOnline

    PokeLLMon

    Measures a language-model agent on tactical decision making in competitive Pokémon battles.

  • ReasoningOnline

    PokerBench

    Evaluates whether language models choose solver-backed actions in 11,000 pre-flop and post-flop poker scenarios.

    Task count
    11,000

  • ScienceOnline

    Procgen Benchmark

    Measures sample efficiency and generalization in reinforcement learning across procedurally generated game environments.

    Task count
    16

  • MathOnline

    ProofNet

    Tests autoformalization and formal proving of undergraduate mathematics in Lean from natural-language statements.

    Task count
    371

  • RetrievalOnline

    RAGChecker

    Diagnoses retrieval-augmented generation systems with claim-level metrics for their retrieval and generation components.

  • RetrievalOnline

    RAGTruth

    Evaluates hallucination detection in retrieval-augmented answers using human annotations at response and word levels.

    Task count
    18,000

  • SafetyOnline

    RealToxicityPrompts

    Measures whether language models continue naturally occurring web prompts with toxic text.

  • CodingOnline

    RepoBench

    Tests repository-level code completion using cross-file dependencies in Python and Java projects.

  • CodingOnline

    RepoQA

    Tests long-context code retrieval by asking models to find described functions inside full repositories.

    Task count
    500

  • ScienceOnline

    RLBench

    Measures robot learning across vision-guided manipulation tasks in a simulated environment.

  • AgentsOnline

    RoboCasa

    Evaluates robot policies on everyday kitchen manipulation tasks in a large-scale simulation framework.

    Task count
    365

  • ScienceOnline

    robomimic

    Measures robot policies learned from offline human demonstrations across standardized datasets and tasks.

  • ScienceOnline

    robosuite

    Provides standardized simulated robot manipulation environments for evaluating learning methods.

  • OtherOnline

    ROUGE

    Scores automatic summaries against references using n-gram overlap and longest-common-subsequence variants.

  • MemoryOnline

    RULER

    Measures effective context length with configurable synthetic tasks that go beyond simple needle retrieval.

    Task count
    13

  • OtherOnline

    SacreBLEU

    Computes reproducible machine-translation metrics while recording a signature for dataset and tokenization settings.

  • SafetyOnline

    SafeBench

    Evaluates autonomous-driving agents in safety-critical simulated scenarios, including adversarially generated cases.

    Task count
    235

  • SafetyOnline

    SafetyBench

    Evaluates large-language-model safety knowledge with Chinese and English multiple-choice questions across seven risk categories.

    Task count
    11,435

  • ScienceOnline

    ScienceWorld

    Measures text-based agents on tasks grounded in an elementary science curriculum.

    Task count
    30

  • ScienceOnline

    SciFact

    Tests retrieval of scientific evidence and classification of research claims as supported or refuted.

  • MultimodalOnline

    SEED-Bench

    Measures image and video comprehension across multimodal ability dimensions using human-annotated multiple-choice questions.

  • AgentsOnline

    SmartPlay

    Measures agent reasoning, planning, spatial understanding, learning, and handling of randomness across six games.

    Task count
    6

  • SafetyOnline

    SORRY-Bench

    Measures the safety-refusal behavior of language models across a balanced taxonomy of unsafe requests and prompt mutations.

    Task count
    450

Categories in Online

Questions

What kinds of benchmarks are in Online?

Categories in Online: Multimodal (51 benchmarks), Reasoning (50 benchmarks), Agents (31 benchmarks), Safety (25 benchmarks), and Science (24 benchmarks).

How many AI benchmarks are in the index?

AI Benchmark Index currently lists 264 benchmarks that link to a primary source.

Can I submit an AI benchmark?

Yes. Submit a public primary source, licence, release date, category, and a one-sentence description of what it measures.

Published by . How we check each benchmark.