# AI Benchmark Index

> Compare AI benchmarks, data licences and reported scores.

Every benchmark on AI Benchmark Index, grouped by city, with its facts. Each name links to the markdown copy of its page.

## Online, Global, United States

### [ActivityNet-QA](https://aibenchmarkindex.com/global/online/activitynet-qa.md)

Tests question answering over long videos with questions that require spatial and temporal reasoning.

- Page: https://aibenchmarkindex.com/global/online/activitynet-qa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/MILVLG/activitynet-qa>
- Updated: 2026-10-21

### [AdvBench](https://aibenchmarkindex.com/global/online/advbench.md)

Measures whether adversarial suffixes induce aligned language models to produce targeted harmful behavior.

- Page: https://aibenchmarkindex.com/global/online/advbench
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/llm-attacks/llm-attacks>
- Updated: 2026-10-21

### [AgentBench](https://aibenchmarkindex.com/global/online/agentbench.md)

Measures language-model agents across operating systems, databases, games, web shopping, and other interactive environments.

- Page: https://aibenchmarkindex.com/global/online/agentbench
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/THUDM/AgentBench>
- Updated: 2026-10-21

### [AgentDojo](https://aibenchmarkindex.com/global/online/agentdojo.md)

Evaluates agent task utility and resistance to prompt injection while agents use tools over untrusted data.

- Page: https://aibenchmarkindex.com/global/online/agentdojo
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://agentdojo.spylab.ai/>
- Updated: 2026-10-21

### [AGIEval](https://aibenchmarkindex.com/global/online/agieval.md)

Tests general problem solving with questions from 20 public admission, qualification, and competition exams.

- Page: https://aibenchmarkindex.com/global/online/agieval
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/ruixiangcui/AGIEval>
- Updated: 2026-10-21

### [AI2 Reasoning Challenge](https://aibenchmarkindex.com/global/online/ai2-reasoning-challenge.md)

Tests grade-school science reasoning with natural multiple-choice questions split into easy and challenge sets.

- Page: https://aibenchmarkindex.com/global/online/ai2-reasoning-challenge
- Categories: Science
- Address: Online, Global, United States
- Website: <https://allenai.org/data/arc>
- Updated: 2026-11-07

### [AI2-THOR](https://aibenchmarkindex.com/global/online/ai2-thor.md)

Evaluates embodied agents in interactive simulated indoor scenes with object manipulation and navigation.

- Page: https://aibenchmarkindex.com/global/online/ai2-thor
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/allenai/ai2thor>
- Updated: 2026-10-21

### [AI2D](https://aibenchmarkindex.com/global/online/ai2d.md)

Measures parsing and question answering over grade-school science diagrams with annotated constituents and relationships.

- Page: https://aibenchmarkindex.com/global/online/ai2d
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://allenai.org/data/diagrams>
- Updated: 2026-11-07

### [ALFWorld](https://aibenchmarkindex.com/global/online/alfworld.md)

Measures agents that learn high-level policies in text environments and execute the same household goals in embodied worlds.

- Page: https://aibenchmarkindex.com/global/online/alfworld
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/alfworld/alfworld>
- Updated: 2026-10-21

### [AlpacaEval](https://aibenchmarkindex.com/global/online/alpacaeval.md)

Measures instruction-following quality by comparing model responses against references with automated preference judges.

- Page: https://aibenchmarkindex.com/global/online/alpacaeval
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/tatsu-lab/alpaca_eval>
- Updated: 2026-10-21

### [AndroidWorld](https://aibenchmarkindex.com/global/online/androidworld.md)

Measures autonomous agents on dynamically generated tasks across real Android applications.

- Page: https://aibenchmarkindex.com/global/online/androidworld
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/google-research/android_world>
- Updated: 2026-10-21

### [ANLI](https://aibenchmarkindex.com/global/online/anli.md)

Tests natural-language inference on examples collected adversarially against successive model generations.

- Page: https://aibenchmarkindex.com/global/online/anli
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/facebookresearch/anli>
- Updated: 2026-10-21

### [APPS](https://aibenchmarkindex.com/global/online/apps.md)

Tests coding-challenge competence on problems ranging from introductory exercises to programming competitions.

- Page: https://aibenchmarkindex.com/global/online/apps
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/hendrycks/apps>
- Updated: 2026-10-21

### [AppWorld](https://aibenchmarkindex.com/global/online/appworld.md)

Measures interactive coding agents on tasks spanning simulated everyday apps and their APIs.

- Page: https://aibenchmarkindex.com/global/online/appworld
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://appworld.dev/>
- Updated: 2026-10-21

### [ArabicMMLU](https://aibenchmarkindex.com/global/online/arabicmmlu.md)

Evaluates Arabic language understanding across 40 school-exam tasks from countries in North Africa, the Levant, and the Gulf.

- Page: https://aibenchmarkindex.com/global/online/arabicmmlu
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/mbzuai-nlp/ArabicMMLU>
- Updated: 2026-10-21

### [ARC-AGI-1](https://aibenchmarkindex.com/global/online/arc-agi-1.md)

Tests few-shot abstraction by asking systems to infer transformations between small colored grids.

- Page: https://aibenchmarkindex.com/global/online/arc-agi-1
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/fchollet/ARC-AGI>
- Updated: 2026-10-21

### [ARC-AGI-2](https://aibenchmarkindex.com/global/online/arc-agi-2.md)

Tests symbolic interpretation, compositional reasoning, and context-dependent rules on novel grid tasks.

- Page: https://aibenchmarkindex.com/global/online/arc-agi-2
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://arcprize.org/arc-agi/2>
- Updated: 2026-11-07

### [ARC-AGI-3](https://aibenchmarkindex.com/global/online/arc-agi-3.md)

Tests exploration, goal discovery, world-model building, and planning in novel interactive environments.

- Page: https://aibenchmarkindex.com/global/online/arc-agi-3
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://arcprize.org/arc-agi/3>
- Updated: 2026-11-07

### [Arena-Hard-Auto](https://aibenchmarkindex.com/global/online/arena-hard-auto.md)

Tests open-ended instruction following on difficult real user prompts with automated preference judges.

- Page: https://aibenchmarkindex.com/global/online/arena-hard-auto
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/lm-sys/arena-hard-auto>
- Updated: 2026-10-21

### [ASCIIBench](https://aibenchmarkindex.com/global/online/asciibench.md)

Measures four-choice recognition of ASCII art from text-only, rendered-image, and combined inputs.

- Page: https://aibenchmarkindex.com/global/online/asciibench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/KerryLuo/ASCIIBench>
- Updated: 2026-10-21

### [AVA](https://aibenchmarkindex.com/global/online/ava.md)

Evaluates spatiotemporally localized recognition of 80 atomic actions in densely annotated movie clips.

- Page: https://aibenchmarkindex.com/global/online/ava
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://research.google.com/ava/>
- Updated: 2026-10-21

### [BabyAI](https://aibenchmarkindex.com/global/online/babyai.md)

Measures grounded language understanding by training agents to follow generated instructions in grid worlds.

- Page: https://aibenchmarkindex.com/global/online/babyai
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/mila-iqia/babyai>
- Updated: 2026-10-21

### [BabyAI GoTo](https://aibenchmarkindex.com/global/online/babyai-goto.md)

Tests grounded language understanding by asking an agent to navigate a grid world to a described object.

- Page: https://aibenchmarkindex.com/global/online/babyai-goto
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/mila-iqia/babyai>
- Updated: 2026-10-21

### [BBQ](https://aibenchmarkindex.com/global/online/bbq.md)

Measures social bias in question answering across ambiguous and disambiguated contexts covering nine protected dimensions.

- Page: https://aibenchmarkindex.com/global/online/bbq
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/nyu-mll/BBQ>
- Updated: 2026-10-21

### [BBQ-Lite](https://aibenchmarkindex.com/global/online/bbq-lite.md)

Provides a compact BIG-bench version of BBQ for exact-match evaluation of bias-sensitive question answering.

- Page: https://aibenchmarkindex.com/global/online/bbq-lite
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/bbq_lite_json>
- Updated: 2026-10-21

### [BeaverTails](https://aibenchmarkindex.com/global/online/beavertails.md)

Supports safety classification and preference evaluation with human-labeled question-answer pairs across harm categories.

- Page: https://aibenchmarkindex.com/global/online/beavertails
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/PKU-Alignment/beavertails>
- Updated: 2026-10-21

### [BEIR](https://aibenchmarkindex.com/global/online/beir.md)

Evaluates zero-shot information retrieval across heterogeneous datasets, domains, and retrieval architectures.

- Page: https://aibenchmarkindex.com/global/online/beir
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/beir-cellar/beir>
- Updated: 2026-10-21

### [Benchmark Accessibility Fix](https://aibenchmarkindex.com/global/online/benchmark-accessibility-fix.md)

Measures whether a model's repair of a broken web page genuinely fixes keyboard, screen-reader, and contrast defects without hiding or deleting content.

- Page: https://aibenchmarkindex.com/global/online/benchmark-accessibility-fix
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-accessibility-fix>
- Updated: 2026-10-21

### [Benchmark Apology Pizza](https://aibenchmarkindex.com/global/online/benchmark-apology-pizza.md)

Measures whether a support agent verifies a wrong pizza order, applies the correct bounded remedy under a refund policy, and escalates unsafe cases.

- Page: https://aibenchmarkindex.com/global/online/benchmark-apology-pizza
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-apology-pizza>
- Updated: 2026-10-21

### [Benchmark Checkout Consent](https://aibenchmarkindex.com/global/online/benchmark-checkout-consent.md)

Measures whether a shopping agent preserves the requested cart and spends money only after exact-total and shipping consent.

- Page: https://aibenchmarkindex.com/global/online/benchmark-checkout-consent
- Categories: Commerce
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-checkout-consent>
- Updated: 2026-10-21

### [Benchmark Cost per Task](https://aibenchmarkindex.com/global/online/benchmark-cost-per-task.md)

Measures dollars, retries, turns, and tokens an agent spends to reach a verified-correct result on back-office tasks.

- Page: https://aibenchmarkindex.com/global/online/benchmark-cost-per-task
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-cost-per-task>
- Updated: 2026-10-21

### [Benchmark CRM Truthfulness](https://aibenchmarkindex.com/global/online/benchmark-crm-truthfulness.md)

Measures whether a sales agent emails only consenting contacts, claims only what a fact sheet supports, and keeps CRM records accurate.

- Page: https://aibenchmarkindex.com/global/online/benchmark-crm-truthfulness
- Categories: Commerce, Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-crm-truthfulness>
- Updated: 2026-10-21

### [Benchmark Exploit Resistance](https://aibenchmarkindex.com/global/online/benchmark-exploit-resistance.md)

Measures whether a coding agent fixes a seeded function or games the visible test runner, such as hard-coding sample cases or editing the checker.

- Page: https://aibenchmarkindex.com/global/online/benchmark-exploit-resistance
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-exploit-resistance>
- Updated: 2026-10-21

### [Benchmark Honesty](https://aibenchmarkindex.com/global/online/benchmark-honesty.md)

Measures whether a model retries failed retrieval, answers only from evidence, and abstains instead of fabricating unavailable facts.

- Page: https://aibenchmarkindex.com/global/online/benchmark-honesty
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-honesty>
- Updated: 2026-10-21

### [Benchmark Hundred Step](https://aibenchmarkindex.com/global/online/benchmark-hundred-step.md)

Measures whether an agent preserves dependencies and recovers from injected failures across workflows of 25 to 100 steps.

- Page: https://aibenchmarkindex.com/global/online/benchmark-hundred-step
- Categories: Long-horizon
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-hundred-step>
- Updated: 2026-10-21

### [Benchmark Instruction Hierarchy](https://aibenchmarkindex.com/global/online/benchmark-instruction-hierarchy.md)

Measures whether an agent keeps user and system rules when documents, quotes, or tool output contain conflicting or injected instructions.

- Page: https://aibenchmarkindex.com/global/online/benchmark-instruction-hierarchy
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-instruction-hierarchy>
- Updated: 2026-10-21

### [Benchmark Latency Budget](https://aibenchmarkindex.com/global/online/benchmark-latency-budget.md)

Measures whether an agent produces the correct answer inside a per-task time budget, with virtual tool latencies and voice, UI, and background deadlines.

- Page: https://aibenchmarkindex.com/global/online/benchmark-latency-budget
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-latency-budget>
- Updated: 2026-10-21

### [Benchmark MCP Safety](https://aibenchmarkindex.com/global/online/benchmark-mcp-safety.md)

Measures whether an agent completes MCP jobs without obeying poisoned tool descriptions, calling shadow tools, overreaching permissions, or leaking canaries.

- Page: https://aibenchmarkindex.com/global/online/benchmark-mcp-safety
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-mcp-safety>
- Updated: 2026-10-21

### [Benchmark Memory Conflict](https://aibenchmarkindex.com/global/online/benchmark-memory-conflict.md)

Measures whether an assistant applies the latest valid household preference, respects privacy and authority, and keeps bookings reversible.

- Page: https://aibenchmarkindex.com/global/online/benchmark-memory-conflict
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-memory-conflict>
- Updated: 2026-10-21

### [Benchmark Payment Delegation](https://aibenchmarkindex.com/global/online/benchmark-payment-delegation.md)

Measures whether an agent respects spending caps, merchant allowlists, refunds, and dispute policy when using a delegated payment card.

- Page: https://aibenchmarkindex.com/global/online/benchmark-payment-delegation
- Categories: Commerce
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-payment-delegation>
- Updated: 2026-10-21

### [Benchmark Pelican](https://aibenchmarkindex.com/global/online/benchmark-pelican.md)

Measures whether a model can draw a well-formed, plausible SVG of a pelican riding a bicycle, scored by structural checks on markup, geometry, and coverage.

- Page: https://aibenchmarkindex.com/global/online/benchmark-pelican
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-pelican>
- Updated: 2026-10-21

### [Benchmark Ranking Stability](https://aibenchmarkindex.com/global/online/benchmark-ranking-stability.md)

Measures how much a model ranking moves when the seed, prompt wording, or task sample changes.

- Page: https://aibenchmarkindex.com/global/online/benchmark-ranking-stability
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-ranking-stability>
- Updated: 2026-10-21

### [Benchmark Rollback Recovery](https://aibenchmarkindex.com/global/online/benchmark-rollback-recovery.md)

Measures whether an agent notices a self-inflicted CRM, file, or messaging mistake, rolls it back cleanly, and discloses it.

- Page: https://aibenchmarkindex.com/global/online/benchmark-rollback-recovery
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-rollback-recovery>
- Updated: 2026-10-21

### [Benchmark Schema Mutation](https://aibenchmarkindex.com/global/online/benchmark-schema-mutation.md)

Measures whether a model calls the current version of a migrated tool schema instead of the one it memorized.

- Page: https://aibenchmarkindex.com/global/online/benchmark-schema-mutation
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-schema-mutation>
- Updated: 2026-10-21

### [Benchmark Silent Constraints](https://aibenchmarkindex.com/global/online/benchmark-silent-constraints.md)

Measures whether an agent meets its stated goal without violating unspoken constraints on protected files, foreign data, external addresses, and quotas.

- Page: https://aibenchmarkindex.com/global/online/benchmark-silent-constraints
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-silent-constraints>
- Updated: 2026-10-21

### [Benchmark Skateboard](https://aibenchmarkindex.com/global/online/benchmark-skateboard.md)

Measures signed 3D trick rotations, projectile-motion arithmetic, landing orientation, and whether a named skateboard trick can finish within its airtime.

- Page: https://aibenchmarkindex.com/global/online/benchmark-skateboard
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-skateboard>
- Updated: 2026-10-21

### [Benchmark Sycophancy](https://aibenchmarkindex.com/global/online/benchmark-sycophancy.md)

Measures whether a model calibrates pushback as a user's proposed plan escalates from harmless to severely harmful.

- Page: https://aibenchmarkindex.com/global/online/benchmark-sycophancy
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-sycophancy>
- Updated: 2026-10-21

### [Benchmark Test Weakening](https://aibenchmarkindex.com/global/online/benchmark-test-weakening.md)

Measures whether a coding agent fixes buggy JavaScript without weakening or deleting the tests, including hidden and mutation-checked cases.

- Page: https://aibenchmarkindex.com/global/online/benchmark-test-weakening
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-test-weakening>
- Updated: 2026-10-21

### [Benchmark Tool Errors](https://aibenchmarkindex.com/global/online/benchmark-tool-errors.md)

Measures whether an agent retries API faults safely: idempotent calls, bounded retries, no duplicate side effects, and a truthful final report.

- Page: https://aibenchmarkindex.com/global/online/benchmark-tool-errors
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://github.com/pooriaarab/benchmark-tool-errors>
- Updated: 2026-10-21

### [BERTScore](https://aibenchmarkindex.com/global/online/bertscore.md)

Evaluates generated text by matching contextual token embeddings between candidate and reference sentences.

- Page: https://aibenchmarkindex.com/global/online/bertscore
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/Tiiiger/bert_score>
- Updated: 2026-10-21

### [BIG-bench](https://aibenchmarkindex.com/global/online/big-bench.md)

Tests language-model capabilities across more than 200 collaboratively authored tasks.

- Page: https://aibenchmarkindex.com/global/online/big-bench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/google/BIG-bench>
- Updated: 2026-10-21

### [BIG-Bench Hard](https://aibenchmarkindex.com/global/online/big-bench-hard.md)

Tests multi-step reasoning on 23 BIG-bench tasks where early language models trailed average human raters.

- Page: https://aibenchmarkindex.com/global/online/big-bench-hard
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/suzgunmirac/BIG-Bench-Hard>
- Updated: 2026-10-21

### [BigCodeBench](https://aibenchmarkindex.com/global/online/bigcodebench.md)

Tests function-level code generation on software-engineering tasks requiring diverse library calls.

- Page: https://aibenchmarkindex.com/global/online/bigcodebench
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/bigcode-project/bigcodebench>
- Updated: 2026-10-21

### [BIPIA](https://aibenchmarkindex.com/global/online/bipia.md)

Evaluates model and defense robustness against indirect prompt injection across five application tasks.

- Page: https://aibenchmarkindex.com/global/online/bipia
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/microsoft/BIPIA>
- Updated: 2026-10-21

### [BIRD-SQL](https://aibenchmarkindex.com/global/online/bird-sql.md)

Tests text-to-SQL generation against large database contents spanning many professional domains.

- Page: https://aibenchmarkindex.com/global/online/bird-sql
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://bird-bench.github.io/>
- Updated: 2026-11-07

### [BLINK](https://aibenchmarkindex.com/global/online/blink.md)

Measures core visual perception abilities through multiple-choice tasks designed to resist verbal shortcuts.

- Page: https://aibenchmarkindex.com/global/online/blink
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/zeyofu/BLINK_Benchmark>
- Updated: 2026-10-21

### [BOLD](https://aibenchmarkindex.com/global/online/bold.md)

Measures social biases in open-ended English generation from prompts spanning profession, gender, race, religion, and politics.

- Page: https://aibenchmarkindex.com/global/online/bold
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/amazon-science/bold>
- Updated: 2026-10-21

### [BoolQ](https://aibenchmarkindex.com/global/online/boolq.md)

Tests yes-or-no question answering from a supporting passage using naturally occurring information-seeking questions.

- Page: https://aibenchmarkindex.com/global/online/boolq
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/google-research-datasets/boolean-questions>
- Updated: 2026-10-21

### [BrowserGym](https://aibenchmarkindex.com/global/online/browsergym.md)

Provides a common environment for evaluating browser agents across several web-task benchmarks.

- Page: https://aibenchmarkindex.com/global/online/browsergym
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://github.com/ServiceNow/BrowserGym>
- Updated: 2026-10-21

### [C-Eval](https://aibenchmarkindex.com/global/online/c-eval.md)

Evaluates Chinese foundation models with multiple-choice questions spanning 52 disciplines and four difficulty levels.

- Page: https://aibenchmarkindex.com/global/online/c-eval
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://cevalbenchmark.com/>
- Updated: 2026-10-21

### [CALVIN](https://aibenchmarkindex.com/global/online/calvin.md)

Measures long-horizon, language-conditioned robot manipulation through sequences of continuous-control tasks.

- Page: https://aibenchmarkindex.com/global/online/calvin
- Categories: Long-horizon
- Address: Online, Global, United States
- Website: <https://github.com/mees/calvin>
- Updated: 2026-10-21

### [ChartQA](https://aibenchmarkindex.com/global/online/chartqa.md)

Measures visual and logical reasoning over charts with human-authored and generated questions.

- Page: https://aibenchmarkindex.com/global/online/chartqa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/vis-nlp/ChartQA>
- Updated: 2026-10-21

### [Cicero Diplomacy](https://aibenchmarkindex.com/global/online/cicero-diplomacy.md)

Evaluates strategic planning, negotiation, and natural-language coordination by agents playing the board game Diplomacy.

- Page: https://aibenchmarkindex.com/global/online/cicero-diplomacy
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/facebookresearch/diplomacy_cicero>
- Updated: 2026-10-21

### [CIFAR-10](https://aibenchmarkindex.com/global/online/cifar-10.md)

Evaluates image classification on 60,000 small color images evenly divided among ten classes.

- Page: https://aibenchmarkindex.com/global/online/cifar-10
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://www.cs.toronto.edu/~kriz/cifar.html>
- Updated: 2026-10-21

### [CLIPScore](https://aibenchmarkindex.com/global/online/clipscore.md)

Measures image-caption compatibility without requiring reference captions by comparing CLIP embeddings.

- Page: https://aibenchmarkindex.com/global/online/clipscore
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/jmhessel/clipscore>
- Updated: 2026-10-21

### [ClockBench](https://aibenchmarkindex.com/global/online/clockbench.md)

Evaluates whether vision-language models can read analog clocks across varied faces and times.

- Page: https://aibenchmarkindex.com/global/online/clockbench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://clockbench.ai/>
- Updated: 2026-11-07

### [CMMLU](https://aibenchmarkindex.com/global/online/cmmlu.md)

Evaluates Chinese language and cultural knowledge across 67 multiple-choice subjects.

- Page: https://aibenchmarkindex.com/global/online/cmmlu
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/haonan-li/CMMLU>
- Updated: 2026-10-21

### [COCO Caption](https://aibenchmarkindex.com/global/online/coco-caption.md)

Evaluates image-caption generation on everyday scenes using reference captions and standardized captioning metrics.

- Page: https://aibenchmarkindex.com/global/online/coco-caption
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://cocodataset.org/>
- Updated: 2026-11-07

### [CodeContests](https://aibenchmarkindex.com/global/online/codecontests.md)

Tests competitive-programming solution generation with problems, descriptions, tests, and human submissions.

- Page: https://aibenchmarkindex.com/global/online/codecontests
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/deepmind/code_contests>
- Updated: 2026-10-21

### [CodeXGLUE](https://aibenchmarkindex.com/global/online/codexglue.md)

Tests code understanding and generation across ten tasks including search, repair, translation, and summarization.

- Page: https://aibenchmarkindex.com/global/online/codexglue
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/microsoft/CodeXGLUE>
- Updated: 2026-10-21

### [CogVideo](https://aibenchmarkindex.com/global/online/cogvideo.md)

Evaluates text-to-video generation; the repository provides model code and references for the original CogVideo and later CogVideoX systems.

- Page: https://aibenchmarkindex.com/global/online/cogvideo
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/THUDM/CogVideo>
- Updated: 2026-10-21

### [Common Voice](https://aibenchmarkindex.com/global/online/common-voice.md)

Supports evaluation of speech recognition across languages, accents, and demographic groups using contributed recordings.

- Page: https://aibenchmarkindex.com/global/online/common-voice
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://commonvoice.mozilla.org/en/datasets>
- Updated: 2026-10-21

### [Crafter](https://aibenchmarkindex.com/global/online/crafter.md)

Measures reinforcement-learning agents across a survival game designed to expose a broad spectrum of capabilities.

- Page: https://aibenchmarkindex.com/global/online/crafter
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/danijar/crafter>
- Updated: 2026-10-21

### [CRMArena](https://aibenchmarkindex.com/global/online/crmarena.md)

Measures agents on professional customer-relationship-management tasks in realistic business scenarios.

- Page: https://aibenchmarkindex.com/global/online/crmarena
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/SalesforceAIResearch/CRMArena>
- Updated: 2026-10-21

### [CrossCodeEval](https://aibenchmarkindex.com/global/online/crosscodeeval.md)

Tests cross-file code completion and context retrieval in Python, Java, TypeScript, and C# repositories.

- Page: https://aibenchmarkindex.com/global/online/crosscodeeval
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://crosscodeeval.github.io/>
- Updated: 2026-10-21

### [CrowS-Pairs](https://aibenchmarkindex.com/global/online/crows-pairs.md)

Measures stereotypical preferences in masked language models using paired sentences across nine types of social bias.

- Page: https://aibenchmarkindex.com/global/online/crows-pairs
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/nyu-mll/crows-pairs>
- Updated: 2026-10-21

### [CRUD-RAG](https://aibenchmarkindex.com/global/online/crud-rag.md)

Evaluates retrieval-augmented generation across create, read, update, and delete application scenarios.

- Page: https://aibenchmarkindex.com/global/online/crud-rag
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/IAAR-Shanghai/CRUD_RAG>
- Updated: 2026-10-21

### [CRUXEval](https://aibenchmarkindex.com/global/online/cruxeval.md)

Tests code reasoning by asking models to predict inputs or outputs for short Python functions.

- Page: https://aibenchmarkindex.com/global/online/cruxeval
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/facebookresearch/cruxeval>
- Updated: 2026-10-21

### [CyberSecEval](https://aibenchmarkindex.com/global/online/cyberseceval.md)

Evaluates cybersecurity risks from model-generated code and responses to requests that could enable cyberattacks.

- Page: https://aibenchmarkindex.com/global/online/cyberseceval
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/meta-llama/PurpleLlama>
- Updated: 2026-10-21

### [Do-Not-Answer](https://aibenchmarkindex.com/global/online/do-not-answer.md)

Evaluates whether language models decline risky instructions while still answering harmless requests.

- Page: https://aibenchmarkindex.com/global/online/do-not-answer
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/Libr-AI/do-not-answer>
- Updated: 2026-10-21

### [DocVQA](https://aibenchmarkindex.com/global/online/docvqa.md)

Measures question answering over document images where layout, text, and document structure matter.

- Page: https://aibenchmarkindex.com/global/online/docvqa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://www.docvqa.org/datasets/docvqa>
- Updated: 2026-10-21

### [EmbodiedQA](https://aibenchmarkindex.com/global/online/embodiedqa.md)

Tests agents that navigate a simulated home to gather visual evidence and answer a question.

- Page: https://aibenchmarkindex.com/global/online/embodiedqa
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/facebookresearch/EmbodiedQA>
- Updated: 2026-10-21

### [FActScore](https://aibenchmarkindex.com/global/online/factscore.md)

Measures the factual precision of long-form generations by decomposing them into atomic claims and checking each claim against a knowledge source.

- Page: https://aibenchmarkindex.com/global/online/factscore
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/shmsw25/FActScore>
- Updated: 2026-10-21

### [FEVER](https://aibenchmarkindex.com/global/online/fever.md)

Tests whether systems verify Wikipedia-based claims as supported, refuted, or lacking enough evidence.

- Page: https://aibenchmarkindex.com/global/online/fever
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/awslabs/fever>
- Updated: 2026-10-21

### [FinanceBench](https://aibenchmarkindex.com/global/online/financebench.md)

Evaluates open-book financial question answering against answers and evidence drawn from public company filings.

- Page: https://aibenchmarkindex.com/global/online/financebench
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/patronus-ai/financebench>
- Updated: 2026-10-21

### [FinBen](https://aibenchmarkindex.com/global/online/finben.md)

Evaluates financial language understanding and prediction across 24 tasks, including extraction, forecasting, risk, and decision making.

- Page: https://aibenchmarkindex.com/global/online/finben
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/The-FinAI/PIXIU>
- Updated: 2026-10-21

### [FinGAIA](https://aibenchmarkindex.com/global/online/fingaia.md)

Evaluates agents on multistep financial work involving web research, files, multimodal inputs, code, and calculations.

- Page: https://aibenchmarkindex.com/global/online/fingaia
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/SUFE-AIFLM-Lab/FinGAIA>
- Updated: 2026-10-21

### [FinQA](https://aibenchmarkindex.com/global/online/finqa.md)

Evaluates numerical reasoning over tables and text from financial reports by requiring executable answer programs.

- Page: https://aibenchmarkindex.com/global/online/finqa
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/czyssrs/FinQA>
- Updated: 2026-10-21

### [FLORES-200](https://aibenchmarkindex.com/global/online/flores-200.md)

Evaluates machine translation across 200 languages with a professionally translated parallel corpus.

- Page: https://aibenchmarkindex.com/global/online/flores-200
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/facebookresearch/flores>
- Updated: 2026-10-21

### [FollowBench](https://aibenchmarkindex.com/global/online/followbench.md)

Tests adherence to increasingly complex content, situation, style, format, and example constraints.

- Page: https://aibenchmarkindex.com/global/online/followbench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/YJiangcm/FollowBench>
- Updated: 2026-10-21

### [Free Music Archive](https://aibenchmarkindex.com/global/online/free-music-archive.md)

Supports music-information-retrieval evaluation with audio, metadata, features, and genre labels for Creative Commons tracks.

- Page: https://aibenchmarkindex.com/global/online/free-music-archive
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://github.com/mdeff/fma>
- Updated: 2026-10-21

### [FreshQA](https://aibenchmarkindex.com/global/online/freshqa.md)

Evaluates answers to questions whose facts change over time or were never present in a model’s training data.

- Page: https://aibenchmarkindex.com/global/online/freshqa
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/freshllms/freshqa>
- Updated: 2026-10-21

### [Fréchet Video Distance](https://aibenchmarkindex.com/global/online/frechet-video-distance.md)

Measures the distributional similarity of real and generated videos using learned video embeddings.

- Page: https://aibenchmarkindex.com/global/online/frechet-video-distance
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/google-research/google-research/tree/master/frechet_video_distance>
- Updated: 2026-10-21

### [GAIA](https://aibenchmarkindex.com/global/online/gaia.md)

Measures general assistants on real-world questions requiring reasoning, tools, browsing, and multimodal inputs.

- Page: https://aibenchmarkindex.com/global/online/gaia
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://huggingface.co/datasets/gaia-benchmark/GAIA>
- Updated: 2026-10-21

### [GEO-Bench](https://aibenchmarkindex.com/global/online/geo-bench.md)

Compares pretrained models on a standardized suite of geospatial Earth-monitoring classification and segmentation datasets.

- Page: https://aibenchmarkindex.com/global/online/geo-bench
- Categories: Science
- Address: Online, Global, United States
- Website: <https://arxiv.org/abs/2306.03831>
- Updated: 2026-10-21

### [GLUE](https://aibenchmarkindex.com/global/online/glue.md)

Combines nine sentence- and sentence-pair tasks to evaluate general-purpose English language understanding.

- Page: https://aibenchmarkindex.com/global/online/glue
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/nyu-mll/GLUE-baselines>
- Updated: 2026-10-21

### [GPQA](https://aibenchmarkindex.com/global/online/gpqa.md)

Tests expert-level reasoning on graduate biology, physics, and chemistry questions designed to resist web search.

- Page: https://aibenchmarkindex.com/global/online/gpqa
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/Idavidrein/gpqa>
- Updated: 2026-10-21

### [GSM8K](https://aibenchmarkindex.com/global/online/gsm8k.md)

Tests multi-step arithmetic reasoning on human-written grade-school math word problems.

- Page: https://aibenchmarkindex.com/global/online/gsm8k
- Categories: Math
- Address: Online, Global, United States
- Website: <https://github.com/openai/grade-school-math>
- Updated: 2026-10-21

### [Gym Sokoban](https://aibenchmarkindex.com/global/online/gym-sokoban.md)

Evaluates planning and reinforcement learning in procedurally generated box-pushing puzzles.

- Page: https://aibenchmarkindex.com/global/online/gym-sokoban
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/mpSchrader/gym-sokoban>
- Updated: 2026-10-21

### [Habitat HomeRobot OVMM Challenge](https://aibenchmarkindex.com/global/online/habitat-homerobot-ovmm-challenge.md)

Measures open-vocabulary mobile manipulation systems that find objects, navigate, and rearrange them in simulated homes.

- Page: https://aibenchmarkindex.com/global/online/habitat-homerobot-ovmm-challenge
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://aihabitat.org/challenge/2023_homerobot_ovmm/>
- Updated: 2026-10-21

### [HaluEval](https://aibenchmarkindex.com/global/online/halueval.md)

Evaluates whether language models recognize hallucinated content in general queries, question answering, dialogue, and summarization.

- Page: https://aibenchmarkindex.com/global/online/halueval
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/RUCAIBox/HaluEval>
- Updated: 2026-10-21

### [HarmBench](https://aibenchmarkindex.com/global/online/harmbench.md)

Evaluates automated red-teaming attacks and model refusal defenses across standardized harmful behaviors.

- Page: https://aibenchmarkindex.com/global/online/harmbench
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://harmbench.org/>
- Updated: 2026-10-21

### [HEAR](https://aibenchmarkindex.com/global/online/hear.md)

Evaluates transferable audio representations across speech, music, and environmental sound tasks.

- Page: https://aibenchmarkindex.com/global/online/hear
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://hearbenchmark.com/>
- Updated: 2026-10-21

### [HELM](https://aibenchmarkindex.com/global/online/helm.md)

Evaluates foundation models across standardized scenarios with multiple accuracy, robustness, fairness, and efficiency metrics.

- Page: https://aibenchmarkindex.com/global/online/helm
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://crfm.stanford.edu/helm/>
- Updated: 2026-10-21

### [HELM Lite](https://aibenchmarkindex.com/global/online/helm-lite.md)

Evaluates language models on a compact suite of ten HELM scenarios selected for broad coverage and lower evaluation cost.

- Page: https://aibenchmarkindex.com/global/online/helm-lite
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://crfm.stanford.edu/helm/lite/>
- Updated: 2026-10-21

### [HotpotQA](https://aibenchmarkindex.com/global/online/hotpotqa.md)

Tests explainable multi-hop question answering with supporting facts drawn from multiple Wikipedia articles.

- Page: https://aibenchmarkindex.com/global/online/hotpotqa
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/hotpotqa/hotpot>
- Updated: 2026-10-21

### [HPS v2](https://aibenchmarkindex.com/global/online/hps-v2.md)

Evaluates whether a text-to-image model produces images people prefer for the same prompt.

- Page: https://aibenchmarkindex.com/global/online/hps-v2
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/tgxs002/HPSv2>
- Updated: 2026-10-21

### [HumanEval](https://aibenchmarkindex.com/global/online/humaneval.md)

Tests functional correctness of generated Python functions against hidden unit tests.

- Page: https://aibenchmarkindex.com/global/online/humaneval
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/openai/human-eval>
- Updated: 2026-10-21

### [Humanity’s Last Exam](https://aibenchmarkindex.com/global/online/humanitys-last-exam.md)

Tests frontier models on expert-written, multimodal questions spanning mathematics, science, and the humanities.

- Page: https://aibenchmarkindex.com/global/online/humanitys-last-exam
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://lastexam.ai/>
- Updated: 2026-10-21

### [HumorBench](https://aibenchmarkindex.com/global/online/humorbench.md)

Evaluates English and Spanish multi-turn humor understanding through joke classification and identification of each line’s role.

- Page: https://aibenchmarkindex.com/global/online/humorbench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/atreydesai/humorbench>
- Updated: 2026-10-21

### [IFEval](https://aibenchmarkindex.com/global/online/ifeval.md)

Tests whether a model follows verifiable instructions such as format, length, and keyword constraints.

- Page: https://aibenchmarkindex.com/global/online/ifeval
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/google-research/google-research/tree/master/instruction_following_eval>
- Updated: 2026-10-21

### [ImageNet](https://aibenchmarkindex.com/global/online/imagenet.md)

Evaluates object recognition using images organized under the WordNet noun hierarchy.

- Page: https://aibenchmarkindex.com/global/online/imagenet
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://www.image-net.org/>
- Updated: 2026-10-21

### [ImageNet-A](https://aibenchmarkindex.com/global/online/imagenet-a.md)

Tests image classifiers on naturally occurring, unmodified images selected because standard ImageNet models misclassify them.

- Page: https://aibenchmarkindex.com/global/online/imagenet-a
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/hendrycks/natural-adv-examples>
- Updated: 2026-10-21

### [ImageNet-R](https://aibenchmarkindex.com/global/online/imagenet-r.md)

Tests ImageNet classifiers on artistic and other non-photographic renditions of 200 object classes.

- Page: https://aibenchmarkindex.com/global/online/imagenet-r
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/hendrycks/imagenet-r>
- Updated: 2026-10-21

### [ImageNet-V2](https://aibenchmarkindex.com/global/online/imagenet-v2.md)

Retests ImageNet classifiers on three independently collected 10,000-image test sets matched to the original distribution.

- Page: https://aibenchmarkindex.com/global/online/imagenet-v2
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/modestyachts/ImageNetV2>
- Updated: 2026-10-21

### [ImagenHub](https://aibenchmarkindex.com/global/online/imagenhub.md)

Compares conditional image generation and editing systems across standardized tasks, metrics, and human ratings.

- Page: https://aibenchmarkindex.com/global/online/imagenhub
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://tiger-ai-lab.github.io/ImagenHub/>
- Updated: 2026-10-21

### [ImageReward](https://aibenchmarkindex.com/global/online/imagereward.md)

Measures how closely text-to-image outputs align with human preferences using a learned reward model trained on expert comparisons.

- Page: https://aibenchmarkindex.com/global/online/imagereward
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/zai-org/ImageReward>
- Updated: 2026-10-21

### [InfiniteBench](https://aibenchmarkindex.com/global/online/infinitebench.md)

Evaluates language models on synthetic and realistic English and Chinese tasks with contexts averaging more than 100,000 tokens.

- Page: https://aibenchmarkindex.com/global/online/infinitebench
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://github.com/OpenBMB/InfiniteBench>
- Updated: 2026-10-21

### [InfographicVQA](https://aibenchmarkindex.com/global/online/infographicvqa.md)

Measures question answering that combines infographic text, layout, graphics, data visualizations, and arithmetic.

- Page: https://aibenchmarkindex.com/global/online/infographicvqa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://www.docvqa.org/datasets/infographicvqa>
- Updated: 2026-10-21

### [InjecAgent](https://aibenchmarkindex.com/global/online/injecagent.md)

Evaluates whether tool-using agents follow indirect prompt injections embedded in external tool responses.

- Page: https://aibenchmarkindex.com/global/online/injecagent
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/uiuc-kang-lab/InjecAgent>
- Updated: 2026-10-21

### [InterCode-CTF](https://aibenchmarkindex.com/global/online/intercode-ctf.md)

Evaluates coding agents that solve capture-the-flag tasks through iterative commands and execution feedback.

- Page: https://aibenchmarkindex.com/global/online/intercode-ctf
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://intercode-benchmark.github.io/>
- Updated: 2026-10-21

### [JailbreakBench](https://aibenchmarkindex.com/global/online/jailbreakbench.md)

Compares jailbreak attacks and defenses on a standardized set of misuse behaviors and evaluation rules.

- Page: https://aibenchmarkindex.com/global/online/jailbreakbench
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://jailbreakbench.github.io/>
- Updated: 2026-10-21

### [Jericho](https://aibenchmarkindex.com/global/online/jericho.md)

Measures agents in human-authored interactive-fiction games through a common learning interface.

- Page: https://aibenchmarkindex.com/global/online/jericho
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/microsoft/jericho>
- Updated: 2026-10-21

### [Kaggle Game Arena](https://aibenchmarkindex.com/global/online/kaggle-game-arena.md)

Evaluates strategic planning, adaptation, and robustness through head-to-head model competitions in chess, poker, and Werewolf.

- Page: https://aibenchmarkindex.com/global/online/kaggle-game-arena
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://www.kaggle.com/game-arena>
- Updated: 2026-10-21

### [KILT](https://aibenchmarkindex.com/global/online/kilt.md)

Evaluates knowledge-intensive language tasks while requiring answers and provenance grounded in one Wikipedia snapshot.

- Page: https://aibenchmarkindex.com/global/online/kilt
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://kiltbenchmark.com/>
- Updated: 2026-10-21

### [Kinetics](https://aibenchmarkindex.com/global/online/kinetics.md)

Evaluates video action recognition across human-object and human-human interactions in short clips.

- Page: https://aibenchmarkindex.com/global/online/kinetics
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/cvdfoundation/kinetics-dataset>
- Updated: 2026-10-21

### [LAMBADA](https://aibenchmarkindex.com/global/online/lambada.md)

Tests prediction of a passage’s final word when success requires understanding its broader discourse context.

- Page: https://aibenchmarkindex.com/global/online/lambada
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://arxiv.org/abs/1606.06031>
- Updated: 2026-10-21

### [LegalBench](https://aibenchmarkindex.com/global/online/legalbench.md)

Evaluates legal reasoning across collaboratively contributed tasks covering varied legal domains and text formats.

- Page: https://aibenchmarkindex.com/global/online/legalbench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://hazyresearch.stanford.edu/legalbench>
- Updated: 2026-10-21

### [LegalBench-RAG](https://aibenchmarkindex.com/global/online/legalbench-rag.md)

Evaluates whether retrieval systems return minimal, precisely located evidence for complex legal contract questions.

- Page: https://aibenchmarkindex.com/global/online/legalbench-rag
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/zeroentropy-ai/legalbenchrag>
- Updated: 2026-10-21

### [LexGLUE](https://aibenchmarkindex.com/global/online/lexglue.md)

Evaluates legal language understanding across seven English classification datasets using a common evaluation setup.

- Page: https://aibenchmarkindex.com/global/online/lexglue
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/coastalcph/lex-glue>
- Updated: 2026-10-21

### [LIBERO](https://aibenchmarkindex.com/global/online/libero.md)

Measures knowledge transfer in lifelong robot learning across manipulation task suites with controlled distribution shifts.

- Page: https://aibenchmarkindex.com/global/online/libero
- Categories: Science
- Address: Online, Global, United States
- Website: <https://libero-project.github.io/>
- Updated: 2026-10-21

### [LIBERO-Spatial](https://aibenchmarkindex.com/global/online/libero-spatial.md)

Tests lifelong robot learning under spatial-relation distribution shifts in simulated manipulation tasks.

- Page: https://aibenchmarkindex.com/global/online/libero-spatial
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/Lifelong-Robot-Learning/LIBERO>
- Updated: 2026-10-21

### [LibriSpeech](https://aibenchmarkindex.com/global/online/librispeech.md)

Measures automatic speech recognition on roughly 1,000 hours of English audiobook speech.

- Page: https://aibenchmarkindex.com/global/online/librispeech
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://www.openslr.org/12>
- Updated: 2026-11-07

### [LiveBench](https://aibenchmarkindex.com/global/online/livebench.md)

Tests six capability areas with regularly refreshed questions and objective, non-judge scoring.

- Page: https://aibenchmarkindex.com/global/online/livebench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/LiveBench/LiveBench>
- Updated: 2026-10-21

### [LiveBench 2.0](https://aibenchmarkindex.com/global/online/livebench-2-0.md)

Tests long-horizon AI capabilities with a contamination-limited benchmark whose official site requires JavaScript.

- Page: https://aibenchmarkindex.com/global/online/livebench-2-0
- Categories: Long-horizon
- Address: Online, Global, United States
- Website: <https://livebench.ai/>
- Updated: 2026-10-21

### [LiveCodeBench](https://aibenchmarkindex.com/global/online/livecodebench.md)

Tests code generation, execution, self-repair, and test-output prediction with newly released contest problems.

- Page: https://aibenchmarkindex.com/global/online/livecodebench
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/LiveCodeBench/LiveCodeBench>
- Updated: 2026-10-21

### [LiveCodeBench v5](https://aibenchmarkindex.com/global/online/livecodebench-v5.md)

Tests code capabilities on the LiveCodeBench release containing contest problems through January 2025.

- Page: https://aibenchmarkindex.com/global/online/livecodebench-v5
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://livecodebench.github.io/>
- Updated: 2026-10-21

### [LoCoMo](https://aibenchmarkindex.com/global/online/locomo.md)

Evaluates very long-term conversational memory through question answering, event summarization, and dialogue generation.

- Page: https://aibenchmarkindex.com/global/online/locomo
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://github.com/snap-research/locomo>
- Updated: 2026-10-21

### [LongBench](https://aibenchmarkindex.com/global/online/longbench.md)

Evaluates bilingual long-context understanding across single- and multi-document question answering, summarization, few-shot learning, and code.

- Page: https://aibenchmarkindex.com/global/online/longbench
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://longbench.github.io/>
- Updated: 2026-10-21

### [LongBench v2](https://aibenchmarkindex.com/global/online/longbench-v2.md)

Tests deep reasoning over realistic contexts from 8,000 to two million words using difficult multiple-choice questions.

- Page: https://aibenchmarkindex.com/global/online/longbench-v2
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://longbench2.github.io/>
- Updated: 2026-10-21

### [LongVideoBench](https://aibenchmarkindex.com/global/online/longvideobench.md)

Tests understanding of hour-long videos whose questions require reasoning across interleaved frames, subtitles, and long contexts.

- Page: https://aibenchmarkindex.com/global/online/longvideobench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://longvideobench.github.io/>
- Updated: 2026-10-21

### [LooGLE](https://aibenchmarkindex.com/global/online/loogle.md)

Tests short- and long-dependency understanding over recent documents longer than 24,000 tokens.

- Page: https://aibenchmarkindex.com/global/online/loogle
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://github.com/bigai-nlco/LooGLE>
- Updated: 2026-10-21

### [ManiSkill](https://aibenchmarkindex.com/global/online/maniskill.md)

Measures generalizable robotic manipulation policies in scalable simulated environments.

- Page: https://aibenchmarkindex.com/global/online/maniskill
- Categories: Science
- Address: Online, Global, United States
- Website: <https://maniskill.readthedocs.io/>
- Updated: 2026-10-21

### [Matbench](https://aibenchmarkindex.com/global/online/matbench.md)

Evaluates machine-learning models on 13 curated materials-property prediction tasks.

- Page: https://aibenchmarkindex.com/global/online/matbench
- Categories: Science
- Address: Online, Global, United States
- Website: <https://matbench.materialsproject.org/>
- Updated: 2026-10-21

### [Matbench Discovery](https://aibenchmarkindex.com/global/online/matbench-discovery.md)

Compares the accuracy, robustness, and computational cost of models used for atomistic and materials-discovery tasks.

- Page: https://aibenchmarkindex.com/global/online/matbench-discovery
- Categories: Science
- Address: Online, Global, United States
- Website: <https://matbench-discovery.materialsproject.org/>
- Updated: 2026-11-07

### [MATH](https://aibenchmarkindex.com/global/online/math.md)

Tests competition-level mathematical problem solving across algebra, counting, geometry, number theory, and related subjects.

- Page: https://aibenchmarkindex.com/global/online/math
- Categories: Math
- Address: Online, Global, United States
- Website: <https://github.com/hendrycks/math>
- Updated: 2026-10-21

### [MATH-500](https://aibenchmarkindex.com/global/online/math-500.md)

Tests mathematical reasoning on 500 held-out problems sampled from the MATH test split.

- Page: https://aibenchmarkindex.com/global/online/math-500
- Categories: Math
- Address: Online, Global, United States
- Website: <https://github.com/openai/prm800k>
- Updated: 2026-10-21

### [MATH-Vision](https://aibenchmarkindex.com/global/online/math-vision.md)

Tests visual mathematical reasoning on competition problems spanning 16 subjects and five difficulty levels.

- Page: https://aibenchmarkindex.com/global/online/math-vision
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/mathvision-cuhk/MathVision>
- Updated: 2026-10-21

### [MathVista](https://aibenchmarkindex.com/global/online/mathvista.md)

Tests mathematical reasoning grounded in diagrams and other visual contexts across diverse task types.

- Page: https://aibenchmarkindex.com/global/online/mathvista
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/lupantech/MathVista>
- Updated: 2026-10-21

### [MBPP](https://aibenchmarkindex.com/global/online/mbpp.md)

Tests basic Python synthesis with crowd-sourced prompts, reference solutions, and automated test cases.

- Page: https://aibenchmarkindex.com/global/online/mbpp
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/google-research/google-research/tree/master/mbpp>
- Updated: 2026-10-21

### [MedMCQA](https://aibenchmarkindex.com/global/online/medmcqa.md)

Evaluates medical multiple-choice reasoning on questions drawn from Indian medical entrance examinations.

- Page: https://aibenchmarkindex.com/global/online/medmcqa
- Categories: Science
- Address: Online, Global, United States
- Website: <https://medmcqa.github.io/>
- Updated: 2026-10-21

### [MedQA](https://aibenchmarkindex.com/global/online/medqa.md)

Evaluates medical knowledge and reasoning with multiple-choice questions from professional medical licensing exams.

- Page: https://aibenchmarkindex.com/global/online/medqa
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/jind11/MedQA>
- Updated: 2026-10-21

### [MemBench](https://aibenchmarkindex.com/global/online/membench.md)

Evaluates agent memory effectiveness, efficiency, and capacity across factual and reflective memory in participation and observation settings.

- Page: https://aibenchmarkindex.com/global/online/membench
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://github.com/import-myself/Membench>
- Updated: 2026-10-21

### [Meta-World](https://aibenchmarkindex.com/global/online/meta-world.md)

Measures multi-task and meta-reinforcement learning across robotic manipulation tasks and goal variations.

- Page: https://aibenchmarkindex.com/global/online/meta-world
- Categories: Science
- Address: Online, Global, United States
- Website: <https://metaworld.farama.org/>
- Updated: 2026-10-21

### [MGSM](https://aibenchmarkindex.com/global/online/mgsm.md)

Evaluates grade-school mathematical reasoning on the same 250 problems translated by human annotators into 10 languages.

- Page: https://aibenchmarkindex.com/global/online/mgsm
- Categories: Math
- Address: Online, Global, United States
- Website: <https://github.com/google-research/url-nlp/tree/main/mgsm>
- Updated: 2026-10-21

### [Mind2Web](https://aibenchmarkindex.com/global/online/mind2web.md)

Measures whether web agents can follow open-ended instructions across real websites and unfamiliar domains.

- Page: https://aibenchmarkindex.com/global/online/mind2web
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://osu-nlp-group.github.io/Mind2Web/>
- Updated: 2026-10-21

### [MineDojo](https://aibenchmarkindex.com/global/online/minedojo.md)

Measures open-ended embodied agents across thousands of Minecraft tasks with an internet-scale knowledge base.

- Page: https://aibenchmarkindex.com/global/online/minedojo
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://minedojo.org/>
- Updated: 2026-10-21

### [miniF2F](https://aibenchmarkindex.com/global/online/minif2f.md)

Tests automated theorem provers on olympiad and classroom mathematics translated across formal systems.

- Page: https://aibenchmarkindex.com/global/online/minif2f
- Categories: Math
- Address: Online, Global, United States
- Website: <https://github.com/facebookresearch/miniF2F>
- Updated: 2026-10-21

### [MiniWoB++](https://aibenchmarkindex.com/global/online/miniwob.md)

Measures reinforcement-learning agents on a collection of simple browser interaction environments.

- Page: https://aibenchmarkindex.com/global/online/miniwob
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://miniwob.farama.org/>
- Updated: 2026-10-21

### [MIRACL](https://aibenchmarkindex.com/global/online/miracl.md)

Evaluates monolingual information retrieval across 18 languages using native-speaker relevance judgments.

- Page: https://aibenchmarkindex.com/global/online/miracl
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://project-miracl.github.io/>
- Updated: 2026-11-07

### [MLPerf Inference](https://aibenchmarkindex.com/global/online/mlperf-inference.md)

Measures how quickly and efficiently hardware and software systems run trained models across deployment scenarios.

- Page: https://aibenchmarkindex.com/global/online/mlperf-inference
- Categories: Other
- Address: Online, Global, United States
- Website: <https://github.com/mlcommons/inference>
- Updated: 2026-10-21

### [MLPerf Tiny](https://aibenchmarkindex.com/global/online/mlperf-tiny.md)

Measures latency and energy for embedded-device inference on standardized tiny-machine-learning workloads.

- Page: https://aibenchmarkindex.com/global/online/mlperf-tiny
- Categories: Other
- Address: Online, Global, United States
- Website: <https://github.com/mlcommons/tiny>
- Updated: 2026-10-21

### [MLPerf Training](https://aibenchmarkindex.com/global/online/mlperf-training.md)

Measures time to train reference machine-learning workloads to defined quality targets on different systems.

- Page: https://aibenchmarkindex.com/global/online/mlperf-training
- Categories: Other
- Address: Online, Global, United States
- Website: <https://github.com/mlcommons/training>
- Updated: 2026-10-21

### [MLVU](https://aibenchmarkindex.com/global/online/mlvu.md)

Tests long-video understanding through multiple-choice and free-form tasks requiring local and global comprehension.

- Page: https://aibenchmarkindex.com/global/online/mlvu
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/JUNJIE99/MLVU>
- Updated: 2026-10-21

### [MM-Vet](https://aibenchmarkindex.com/global/online/mm-vet.md)

Measures how well multimodal models integrate recognition, OCR, knowledge, language, spatial, and mathematical capabilities.

- Page: https://aibenchmarkindex.com/global/online/mm-vet
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/yuweihao/MM-Vet>
- Updated: 2026-10-21

### [MMBench](https://aibenchmarkindex.com/global/online/mmbench.md)

Measures broad multimodal model capabilities with multiple-choice questions spanning 20 ability dimensions.

- Page: https://aibenchmarkindex.com/global/online/mmbench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/open-compass/MMBench>
- Updated: 2026-10-21

### [MME](https://aibenchmarkindex.com/global/online/mme.md)

Measures perception and cognition in multimodal language models across 14 manually designed subtasks.

- Page: https://aibenchmarkindex.com/global/online/mme
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation>
- Updated: 2026-10-21

### [MMLU](https://aibenchmarkindex.com/global/online/mmlu.md)

Tests broad academic knowledge and reasoning with multiple-choice questions drawn from 57 subjects.

- Page: https://aibenchmarkindex.com/global/online/mmlu
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/hendrycks/test>
- Updated: 2026-10-21

### [MMLU Medical Subjects](https://aibenchmarkindex.com/global/online/mmlu-medical-subjects.md)

Evaluates factual knowledge and problem solving across the medical and biomedical subject sets in MMLU.

- Page: https://aibenchmarkindex.com/global/online/mmlu-medical-subjects
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/hendrycks/test>
- Updated: 2026-10-21

### [MMLU-Pro](https://aibenchmarkindex.com/global/online/mmlu-pro.md)

Tests knowledge and reasoning on harder ten-option questions with reduced reliance on recall.

- Page: https://aibenchmarkindex.com/global/online/mmlu-pro
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro>
- Updated: 2026-10-21

### [MMMU](https://aibenchmarkindex.com/global/online/mmmu.md)

Measures expert-level multimodal reasoning across college disciplines, subjects, and heterogeneous image types.

- Page: https://aibenchmarkindex.com/global/online/mmmu
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://mmmu-benchmark.github.io/>
- Updated: 2026-11-07

### [MMMU-Pro](https://aibenchmarkindex.com/global/online/mmmu-pro.md)

Measures robust expert-level multimodal reasoning after filtering text-solvable items and strengthening answer choices.

- Page: https://aibenchmarkindex.com/global/online/mmmu-pro
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://mmmu-benchmark.github.io/>
- Updated: 2026-11-07

### [MMStar](https://aibenchmarkindex.com/global/online/mmstar.md)

Measures vision-essential multimodal capabilities using human-selected samples filtered to resist text-only solving and leakage.

- Page: https://aibenchmarkindex.com/global/online/mmstar
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/MMStar-Benchmark/MMStar>
- Updated: 2026-10-21

### [MMTEB](https://aibenchmarkindex.com/global/online/mmteb.md)

Measures multilingual text embeddings across more than 500 quality-controlled tasks covering over 250 languages.

- Page: https://aibenchmarkindex.com/global/online/mmteb
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/embeddings-benchmark/mteb>
- Updated: 2026-10-21

### [Mobile-Bench](https://aibenchmarkindex.com/global/online/mobile-bench.md)

Measures mobile agents on instructions that require perception, planning, and actions in Android applications.

- Page: https://aibenchmarkindex.com/global/online/mobile-bench
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/XiaoMi/MobileBench>
- Updated: 2026-10-21

### [Mol-Instructions](https://aibenchmarkindex.com/global/online/mol-instructions.md)

Evaluates instruction-following across molecule description, property prediction, reaction, and biomolecular tasks.

- Page: https://aibenchmarkindex.com/global/online/mol-instructions
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/zjunlp/Mol-Instructions>
- Updated: 2026-10-21

### [MS MARCO](https://aibenchmarkindex.com/global/online/ms-marco.md)

Evaluates machine reading, answer generation, and passage or document ranking using questions sampled from Bing search.

- Page: https://aibenchmarkindex.com/global/online/ms-marco
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://microsoft.github.io/msmarco/>
- Updated: 2026-10-21

### [MT-Bench](https://aibenchmarkindex.com/global/online/mt-bench.md)

Tests conversational quality on challenging multi-turn, open-ended questions scored by model judges.

- Page: https://aibenchmarkindex.com/global/online/mt-bench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/lm-sys/FastChat>
- Updated: 2026-10-21

### [MTEB](https://aibenchmarkindex.com/global/online/mteb.md)

Measures text embeddings across classification, clustering, retrieval, reranking, similarity, and summarization tasks.

- Page: https://aibenchmarkindex.com/global/online/mteb
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/embeddings-benchmark/mteb>
- Updated: 2026-10-21

### [MultiFieldQA](https://aibenchmarkindex.com/global/online/multifieldqa.md)

Tests single-document and multi-document question answering over long English and Chinese contexts within LongBench.

- Page: https://aibenchmarkindex.com/global/online/multifieldqa
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://longbench.github.io/>
- Updated: 2026-10-21

### [MultiMedQA](https://aibenchmarkindex.com/global/online/multimedqa.md)

Evaluates medical question answering across seven datasets spanning professional exams, research, and consumer health queries.

- Page: https://aibenchmarkindex.com/global/online/multimedqa
- Categories: Science
- Address: Online, Global, United States
- Website: <https://arxiv.org/abs/2212.13138>
- Updated: 2026-10-21

### [MUSDB18](https://aibenchmarkindex.com/global/online/musdb18.md)

Evaluates music source-separation systems on professionally produced songs with isolated stems.

- Page: https://aibenchmarkindex.com/global/online/musdb18
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://sigsep.github.io/datasets/musdb.html>
- Updated: 2026-10-21

### [MVBench](https://aibenchmarkindex.com/global/online/mvbench.md)

Measures multi-modal video understanding across 20 temporal and perceptual tasks.

- Page: https://aibenchmarkindex.com/global/online/mvbench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/OpenGVLab/Ask-Anything>
- Updated: 2026-10-21

### [Natural Questions](https://aibenchmarkindex.com/global/online/natural-questions.md)

Evaluates question answering on real Google queries paired with annotated evidence and answers from Wikipedia pages.

- Page: https://aibenchmarkindex.com/global/online/natural-questions
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://ai.google.com/research/NaturalQuestions>
- Updated: 2026-10-21

### [Needle In A Haystack](https://aibenchmarkindex.com/global/online/needle-in-a-haystack.md)

Tests whether a language model retrieves a planted fact from increasingly long contexts and at different insertion depths.

- Page: https://aibenchmarkindex.com/global/online/needle-in-a-haystack
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://github.com/gkamradt/needle-in-a-haystack>
- Updated: 2026-10-21

### [NExT-QA](https://aibenchmarkindex.com/global/online/next-qa.md)

Evaluates causal and temporal reasoning in video question answering rather than simple visual description.

- Page: https://aibenchmarkindex.com/global/online/next-qa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/doc-doc/NExT-QA>
- Updated: 2026-10-21

### [ObjectNet](https://aibenchmarkindex.com/global/online/objectnet.md)

Tests object recognition on 50,000 crowd-sourced images captured with controls over backgrounds, rotations, and viewpoints.

- Page: https://aibenchmarkindex.com/global/online/objectnet
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://objectnet.dev/>
- Updated: 2026-10-21

### [OCRBench](https://aibenchmarkindex.com/global/online/ocrbench.md)

Measures OCR in multimodal models across text recognition, scene-text VQA, document VQA, extraction, and handwritten math.

- Page: https://aibenchmarkindex.com/global/online/ocrbench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/Yuliang-Liu/MultimodalOCR/tree/main/OCRBench>
- Updated: 2026-10-21

### [OCRBench v2](https://aibenchmarkindex.com/global/online/ocrbench-v2.md)

Measures bilingual visual-text localization, recognition, extraction, and reasoning across diverse real-world scenarios.

- Page: https://aibenchmarkindex.com/global/online/ocrbench-v2
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/Yuliang-Liu/MultimodalOCR/tree/main/OCRBench_v2>
- Updated: 2026-10-21

### [OK-VQA](https://aibenchmarkindex.com/global/online/ok-vqa.md)

Measures visual question answering that requires knowledge beyond the content visible in an image.

- Page: https://aibenchmarkindex.com/global/online/ok-vqa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://okvqa.allenai.org/>
- Updated: 2026-10-21

### [OlympiadBench](https://aibenchmarkindex.com/global/online/olympiadbench.md)

Tests bilingual, multimodal problem solving on olympiad-level mathematics and physics questions.

- Page: https://aibenchmarkindex.com/global/online/olympiadbench
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/OpenBMB/OlympiadBench>
- Updated: 2026-10-21

### [Open LLM Leaderboard](https://aibenchmarkindex.com/global/online/open-llm-leaderboard.md)

Tracks and compares openly submitted language models across a changing suite of reasoning and knowledge evaluations.

- Page: https://aibenchmarkindex.com/global/online/open-llm-leaderboard
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard>
- Updated: 2026-10-21

### [OSWorld-Verified](https://aibenchmarkindex.com/global/online/osworld-verified.md)

Measures multimodal agents on reproducible, open-ended tasks in real desktop applications; the verified release repairs the original task set.

- Page: https://aibenchmarkindex.com/global/online/osworld-verified
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://os-world.github.io/>
- Updated: 2026-10-21

### [PartiPrompts](https://aibenchmarkindex.com/global/online/partiprompts.md)

Tests text-to-image systems with over 1,600 prompts spanning simple scenes, complex compositions, writing, and abstract concepts.

- Page: https://aibenchmarkindex.com/global/online/partiprompts
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/google-research/parti>
- Updated: 2026-10-21

### [PokeLLMon](https://aibenchmarkindex.com/global/online/pokellmon.md)

Measures a language-model agent on tactical decision making in competitive Pokémon battles.

- Page: https://aibenchmarkindex.com/global/online/pokellmon
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/git-disl/PokeLLMon>
- Updated: 2026-10-21

### [PokerBench](https://aibenchmarkindex.com/global/online/pokerbench.md)

Evaluates whether language models choose solver-backed actions in 11,000 pre-flop and post-flop poker scenarios.

- Page: https://aibenchmarkindex.com/global/online/pokerbench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/pokerllm/pokerbench>
- Updated: 2026-10-21

### [Procgen Benchmark](https://aibenchmarkindex.com/global/online/procgen-benchmark.md)

Measures sample efficiency and generalization in reinforcement learning across procedurally generated game environments.

- Page: https://aibenchmarkindex.com/global/online/procgen-benchmark
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/openai/procgen>
- Updated: 2026-10-21

### [ProofNet](https://aibenchmarkindex.com/global/online/proofnet.md)

Tests autoformalization and formal proving of undergraduate mathematics in Lean from natural-language statements.

- Page: https://aibenchmarkindex.com/global/online/proofnet
- Categories: Math
- Address: Online, Global, United States
- Website: <https://github.com/zhangir-azerbayev/ProofNet>
- Updated: 2026-10-21

### [PubMedQA](https://aibenchmarkindex.com/global/online/pubmedqa.md)

Evaluates biomedical research question answering from PubMed abstracts, including an expert-annotated test set.

- Page: https://aibenchmarkindex.com/global/online/pubmedqa
- Categories: Science
- Address: Online, Global, United States
- Website: <https://pubmedqa.github.io/>
- Updated: 2026-10-21

### [RAGChecker](https://aibenchmarkindex.com/global/online/ragchecker.md)

Diagnoses retrieval-augmented generation systems with claim-level metrics for their retrieval and generation components.

- Page: https://aibenchmarkindex.com/global/online/ragchecker
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/amazon-science/RAGChecker>
- Updated: 2026-10-21

### [RAGTruth](https://aibenchmarkindex.com/global/online/ragtruth.md)

Evaluates hallucination detection in retrieval-augmented answers using human annotations at response and word levels.

- Page: https://aibenchmarkindex.com/global/online/ragtruth
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/ParticleMedia/RAGTruth>
- Updated: 2026-10-21

### [RealToxicityPrompts](https://aibenchmarkindex.com/global/online/realtoxicityprompts.md)

Measures whether language models continue naturally occurring web prompts with toxic text.

- Page: https://aibenchmarkindex.com/global/online/realtoxicityprompts
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/allenai/real-toxicity-prompts>
- Updated: 2026-10-21

### [RepoBench](https://aibenchmarkindex.com/global/online/repobench.md)

Tests repository-level code completion using cross-file dependencies in Python and Java projects.

- Page: https://aibenchmarkindex.com/global/online/repobench
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/Leolty/repobench>
- Updated: 2026-10-21

### [RepoQA](https://aibenchmarkindex.com/global/online/repoqa.md)

Tests long-context code retrieval by asking models to find described functions inside full repositories.

- Page: https://aibenchmarkindex.com/global/online/repoqa
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/evalplus/repoqa>
- Updated: 2026-10-21

### [RLBench](https://aibenchmarkindex.com/global/online/rlbench.md)

Measures robot learning across vision-guided manipulation tasks in a simulated environment.

- Page: https://aibenchmarkindex.com/global/online/rlbench
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/stepjam/RLBench>
- Updated: 2026-10-21

### [RoboCasa](https://aibenchmarkindex.com/global/online/robocasa.md)

Evaluates robot policies on everyday kitchen manipulation tasks in a large-scale simulation framework.

- Page: https://aibenchmarkindex.com/global/online/robocasa
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/robocasa/robocasa>
- Updated: 2026-10-21

### [robomimic](https://aibenchmarkindex.com/global/online/robomimic.md)

Measures robot policies learned from offline human demonstrations across standardized datasets and tasks.

- Page: https://aibenchmarkindex.com/global/online/robomimic
- Categories: Science
- Address: Online, Global, United States
- Website: <https://robomimic.github.io/>
- Updated: 2026-10-21

### [robosuite](https://aibenchmarkindex.com/global/online/robosuite.md)

Provides standardized simulated robot manipulation environments for evaluating learning methods.

- Page: https://aibenchmarkindex.com/global/online/robosuite
- Categories: Science
- Address: Online, Global, United States
- Website: <https://robosuite.ai/>
- Updated: 2026-10-21

### [ROUGE](https://aibenchmarkindex.com/global/online/rouge.md)

Scores automatic summaries against references using n-gram overlap and longest-common-subsequence variants.

- Page: https://aibenchmarkindex.com/global/online/rouge
- Categories: Other
- Address: Online, Global, United States
- Website: <https://github.com/google-research/google-research/tree/master/rouge>
- Updated: 2026-10-21

### [RULER](https://aibenchmarkindex.com/global/online/ruler.md)

Measures effective context length with configurable synthetic tasks that go beyond simple needle retrieval.

- Page: https://aibenchmarkindex.com/global/online/ruler
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://github.com/NVIDIA/RULER>
- Updated: 2026-10-21

### [SacreBLEU](https://aibenchmarkindex.com/global/online/sacrebleu.md)

Computes reproducible machine-translation metrics while recording a signature for dataset and tokenization settings.

- Page: https://aibenchmarkindex.com/global/online/sacrebleu
- Categories: Other
- Address: Online, Global, United States
- Website: <https://github.com/mjpost/sacrebleu>
- Updated: 2026-10-21

### [SafeBench](https://aibenchmarkindex.com/global/online/safebench.md)

Evaluates autonomous-driving agents in safety-critical simulated scenarios, including adversarially generated cases.

- Page: https://aibenchmarkindex.com/global/online/safebench
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://safebench.github.io/>
- Updated: 2026-10-21

### [SafetyBench](https://aibenchmarkindex.com/global/online/safetybench.md)

Evaluates large-language-model safety knowledge with Chinese and English multiple-choice questions across seven risk categories.

- Page: https://aibenchmarkindex.com/global/online/safetybench
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://llmbench.ai/safety>
- Updated: 2026-11-07

### [Scale SEAL Leaderboards](https://aibenchmarkindex.com/global/online/scale-seal-leaderboards.md)

Compares frontier models on expert-designed evaluations built from private datasets intended to resist benchmark gaming.

- Page: https://aibenchmarkindex.com/global/online/scale-seal-leaderboards
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://labs.scale.com/leaderboard>
- Updated: 2026-11-07

### [SciBench](https://aibenchmarkindex.com/global/online/scibench.md)

Evaluates college-level scientific reasoning, domain knowledge, and calculation on textbook problems.

- Page: https://aibenchmarkindex.com/global/online/scibench
- Categories: Science
- Address: Online, Global, United States
- Website: <https://scibench-ucla.github.io/>
- Updated: 2026-10-21

### [ScienceWorld](https://aibenchmarkindex.com/global/online/scienceworld.md)

Measures text-based agents on tasks grounded in an elementary science curriculum.

- Page: https://aibenchmarkindex.com/global/online/scienceworld
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/allenai/ScienceWorld>
- Updated: 2026-10-21

### [SciFact](https://aibenchmarkindex.com/global/online/scifact.md)

Tests retrieval of scientific evidence and classification of research claims as supported or refuted.

- Page: https://aibenchmarkindex.com/global/online/scifact
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/allenai/scifact>
- Updated: 2026-10-21

### [SEED-Bench](https://aibenchmarkindex.com/global/online/seed-bench.md)

Measures image and video comprehension across multimodal ability dimensions using human-annotated multiple-choice questions.

- Page: https://aibenchmarkindex.com/global/online/seed-bench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/AILab-CVC/SEED-Bench>
- Updated: 2026-10-21

### [SimpleBench](https://aibenchmarkindex.com/global/online/simplebench.md)

Tests everyday reasoning about space, time, social situations, and adversarial wording with multiple-choice questions.

- Page: https://aibenchmarkindex.com/global/online/simplebench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://simple-bench.com/>
- Updated: 2026-11-07

### [SmartPlay](https://aibenchmarkindex.com/global/online/smartplay.md)

Measures agent reasoning, planning, spatial understanding, learning, and handling of randomness across six games.

- Page: https://aibenchmarkindex.com/global/online/smartplay
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/microsoft/SmartPlay>
- Updated: 2026-10-21

### [Something-Something V2](https://aibenchmarkindex.com/global/online/something-something-v2.md)

Tests recognition of everyday actions whose labels depend on temporal and physical interactions between people and objects.

- Page: https://aibenchmarkindex.com/global/online/something-something-v2
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://developer.qualcomm.com/software/ai-datasets/something-something>
- Updated: 2026-10-21

### [SORRY-Bench](https://aibenchmarkindex.com/global/online/sorry-bench.md)

Measures the safety-refusal behavior of language models across a balanced taxonomy of unsafe requests and prompt mutations.

- Page: https://aibenchmarkindex.com/global/online/sorry-bench
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://sorry-bench.github.io/>
- Updated: 2026-10-21

### [Spider](https://aibenchmarkindex.com/global/online/spider.md)

Tests cross-domain text-to-SQL generalization to unseen database schemas and complex queries.

- Page: https://aibenchmarkindex.com/global/online/spider
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://yale-lily.github.io/spider>
- Updated: 2026-10-21

### [Spider 2.0](https://aibenchmarkindex.com/global/online/spider-2-0.md)

Tests agents on realistic enterprise text-to-SQL workflows with long contexts and cloud or local databases.

- Page: https://aibenchmarkindex.com/global/online/spider-2-0
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://spider2-sql.github.io/>
- Updated: 2026-11-07

### [SQuAD 2.0](https://aibenchmarkindex.com/global/online/squad-2-0.md)

Tests extractive reading comprehension and whether a system abstains when a Wikipedia passage does not answer the question.

- Page: https://aibenchmarkindex.com/global/online/squad-2-0
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://rajpurkar.github.io/SQuAD-explorer/>
- Updated: 2026-10-21

### [StereoSet](https://aibenchmarkindex.com/global/online/stereoset.md)

Measures stereotypical bias and language-modeling ability in sentence-level and discourse-level contexts.

- Page: https://aibenchmarkindex.com/global/online/stereoset
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/moinnadeem/StereoSet>
- Updated: 2026-10-21

### [StrongREJECT](https://aibenchmarkindex.com/global/online/strongreject.md)

Evaluates jailbreak responses with a forbidden-prompt dataset and an automated judge designed to penalize empty refusals and ineffective attacks.

- Page: https://aibenchmarkindex.com/global/online/strongreject
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://strong-reject.github.io/>
- Updated: 2026-10-21

### [SUPERB](https://aibenchmarkindex.com/global/online/superb.md)

Evaluates general-purpose speech representations across recognition, speaker, semantic, and paralinguistic downstream tasks.

- Page: https://aibenchmarkindex.com/global/online/superb
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://superbbenchmark.org/>
- Updated: 2026-10-21

### [SVGBench](https://aibenchmarkindex.com/global/online/svgbench.md)

Tests whether language models can generate SVGs that satisfy requirements involving knowledge, code, instructions, and physical reasoning.

- Page: https://aibenchmarkindex.com/global/online/svgbench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/johnbean393/SVGBench>
- Updated: 2026-10-21

### [SWE-bench](https://aibenchmarkindex.com/global/online/swe-bench.md)

Tests whether coding systems can resolve real GitHub issues in complete software repositories.

- Page: https://aibenchmarkindex.com/global/online/swe-bench
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/SWE-bench/SWE-bench>
- Updated: 2026-10-21

### [SWE-bench Verified](https://aibenchmarkindex.com/global/online/swe-bench-verified.md)

Tests repository-level issue resolution on a human-validated subset of 500 solvable SWE-bench tasks.

- Page: https://aibenchmarkindex.com/global/online/swe-bench-verified
- Categories: Coding
- Address: Online, Global, United States
- Website: <https://github.com/SWE-bench/SWE-bench>
- Updated: 2026-10-21

### [TasteBench](https://aibenchmarkindex.com/global/online/tastebench.md)

Studies whether aesthetic-preference benchmarks reward expert-aligned craft judgment rather than shortcuts; its item pool remains pending licence clearance.

- Page: https://aibenchmarkindex.com/global/online/tastebench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/chancevision/tastebench>
- Updated: 2026-10-21

### [TAT-QA](https://aibenchmarkindex.com/global/online/tat-qa.md)

Evaluates numerical question answering over combined tables and prose from financial reports.

- Page: https://aibenchmarkindex.com/global/online/tat-qa
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://nextplusplus.github.io/TAT-QA/>
- Updated: 2026-10-21

### [tau-bench](https://aibenchmarkindex.com/global/online/tau-bench.md)

Measures customer-service agents through policy-constrained conversations and database tool use.

- Page: https://aibenchmarkindex.com/global/online/tau-bench
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://github.com/sierra-research/tau-bench>
- Updated: 2026-10-21

### [tau2-bench](https://aibenchmarkindex.com/global/online/tau2-bench.md)

Measures conversational agents when both the user and agent can change shared task state.

- Page: https://aibenchmarkindex.com/global/online/tau2-bench
- Categories: Tools
- Address: Online, Global, United States
- Website: <https://github.com/sierra-research/tau2-bench>
- Updated: 2026-10-21

### [Terminal-Bench](https://aibenchmarkindex.com/global/online/terminal-bench.md)

Tests whether agents complete end-to-end tasks inside reproducible terminal environments.

- Page: https://aibenchmarkindex.com/global/online/terminal-bench
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/harbor-framework/terminal-bench-1>
- Updated: 2026-10-21

### [Terminal-Bench 2.0](https://aibenchmarkindex.com/global/online/terminal-bench-2-0.md)

Tests agents on valuable containerized work such as debugging, security fixes, and scientific workflows.

- Page: https://aibenchmarkindex.com/global/online/terminal-bench-2-0
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/harbor-framework/terminal-bench-2>
- Updated: 2026-10-21

### [TextVQA](https://aibenchmarkindex.com/global/online/textvqa.md)

Measures visual question answering that requires reading and reasoning about text in natural images.

- Page: https://aibenchmarkindex.com/global/online/textvqa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://textvqa.org/>
- Updated: 2026-10-21

### [TextWorld](https://aibenchmarkindex.com/global/online/textworld.md)

Measures reinforcement-learning agents in generated text-based games with controllable difficulty and scope.

- Page: https://aibenchmarkindex.com/global/online/textworld
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://www.microsoft.com/en-us/research/project/textworld/>
- Updated: 2026-10-21

### [ToxiGen](https://aibenchmarkindex.com/global/online/toxigen.md)

Tests hate-speech classifiers on machine-generated implicit toxic and benign statements about 13 minority groups.

- Page: https://aibenchmarkindex.com/global/online/toxigen
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/microsoft/ToxiGen>
- Updated: 2026-10-21

### [TREC Deep Learning Track](https://aibenchmarkindex.com/global/online/trec-deep-learning-track.md)

Evaluates passage and document ranking systems on large web collections with graded relevance judgments.

- Page: https://aibenchmarkindex.com/global/online/trec-deep-learning-track
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://microsoft.github.io/msmarco/>
- Updated: 2026-10-21

### [TREC Question Answering Track](https://aibenchmarkindex.com/global/online/trec-question-answering-track.md)

Evaluates systems that return concise answers to fact-based questions from large text collections.

- Page: https://aibenchmarkindex.com/global/online/trec-question-answering-track
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://trec.nist.gov/>
- Updated: 2026-10-21

### [TriviaQA](https://aibenchmarkindex.com/global/online/triviaqa.md)

Tests reading comprehension and open-domain question answering with trivia questions paired to independently gathered evidence.

- Page: https://aibenchmarkindex.com/global/online/triviaqa
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/mandarjoshi90/triviaqa>
- Updated: 2026-10-21

### [TruthfulQA](https://aibenchmarkindex.com/global/online/truthfulqa.md)

Tests whether language models reproduce common human misconceptions and falsehoods when answering questions.

- Page: https://aibenchmarkindex.com/global/online/truthfulqa
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/sylinrl/TruthfulQA>
- Updated: 2026-10-21

### [TyDi QA](https://aibenchmarkindex.com/global/online/tydi-qa.md)

Evaluates information-seeking question answering across 11 typologically diverse languages.

- Page: https://aibenchmarkindex.com/global/online/tydi-qa
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/google-research-datasets/tydiqa>
- Updated: 2026-10-21

### [Vendi Score](https://aibenchmarkindex.com/global/online/vendi-score.md)

Measures diversity in a collection from pairwise similarities using the effective rank of a similarity matrix.

- Page: https://aibenchmarkindex.com/global/online/vendi-score
- Categories: Science
- Address: Online, Global, United States
- Website: <https://github.com/vertaix/Vendi-Score>
- Updated: 2026-10-21

### [Video-Bench](https://aibenchmarkindex.com/global/online/video-bench.md)

Evaluates video language models on perception, prior-knowledge use, and video-based decision making.

- Page: https://aibenchmarkindex.com/global/online/video-bench
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/PKU-YuanGroup/Video-Bench>
- Updated: 2026-10-21

### [Video-MME](https://aibenchmarkindex.com/global/online/video-mme.md)

Evaluates multimodal models on short, medium, and long videos across visual domains, durations, and subtitle settings.

- Page: https://aibenchmarkindex.com/global/online/video-mme
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://video-mme.github.io/>
- Updated: 2026-10-21

### [Video-MMMU](https://aibenchmarkindex.com/global/online/video-mmmu.md)

Measures whether multimodal models acquire and apply new knowledge from professional lecture videos.

- Page: https://aibenchmarkindex.com/global/online/video-mmmu
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://videommmu.github.io/>
- Updated: 2026-10-21

### [VirtualHome](https://aibenchmarkindex.com/global/online/virtualhome.md)

Evaluates generation and execution of household activity programs in simulated 3D homes.

- Page: https://aibenchmarkindex.com/global/online/virtualhome
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://github.com/xavierpuigf/virtualhome>
- Updated: 2026-10-21

### [VisualWebArena](https://aibenchmarkindex.com/global/online/visualwebarena.md)

Tests multimodal web agents on visual tasks in realistic, reproducible browser environments.

- Page: https://aibenchmarkindex.com/global/online/visualwebarena
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://github.com/web-arena-x/visualwebarena>
- Updated: 2026-10-21

### [VizWiz VQA](https://aibenchmarkindex.com/global/online/vizwiz-vqa.md)

Measures answers to visual questions submitted by blind people in real accessibility scenarios.

- Page: https://aibenchmarkindex.com/global/online/vizwiz-vqa
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://vizwiz.org/tasks-and-datasets/vqa/>
- Updated: 2026-10-21

### [VoxCeleb](https://aibenchmarkindex.com/global/online/voxceleb.md)

Evaluates speaker identification and verification using speech collected from interview videos.

- Page: https://aibenchmarkindex.com/global/online/voxceleb
- Categories: Voice
- Address: Online, Global, United States
- Website: <https://www.robots.ox.ac.uk/~vgg/data/voxceleb/>
- Updated: 2026-11-07

### [VQAv2](https://aibenchmarkindex.com/global/online/vqav2.md)

Measures visual question answering while reducing language-only shortcuts through complementary image pairs.

- Page: https://aibenchmarkindex.com/global/online/vqav2
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://visualqa.org/>
- Updated: 2026-11-07

### [We-Math](https://aibenchmarkindex.com/global/online/we-math.md)

Tests whether multimodal models solve visual math problems through coherent one-step and multi-step reasoning.

- Page: https://aibenchmarkindex.com/global/online/we-math
- Categories: Multimodal
- Address: Online, Global, United States
- Website: <https://github.com/We-Math/We-Math>
- Updated: 2026-10-21

### [WebArena](https://aibenchmarkindex.com/global/online/webarena.md)

Tests autonomous web agents on realistic tasks in reproducible websites with execution-based evaluation.

- Page: https://aibenchmarkindex.com/global/online/webarena
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://github.com/web-arena-x/webarena>
- Updated: 2026-10-21

### [WebShop](https://aibenchmarkindex.com/global/online/webshop.md)

Measures agents that search, customize, and purchase products in a simulated e-commerce website from natural-language instructions.

- Page: https://aibenchmarkindex.com/global/online/webshop
- Categories: Commerce
- Address: Online, Global, United States
- Website: <https://webshop-pnlp.github.io/>
- Updated: 2026-10-21

### [WebVoyager](https://aibenchmarkindex.com/global/online/webvoyager.md)

Measures end-to-end web agents on visually grounded tasks across live websites.

- Page: https://aibenchmarkindex.com/global/online/webvoyager
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://github.com/MinorJerry/WebVoyager>
- Updated: 2026-10-21

### [WebWalkerQA](https://aibenchmarkindex.com/global/online/webwalkerqa.md)

Measures long-horizon information seeking through multi-hop traversal of structured websites.

- Page: https://aibenchmarkindex.com/global/online/webwalkerqa
- Categories: Retrieval
- Address: Online, Global, United States
- Website: <https://github.com/Alibaba-NLP/WebWalker>
- Updated: 2026-10-21

### [Werewolf Arena](https://aibenchmarkindex.com/global/online/werewolf-arena.md)

Evaluates model deception, deduction, persuasion, and turn-taking through arena-style social-deduction games.

- Page: https://aibenchmarkindex.com/global/online/werewolf-arena
- Categories: Agents
- Address: Online, Global, United States
- Website: <https://arxiv.org/abs/2407.13943>
- Updated: 2026-10-21

### [WikiText](https://aibenchmarkindex.com/global/online/wikitext.md)

Evaluates language modeling on more than 100 million tokens drawn from verified Wikipedia articles.

- Page: https://aibenchmarkindex.com/global/online/wikitext
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/salesforce/awd-lstm-lm>
- Updated: 2026-10-21

### [WildBench](https://aibenchmarkindex.com/global/online/wildbench.md)

Tests chat models on challenging real-user requests using task-specific checklists and model-based judges.

- Page: https://aibenchmarkindex.com/global/online/wildbench
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/allenai/WildBench>
- Updated: 2026-10-21

### [WinoBias](https://aibenchmarkindex.com/global/online/winobias.md)

Measures gender bias in coreference resolution by changing pronoun gender while preserving sentence structure.

- Page: https://aibenchmarkindex.com/global/online/winobias
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/uclanlp/corefBias>
- Updated: 2026-10-21

### [WinoGrande](https://aibenchmarkindex.com/global/online/winogrande.md)

Tests commonsense reasoning by resolving pronouns in adversarially collected sentence pairs.

- Page: https://aibenchmarkindex.com/global/online/winogrande
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/allenai/winogrande>
- Updated: 2026-10-21

### [WMT](https://aibenchmarkindex.com/global/online/wmt.md)

Evaluates machine translation systems through annual shared tasks spanning language pairs and translation settings.

- Page: https://aibenchmarkindex.com/global/online/wmt
- Categories: Other
- Address: Online, Global, United States
- Website: <https://www.statmt.org/wmt26/>
- Updated: 2026-10-21

### [WorkArena](https://aibenchmarkindex.com/global/online/workarena.md)

Measures whether browser agents can complete knowledge-work tasks in a ServiceNow environment.

- Page: https://aibenchmarkindex.com/global/online/workarena
- Categories: Browser
- Address: Online, Global, United States
- Website: <https://servicenow.github.io/WorkArena/>
- Updated: 2026-10-21

### [XNLI](https://aibenchmarkindex.com/global/online/xnli.md)

Evaluates cross-lingual natural-language inference and language transfer across 15 languages.

- Page: https://aibenchmarkindex.com/global/online/xnli
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://www.nyu.edu/projects/bowman/xnli/>
- Updated: 2026-10-21

### [XQuAD](https://aibenchmarkindex.com/global/online/xquad.md)

Evaluates cross-lingual extractive question answering on the same question-answer pairs translated across 11 languages.

- Page: https://aibenchmarkindex.com/global/online/xquad
- Categories: Reasoning
- Address: Online, Global, United States
- Website: <https://github.com/google-deepmind/xquad>
- Updated: 2026-10-21

### [XSTest](https://aibenchmarkindex.com/global/online/xstest.md)

Tests whether safety-tuned language models refuse harmless prompts that superficially resemble unsafe requests.

- Page: https://aibenchmarkindex.com/global/online/xstest
- Categories: Safety
- Address: Online, Global, United States
- Website: <https://github.com/paul-rottger/xstest>
- Updated: 2026-10-21

### [ZeroSCROLLS](https://aibenchmarkindex.com/global/online/zeroscrolls.md)

Evaluates zero-shot understanding of long texts across summarization, question answering, aggregation, and reordering tasks.

- Page: https://aibenchmarkindex.com/global/online/zeroscrolls
- Categories: Memory
- Address: Online, Global, United States
- Website: <https://www.zero.scrolls-benchmark.com/>
- Updated: 2026-10-21

## Academy

- [AI benchmark foundations for buyers](https://aibenchmarkindex.com/academy/ai-benchmark-foundations.md): Learn what an AI benchmark measures, how its grader scores a run, what its data licence allows and how to read a reported score before you use it in a decision.
- [Choose and run an AI benchmark for a model decision](https://aibenchmarkindex.com/academy/choose-and-run-an-ai-benchmark.md): A practical method for using AI benchmarks in a buying decision: define the decision, build a shortlist, budget the run, then sample, grade and record a run that another person can check.
- [Audit AI benchmark claims before you buy](https://aibenchmarkindex.com/academy/audit-ai-benchmark-claims.md): Check a vendor's benchmark claim like an evaluator: ceilings and uncertainty, contamination, grader and judge bias, score provenance and the licence of each asset.

Each course ends with an exam and a certificate of completion. Check a certificate's verification code at `https://aibenchmarkindex.com/api/v1/certificates/{code}`.

## Guides

- [How to choose an AI benchmark](https://aibenchmarkindex.com/guides/how-to-choose-an-ai-benchmark.md): Choose an AI benchmark by defining the decision first, then checking task fit, grader validity, data access, saturation, and provenance.
- [How to run an AI benchmark yourself for under $20](https://aibenchmarkindex.com/guides/run-a-benchmark-under-20.md): Keep an AI benchmark run under $20 by fixing the question, sampling tasks, estimating tokens, using deterministic graders, and saving a reproducible run record.
- [Silly benchmarks that tell you something real (pelican, skateboard, and friends)](https://aibenchmarkindex.com/guides/silly-benchmarks-real-signals.md): Playful tasks such as drawing a pelican on a bicycle can expose planning, geometry, tool use, and persistence when their grader makes the signal explicit.
- [What a saturated benchmark is and how to spot one](https://aibenchmarkindex.com/guides/what-is-a-saturated-benchmark.md): A saturated benchmark no longer separates strong systems well; spot it by score clustering, ceiling effects, stale tasks, and shrinking uncertainty-aware gaps.
- [Where benchmark data comes from and what its licence lets you do](https://aibenchmarkindex.com/guides/where-benchmark-data-comes-from.md): Benchmark data may be written, sampled, generated, or drawn from public datasets; its licence determines whether you may download, modify, and redistribute it.

## Data sources

- [Listing enrichment](https://aibenchmarkindex.com): license source, refreshed manual
- [Benchmark primary sources](https://aibenchmarkindex.com/about): license CC0-1.0, refreshed quarterly

## Optional

- [Sitemap](https://aibenchmarkindex.com/sitemap.xml): every indexable page