# Coding benchmarks

There are 17 coding in the index.

17 benchmarks. Also on the [web page](https://aibenchmarkindex.com/global/online/category/coding).

## Benchmarks

1. [APPS](https://aibenchmarkindex.com/global/online/apps.md), Coding, Online: Tests coding-challenge competence on problems ranging from introductory exercises to programming competitions.
2. [BigCodeBench](https://aibenchmarkindex.com/global/online/bigcodebench.md), Coding, Online: Tests function-level code generation on software-engineering tasks requiring diverse library calls.
3. [CodeContests](https://aibenchmarkindex.com/global/online/codecontests.md), Coding, Online: Tests competitive-programming solution generation with problems, descriptions, tests, and human submissions.
4. [CodeXGLUE](https://aibenchmarkindex.com/global/online/codexglue.md), Coding, Online: Tests code understanding and generation across ten tasks including search, repair, translation, and summarization.
5. [CrossCodeEval](https://aibenchmarkindex.com/global/online/crosscodeeval.md), Coding, Online: Tests cross-file code completion and context retrieval in Python, Java, TypeScript, and C\# repositories.
6. [CRUXEval](https://aibenchmarkindex.com/global/online/cruxeval.md), Coding, Online: Tests code reasoning by asking models to predict inputs or outputs for short Python functions.
7. [HumanEval](https://aibenchmarkindex.com/global/online/humaneval.md), Coding, Online: Tests functional correctness of generated Python functions against hidden unit tests.
8. [InterCode-CTF](https://aibenchmarkindex.com/global/online/intercode-ctf.md), Coding, Online: Evaluates coding agents that solve capture-the-flag tasks through iterative commands and execution feedback.
9. [LiveCodeBench](https://aibenchmarkindex.com/global/online/livecodebench.md), Coding, Online: Tests code generation, execution, self-repair, and test-output prediction with newly released contest problems.
10. [LiveCodeBench v5](https://aibenchmarkindex.com/global/online/livecodebench-v5.md), Coding, Online: Tests code capabilities on the LiveCodeBench release containing contest problems through January 2025.
11. [MBPP](https://aibenchmarkindex.com/global/online/mbpp.md), Coding, Online: Tests basic Python synthesis with crowd-sourced prompts, reference solutions, and automated test cases.
12. [RepoBench](https://aibenchmarkindex.com/global/online/repobench.md), Coding, Online: Tests repository-level code completion using cross-file dependencies in Python and Java projects.
13. [RepoQA](https://aibenchmarkindex.com/global/online/repoqa.md), Coding, Online: Tests long-context code retrieval by asking models to find described functions inside full repositories.
14. [SWE-bench](https://aibenchmarkindex.com/global/online/swe-bench.md), Coding, Online: Tests whether coding systems can resolve real GitHub issues in complete software repositories.
15. [SWE-bench Verified](https://aibenchmarkindex.com/global/online/swe-bench-verified.md), Coding, Online: Tests repository-level issue resolution on a human-validated subset of 500 solvable SWE-bench tasks.
16. [Benchmark Test Weakening](https://aibenchmarkindex.com/global/online/benchmark-test-weakening.md), Coding, Online: Tests whether a coding agent makes tests pass by weakening the tests instead of fixing the code.
17. [Benchmark Skateboard](https://aibenchmarkindex.com/global/online/benchmark-skateboard.md), Coding, Online: Tests whether a coding model can build a playable skateboard trick scene.

## Other categories in Online

- [Multimodal](https://aibenchmarkindex.com/global/online/category/multimodal.md): 51 benchmarks
- [Reasoning](https://aibenchmarkindex.com/global/online/category/reasoning.md): 50 benchmarks
- [Agents](https://aibenchmarkindex.com/global/online/category/agents.md): 31 benchmarks
- [Safety](https://aibenchmarkindex.com/global/online/category/safety.md): 25 benchmarks
- [Science](https://aibenchmarkindex.com/global/online/category/science.md): 24 benchmarks
- [Retrieval](https://aibenchmarkindex.com/global/online/category/retrieval.md): 17 benchmarks
- [Memory](https://aibenchmarkindex.com/global/online/category/memory.md): 10 benchmarks
- [Browser](https://aibenchmarkindex.com/global/online/category/browser.md): 7 benchmarks
- [Voice](https://aibenchmarkindex.com/global/online/category/voice.md): 7 benchmarks
- [Math](https://aibenchmarkindex.com/global/online/category/math.md): 6 benchmarks
- [Other](https://aibenchmarkindex.com/global/online/category/other.md): 6 benchmarks
- [Tools](https://aibenchmarkindex.com/global/online/category/tools.md): 6 benchmarks
- [Commerce](https://aibenchmarkindex.com/global/online/category/commerce.md): 4 benchmarks
- [Long-horizon](https://aibenchmarkindex.com/global/online/category/long-horizon.md): 3 benchmarks

> **Check before you compare.** Benchmark methods and leaderboards change. Follow the cited primary source before using a result in a decision.