Skip to content

Coding benchmarks

There are 17 coding in the index.

Results

17 benchmarks

  • CodingOnline

    APPS

    Tests coding-challenge competence on problems ranging from introductory exercises to programming competitions.

    Task count
    10,000

  • CodingOnline

    BigCodeBench

    Tests function-level code generation on software-engineering tasks requiring diverse library calls.

    Task count
    1,140

  • CodingOnline

    CodeContests

    Tests competitive-programming solution generation with problems, descriptions, tests, and human submissions.

    Task count
    13,328

  • CodingOnline

    CodeXGLUE

    Tests code understanding and generation across ten tasks including search, repair, translation, and summarization.

    Task count
    10

  • CodingOnline

    CrossCodeEval

    Tests cross-file code completion and context retrieval in Python, Java, TypeScript, and C# repositories.

  • CodingOnline

    CRUXEval

    Tests code reasoning by asking models to predict inputs or outputs for short Python functions.

    Task count
    800

  • CodingOnline

    HumanEval

    Tests functional correctness of generated Python functions against hidden unit tests.

    Task count
    164

  • CodingOnline

    InterCode-CTF

    Evaluates coding agents that solve capture-the-flag tasks through iterative commands and execution feedback.

  • CodingOnline

    LiveCodeBench

    Tests code generation, execution, self-repair, and test-output prediction with newly released contest problems.

    Task count
    1,055

  • CodingOnline

    LiveCodeBench v5

    Tests code capabilities on the LiveCodeBench release containing contest problems through January 2025.

    Task count
    880

  • CodingOnline

    MBPP

    Tests basic Python synthesis with crowd-sourced prompts, reference solutions, and automated test cases.

    Task count
    974

  • CodingOnline

    RepoBench

    Tests repository-level code completion using cross-file dependencies in Python and Java projects.

  • CodingOnline

    RepoQA

    Tests long-context code retrieval by asking models to find described functions inside full repositories.

    Task count
    500

  • CodingOnline

    SWE-bench

    Tests whether coding systems can resolve real GitHub issues in complete software repositories.

    Task count
    2,294

  • CodingOnline

    SWE-bench Verified

    Tests repository-level issue resolution on a human-validated subset of 500 solvable SWE-bench tasks.

    Task count
    500

  • CodingOnline

    Benchmark Test Weakening

    Tests whether a coding agent makes tests pass by weakening the tests instead of fixing the code.

  • CodingOnline

    Benchmark Skateboard

    Tests whether a coding model can build a playable skateboard trick scene.

Other categories in Online

More benchmarks

Questions

What other kinds of benchmarks are in Online?

Other categories in Online: Multimodal (51 benchmarks), Reasoning (50 benchmarks), Agents (31 benchmarks), Safety (25 benchmarks), and Science (24 benchmarks).

Published by . How we check each benchmark.