SWE-bench-Live
Collection
The datasets for benchmarking and training LLM coding agents. • 3 items • Updated
repo stringclasses 156
values | pull_number stringlengths 2 5 | instance_id stringlengths 15 55 | issue_numbers listlengths 1 11 | base_commit stringlengths 40 40 | patch stringlengths 356 2.11M | test_patch stringlengths 375 3.22M | problem_statement stringlengths 25 64.6k | hints_text stringlengths 2 107k | all_hints_text stringlengths 2 127k | commit_urls listlengths 1 93 | created_at timestamp[s]date 2024-01-01 02:57:59 2025-08-30 07:39:31 | commit_url stringlengths 74 114 | test_cmds listlengths 1 4 | log_parser stringclasses 1
value | difficulty dict | FAIL_TO_PASS listlengths 1 278 | PASS_TO_PASS listlengths 3 23.9k |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
conan-io/conan | 15377 | conan-io__conan-15377 | [
"15371"
] | c361744fa45be644670a8bf476d2bafccafe7f16 | "diff --git a/conan/tools/files/conandata.py b/conan/tools/files/conandata.py\nindex c6925450826..0b(...TRUNCATED) | "diff --git a/conans/test/functional/tools/scm/test_git.py b/conans/test/functional/tools/scm/test_g(...TRUNCATED) | "[feature] scm to conandata helper\n### What is your suggestion?\n\nIn Conan 1.X there was the possi(...TRUNCATED) | To take into account: the ``trim_conandata`` possible interactions
| To take into account: the ``trim_conandata`` possible interactions
| ["https://github.com/conan-io/conan/commit/f6109258ceefdffce8299756a05a2425c4f21ff9","https://github(...TRUNCATED) | 2024-01-03T09:26:31 | https://github.com/conan-io/conan/tree/c361744fa45be644670a8bf476d2bafccafe7f16 | [
"pytest -rA"
] | pytest | {
"files": 2,
"hunks": 3,
"lines": 13
} | ["conans/test/functional/tools/scm/test_git.py::TestGitBasicSCMFlow::test_full_scm[True]","conans/te(...TRUNCATED) | ["conans/test/functional/only_source_test.py::OnlySourceTest::test_build_policies_in_conanfile","con(...TRUNCATED) |
geopandas/geopandas | 3132 | geopandas__geopandas-3132 | [
"1320"
] | fed9e57a0033dee6c18ce9da40dcd49e1211f876 | "diff --git a/CHANGELOG.md b/CHANGELOG.md\nindex fadd1dd85e..9405275ef2 100644\n--- a/CHANGELOG.md\n(...TRUNCATED) | "diff --git a/geopandas/tests/test_geodataframe.py b/geopandas/tests/test_geodataframe.py\nindex 85c(...TRUNCATED) | "Make `GeoDataFrame._to_geo` \"public\"\nCurrently `GeoDataFrame.to_json` uses `GeoDataFrame._to_geo(...TRUNCATED) | "What would be the use case of that? I am just curious, I have nothing against it. Feel free to make(...TRUNCATED) | "What would be the use case of that? I am just curious, I have nothing against it. Feel free to make(...TRUNCATED) | ["https://github.com/geopandas/geopandas/commit/67ef3eb2fe48d32c5ef809305d581c68a269d114","https://g(...TRUNCATED) | 2024-01-05T17:09:25 | https://github.com/geopandas/geopandas/tree/fed9e57a0033dee6c18ce9da40dcd49e1211f876 | [
"pytest -rA"
] | pytest | {
"files": 3,
"hunks": 10,
"lines": 76
} | ["geopandas/tests/test_geodataframe.py::TestDataFrame::test_geodataframe_geojson_no_bbox","geopandas(...TRUNCATED) | ["geopandas/io/tests/test_file.py::test_to_file[fiona-ESRI","geopandas/io/tests/test_file.py::test_t(...TRUNCATED) |
reata/sqllineage | 524 | reata__sqllineage-524 | [
"487"
] | 4adcc8f46fe9c5f919c9d942e1e1a0cf7d342cc2 | "diff --git a/sqllineage/core/holders.py b/sqllineage/core/holders.py\nindex fe663d00..688833fe 1006(...TRUNCATED) | "diff --git a/tests/sql/column/test_column_update.py b/tests/sql/column/test_column_update.py\nnew f(...TRUNCATED) | "update statement column lineage\n```sql\r\nupdate \r\n public.tgt_tbl1\r\nset \r\n email = s.(...TRUNCATED) | "Column lineage for UPDATE statement is not supported as of v1.4.8.\r\n\r\nIt should be quite simila(...TRUNCATED) | "Column lineage for UPDATE statement is not supported as of v1.4.8.\r\n\r\nIt should be quite simila(...TRUNCATED) | ["https://github.com/reata/sqllineage/commit/2ec1af27c66a00fc169744cf6a892d7e528f0376","https://gith(...TRUNCATED) | 2024-01-02T02:25:22 | https://github.com/reata/sqllineage/tree/4adcc8f46fe9c5f919c9d942e1e1a0cf7d342cc2 | [
"pytest -rA"
] | pytest | {
"files": 7,
"hunks": 16,
"lines": 341
} | ["tests/sql/column/test_column_update.py::test_column_update_with_single_table","tests/sql/column/te(...TRUNCATED) | ["tests/core/test_cli.py::test_cli_dummy","tests/core/test_cli.py::test_file_exception","tests/core/(...TRUNCATED) |
privacyidea/privacyidea | 3852 | privacyidea__privacyidea-3852 | [
"3281"
] | 969e5f55d603ea86b3f0832e23a69e0c58621d3e | "diff --git a/doc/policies/index.rst b/doc/policies/index.rst\nindex 7e663e0809..dabdacc572 100644\n(...TRUNCATED) | "diff --git a/tests/test_lib_policy.py b/tests/test_lib_policy.py\nindex 158f25ee9f..11b39642e7 1006(...TRUNCATED) | "Allow case insensitive usersnames in Policies\n**Is your feature request related to a problem? Plea(...TRUNCATED) | "Thank you for filing an issue and sharing your observations or ideas. Please be sure to provide as (...TRUNCATED) | "Thank you for filing an issue and sharing your observations or ideas. Please be sure to provide as (...TRUNCATED) | ["https://github.com/privacyidea/privacyidea/commit/19d2d710f90aacb2b2fcf1c64254c5a65d6af25d","https(...TRUNCATED) | 2024-01-29T09:38:26 | https://github.com/privacyidea/privacyidea/tree/969e5f55d603ea86b3f0832e23a69e0c58621d3e | [
"pytest -v -rA tests/"
] | pytest | {
"files": 6,
"hunks": 15,
"lines": 92
} | [
"tests/test_lib_policy.py::PolicyTestCase::test_02_update_policies"
] | ["tests/test_api_2stepinit.py::TwoStepInitTestCase::test_01_init_token","tests/test_api_2stepinit.py(...TRUNCATED) |
matplotlib/matplotlib | 27613 | matplotlib__matplotlib-27613 | [
"27596"
] | 83aa3e41bf9d4de308e306521e0cb3f688952942 | "diff --git a/lib/matplotlib/rcsetup.py b/lib/matplotlib/rcsetup.py\nindex cf4dd07a6b98..f730db0dee3(...TRUNCATED) | "diff --git a/lib/matplotlib/tests/test_cycles.py b/lib/matplotlib/tests/test_cycles.py\nindex 9bbb9(...TRUNCATED) | "[Bug]: Markers with numeric name like CARETLEFT cannot be specified using a cycler\n### Bug summary(...TRUNCATED) | "Looks like the validator for the `marker` property is a stringlist\r\nvalidator, so it is trying to(...TRUNCATED) | "Looks like the validator for the `marker` property is a stringlist\r\nvalidator, so it is trying to(...TRUNCATED) | [
"https://github.com/matplotlib/matplotlib/commit/8ca4d7938aae134d8394f8eb26efa03f5cf691de"
] | 2024-01-08T23:53:11 | https://github.com/matplotlib/matplotlib/tree/83aa3e41bf9d4de308e306521e0cb3f688952942 | [
"pytest -rA"
] | pytest | {
"files": 2,
"hunks": 7,
"lines": 26
} | [
"lib/matplotlib/tests/test_cycles.py::test_valid_marker_cycles"
] | ["lib/matplotlib/tests/test_afm.py::test_nonascii_str","lib/matplotlib/tests/test_afm.py::test_parse(...TRUNCATED) |
reata/sqllineage | 565 | reata__sqllineage-565 | [
"564"
] | 4c6c65e7c94b87b50f26ad7f35ac26ed26a54cee | "diff --git a/sqllineage/core/parser/sqlfluff/extractors/create_insert.py b/sqllineage/core/parser/s(...TRUNCATED) | "diff --git a/tests/sql/column/test_column_select_column_specified_in_dml.py b/tests/sql/column/test(...TRUNCATED) | "Not fully processed top-level subquery in DML\n**SQL**\r\n```sql\r\nINSERT INTO tab1 (col1, col2) ((...TRUNCATED) | [
"https://github.com/reata/sqllineage/commit/9060655fb8271c508348c83d6fa3f617c243af92"
] | 2024-01-23T14:46:47 | https://github.com/reata/sqllineage/tree/4c6c65e7c94b87b50f26ad7f35ac26ed26a54cee | [
"pytest -rA"
] | pytest | {
"files": 2,
"hunks": 3,
"lines": 28
} | ["tests/sql/column/test_column_select_column_specified_in_dml.py::test_view_with_subquery_custom_col(...TRUNCATED) | ["tests/core/test_cli.py::test_cli_dummy","tests/core/test_cli.py::test_file_exception","tests/core/(...TRUNCATED) | ||
sphinx-doc/sphinx | 11888 | sphinx-doc__sphinx-11888 | [
"11887"
] | 882a174e48a4dfd22d4fab4b2e3b74f091b3f98e | "diff --git a/sphinx/search/__init__.py b/sphinx/search/__init__.py\nindex 99c934a0f06..42b45e113ad (...TRUNCATED) | "diff --git a/tests/test_search.py b/tests/test_search.py\nindex d4bbef5b3b3..f13c9a2468e 100644\n--(...TRUNCATED) | "searchindex.js: parallel HTML builds may produce non-deterministic index\n### Describe the bug\n\nT(...TRUNCATED) | ["https://github.com/sphinx-doc/sphinx/commit/44d610a60d6e4375d996f4d3f1c902ca4dc2b9ad","https://git(...TRUNCATED) | 2024-01-17T00:57:52 | https://github.com/sphinx-doc/sphinx/tree/882a174e48a4dfd22d4fab4b2e3b74f091b3f98e | [
"python -X dev -X warn_default_encoding -m pytest -v -rA"
] | pytest | {
"files": 1,
"hunks": 1,
"lines": 2
} | [
"tests/test_search.py::test_IndexBuilder"
] | ["tests/test_addnodes.py::test_desc_sig_element_nodes","tests/test_api_translator.py::test_html_tran(...TRUNCATED) | ||
conan-io/conan | 15571 | conan-io__conan-15571 | [
"14764",
"15519"
] | d4e7b3e1edda188c13f7c404e064cb3a98d58965 | "diff --git a/conans/model/options.py b/conans/model/options.py\nindex 9384fb08e0e..62aedac2132 1006(...TRUNCATED) | "diff --git a/conans/test/integration/options/options_test.py b/conans/test/integration/options/opti(...TRUNCATED) | "[feature] Recipe definition of options with medium precedence over profiles\n### What is your sugge(...TRUNCATED) | "We agreed to give this a try for a POC, and see how it looks\n\nHi @frogarian \r\n\r\nThanks for y(...TRUNCATED) | "We agreed to give this a try for a POC, and see how it looks\n\nHi @frogarian \r\n\r\nThanks for y(...TRUNCATED) | ["https://github.com/conan-io/conan/commit/a3c7bccbd97b93220151cee29f526507a8194f94","https://github(...TRUNCATED) | 2024-01-31T13:08:27 | https://github.com/conan-io/conan/tree/d4e7b3e1edda188c13f7c404e064cb3a98d58965 | [
"python -m pytest -rA ."
] | pytest | {
"files": 1,
"hunks": 3,
"lines": 15
} | ["conans/test/integration/options/options_test.py::TestImportantOptions::test_important_options[liba(...TRUNCATED) | ["conans/test/functional/only_source_test.py::OnlySourceTest::test_build_policies_in_conanfile","con(...TRUNCATED) |
conan-io/conan | 15504 | conan-io__conan-15504 | [
"15502"
] | c361744fa45be644670a8bf476d2bafccafe7f16 | "diff --git a/conan/api/subapi/export.py b/conan/api/subapi/export.py\nindex 9c3a59c9525..ab8d410cdd(...TRUNCATED) | "diff --git a/conans/test/integration/command/export_pkg_test.py b/conans/test/integration/command/e(...TRUNCATED) | "[feature] upload the binary package just created with `create`/`export-pkg`\n### What is your sugge(...TRUNCATED) | "Hi @X1aomu \r\n\r\nThanks for your suggestions.\r\n\r\nI'd like to clarify some issues first:\r\n\r(...TRUNCATED) | "Hi @X1aomu \r\n\r\nThanks for your suggestions.\r\n\r\nI'd like to clarify some issues first:\r\n\r(...TRUNCATED) | [
"https://github.com/conan-io/conan/commit/2ad5a75c24b8e5d151ccffaab62a6b9754047aba"
] | 2024-01-23T14:06:48 | https://github.com/conan-io/conan/tree/c361744fa45be644670a8bf476d2bafccafe7f16 | [
"python -m pytest -rA ."
] | pytest | {
"files": 2,
"hunks": 3,
"lines": 10
} | [
"conans/test/integration/command/export_pkg_test.py::ExportPkgTest::test_export_pkg_json"
] | ["conans/test/functional/only_source_test.py::OnlySourceTest::test_build_policies_in_conanfile","con(...TRUNCATED) |
reata/sqllineage | 557 | reata__sqllineage-557 | [
"528"
] | 942fcf4e5578da3f3ce22501b17fbceb5ea9c350 | "diff --git a/sqllineage/core/parser/sqlfluff/extractors/create_insert.py b/sqllineage/core/parser/s(...TRUNCATED) | "diff --git a/tests/sql/column/test_metadata_target_column.py b/tests/sql/column/test_metadata_targe(...TRUNCATED) | "set target table column name from MetaDataProvider\n**Is your feature request related to a problem?(...TRUNCATED) | "This is a valid case that we definitely want to improve with MetaDataProvider.\r\n\r\nI'll come bac(...TRUNCATED) | "This is a valid case that we definitely want to improve with MetaDataProvider.\r\n\r\nI'll come bac(...TRUNCATED) | ["https://github.com/reata/sqllineage/commit/9bb8b4a812f2e950b5cb2fcde7c760e8a3112e4b","https://gith(...TRUNCATED) | 2024-01-17T04:54:29 | https://github.com/reata/sqllineage/tree/942fcf4e5578da3f3ce22501b17fbceb5ea9c350 | [
"pytest -rA"
] | pytest | {
"files": 1,
"hunks": 2,
"lines": 11
} | ["tests/sql/column/test_metadata_target_column.py::test_metadata_target_column[provider0]","tests/sq(...TRUNCATED) | ["tests/core/test_cli.py::test_cli_dummy","tests/core/test_cli.py::test_file_exception","tests/core/(...TRUNCATED) |
End of preview. Expand in Data Studio
A brand-new, continuously updated SWE-bench-like dataset powered by an automated curation pipeline.
For the official data release page, please see microsoft/SWE-bench-Live.
Dataset Summary
SWE-bench-Live is a live benchmark for issue resolving, designed to evaluate an AI system’s ability to complete real-world software engineering tasks. Thanks to our automated dataset curation pipeline, we plan to update SWE-bench-Live on a monthly basis to provide the community with up-to-date task instances and support rigorous and contamination-free evaluation.
News
- 9/17/2025: Dataset updated! We’ve finalized the update process for SWE-bench-Live: Each month, we will add 50 newly verified, high-quality issues to the dataset. The
liteandverifiedsplits will remain frozen, ensuring fair leaderboard comparisons and keeping evaluation costs manageable. To access the latest issues, please refer to thefullsplit! - 07/19/2025: We've employed a LLM filter to automatically filter full dataset to create SWE-bench-Live Verified. The initial Verified subset contains 500 instances from 2024-07 to 2025-04.
- 06/30/2025: We’ve updated the dataset — it now includes a total of 1,565 task instances across 164 repositories!
- 05/21/2025: The initial release of SWE-bench-Live includes 1,319 latest (created after 2024) task instances, each paired with an instance-level Docker image for test execution, covering 93 repositories.
SWE-bench-Live Curation Pipeline
Dataset Structure
| Field | Type | Description |
|---|---|---|
repo |
str |
Repository full name |
pull_number |
str |
Pull request number |
instance_id |
str |
Unique identifier of the task instance |
issue_numbers |
str |
Issues that are resolved by the pull request |
base_commit |
str |
The commit on which the PR is based |
patch |
str |
Gold patch in .diff format |
test_patch |
str |
Test suite modifications |
problem_statement |
str |
Issue description text |
FAIL_TO_PASS |
List[str] |
Tests that should transition from failing to passing |
PASS_TO_PASS |
List[str] |
Tests that should remain passing |
image_key |
str |
Instance-level docker image |
test_cmds |
List[str] |
Commands to run test suite |
log_parser |
str |
Type of log parser, pytest by default |
Evaluation Protocol
SWE-bench-Live evaluation strictly follows the original
SWE-benchprotocol:
- During a rollout, the agent may access only the
problem_statementfield of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such ashint,FAIL_TO_PASS, ortest_patch. Thetest_patchmust not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on theproblem_statementon the docker container started from the image of the task instance.- Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution.
Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions.
When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.
- Downloads last month
- 45,637
