Skip to content

dataset: verify every feature against combined.patch, on both architectures #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akhatua2 merged 1 commit into from
Sep 5, 2026
Merged

dataset: verify every feature against combined.patch, on both architectures #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,4 @@ workspace/

# Regenerated by scripts/check_gradeable.py on every run
dataset/gradeable_report*.json
dataset/combined_report*.json
31 changes: 31 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed

- **Every feature's tests now pass against its task's `combined.patch`** — the third dataset
invariant, alongside fail-on-base / pass-on-gold, and the one the coop eval actually depends on:
`test_merged` runs each feature's `tests.patch` against the merged tree, so a feature whose tests
fail once a sibling's gold is present is unwinnable for every pair containing that sibling.
Swept all 199 on Modal (linux/amd64) and local Docker (linux/arm64): 19 features in 5 tasks failed.
`dspy/8563` f2–f6 (f1 changes `ToolCalls.format()`'s return shape; the pre-existing shape tests
in the shared file failed in any tree with f1 — **8/15 pairs were hard zeros**) and `dspy/8635`
f1–f6 (f6's `min_instr_chars=30` replaces the base tests' 11-char dummy instruction — **5/15
pairs**; March's #40 had papered over it by setting the default to 0 in combined.patch only,
contradicting f6's spec and gold) are fixed by making the pre-existing assertions in the sibling
`tests.patch` files tolerant of the sibling behaviour; `openai_tiktoken/0` f3 (combined.patch
still used the pre-#46 parameter name) and `pallets_jinja/1559` f3 (combined.patch did not
implement `priority=` at all) and `pallets_jinja/1465` all 10 (combined.patch carried the PR's test-file diffs, so it could not apply on top of any `tests.patch`) are fixed in combined.patch. No `feature.patch` changed. Reasoning
in `dataset/SPEC_AUDIT.md` ("Second pass").
- Re-verified fail-on-base / pass-on-gold for all 199 features on both architectures: 199/199.
- **Three task images had no linux/arm64 manifest** (`react-hook-form:task153`, `:task85`,
`huggingface-datasets:task3997`), so those 13 features could not run on an arm64 Docker host at
all. arm64 images were built natively and added to the existing indexes alongside the untouched
amd64 manifests.

### Added

- **`scripts/check_combined.py`** — the sweep above; one sandbox per feature through the image's
own `runner.sh`, `--backend modal|docker`.
- **`scripts/check_gradeable.py --backend docker`** — run the fail-on-base / pass-on-gold sweep on
the local daemon (arm64 on Apple Silicon) as well as Modal (amd64). Both scripts now write
sandbox files in 32 KB chunks; a single `echo` of `pallets_jinja/1465`'s 107 KB combined.patch
exceeded Modal's 64 KiB `ARG_MAX` and showed up as 10 spurious `ERROR`s.

## [0.0.29] - 2026-08-14

### Fixed
Expand Down
108 changes: 54 additions & 54 deletions dataset/.spec_audit_verdicts.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
],
"dottxt_ai_outlines_task/task1655|feature10": [
"OK",
"over-complete \u2014 spec contains the answer verbatim"
"over-complete — spec contains the answer verbatim"
],
"dottxt_ai_outlines_task/task1655|feature2": [
"OK",
Expand Down Expand Up @@ -49,15 +49,15 @@
],
"dottxt_ai_outlines_task/task1655|feature8": [
"OK",
"format-only is explicit ('format' x3, XXX-XX-XXXX given) \u2014 note 000-00-0000 and 999-99-9999 are valid, so real SSN area rules must NOT be applied"
"format-only is explicit ('format' x3, XXX-XX-XXXX given) — note 000-00-0000 and 999-99-9999 are valid, so real SSN area rules must NOT be applied"
],
"dottxt_ai_outlines_task/task1655|feature9": [
"OK",
"regex type: format described and the type name given"
],
"dottxt_ai_outlines_task/task1706|feature1": [
"OK",
"'\u2026is not available' is pre-existing outlines text (0 occurrences in gold)"
"'…is not available' is pre-existing outlines text (0 occurrences in gold)"
],
"dottxt_ai_outlines_task/task1706|feature2": [
"OK",
Expand All @@ -73,7 +73,7 @@
],
"dottxt_ai_outlines_task/task1706|feature5": [
"SPEC",
"exact custom_adapter ValueError texts \u2014 conditions stated, wording was not"
"exact custom_adapter ValueError texts — conditions stated, wording was not"
],
"dottxt_ai_outlines_task/task1706|feature6": [
"OK",
Expand All @@ -93,7 +93,7 @@
],
"dspy_task/task8394|feature2": [
"OK",
"over-complete \u2014 gives the exact ns_hash formula and key format"
"over-complete — gives the exact ns_hash formula and key format"
],
"dspy_task/task8394|feature3": [
"OK",
Expand All @@ -112,72 +112,72 @@
"ToolCall / convert_input_schema_to_tool_args are pre-existing; 'Arg X is invalid' is pre-existing dspy validation"
],
"dspy_task/task8563|feature2": [
"OK",
"ToolCall / convert_input_schema_to_tool_args are pre-existing; 'Arg X is invalid' is pre-existing dspy validation"
"TEST",
"pre-existing format() tests made shape-tolerant: f1 switches format() to the compact dict shape, which failed these tests in any tree containing f1 (combined, or any coop pair with f1)"
],
"dspy_task/task8563|feature3": [
"OK",
"ToolCall / convert_input_schema_to_tool_args are pre-existing; 'Arg X is invalid' is pre-existing dspy validation"
"TEST",
"pre-existing format() tests made shape-tolerant: f1 switches format() to the compact dict shape, which failed these tests in any tree containing f1 (combined, or any coop pair with f1)"
],
"dspy_task/task8563|feature4": [
"OK",
"ToolCall / convert_input_schema_to_tool_args are pre-existing; 'Arg X is invalid' is pre-existing dspy validation"
"TEST",
"pre-existing format() tests made shape-tolerant: f1 switches format() to the compact dict shape, which failed these tests in any tree containing f1 (combined, or any coop pair with f1)"
],
"dspy_task/task8563|feature5": [
"OK",
"ToolCall / convert_input_schema_to_tool_args are pre-existing; 'Arg X is invalid' is pre-existing dspy validation"
"TEST",
"pre-existing format() tests made shape-tolerant: f1 switches format() to the compact dict shape, which failed these tests in any tree containing f1 (combined, or any coop pair with f1)"
],
"dspy_task/task8563|feature6": [
"OK",
"ToolCall / convert_input_schema_to_tool_args are pre-existing; 'Arg X is invalid' is pre-existing dspy validation"
"TEST",
"pre-existing format() tests made shape-tolerant: f1 switches format() to the compact dict shape, which failed these tests in any tree containing f1 (combined, or any coop pair with f1)"
],
"dspy_task/task8587|feature1": [
"OK",
"nothing flagged \u2014 streaming API fully named in each spec"
"nothing flagged — streaming API fully named in each spec"
],
"dspy_task/task8587|feature2": [
"OK",
"nothing flagged \u2014 streaming API fully named in each spec"
"nothing flagged — streaming API fully named in each spec"
],
"dspy_task/task8587|feature3": [
"OK",
"nothing flagged \u2014 streaming API fully named in each spec"
"nothing flagged — streaming API fully named in each spec"
],
"dspy_task/task8587|feature4": [
"OK",
"nothing flagged \u2014 streaming API fully named in each spec"
"nothing flagged — streaming API fully named in each spec"
],
"dspy_task/task8587|feature5": [
"OK",
"nothing flagged \u2014 streaming API fully named in each spec"
"nothing flagged — streaming API fully named in each spec"
],
"dspy_task/task8587|feature6": [
"OK",
"nothing flagged \u2014 streaming API fully named in each spec"
"nothing flagged — streaming API fully named in each spec"
],
"dspy_task/task8635|feature1": [
"OK",
"nothing flagged \u2014 proposer params fully named in each spec"
"TEST",
"pre-existing proposer tests accept the signature-default instruction too: f6's min_instr_chars=30 replaces the 11-char dummy proposal, which failed these tests in any tree containing f6"
],
"dspy_task/task8635|feature2": [
"OK",
"nothing flagged \u2014 proposer params fully named in each spec"
"TEST",
"pre-existing proposer tests accept the signature-default instruction too: f6's min_instr_chars=30 replaces the 11-char dummy proposal, which failed these tests in any tree containing f6"
],
"dspy_task/task8635|feature3": [
"OK",
"nothing flagged \u2014 proposer params fully named in each spec"
"TEST",
"pre-existing proposer tests accept the signature-default instruction too: f6's min_instr_chars=30 replaces the 11-char dummy proposal, which failed these tests in any tree containing f6"
],
"dspy_task/task8635|feature4": [
"OK",
"nothing flagged \u2014 proposer params fully named in each spec"
"TEST",
"pre-existing proposer tests accept the signature-default instruction too: f6's min_instr_chars=30 replaces the 11-char dummy proposal, which failed these tests in any tree containing f6"
],
"dspy_task/task8635|feature5": [
"OK",
"nothing flagged \u2014 proposer params fully named in each spec"
"TEST",
"pre-existing proposer tests accept the signature-default instruction too: f6's min_instr_chars=30 replaces the 11-char dummy proposal, which failed these tests in any tree containing f6"
],
"dspy_task/task8635|feature6": [
"OK",
"nothing flagged \u2014 proposer params fully named in each spec"
"nothing flagged — proposer params fully named in each spec; combined.patch min_instr_chars default restored to 30 (spec + gold), reverting #40"
],
"go_chi_task/task26|feature1": [
"OK",
Expand All @@ -189,7 +189,7 @@
],
"go_chi_task/task26|feature3": [
"SPEC",
"added RouteSelector API \u2014 spec named no identifiers at all"
"added RouteSelector API — spec named no identifiers at all"
],
"go_chi_task/task26|feature4": [
"SPEC",
Expand All @@ -205,7 +205,7 @@
],
"go_chi_task/task27|feature3": [
"SPEC",
"added EnableDebugLogging() \u2014 gold introduces it, spec named nothing"
"added EnableDebugLogging() — gold introduces it, spec named nothing"
],
"go_chi_task/task27|feature4": [
"OK",
Expand Down Expand Up @@ -237,7 +237,7 @@
],
"huggingface_datasets_task/task3997|feature2": [
"SPEC",
"added set_custom_decoding_criteria \u2014 gold introduces it, spec named nothing"
"added set_custom_decoding_criteria — gold introduces it, spec named nothing"
],
"huggingface_datasets_task/task3997|feature3": [
"SPEC",
Expand Down Expand Up @@ -329,7 +329,7 @@
],
"llama_index_task/task18813|feature2": [
"SPEC",
"exact 'exceeds maximum allowed size' message \u2014 spec gave only the condition"
"exact 'exceeds maximum allowed size' message — spec gave only the condition"
],
"llama_index_task/task18813|feature3": [
"OK",
Expand Down Expand Up @@ -361,7 +361,7 @@
],
"openai_tiktoken_task/task0|feature3": [
"OK",
"encode() parameter named with its post-processing semantics stated"
"encode() parameter named with its post-processing semantics stated; combined.patch renamed return_frequency -> analyze_frequency to match spec/gold (#46 missed it)"
],
"openai_tiktoken_task/task0|feature4": [
"OK",
Expand All @@ -381,7 +381,7 @@
],
"openai_tiktoken_task/task0|feature8": [
"OK",
"recomputed the dict \u2014 exactly \"top 3 adjacent pairs\""
"recomputed the dict — exactly \"top 3 adjacent pairs\""
],
"openai_tiktoken_task/task0|feature9": [
"OK",
Expand All @@ -405,7 +405,7 @@
],
"pallets_click_task/task2068|feature2": [
"SPEC",
"exact timeout message \u2014 and gold does not pluralise, so \"after 1 seconds\""
"exact timeout message — and gold does not pluralise, so \"after 1 seconds\""
],
"pallets_click_task/task2068|feature3": [
"SPEC",
Expand Down Expand Up @@ -437,7 +437,7 @@
],
"pallets_click_task/task2800|feature1": [
"OK",
"over-complete \u2014 names function, call sites, construct"
"over-complete — names function, call sites, construct"
],
"pallets_click_task/task2800|feature2": [
"OK",
Expand Down Expand Up @@ -549,11 +549,11 @@
],
"pallets_jinja_task/task1559|feature3": [
"OK",
"i18n extension API and tag syntax fully named in the spec"
"i18n extension API and tag syntax fully named in the spec; combined.patch gained the priority= implementation, which it did not contain at all"
],
"pallets_jinja_task/task1559|feature4": [
"OK",
"over-complete \u2014 error strings quoted verbatim"
"over-complete — error strings quoted verbatim"
],
"pallets_jinja_task/task1559|feature5": [
"OK",
Expand Down Expand Up @@ -677,47 +677,47 @@
],
"react_hook_form_task/task153|feature1": [
"OK",
"over-complete \u2014 callback signature, ordering and target file all stated"
"over-complete — callback signature, ordering and target file all stated"
],
"react_hook_form_task/task153|feature2": [
"OK",
"over-complete \u2014 callback signature, ordering and target file all stated"
"over-complete — callback signature, ordering and target file all stated"
],
"react_hook_form_task/task153|feature3": [
"OK",
"over-complete \u2014 callback signature, ordering and target file all stated"
"over-complete — callback signature, ordering and target file all stated"
],
"react_hook_form_task/task153|feature4": [
"OK",
"over-complete \u2014 callback signature, ordering and target file all stated"
"over-complete — callback signature, ordering and target file all stated"
],
"react_hook_form_task/task153|feature5": [
"OK",
"over-complete \u2014 callback signature, ordering and target file all stated"
"over-complete — callback signature, ordering and target file all stated"
],
"react_hook_form_task/task153|feature6": [
"OK",
"over-complete \u2014 callback signature, ordering and target file all stated"
"over-complete — callback signature, ordering and target file all stated"
],
"react_hook_form_task/task85|feature1": [
"OK",
"over-complete \u2014 names the prop, its type, the file and often the implementation shape"
"over-complete — names the prop, its type, the file and often the implementation shape"
],
"react_hook_form_task/task85|feature2": [
"OK",
"over-complete \u2014 names the prop, its type, the file and often the implementation shape"
"over-complete — names the prop, its type, the file and often the implementation shape"
],
"react_hook_form_task/task85|feature3": [
"OK",
"over-complete \u2014 names the prop, its type, the file and often the implementation shape"
"over-complete — names the prop, its type, the file and often the implementation shape"
],
"react_hook_form_task/task85|feature4": [
"OK",
"over-complete \u2014 names the prop, its type, the file and often the implementation shape"
"over-complete — names the prop, its type, the file and often the implementation shape"
],
"react_hook_form_task/task85|feature5": [
"OK",
"over-complete \u2014 names the prop, its type, the file and often the implementation shape"
"over-complete — names the prop, its type, the file and often the implementation shape"
],
"samuelcolvin_dirty_equals_task/task43|feature1": [
"OK",
Expand All @@ -737,7 +737,7 @@
],
"samuelcolvin_dirty_equals_task/task43|feature5": [
"SPEC",
"country codes are case-insensitive \u2014 gold does .upper(), spec silent"
"country codes are case-insensitive — gold does .upper(), spec silent"
],
"samuelcolvin_dirty_equals_task/task43|feature6": [
"SPEC",
Expand Down
25 changes: 21 additions & 4 deletions dataset/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@ size_categories:

This dataset contains the benchmark tasks for evaluating multi-agent coordination in code collaboration.

**Paper**: [CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://arxiv.org/abs/2601.13295)
**Code**: [github.com/cooperbench/CooperBench](https://github.com/cooperbench/CooperBench)
**Website**: [cooperbench.com](https://cooperbench.com)
**Run it with the official harness:** [github.com/cooperbench/CooperBench](https://github.com/cooperbench/CooperBench) (`pip install cooperbench`, then `cooperbench prepare` downloads this dataset). The harness, the Docker images, and this dataset are versioned together — use the GitHub repo's `dataset/` tree or this mirror at the matching tag.

- **Paper**: [CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://arxiv.org/abs/2601.13295)
- **Code**: [github.com/cooperbench/CooperBench](https://github.com/cooperbench/CooperBench)
- **Website**: [cooperbench.com](https://cooperbench.com)
- **Task images**: `akhatua/cooperbench-<repo>:task<id>` on Docker Hub, multi-arch (linux/amd64 + linux/arm64)

## Structure

Expand All @@ -42,6 +45,20 @@ dataset/
```


## Patches: what each one is for

Each task is one real pull request split into N independent features. The eval never merges gold patches — it tests **an agent's** patch (or the merge of two agents' patches) against each feature's hidden tests. The three patch files mean different things:

| file | what it is | property it must satisfy |
|---|---|---|
| `feature{N}/feature.patch` | the gold implementation of feature N **alone** | passes `feature{N}/tests.patch` on the base commit |
| `feature{N}/tests.patch` | feature N's hidden tests | **fails** on the untouched base commit |
| `combined.patch` | the whole PR — every feature landed in one tree, conflicts already resolved | passes **every** feature's `tests.patch` |

**Gold patches for two different features are not meant to be merged.** They were carved out of the same PR and edit the same files, so `git merge` of `feature1.patch` and `feature2.patch` conflicts for 499 of the 652 pairs (see `gold_conflict_report.json`). That is the coordination challenge the benchmark measures, not a defect: two agents each implementing one feature must produce patches that *do* merge. The resolved oracle for any pair is `combined.patch` — it contains both features and passes both test suites.

All three properties are verified for all 199 features on both architectures with the harness's `scripts/check_gradeable.py` and `scripts/check_combined.py`; every fix and its reasoning is logged in [`SPEC_AUDIT.md`](SPEC_AUDIT.md).

## Repositories

| Directory | Repository | Tasks | Features |
Expand All @@ -57,7 +74,7 @@ dataset/
| `pillow_task` | [python-pillow/Pillow](https://github.com/python-pillow/Pillow) | 3 | 15 |
| `react_hook_form_task` | [react-hook-form/react-hook-form](https://github.com/react-hook-form/react-hook-form) | 2 | 11 |
| `samuelcolvin_dirty_equals_task` | [samuelcolvin/dirty-equals](https://github.com/samuelcolvin/dirty-equals) | 1 | 9 |
| `typst` | [typst/typst](https://github.com/typst/typst) | 1 | 10 |
| `typst_task` | [typst/typst](https://github.com/typst/typst) | 1 | 10 |

## Subsets

Expand Down
Loading
Loading