dataset: verify every feature against combined.patch, on both architectures - #85
Merged
Conversation
…ctures
Second verification pass over all 199 features. Adds the third dataset invariant next to
fail-on-base / pass-on-gold: each feature's tests must pass against its task's combined.patch.
That is the property the coop eval depends on -- test_merged runs a feature's tests against
the merged tree -- and it had never been measured. 19 features in 5 tasks failed it:
dspy/8563 f2-f6 f1 changes ToolCalls.format()'s return shape; two pre-existing tests in
the shared file asserted the old shape, so every tree containing f1 failed
them. 8 of 15 pairs were unwinnable. Sibling tests.patch files now accept
both shapes.
dspy/8635 f1-f6 f6's min_instr_chars=30 replaces the base tests' 11-char dummy. #40 had
set the default to 0 in combined.patch only, contradicting f6's spec and
gold. Default restored; sibling assertions accept either value. 5 of 15
pairs were unwinnable.
tiktoken/0 f3 combined.patch still used the pre-#46 parameter name.
jinja/1559 f3 combined.patch did not implement priority= at all; f3's gold merged in.
jinja/1465 all combined.patch carried the PR's test-file diffs and could not apply.
No feature.patch changed. Reasoning per feature in dataset/SPEC_AUDIT.md ("Second pass").
Also: three task images (react-hook-form 153/85, huggingface-datasets 3997) had no arm64
manifest; arm64 builds were added next to the unchanged amd64 ones. task85's Dockerfile pins
pnpm@8 (unpinned resolves to v10, which refuses to build). scripts/check_combined.py is the
new sweep; check_gradeable.py gains --backend docker, chunked sandbox writes (Modal ARG_MAX),
and a cached digest lookup (Docker Hub anonymous rate limit).
Verified: all 199 pass all three checks on Modal (linux/amd64) and local Docker (linux/arm64).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second verification pass over all 199 features, adding the third dataset invariant next to fail-on-base / pass-on-gold: each feature's tests must pass against its task's
combined.patch. That is the property the coop eval depends on (test_mergedruns a feature's tests against the merged tree) and it had never been measured.19 features in 5 tasks failed it; all fixed, no
feature.patchchanged:ToolCalls.format()'s return shape; pre-existing tests in the shared file asserted the old shape, so every tree with f1 failed them. 8/15 pairs were unwinnable.tests.patchaccept both shapesmin_instr_chars=30replaces the base tests' dummy instruction; #40 had set the default to 0 in combined.patch only. 5/15 pairs were unwinnable.priority=Also
react_hook_form_task/task85/Dockerfilepinspnpm@8(unpinned resolves to v10, which refuses to build).scripts/check_combined.py(new),scripts/check_gradeable.py --backend docker, chunked sandbox writes (Modal ARG_MAX), cached digest lookup (Docker Hub rate limit).Verified: all 199 features pass all three checks on Modal (linux/amd64) and local Docker (linux/arm64). Under 4–16-way concurrency, dspy/8587 (10 s server-readiness wait in dspy's own test infra) and typst (parallel cargo builds) can fail transiently; both pass serially. Reasoning per feature in
dataset/SPEC_AUDIT.md. HF dataset already updated.