docs(tasks): scope the record protocol to both sides of the contract - #82
Merged
Conversation
Not a compression change, despite moving 462 words -- for a task author the loaded total goes 1264 -> 1239, which is noise. Three separate things: **The record protocol was visible to only one party.** `sieval/tasks/CLAUDE.md` held the vocabulary for types defined in `sieval/core/tasks/records.py`, and CLAUDE.md loads by directory proximity -- so someone editing `records.py`, `profiler.py` or `runner.py` never saw it. That matters concretely for `grader_output`: the name is a contract the runner reads back to route grader spend into `profile.json`, and the side most likely to rename it is the side that could not see the rule. Moved to `.claude/rules/records.md`, whose frontmatter globs `sieval/tasks/**` *and* `sieval/core/tasks/**`. Task authors load the same content as before; core authors now load it too. **`## Data Flow — Async & Concurrency` was byte-identical in `.claude/rules/tasks.md`**, which globs the same `sieval/tasks/**/*.py`. Not the intended rules-are-a-checklist / CLAUDE.md-is-principles overlap -- the same three bullets twice, loaded together. Dropped here, kept in the rules file, which is where checklist-shaped text belongs. **`Constructor knobs` restated what an enforcer already checks.** The spellings are machine-verified by `check_task_shot_knobs` across every constructor under `sieval/tasks/`; repeating the accepted-name list in prose adds nothing a violation would not surface. Trimmed 234 -> 172 words, keeping only what the checker cannot say: assigning `self._n_shot` instead of `self.n_shot` leaves the class default standing, so `meta.json` reports the declared shot count rather than the one the run used. Net effect on `sieval/tasks/CLAUDE.md` is a coherent charter -- how a task is declared and where it lives -- at 727 words instead of 1264. No coupled site breaks: the inbound references from `.claude/rules/tasks.md` and `sieval/tasks/arc/_base.py` point at the "Task Metadata" and "ppl vs clp" sections, both kept, and `CONTRIBUTING.md` does not mirror the protocol. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ethan-scitix
force-pushed
the
docs/tasks-record-protocol-scope
branch
from
August 8, 2026 08:30
7aa2329 to
ec03e2b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
Follow-up to #68, which is where the sizes below were noticed. This is a scoping fix, not a compression one — for a task author the loaded total goes 1485 → 1460 words, which is noise. Three separate things:
sieval/tasks/CLAUDE.mdheld the vocabulary for types defined insieval/core/tasks/records.py, and CLAUDE.md loads by directory proximity — so someone editingrecords.py,profiler.pyorrunner.pynever saw it. That is concrete forgrader_output: the key name is a contract the runner reads back to route grader spend intoprofile.json, and the side most likely to rename it is the side that could not see the rule. Moved to.claude/rules/records.md, whose frontmatter globssieval/tasks/**andsieval/core/tasks/**.## Data Flow — Async & Concurrencywas byte-identical in.claude/rules/tasks.md, which globs the samesieval/tasks/**/*.py— so both copies load together. That is not the intended rules-are-a-checklist / CLAUDE.md-is-principles overlap; it is the same three bullets twice. Dropped from CLAUDE.md, kept in the rules file.Constructor knobsrestated what an enforcer already checks.check_task_shot_knobsmachine-verifies the spellings across every constructor undersieval/tasks/. Trimmed 234 → 172 words, keeping only what the checker cannot say: assigningself._n_shotinstead ofself.n_shotleaves the class default standing, someta.jsonreports the declared shot count rather than the one the run used.Net:
sieval/tasks/CLAUDE.mdgets a coherent charter — how a task is declared and where it lives — at 948 words instead of 1485.Related Issues
Refs #68.
Ordering: #68 has since merged (
1c15c00c) and this branch is rebased onto it. The prediction held — #68 touched only the## Naming Conventions/### Variantsregion ofsieval/tasks/CLAUDE.md, this PR touches the sections below it, and the rebase was conflict-free. Word counts above are against post-#68main.Test Plan
Automated
markdownlintpasses (pre-commit)check_preflight.py --level quick— 22 PASS, 0 FAIL (only the deep-link reachability SKIP)Manual
.claude/rules/engineering-infra.md: the inbound references tosieval/tasks/CLAUDE.md(.claude/rules/tasks.md×2,sieval/tasks/arc/_base.py,sieval/core/tasks/meta.py,scripts/check_preflight.py) point at the "Task Metadata" and "ppl vs clp" sections, both kept.CONTRIBUTING.mddoes not mirror the record protocol..claude/rules/tasks.md.paths:glob list, as intasks.md/datasets.md).check_record_key_accessandcheck_task_shot_knobsare untouched and still enforce what the prose now defers to.Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstring — N/A, no codecore/.claude/rules/tasks.mdor a restatement of a machine-checked rule🤖 Generated with Claude Code