Skip to content

docs(tasks): scope the record protocol to both sides of the contract - #82

Merged
ethan-scitix merged 1 commit into
mainfrom
docs/tasks-record-protocol-scope
Aug 8, 2026
Merged

docs(tasks): scope the record protocol to both sides of the contract#82
ethan-scitix merged 1 commit into
mainfrom
docs/tasks-record-protocol-scope

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Type

  • docs — documentation only

Summary

Follow-up to #68, which is where the sizes below were noticed. This is a scoping fix, not a compression one — for a task author the loaded total goes 1485 → 1460 words, which is noise. Three separate things:

  • The record protocol was visible to only one party. sieval/tasks/CLAUDE.md held the vocabulary for types defined in sieval/core/tasks/records.py, and CLAUDE.md loads by directory proximity — so someone editing records.py, profiler.py or runner.py never saw it. That is concrete for grader_output: the key name is a contract the runner reads back to route grader spend into profile.json, and the side most likely to rename it is the side that could not see the rule. Moved to .claude/rules/records.md, whose frontmatter globs sieval/tasks/** and sieval/core/tasks/**.
  • ## Data Flow — Async & Concurrency was byte-identical in .claude/rules/tasks.md, which globs the same sieval/tasks/**/*.py — so both copies load together. That is not the intended rules-are-a-checklist / CLAUDE.md-is-principles overlap; it is the same three bullets twice. Dropped from CLAUDE.md, kept in the rules file.
  • Constructor knobs restated what an enforcer already checks. check_task_shot_knobs machine-verifies the spellings across every constructor under sieval/tasks/. Trimmed 234 → 172 words, keeping only what the checker cannot say: assigning self._n_shot instead of self.n_shot leaves the class default standing, so meta.json reports the declared shot count rather than the one the run used.

Net: sieval/tasks/CLAUDE.md gets a coherent charter — how a task is declared and where it lives — at 948 words instead of 1485.

Related Issues

Refs #68.

Ordering: #68 has since merged (1c15c00c) and this branch is rebased onto it. The prediction held — #68 touched only the ## Naming Conventions / ### Variants region of sieval/tasks/CLAUDE.md, this PR touches the sections below it, and the rebase was conflict-free. Word counts above are against post-#68 main.

Test Plan

Automated

  • Lint/format clean — markdownlint passes (pre-commit)
  • check_preflight.py --level quick — 22 PASS, 0 FAIL (only the deep-link reachability SKIP)
  • No code changed, so no test impact

Manual

  • Coupled sites checked per .claude/rules/engineering-infra.md: the inbound references to sieval/tasks/CLAUDE.md (.claude/rules/tasks.md ×2, sieval/tasks/arc/_base.py, sieval/core/tasks/meta.py, scripts/check_preflight.py) point at the "Task Metadata" and "ppl vs clp" sections, both kept. CONTRIBUTING.md does not mirror the record protocol.
  • Duplication verified before deleting — the three Data Flow bullets compared byte-for-byte against .claude/rules/tasks.md.
  • New rule frontmatter matches the existing convention (paths: glob list, as in tasks.md / datasets.md).
  • No enforcer behaviour changes — check_record_key_access and check_task_shot_knobs are untouched and still enforce what the prose now defers to.

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — N/A, no code
  • No new upper-layer dependencies added to core/
  • Deleted code verified — prose only; every deletion is either a verbatim duplicate that survives in .claude/rules/tasks.md or a restatement of a machine-checked rule

🤖 Generated with Claude Code

Not a compression change, despite moving 462 words -- for a task author the
loaded total goes 1264 -> 1239, which is noise. Three separate things:

**The record protocol was visible to only one party.** `sieval/tasks/CLAUDE.md`
held the vocabulary for types defined in `sieval/core/tasks/records.py`, and
CLAUDE.md loads by directory proximity -- so someone editing `records.py`,
`profiler.py` or `runner.py` never saw it. That matters concretely for
`grader_output`: the name is a contract the runner reads back to route grader
spend into `profile.json`, and the side most likely to rename it is the side
that could not see the rule. Moved to `.claude/rules/records.md`, whose
frontmatter globs `sieval/tasks/**` *and* `sieval/core/tasks/**`. Task authors
load the same content as before; core authors now load it too.

**`## Data Flow — Async & Concurrency` was byte-identical in
`.claude/rules/tasks.md`**, which globs the same `sieval/tasks/**/*.py`. Not the
intended rules-are-a-checklist / CLAUDE.md-is-principles overlap -- the same
three bullets twice, loaded together. Dropped here, kept in the rules file,
which is where checklist-shaped text belongs.

**`Constructor knobs` restated what an enforcer already checks.** The spellings
are machine-verified by `check_task_shot_knobs` across every constructor under
`sieval/tasks/`; repeating the accepted-name list in prose adds nothing a
violation would not surface. Trimmed 234 -> 172 words, keeping only what the
checker cannot say: assigning `self._n_shot` instead of `self.n_shot` leaves the
class default standing, so `meta.json` reports the declared shot count rather
than the one the run used.

Net effect on `sieval/tasks/CLAUDE.md` is a coherent charter -- how a task is
declared and where it lives -- at 727 words instead of 1264.

No coupled site breaks: the inbound references from `.claude/rules/tasks.md` and
`sieval/tasks/arc/_base.py` point at the "Task Metadata" and "ppl vs clp"
sections, both kept, and `CONTRIBUTING.md` does not mirror the protocol.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ethan-scitix
ethan-scitix force-pushed the docs/tasks-record-protocol-scope branch from 7aa2329 to ec03e2b Compare August 8, 2026 08:30
@ethan-scitix
ethan-scitix merged commit 23d73b1 into main Aug 8, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the docs/tasks-record-protocol-scope branch August 8, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant