Skip to content

feat(training): model search harness with a fixed protocol and one logged holdout final - #34

Merged
ng merged 4 commits into
mainfrom
feat/search-harness
Nov 16, 2026
Merged

ng merged 4 commits into
mainfrom
feat/search-harness

Conversation

@ng

@ng ng commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Plan F (training-15): scripts/search.py, the model search harness. Based on main, after #27 (derived snapshots and model sets).

search.py plan   SPEC.json --out RUN     # expand the space, prepare and cache inputs, print cost
search.py run    RUN [--jobs N]          # score every candidate on development nights
search.py report RUN                     # ranking, layer scores, the pre-registered comparison
search.py final  RUN --candidate ID      # one locked-holdout read; logs it; writes model/set.json
  • Candidates. A candidate is one choice per node:

    • vitals: live rows, or one replayed engine and profile, named by hashes or by a profile config;
    • movementModel: none, or a movement_model.py model with its cross-fitted epochs;
    • stage: a podNative/podContext contract plus the Python groups context/movement/arousal/quality, and the model balanced_leaf5 or hierarchical.

    Candidates are named vitals+movementModel+stage, and the space is capped by budget.maxCandidates.

  • Caches. Each node's output is cached once per version under data/cache/: vitals/ (replay readings), snapshots/ (derived databases), features/, and movementModel/<model12>/<session>.json.

    • When a replayed profile isn't cached, plan prints the exact pnpm biometrics:local-fit replay --archive … --compare-stored and derive_snapshot.py ingest commands, and run refuses to start. Missing movement epochs get the same treatment, with the movement_model.py epochs command.
    • Search never shells out to core.
  • Protocol, fixed in code:

    • night groups are the unit, and every candidate is scored on the same sessions (those every choice covers); excluded sessions are listed with reasons;
    • locked nights are dropped by plan, run and report;
    • stages walk forward with MIN_TRAIN_GROUPS;
    • movement columns are cross-fitted through fusion.movement_rows, and a movement model taught on a locked night is refused;
    • comparisons are paired per night against the named baseline, with a night bootstrap (10,000 resamples, seed 42) and a sign test;
    • only primary is pre-registered; the winner is printed with the candidate count and labelled exploratory.
  • Gated replays. A replayed night is used only if core's training gate passed it, or, when the vitals choice sets "allowUngated": true, if core could not judge it. A rejected night is excluded for every candidate, with core's reason. The E2 spec sets allowUngated, because most of its nights stored a different profile; the reason is in docs/search.md.

  • Layer scores:

    • vitals: per-night HR/BR MAE and HRV Spearman against Watch vitals, with paired error reductions;
    • movement: the model's own leave-one-night-out strict recall at 1% still false alarms, against the pod score;
    • stages end to end: macro F1, κ, per-class F1, confusion, and TST/WASO error.
  • final.

    • Refuses a second final on the run, a candidate not in the run, and a run that isn't fully scored.
    • Fits on development nights, and exports and parity-checks the portable model (verify-portable.ts) before any locked night is read.
    • Then calls holdout.log_access(purpose='final'), scores the locked nights once, and writes final.json, model/candidate.json and model/set.json (modelset-v1; stage pinned by core's hash, vitals expected, training.inputs replayed|live). It never uploads.
    • Holdout scores go to final.json and the terminal only; report.json, which the dashboard reads, holds only the read count (holdout.accesses()).
  • Provenance. plan.json records the CLI command, script SHA-256s, input hashes (snapshot, Watch, readings per night, epochs and model, archive cursor), night lists per split, and the protocol with its SHA-256. Runs under data/ are registered in the run catalog as kind search, through run_history.register_output. The dashboard maps search to its own task, so these runs stay out of the sleep list until B5.

  • Docs. docs/search.md covers the spec reference, protocol, caches and the fusion arm mapping. It also has the prepared E2 spec (docs/search/e2-replayed-vs-live.json) and its command sequence; E2 is not run.

Reproduction and cost

  • Fusion: docs/search/fusion-xfit.json lists the fusion.py arms as candidates, on the same features, snapshot and cross-fitted movement epochs. On every arm, and on pod_live, the walk-forward pooled macro F1 is identical to data/runs/fusion-20261115-xfit/report.json to four decimals (difference 0.0000 against the 0.005 tolerance), on the same 1,165 epochs and 24 night groups. The same input and movement-model hashes were checked.
  • Cost: 24 candidates (six group sets × two stage models × {none, movementRaw-v1}) on those nights took 57 s wall with --jobs 4 (Apple M4): 8.3 s mean and 12 s max per candidate, with 576 forest fits. Vitals were cached; the replay itself is core's and runs once per profile.

Test plan

  • uv run --locked python -m unittest discover -s scripts -p 'test_*.py' (282 tests). New tests cover:
    • spec expansion and refusals;
    • the fixed-seed paired bootstrap;
    • movement cross-fitting (night T's training rows come from excluding[T]);
    • a synthetic plan → blocked → ingest → plan → run → report → final: one access-log entry, a valid set.json, the second final refused, and no locked night in any scored list.
  • All of README "Validate without personal data", including the new synthetic search steps (also in .github/workflows/validate.yml; the synthetic final writes its own access log).
  • Stage hash of an exported model checked against core's stageModelSha256(stageModelSchema.parse(model)): equal.
  • E2, once core's replay --archive lands and the archive is populated (sequence in docs/search.md).

No real health data in git or CI; never contacts a pod, uploads or activates.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 48 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 415b95a3-7635-44d1-b6f1-c689917e5223
Commits

Reviewing files that changed from the base of the PR and between 68243f0 and d262b1b.

Files selected for processing: 12

  • .github/workflows/validate.yml
  • README.md
  • docs/search.md
  • docs/search/e2-replayed-vs-live.json
  • docs/search/fusion-xfit.json
  • docs/search/synthetic.json
  • scripts/make-synthetic-inputs.py
  • scripts/run_history.py
  • scripts/search.py
  • scripts/test-dashboard.mjs
  • scripts/test_search.py
  • scripts/ui/dashboard.js
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ng
ng force-pushed the feat/derived-snapshot branch from 3f88bf5 to d9e4227 Compare November 16, 2026 00:22
@ng
ng force-pushed the feat/search-harness branch from 9d3d300 to 8a4c301 Compare November 16, 2026 00:22
@ng
ng force-pushed the feat/search-harness branch from 8a4c301 to d1d4be0 Compare November 16, 2026 00:34
@ng
ng force-pushed the feat/derived-snapshot branch from d9e4227 to 4acfc52 Compare November 16, 2026 00:37
@ng
ng force-pushed the feat/search-harness branch from d1d4be0 to 82130c0 Compare November 16, 2026 00:37
ng and others added 4 commits November 15, 2026 16:42
…ogged holdout final so candidate comparisons stay honest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and how long 24 candidates take

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…from every candidate so search compares on gated inputs only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onical-JSON port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ng
ng force-pushed the feat/search-harness branch from 82130c0 to d262b1b Compare November 16, 2026 00:44
@ng
ng changed the base branch from feat/derived-snapshot to main November 16, 2026 00:44
@ng
ng merged commit d21cff7 into main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant