From e4a8f535817ece42b8ec1b012e862ee1135d6a51 Mon Sep 17 00:00:00 2001 From: Jonathan Ng Date: Sun, 15 Nov 2026 17:37:25 -0800 Subject: [PATCH] docs: record E2's tie and the plan's implementation status so the next step starts from what landed Co-Authored-By: Claude Opus 5.5 --- docs/plans/model-graph-plan.md | 40 +++++++++++++++++-- .../e2-replayed-vitals-2026-11-16.md | 40 +++++++++++++++++++ knowledge/evaluation/index.md | 1 + knowledge/log.md | 4 ++ 4 files changed, 81 insertions(+), 4 deletions(-) create mode 100644 knowledge/evaluation/e2-replayed-vitals-2026-11-16.md diff --git a/docs/plans/model-graph-plan.md b/docs/plans/model-graph-plan.md index 3637a43..a283baa 100644 --- a/docs/plans/model-graph-plan.md +++ b/docs/plans/model-graph-plan.md @@ -1,7 +1,7 @@ # Pod model graph: plan Drafted 2026-11-15 from the 2026-11-14/15 training session; built out the same day after -checking core `dev`, the pod and local data. Status: accepted 2026-11-15. The owner took the +checking core `dev`, the pod and local data. Status: accepted 2026-11-15; implemented 2026-11-16 (see Implementation status). The owner took the recommended answer to every decision (Q1–Q7), and the other sleeper consented to archiving (Q2). Results quoted below are one participant's retrospective aggregates, not validation. @@ -35,6 +35,37 @@ a laptop replay of the pod's own vitals engine. The new work is mostly glue (an index, replayed training inputs), a model-set contract, and a search harness with a locked holdout. The urgent item is A1: the pod is pruning about one night of raw per day. +## Implementation status (2026-11-16) + +Every item not gated by data or a pending result is implemented. Training PRs are merged; +core and iOS PRs are open for review (they change what the pod and phone run). + +| Item | State | Where | +| --- | --- | --- | +| A1 + B3 archive | Done: first pull 2,477 RAW, 1.1M rows; nightly launchd agent at 12:30 | core #925 (`--curl`), training #25 | +| A1b, A3 index, lineage, Watch coverage | Done; lineage from processing runs (core #930 identity) | training #25, #30 | +| A2 nightly Watch motion | Code done; 7-night on-device check pending | iOS #109 | +| A5 locked holdout | Done | training #26, #38 | +| A6 archive inputs | Done | training #28, #31 | +| B1/B2/B4/B5 dashboard | Done | training #32, #35, #37 | +| C1 profile rebase (Q1: auto-activate) | PR open | core #927 | +| C2 re-pin | Likely unneeded: #927 restores the saved overrides on the next deploy | — | +| C3 robust breathing display | PR open; floor may read ~0.7/min low once #927 deploys | core #926 | +| C4 breathing ranking | PR open | core #929 | +| C5 replay from archive + derived snapshot | Core PR open; training merged | core #928, training #27 | +| D1 movement export | Done | training #24 | +| D2–D4, E3 | Gated: ~20 Watch-motion nights | sleepypod-core-132/133/139, training-4 | +| E1 weekly trial | Tooling done; first scorable night 11-15; Q5 on 11-28 | training #29, training-18 | +| E2 replayed vs live | Run 2026-11-16: tie (+0.000 [−0.036, +0.037]); no export | knowledge/evaluation/e2-replayed-vitals-2026-11-16.md | +| E4 two-step format | Gated on an F final confirming it | sleepypod-core-138 | +| F search harness | Done; reproduces the cross-fitted fusion run exactly | training #34 | +| Model set + Q6 (`inputMismatch`) | PR open | core #930 | + +Found while implementing: macOS Local Network privacy blocks node and Python (not `curl`) on +this machine, hence the curl transports (core #925, training #22); per-minute replay parity +cannot reach 0.1 bpm (C5 gate above); the stage-model evaluator compares float64 inputs where +sklearn uses float32 (sleepypod-core-140). + ## Where things stand ### Code changes @@ -331,9 +362,10 @@ worker request in Python: Acceptance: - training gate (core #928 `parity.trainingGate`), judged on nights whose stored rows all ran the - replayed config, over spans where replay had RAW: ≥60 compared minutes, replayed/stored HR - minutes 0.90–1.15, |bias| ≤0.25 bpm, median |Δ| ≤0.5 bpm, ≥70% of minutes within 1 bpm; a - failing night keeps its stored rows. The originally planned 0.1 bpm on ≥95% of minutes cannot + replayed config, inside the night's sleep sessions where replay had RAW: ≥60 compared minutes, + replayed/stored HR minutes 0.90–1.20, |trimmed bias| ≤0.25 bpm (minutes within 5 bpm), median + |Δ| ≤0.6 bpm, ≥70% of minutes with both HRs within 1 bpm, ≤10% jumps over 5 bpm; a failing + night keeps its stored rows. Recalibrated on 17 archived nights (16 pass with defaults). The originally planned 0.1 bpm on ≥95% of minutes cannot pass: live and replay average windows offset by up to 30 s each minute (2026-10-19: 13% within 0.1, 75% within 1, bias −0.11). Thresholds rest on two nights; confirm on the first five gated archived nights. The 0.1 bpm check stays for pod-vs-laptop backtest parity (`local-fit parity`) diff --git a/knowledge/evaluation/e2-replayed-vitals-2026-11-16.md b/knowledge/evaluation/e2-replayed-vitals-2026-11-16.md new file mode 100644 index 0000000..0e5c9bf --- /dev/null +++ b/knowledge/evaluation/e2-replayed-vitals-2026-11-16.md @@ -0,0 +1,40 @@ +--- +type: "Experimental finding" +title: "Replayed vs live vitals for staging (E2): 2026-11-16" +description: "Retraining podContext on vitals replayed through one pinned engine and profile ties training on the mixed-version live rows; replayed breathing tracks the Watch more closely." +status: "draft" +generated: {"by": "claude/opus-5.5", "at": "2026-11-16T01:30:00+00:00"} +sources: [{"resource": "Local run data/runs/e2-20261116/report.json (ignored)", "title": "search.py report, E2"}, {"resource": "Local replay data/replay/c2-v2-20261116/replay.json (ignored)", "title": "core biometrics-local-fit replay, profile 04f7a165"}] +--- + +# Replayed vs live vitals for staging (E2): 2026-11-16 + +Question (plan E2, pre-registered before the run): does training the stage model on vitals +from one pinned engine and profile beat training on the live rows, which came from at least +seven engine/profile pairs across these nights (archive lineage)? + +Setup: `search.py` spec `e2-replayed-c2-vs-live`, primary comparison +`replayed+none+podContext` vs `live+none+podContext`, macro F1, walk-forward, forest +`balanced_leaf5`, the same 24 night groups (2026-10-18 → 11-13, left side, one participant). +Replayed vitals came from core `biometrics-local-fit replay --archive` (core #928) with engine +`8865a12b` and the left side's saved profile `04f7a165` (the profile core #927 restores after a +deploy). Core's training gate judged the 10 nights that live had run on `04f7a165`; all 10 +passed. The other 18 nights ran another profile and were accepted ungated, as pre-registered. +No locked holdout night had Watch labels yet, so none was read. + +| Candidate | Walk-forward macro F1 | κ | +| --- | ---: | ---: | +| live vitals | 0.524 | 0.488 | +| replayed vitals | 0.524 | 0.497 | + +Primary: +0.000 macro F1, night bootstrap 95% [−0.036, +0.037], 9 of 16 test nights better, +sign test p = 0.80. Under the plan's rule this is a tie: the mixed vitals versions did not +measurably hurt staging, and nothing is exported. Replay stays the input for vitals tuning +(C4) and the search harness (F). + +Vitals against the Watch on the same epochs: HR MAE 1.59 (live) vs 1.60 bpm (replayed); +breathing MAE 1.80 vs 1.11 breaths/min; HRV rank ρ 0.51 for both. The breathing gain fits +the November nights having run on defaults, which read about 2/min above the Watch. + +Watch stages are the reference, not polysomnography. Retrospective; no model was exported, +uploaded or activated. diff --git a/knowledge/evaluation/index.md b/knowledge/evaluation/index.md index ca05071..6c187ef 100644 --- a/knowledge/evaluation/index.md +++ b/knowledge/evaluation/index.md @@ -1,5 +1,6 @@ # Evaluation +- [Replayed vs live vitals for staging (E2): 2026-11-16](e2-replayed-vitals-2026-11-16.md) — One pinned engine and profile ties the mixed-version live rows (macro F1 0.524 both, +0.000 [−0.036, +0.037]); replayed breathing is closer to the Watch (MAE 1.11 vs 1.80). - [Locked holdout](locked-holdout.md) — From the night of 2026-11-14, every fourth night group is locked: only a deployed model's trial and one final read of a finished candidate may score it, and every read is logged. - [Cross-fitted fusion rerun: 2026-11-15](fusion-cross-fitted-2026-11-15.md) — With the movement-teacher leak removed, podContext ties the live pod model on 9 unseen nights (0.545 vs 0.533 macro F1); wake-first is best (+0.030, exploratory); movement-model features add nothing. - [Literature feature ablation: 2026-11-03](feature-ablation-2026-11-03.md) — On 23 night groups, temporal-context and arousal features from pod data raised light F1 from 0.68 to 0.75–0.78 and κ from 0.38 to 0.46–0.48; a chronological backtest kept the gain, with the flat forest the balanced choice. Wake stayed at zero recall. diff --git a/knowledge/log.md b/knowledge/log.md index 9c6c212..79a4def 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -1,5 +1,9 @@ # Knowledge update log +## 2026-11-16 + +- **Creation**: Captured [E2, replayed vs live vitals](evaluation/e2-replayed-vitals-2026-11-16.md): a tie on staging, better breathing agreement. No model export or activation. + ## 2026-11-15 - **Creation**: Set the [locked holdout](evaluation/locked-holdout.md) from the model-graph plan (decision Q4): freeze 2026-11-14, every fourth night group locked, trial and final reads only, each logged in `data/holdout-access.jsonl`. Stage scripts drop locked nights; dashboards hide their scores. No model activation.