Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 36 additions & 4 deletions docs/plans/model-graph-plan.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Pod model graph: plan

Drafted 2026-11-15 from the 2026-11-14/15 training session; built out the same day after
checking core `dev`, the pod and local data. Status: accepted 2026-11-15. The owner took the
checking core `dev`, the pod and local data. Status: accepted 2026-11-15; implemented 2026-11-16 (see Implementation status). The owner took the
recommended answer to every decision (Q1–Q7), and the other sleeper consented to archiving (Q2).
Results quoted below are one participant's retrospective aggregates, not validation.

Expand Down Expand Up @@ -35,6 +35,37 @@ a laptop replay of the pod's own vitals engine. The new work is mostly glue (an
index, replayed training inputs), a model-set contract, and a search harness with a locked
holdout. The urgent item is A1: the pod is pruning about one night of raw per day.

## Implementation status (2026-11-16)

Every item not gated by data or a pending result is implemented. Training PRs are merged;
core and iOS PRs are open for review (they change what the pod and phone run).

| Item | State | Where |
| --- | --- | --- |
| A1 + B3 archive | Done: first pull 2,477 RAW, 1.1M rows; nightly launchd agent at 12:30 | core #925 (`--curl`), training #25 |
| A1b, A3 index, lineage, Watch coverage | Done; lineage from processing runs (core #930 identity) | training #25, #30 |
| A2 nightly Watch motion | Code done; 7-night on-device check pending | iOS #109 |
| A5 locked holdout | Done | training #26, #38 |
| A6 archive inputs | Done | training #28, #31 |
| B1/B2/B4/B5 dashboard | Done | training #32, #35, #37 |
| C1 profile rebase (Q1: auto-activate) | PR open | core #927 |
| C2 re-pin | Likely unneeded: #927 restores the saved overrides on the next deploy | — |
| C3 robust breathing display | PR open; floor may read ~0.7/min low once #927 deploys | core #926 |
| C4 breathing ranking | PR open | core #929 |
| C5 replay from archive + derived snapshot | Core PR open; training merged | core #928, training #27 |
| D1 movement export | Done | training #24 |
| D2–D4, E3 | Gated: ~20 Watch-motion nights | sleepypod-core-132/133/139, training-4 |
| E1 weekly trial | Tooling done; first scorable night 11-15; Q5 on 11-28 | training #29, training-18 |
| E2 replayed vs live | Run 2026-11-16: tie (+0.000 [−0.036, +0.037]); no export | knowledge/evaluation/e2-replayed-vitals-2026-11-16.md |
| E4 two-step format | Gated on an F final confirming it | sleepypod-core-138 |
| F search harness | Done; reproduces the cross-fitted fusion run exactly | training #34 |
| Model set + Q6 (`inputMismatch`) | PR open | core #930 |

Found while implementing: macOS Local Network privacy blocks node and Python (not `curl`) on
this machine, hence the curl transports (core #925, training #22); per-minute replay parity
cannot reach 0.1 bpm (C5 gate above); the stage-model evaluator compares float64 inputs where
sklearn uses float32 (sleepypod-core-140).

## Where things stand

### Code changes
Expand Down Expand Up @@ -331,9 +362,10 @@ worker request in Python:

Acceptance:
- training gate (core #928 `parity.trainingGate`), judged on nights whose stored rows all ran the
replayed config, over spans where replay had RAW: ≥60 compared minutes, replayed/stored HR
minutes 0.90–1.15, |bias| ≤0.25 bpm, median |Δ| ≤0.5 bpm, ≥70% of minutes within 1 bpm; a
failing night keeps its stored rows. The originally planned 0.1 bpm on ≥95% of minutes cannot
replayed config, inside the night's sleep sessions where replay had RAW: ≥60 compared minutes,
replayed/stored HR minutes 0.90–1.20, |trimmed bias| ≤0.25 bpm (minutes within 5 bpm), median
|Δ| ≤0.6 bpm, ≥70% of minutes with both HRs within 1 bpm, ≤10% jumps over 5 bpm; a failing
night keeps its stored rows. Recalibrated on 17 archived nights (16 pass with defaults). The originally planned 0.1 bpm on ≥95% of minutes cannot
pass: live and replay average windows offset by up to 30 s each minute (2026-10-19: 13% within
0.1, 75% within 1, bias −0.11). Thresholds rest on two nights; confirm on the first five gated
archived nights. The 0.1 bpm check stays for pod-vs-laptop backtest parity (`local-fit parity`)
Expand Down
40 changes: 40 additions & 0 deletions knowledge/evaluation/e2-replayed-vitals-2026-11-16.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
type: "Experimental finding"
title: "Replayed vs live vitals for staging (E2): 2026-11-16"
description: "Retraining podContext on vitals replayed through one pinned engine and profile ties training on the mixed-version live rows; replayed breathing tracks the Watch more closely."
status: "draft"
generated: {"by": "claude/opus-5.5", "at": "2026-11-16T01:30:00+00:00"}
sources: [{"resource": "Local run data/runs/e2-20261116/report.json (ignored)", "title": "search.py report, E2"}, {"resource": "Local replay data/replay/c2-v2-20261116/replay.json (ignored)", "title": "core biometrics-local-fit replay, profile 04f7a165"}]
---

# Replayed vs live vitals for staging (E2): 2026-11-16

Question (plan E2, pre-registered before the run): does training the stage model on vitals
from one pinned engine and profile beat training on the live rows, which came from at least
seven engine/profile pairs across these nights (archive lineage)?

Setup: `search.py` spec `e2-replayed-c2-vs-live`, primary comparison
`replayed+none+podContext` vs `live+none+podContext`, macro F1, walk-forward, forest
`balanced_leaf5`, the same 24 night groups (2026-10-18 → 11-13, left side, one participant).
Replayed vitals came from core `biometrics-local-fit replay --archive` (core #928) with engine
`8865a12b` and the left side's saved profile `04f7a165` (the profile core #927 restores after a
deploy). Core's training gate judged the 10 nights that live had run on `04f7a165`; all 10
passed. The other 18 nights ran another profile and were accepted ungated, as pre-registered.
No locked holdout night had Watch labels yet, so none was read.

| Candidate | Walk-forward macro F1 | κ |
| --- | ---: | ---: |
| live vitals | 0.524 | 0.488 |
| replayed vitals | 0.524 | 0.497 |

Primary: +0.000 macro F1, night bootstrap 95% [−0.036, +0.037], 9 of 16 test nights better,
sign test p = 0.80. Under the plan's rule this is a tie: the mixed vitals versions did not
measurably hurt staging, and nothing is exported. Replay stays the input for vitals tuning
(C4) and the search harness (F).

Vitals against the Watch on the same epochs: HR MAE 1.59 (live) vs 1.60 bpm (replayed);
breathing MAE 1.80 vs 1.11 breaths/min; HRV rank ρ 0.51 for both. The breathing gain fits
the November nights having run on defaults, which read about 2/min above the Watch.

Watch stages are the reference, not polysomnography. Retrospective; no model was exported,
uploaded or activated.
1 change: 1 addition & 0 deletions knowledge/evaluation/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Evaluation

- [Replayed vs live vitals for staging (E2): 2026-11-16](e2-replayed-vitals-2026-11-16.md) — One pinned engine and profile ties the mixed-version live rows (macro F1 0.524 both, +0.000 [−0.036, +0.037]); replayed breathing is closer to the Watch (MAE 1.11 vs 1.80).
- [Locked holdout](locked-holdout.md) — From the night of 2026-11-14, every fourth night group is locked: only a deployed model's trial and one final read of a finished candidate may score it, and every read is logged.
- [Cross-fitted fusion rerun: 2026-11-15](fusion-cross-fitted-2026-11-15.md) — With the movement-teacher leak removed, podContext ties the live pod model on 9 unseen nights (0.545 vs 0.533 macro F1); wake-first is best (+0.030, exploratory); movement-model features add nothing.
- [Literature feature ablation: 2026-11-03](feature-ablation-2026-11-03.md) — On 23 night groups, temporal-context and arousal features from pod data raised light F1 from 0.68 to 0.75–0.78 and κ from 0.38 to 0.46–0.48; a chronological backtest kept the gain, with the flat forest the balanced choice. Wake stayed at zero recall.
Expand Down
4 changes: 4 additions & 0 deletions knowledge/log.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Knowledge update log

## 2026-11-16

- **Creation**: Captured [E2, replayed vs live vitals](evaluation/e2-replayed-vitals-2026-11-16.md): a tie on staging, better breathing agreement. No model export or activation.

## 2026-11-15

- **Creation**: Set the [locked holdout](evaluation/locked-holdout.md) from the model-graph plan (decision Q4): freeze 2026-11-14, every fourth night group locked, trial and final reads only, each logged in `data/holdout-access.jsonl`. Stage scripts drop locked nights; dashboards hide their scores. No model activation.
Expand Down
Loading