docs: add a validation protocol for scoring changes

[vishk23]

Why

Six accuracy findings about sleep scoring reversed in a single review when a fresh reader re-measured them instead of inheriting them — a contamination count, three agreement statistics, the sign of a bias, and the direction of a stage error. One of them rested on a mechanism that cannot physically occur, because a single transaction writes both of the streams it claimed had diverged.

None were arithmetic mistakes. They were process failures, and the same process produces them again. This adds docs/VALIDATION_PROTOCOL.md — eight rules and a paste-in PR checklist — so that a scoring change has a defined bar to clear before it is described as an improvement.

The rules, in one line each

1. Pre-register the prediction, threshold, and evaluation domain before measuring.
2. Never score against a reference the scorer produced — run E.-1 every time and use the exclusion list it prints.
3. Re-derive, never cite — a number ships with the command that regenerates it.
4. Match the instrument to the question — signed per-stage bias for fractions, minutes for timing, kappa only next to its marginals.
5. Hold out a temporal split and report the gap.
6. Check the base rate, and check the mechanism is reachable in code at all.
7. Definitions travel with the number — n, instrument, units, measured-vs-inherited.
8. A negative result is a result.

The numbers in the doc were measured, not assumed

Everything quantitative in it was re-derived against live wearable data while writing it:

• Evaluation domain moved sleep/wake kappa from 0.095 to 0.795 — same nights, same reference (the strap's own band sleep_state), same predictor. The only difference was scoring every band epoch versus only epochs inside the detected sleep sessions. That single unstated choice swings kappa further than most effects anyone is trying to measure. Worth knowing: Tools/SleepBench section C uses the in-session domain, so its kappa is not comparable to a whole-record figure.
• Coverage beats statistics for completeness. A coverage test (1 Hz samples banked / 86400) separated a genuinely truncated day from 18 complete ones with zero false positives — complete days at 99.5–100%, the truncated day at 36%. Statistical outlier tests on the scored value flagged 23–32% of all days and the better-specified one still missed the truncated day entirely.
• A remembered cross-source invariant did not hold. "Wrist ticks run 2–6× phone steps", re-derived on 30 qualifying days, held on 57% of them, and every violation was on the low side — the upper bound never bound at all.
• A one-parameter HR threshold is not temporally stable. Fitted on an early window and tested on a later one it lost 5.7 pp of balanced accuracy, and the later window's own optimum sat more than 10 bpm away, inside three weeks, on one person. The scored class balance drifted with it. Nothing in the current process would surface that.

Also recorded

The stored hypnogram vocabulary contains both wake and awake as distinct stage strings. Any consumer written as stage == "wake" silently misfiles the other spelling as sleep. Android already has a canonicalStage() normaliser with tests; Swift does not. The protocol calls this out under R7; the harness fix is filed separately rather than bundled here, since changing the harness mid-audit would change the instrument.

Scope

Docs only — docs/VALIDATION_PROTOCOL.md plus a link from both CONTRIBUTING.md files. No code, no behaviour change. No health data: the doc carries no dates, no absolute biometric values, and no database.

The bar it sets is deliberately narrow — a scoring change with no held-out number can still ship behind a flag; it just cannot be called an improvement in the changelog, a release note, or a source comment.

[ryanbr]

The document is good and I want it in. The PR as it stands cannot be merged, and the reason has nothing to do with its content.

The branch is carrying your fork's main

The body says "Docs only — docs/VALIDATION_PROTOCOL.md plus a link from both CONTRIBUTING.md files. No code, no behaviour change." That is an accurate description of your intent, and of exactly 3 of the 190 files in the diff.

The other 187 are 100 commits of unrelated work — Oura live-API import with OAuth2 and Keychain token storage, Strand/CloudSync/ (16 files), a new Rust/ component with uniffi bindings, plus 30 files under WhoopStore and 29 under StrandAnalytics. The last commits on the branch are fix(cloudsync): … and feat(scoring): …, not the doc. It reads as a branch cut from your fork's main rather than from this repo's, which also explains the six fields #982 named that do not exist here.

One thing to flag plainly, not as an accusation: Strand/CloudSync/ and Rust/ are pure additions — 2,080 and 1,416 lines, neither present on main. An uploader, a background-refresh and a push registration are inside CLAUDE.md's hard scope limits, which bar "a server, account, cloud sync, or sending any data off-device". So this is not merge-with-cleanup; the payload could not land here even deliberately. Worth saying so nobody assumes the diff is only noisy.

It is also CONFLICTING against current main, which has moved a fair amount since the 30th.

The fix

Re-cut from this repo's main and carry only the three files. Everything else in the diff is accidental.

On the document itself

Its repo-specific claims check out — E.-1 SELF-COMPARISON AUDIT really is at Tools/SleepBench/Sources/sleepbench/main.swift:424, and --exclude is a real flag that takes the list it prints. R2 and R3 are the strongest parts, and R3's framing is the one I would most want quoted back at me.

One correction to make before re-cutting. R3 says the benchmark behind the shipped sleep-staging default "exists only as a numeric claim inside a source comment. No dataset loader, no script, and no test in this repository reproduces it." That was true when you wrote it and is not true now — Tools/SleepPSG landed on main and scores SleepStagerV2 against PSG truth reproducibly. Your own #991. The liability R3 names as "the largest standing" is the one you then closed, so the rule should point at the harness rather than assert the gap.

Two smaller notes for the re-cut, neither blocking:

• The wake / awake vocabulary split is a real bug and I would rather see the Swift canonicalStage() normaliser as its own PR than left in a doc — a consumer written stage == "wake" silently misfiles the other spelling as sleep, which is a scoring error hiding in a string comparison.
• R4's note that SleepBench section C uses the in-session domain, and so is not comparable to a whole-record kappa, is worth putting in SleepBench's own header too. That is where someone will be standing when they misread it.

Re-open against current main with the three files and I will merge it.

[vishk23]

Re-cut as asked. The branch is now a single commit on top of current main (403c0b8, the 10.6.1 staging tip) carrying exactly the three files — docs/VALIDATION_PROTOCOL.md plus the link line in each CONTRIBUTING.md. The 187 accidental files are gone and the PR reports MERGEABLE.

The R3 correction is in. The paragraph no longer asserts the gap; it points at Tools/SleepPSG (#991) as the model the rule generalizes — a stage-accuracy claim now ships with its regenerating command (swift run sleeppsg --dataset …), which is the shape R3 asks of every number.

Re-verified the doc's repo-specific claims against today's main rather than the main of Aug 1: E.-1 SELF-COMPARISON AUDIT still prints from Tools/SleepBench/Sources/sleepbench/main.swift:424, --exclude still takes the list it prints, and the wake/awake split is still live (no normalizer on main), so that sentence still describes the tree.

On the two non-blocking notes: both deliberately held out of this diff to keep it docs-only — the canonicalStage() normalizer belongs in its own PR as you say, and the section-C domain caveat belongs in SleepBench's header where the misreading would happen. Follow-ups, not riders.