Skip to content

feat(analysis): freeze the consortium run history into a committed dataset - #560

Merged
Ultimate-Storm merged 1 commit into
mainfrom
feat/run-history-dataset
Sep 10, 2026
Merged

Ultimate-Storm merged 1 commit into
mainfrom
feat/run-history-dataset

Conversation

@Ultimate-Storm

Copy link
Copy Markdown
Contributor

Extracts 120 runs, 2026-03-26 → 2026-09-09 (231 site-run rows) into a committed dataset, so the lessons-learned paper's figures regenerate from a file rather than from a directory that keeps changing.

Done now because the record is decaying: the coordinator retains only 22 jobs and prunes the rest.

Three things this had to get right — each had already produced a wrong answer

The site label is not the site. Internal machines (dl0, dl2, dl3) run under real hospital site names during deploy tests. heartbeat.json records the hostname that actually produced each run; for the third of the archive predating it, the kit path in the console log (deploy_test/ vs the site's own workspace) is the only surviving attribution.

Reading the directory name alone had produced four wrong join dates — including Utrecht joining in April with 122 volumes when it actually joined in July with 6,133.

Neither archive is the record. The live monitor holds 101 jobs of older per-site history; the coordinator holds 22 recent ones, 19 of which the monitor never received — including 87c5bbee, the twenty-round eight-site benchmark. Both are merged.

A log signature is not a failure cause. Validated against the 22 runs whose terminal status is still known:

signature in failures in successes verdict
liveness_timeout 7 0 discriminative
client_disconnected 3 0 discriminative
posix_spawn 2 1 advisory
vpn_or_peer_transfer 1 1 advisory

posix_spawn fires in 70 of 120 jobs including ones that completed cleanly, so counting it as a cause would have dominated a failure Pareto with a warning. Advisory signatures are recorded but never reported as the cause. One failure (831d16b4) matches no signature at all, so the taxonomy states its own incompleteness rather than implying full coverage.

Verification

Reconciles exactly against independently known ground truth — 8 sites, 34,462 training volumes, each site's three class counts summing to its n, and all four reference runs present with correct status and duration (87c5bbee 8 sites COMPLETED 1 d 19:52:47).

🤖 Generated with Claude Code

…taset

120 runs, 2026-03-26 to 2026-09-09, 231 site-run rows. Extracted now because
the record was decaying and partly untrustworthy.

Three things this had to get right, each of which had already produced a
wrong answer:

The site label is not the site. Internal machines (dl0, dl2, dl3) run under
real hospital site names during deploy tests, so records under RUMC_1 may
have come from dl0 and MHA_1 from dl2. heartbeat.json records the hostname
that actually produced each run; for the third of the archive predating it,
the kit path in the console log (deploy_test/ vs the site's own workspace)
is the only surviving attribution. Reading the directory name alone had
produced four wrong join dates, including Utrecht joining in April with 122
volumes when it joined in July with 6,133.

Neither archive is the record. The live monitor holds 101 jobs of older
per-site history; the coordinator holds 22 recent ones and prunes the rest,
19 of which the monitor never received -- including 87c5bbee, the
twenty-round eight-site benchmark. Both are merged.

A log signature is not a failure cause. Validated against the 22 runs whose
terminal status is still known: liveness_timeout (7 failures, 0 successes)
and client_disconnected (3, 0) discriminate; posix_spawn (2 failures but
also 1 success) and vpn_or_peer_transfer (1, 1) do not. posix_spawn fires in
70 of 120 jobs including ones that completed cleanly, so counting it as a
cause would have dominated a failure Pareto with a warning. Advisory
signatures are recorded but never reported as the cause, and one failure
matches no signature at all, so the taxonomy states its own incompleteness.

Reconciles exactly against ground truth: 8 sites, 34,462 training volumes,
each site's three class counts summing to its n, and all four reference runs
present with correct status and duration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants