feat(analysis): freeze the consortium run history into a committed dataset - #560
Merged
Merged
Conversation
…taset 120 runs, 2026-03-26 to 2026-09-09, 231 site-run rows. Extracted now because the record was decaying and partly untrustworthy. Three things this had to get right, each of which had already produced a wrong answer: The site label is not the site. Internal machines (dl0, dl2, dl3) run under real hospital site names during deploy tests, so records under RUMC_1 may have come from dl0 and MHA_1 from dl2. heartbeat.json records the hostname that actually produced each run; for the third of the archive predating it, the kit path in the console log (deploy_test/ vs the site's own workspace) is the only surviving attribution. Reading the directory name alone had produced four wrong join dates, including Utrecht joining in April with 122 volumes when it joined in July with 6,133. Neither archive is the record. The live monitor holds 101 jobs of older per-site history; the coordinator holds 22 recent ones and prunes the rest, 19 of which the monitor never received -- including 87c5bbee, the twenty-round eight-site benchmark. Both are merged. A log signature is not a failure cause. Validated against the 22 runs whose terminal status is still known: liveness_timeout (7 failures, 0 successes) and client_disconnected (3, 0) discriminate; posix_spawn (2 failures but also 1 success) and vpn_or_peer_transfer (1, 1) do not. posix_spawn fires in 70 of 120 jobs including ones that completed cleanly, so counting it as a cause would have dominated a failure Pareto with a warning. Advisory signatures are recorded but never reported as the cause, and one failure matches no signature at all, so the taxonomy states its own incompleteness. Reconciles exactly against ground truth: 8 sites, 34,462 training volumes, each site's three class counts summing to its n, and all four reference runs present with correct status and duration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracts 120 runs, 2026-03-26 → 2026-09-09 (231 site-run rows) into a committed dataset, so the lessons-learned paper's figures regenerate from a file rather than from a directory that keeps changing.
Done now because the record is decaying: the coordinator retains only 22 jobs and prunes the rest.
Three things this had to get right — each had already produced a wrong answer
The site label is not the site. Internal machines (
dl0,dl2,dl3) run under real hospital site names during deploy tests.heartbeat.jsonrecords the hostname that actually produced each run; for the third of the archive predating it, the kit path in the console log (deploy_test/vs the site's own workspace) is the only surviving attribution.Reading the directory name alone had produced four wrong join dates — including Utrecht joining in April with 122 volumes when it actually joined in July with 6,133.
Neither archive is the record. The live monitor holds 101 jobs of older per-site history; the coordinator holds 22 recent ones, 19 of which the monitor never received — including
87c5bbee, the twenty-round eight-site benchmark. Both are merged.A log signature is not a failure cause. Validated against the 22 runs whose terminal status is still known:
liveness_timeoutclient_disconnectedposix_spawnvpn_or_peer_transferposix_spawnfires in 70 of 120 jobs including ones that completed cleanly, so counting it as a cause would have dominated a failure Pareto with a warning. Advisory signatures are recorded but never reported as the cause. One failure (831d16b4) matches no signature at all, so the taxonomy states its own incompleteness rather than implying full coverage.Verification
Reconciles exactly against independently known ground truth — 8 sites, 34,462 training volumes, each site's three class counts summing to its
n, and all four reference runs present with correct status and duration (87c5bbee8 sites COMPLETED 1 d 19:52:47).🤖 Generated with Claude Code