release: 1.8.0 — per-site metrics, per-case predictions, warm-start guard, deploy test on real kits - #577
Merged
Conversation
…in-memory shm race) Both found by the 1.8.0 MVP deploy test and fixed in this PR; both read as something else at first (F11 as an aggregation bug, F12 as F7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 3e951cd0e8678b515c0045a088e8ce8bca42a2e4)
…uard, deploy test on real kits Version bump and changelog only. The image and the :current re-tag are pushed manually after the four MVP deploy tests pass on real kits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Documents what was previously only in the operator's head: MEDISWARM_IMAGE_VERSION must be exported for a clean tag, :current is the only update channel sites have (skipped for 1.7.0), provisioning the production project reuses the root CA in workspace/odelia_allsites/state so new kits are drop-in, and a published GitHub release fires the deploy test on the runners. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One restart for the 1.6.0-kit sites, a drop-in kit for the two 1.5.0-kit sites, and a pre-send checklist tied to the runbook. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…eted run The server deletes the run dir on completion and keeps the workspace in its job store inside the container; download_job is the retrieval path. Found when the deploy test's clean 20-round run saved nothing on 13 Sep. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Version bump and release paperwork for 1.8.0. No code changes.
odelia_image.version→ 1.8.0 (shared with DECADE; only the next build moves)CHANGELOG.md→ the 1.8.0 section: per-site metrics (feat(metrics): publish per-site metrics with class support (#525, #441) #534), opt-in per-case predictions (feat(metrics): return per-case predictions, opt-in per site (#526, #527, #528) #556/feat: wire per-case prediction return end to end (#526, #527, #528) #557), warm-start provenance + structural guard (Label warm-start checkpoints and refuse another model's weights (#535) #545/fix: refuse wrong-architecture warm-start checkpoints structurally; cap DataLoader workers (#574) #575), loader-worker cap (fix: refuse wrong-architecture warm-start checkpoints structurally; cap DataLoader workers (#574) #575), strict exact-client runs (Make strict ODELIA swarm runs reliable and exact-client by default #514), the deliverable groundwork (feat(active-learning): sample-selection strategies for D3.2 (#527) #555/feat: MVPs for active learning (D3.2) and privacy accounting (T3.3 Phase 2) #563/feat(d3.4): robust aggregation rules and a poisoning simulation (#529) #566/fix(d3.2): retract the active-learning null result — it was computed on wrong-architecture predictions #568/feat(ms6): white-hat site-inference probe we can run without partner scheduling (#531) #571), deploy-test fixes (Make the 2-node deploy test able to run at all (ports, min_clients, gitignore) #544/fix(deploy-test): make the 2-node deploy test run on real kits (rename sites, decouple data folder, mirror hygiene, staged jobs) #573), weekly preflight (fix(ci): build startup kits before the weekly all-models preflight #552/539 run preflight check for all odelia models #559/fixed output of runIntegrationTests.sh #572), supply chain (Supply chain: digest-pin the base, keep apt unpinned, scan for CVEs (#395) #446)docs/SWARM_FAILURE_MODES.md→ F11 (unlabelled wrong-architecture mirror) and F12 (pin-memory shm race), both found by the 1.8.0 deploy testsdocs/RELEASE_RUNBOOK.md→ the manual release procedure, verified step by step on 13 Sep:MEDISWARM_IMAGE_VERSIONmust be exported for a clean tag,:currentis the only update channel sites have (it was never moved for 1.7.0 — Docker Hub has no 1.7.0 tag at all), provisioning the allsites project reuses the root CA inworkspace/odelia_allsites/stateso new kits are drop-in, and a published GitHub release fires the deploy test on the runnersdocs/EMAIL_release_1.8.0.md→ announcement draft with per-site actions taken from the sites' own heartbeats (one restart for 1.6.0 kits; a kit or--imagefor the two 1.5.0 kits)Evidence the release is ready
The four MVP deploy tests on real kits (server on dl3, clients dl0 + dl2, image
1.7.0-dev.260913.19f8613= main + #573 + #575) — results recorded on this PR as they complete:loader workers capped 16 -> 9, no pin-memory failure (mvp1_rc2_baseline_260913_0717, 3892 s). Metrics were published but the harness could not save them — see fix(deploy-test): make the 2-node deploy test run on real kits (rename sites, decouple data folder, mirror hygiene, staged jobs) #573's job-store fix.provenance OK -- written by 'MST',structure OK -- 187/187 parameter names shared, warm-started inrequiremode, 20 rounds PASSED (mvp3_guard_accept_260913_0835, 3851 s);cross_val_results.jsonwith both sites and per-class support recovered from the job storeTEST_B_1, that client loggedcarries no provenanceand refused withWARM_START_MODEL_MISMATCHon all three NVFlare retries;TEST_A_1accepted its labelled MST mirror (structure OK -- 187/187); the run aborted as it must (mvp3_guard_refuse_260913_1418, harness verdictFAILED: MST (training fail)= the expected outcome)ODELIA_RETURN_PER_CASE=1on both clients,cross_site_val/per_case_predictions/TEST_A_1.csv(10 rows) andTEST_B_1.csv(26 rows) came back —row_index,ground_truth,prob_class_0..2, no identifiers — matching each site'sval/nincross_val_results.json; 20 rounds PASSED (mvp2_percase_260913_1426, 3821 s)4625edc(fix(deploy-test): make the 2-node deploy test run on real kits (rename sites, decouple data folder, mirror hygiene, staged jobs) #573 + fix: refuse wrong-architecture warm-start checkpoints structurally; cap DataLoader workers (#574) #575 + ci: require both simulated clients to configure before the swarm starts (#576) #578): 34775138617 green, all six models' preflight plus simulation and two-client swarm — the first green weekly since 16 Aug (Nothing validates main after a merge, and the weekly check has been red for a month #554)🤖 Generated with Claude Code