release: 1.8.0 — per-site metrics, per-case predictions, warm-start guard, deploy test on real kits

CHANGELOG.md

@@ -2,6 +2,92 @@
 
 All notable changes to MediSwarm are documented in this file.
 
+## [1.8.0] - UNRELEASED
+
+Per-site evaluation and deploy-test release. A completed swarm run now reports what
+every site measured, can return per-case predictions from sites that opt in, and
+refuses to warm-start from another architecture's weights. The two-node deploy test
+runs end to end on real startup kits again, and it is where most of the fixes below
+were found. Every training-path change was confirmed on a real 2-node run (dl0 + dl2,
+server on dl3), not by unit tests alone.
+
+### Added
+
+- **Per-site metrics reach the coordinator (#534; #525, #441)** — each site's validation
+  metrics, including per-class support counts, are published to the server and returned
+  in `cross_val_results.json`. Previously the file was `{}` although the server logged
+  `Published metrics for N site(s)` (F10). Verified on real kits: both test sites present
+  with per-class support.
+- **Per-case predictions, opt-in per site (#556, #557; #526, #527, #528)** — a site that
+  starts its client with `ODELIA_RETURN_PER_CASE=1` returns per-case class probabilities
+  (no identifiers) with its metrics. Off by default; nothing changes for a site that does
+  not set it. Needed for regional fine-tuning comparison (D2.5), active learning (D3.2)
+  and the testing node (D3.3).
+- **Warm-start provenance and a structural guard (#545, #575; #535)** — the mirrored
+  global checkpoint carries a `.provenance.json` sidecar naming the model that wrote it.
+  A mirror from another architecture is refused by label and, since #575, also by
+  comparing parameter names, so an unlabelled mirror is caught too. A refusal now returns
+  no model instead of falling through and loading the file anyway (F11).
+- **DataLoader worker cap (#575; #574)** — a client never runs more loader workers than its
+  training set can feed (at least four samples per worker). Removes the pin-memory
+  shared-memory race on tiny sets (F12); no effect at consortium data sizes.
+- **Strict swarm runs are exact-client by default (#514)** — large-model result references
+  stay alive for the whole run budget, only missing deliveries are retried, repeated
+  delivery of a round is idempotent, and strict mode rejects timeout-driven partial
+  aggregation.
+- **Active learning (D3.2): sample-selection strategies (#555) and the acquisition
+  experiment (#563, corrected in #568)** — entropy and margin sampling against a random
+  control over the real per-case predictions of the eight-site run. The first result was
+  computed on predictions from the wrong architecture and was retracted; the analysis now
+  refuses predictions whose malignant AUROC is below 0.60 (E2 guard) and computes its
+  verdict instead of stating it.
+- **Robust aggregation and a poisoning simulation (#566; D3.4 #529)** — norm-clipped
+  aggregation measured against a scaled-update attacker. The size-weighted mean every job
+  uses has breakdown point zero; the smallest site can move it arbitrarily.
+- **White-hat site-inference probe (#571; MS6 #531)** — the part of the white-hat attack
+  TUD can run without partner scheduling: whether the three class probabilities of a case
+  reveal which hospital it came from.
+- **Privacy accounting MVP (#563; T3.3 phase 2)**.
+- **Frozen run history (#560, #573)** — `scripts/analysis/extract_run_history.py` and the
+  committed dataset under `workspace/run_history/`. Records are attributed by the
+  heartbeat's hostname, not the site directory name, so TU Dresden test machines no
+  longer masquerade as hospitals.
+- **Supply chain (#446; #395)** — base image pinned by digest, apt left unpinned on purpose,
+  CVE scan in CI.
+- **Docs** — consortium briefing rewritten for a non-technical audience (#564); deck index
+  (#558); evaluation pitfalls E1–E4 (#533, #568); failure modes F11 and F12; manuscript
+  evidence matrix with a mechanically enforced anonymisation rule (#569).
+
+### Fixed
+
+- **`finalize_training` in swarm mode now runs through a callback (#502; #480)** — the
+  post-loop call could not execute because the launcher terminates the training
+  subprocess at job end.
+- **STAMP scheduler horizon follows the job's `num_rounds` (#520; #503)** — the client read
+  `STAMP_NUM_ROUNDS` from its environment while the server ran the job's `num_rounds`;
+  when they disagreed training died with `Tried to step 9 times`.
+- **STAMP installs from a release tarball (#523)** — the git clone in the Dockerfile failed
+  under rate limiting and looked like a permissions error.
+- **The 2-node deploy test can run at all (#544)** — its project sat on the productive
+  server's ports and name, so test clients reached the production server and died on
+  certificates from another provisioning generation; `min_clients` and the gitignore for
+  site configs fixed alongside.
+- **The 2-node deploy test runs on real kits (#573)** — clients are `TEST_A_1`/`TEST_B_1`
+  instead of hospital names (which polluted the live monitor and a published chart), the
+  data folder is decoupled from the FL identity, stale warm-start mirrors are wiped before
+  each model, staged jobs can be submitted by absolute path, and server-side artifacts are
+  saved before cleanup deletes them.
+- **Weekly all-models preflight (#552, #559, #572)** — builds the kits it needs first, covers
+  the ResNet variants, verifies that preflight succeeded rather than started, and prints
+  readable output.
+
+### Changed
+
+- **`jefftud/odelia:current` is re-tagged to 1.8.0 with this release.** (It was not
+  re-tagged for 1.7.0, so sites have been running the 1.6.0-era image since July.)
+- Dependencies: actions/checkout 7, actions/setup-python 7, trivy-action 0.36 (#548, #551,
+  #549).
+
 ## [1.7.0] - 2026-07-29
 
 Swarm evaluation release: a completed swarm run is now scientifically usable.

docs/EMAIL_release_1.8.0.md

@@ -0,0 +1,74 @@
+# Email draft — ODELIA/MediSwarm 1.8.0: what changes at your site (nothing to install for most)
+
+**To:** all ODELIA site contacts (CAM, MHA, RSH, RUMC, UKA, UMCU, USZ, VHIO)
+**Subject:** ODELIA software 1.8.0 — one client restart at your convenience; new kits only for RSH and USZ
+
+---
+
+Hi all,
+
+We have released version 1.8.0 of the ODELIA swarm software. It was tested end to end on
+real startup kits on our own three-machine test bed before publishing, and this is what it
+means for you.
+
+**What is new**
+
+1. **Your site's results now reach the coordinator by themselves.** Each site's validation
+   metrics (AUROC, accuracy, and how many cases of each class they were computed on) are
+   returned with the run. Until now we collected them by logging into each site by hand,
+   which is why the results reports came late.
+
+2. **Optional: per-case predictions.** For the regional fine-tuning comparison and the
+   active-learning work we will need, per validation case, the model's three class
+   probabilities and the label. This is **off by default**. When a run needs it we will ask
+   you first, and switching it on is one environment variable at start
+   (`ODELIA_RETURN_PER_CASE=1`). What leaves the site is a row number, the label and three
+   numbers — no identifiers, no image data.
+
+3. **Safer resumes.** A run that continues from an earlier model now checks that the saved
+   weights really belong to the model being trained, and refuses otherwise. Previously a
+   leftover file from a different model could be loaded silently.
+
+4. Robustness fixes found on our test bed, and the groundwork for the active-learning,
+   robustness and privacy deliverables — none of which needs anything from you yet.
+
+**What you need to do**
+
+- **CAM, MHA, RUMC, UMCU, VHIO, UKA** (1.6.0 kits): restart your client once at a convenient
+  moment — stop the running client container, then start it again as usual with
+  `./docker.sh --start_client`. It picks up the new version on start; there is nothing to
+  download or install. Please do not restart while a training run is in progress; I will
+  announce runs before they start, as before.
+
+- **RSH and USZ** (1.5.0 kits): your kit does not follow the release channel, so a restart
+  alone would keep the old version. I will send you a 1.8.0 kit through the usual folder.
+  It is a drop-in: same certificates, unpack, copy your `sync.conf` over if you have one, and
+  start. Until then you can also start the current kit with
+  `./docker.sh --image jefftud/odelia:1.8.0 --start_client`.
+
+That is all. As before, a full twenty-round run takes about two days across the eight sites
+and I will announce each one before it starts.
+
+Best regards,
+Jeff
+
+---
+
+## Notes before sending
+
+- **Do not send before `:current` is re-tagged and the coordinator server is restarted on
+  1.8.0** (`docs/RELEASE_RUNBOOK.md` §3 and §5). A site that restarts before the re-tag
+  simply gets the old image again and will believe it has updated.
+- Verified 2026-09-13 from the sites' own heartbeats: CAM, MHA, UMCU, VHIO and UKA run 1.6.0
+  kits resolving `jefftud/odelia:current` (image id `b8b918541293`, which is the 1.6.0 image —
+  `:current` was never moved for 1.7.0). RSH's heartbeat comes from a 1.5.0 kit
+  (`/home/asoro/rsh_v150/RSH_1`) although a 1.6.0 RSH record was last seen on 9 Sep — RSH may
+  have two installs; ask which one is meant to run. USZ's monitoring feed has been silent
+  since 22 Jul (1.5.0 kit). UKA has been unreachable since 4 Sep.
+- RUMC's line in the live monitor is currently masked by a deploy-test record from dl0
+  (fixed going forward by #573); the real RUMC node is on a 1.6.0 kit per the run history.
+- The 1.5.0-kit sites are the only ones that need a kit. New kits come from
+  `workspace/odelia_allsites/prod_NN` and reuse the production root CA, so they coexist with
+  everyone else's 1.6.0 kits — no re-registration, no server change.
+- If UKA's kernel problem (crashes on 6.8.0-138) is still open, say so separately rather
+  than here; this email should stay a one-action message.

docs/RELEASE_RUNBOOK.md

@@ -0,0 +1,104 @@
+# Release runbook (ODELIA image + startup kits)
+
+Everything here is manual; nothing in CI builds, pushes or re-tags a release. Each step
+was verified on 2026-09-13 against the repository and the live consortium, not assumed.
+Written for the 1.8.0 release; the procedure is version-independent.
+
+## 0. What a release changes, and what it does not
+
+| Artefact | Who consumes it | Changes on release? |
+|---|---|---|
+| `jefftud/odelia:<version>` | nobody directly | yes, built and pushed |
+| `jefftud/odelia:current` | every site whose kit ships `startup/image.conf` (1.6.0 kits and later): `docker.sh` sources it and **pulls on every start** | **yes — this is the only update channel sites have.** It was skipped for 1.7.0, so sites ran the 1.6.0 image until 1.8.0 |
+| Startup kits (`workspace/odelia_allsites/prod_NN`) | sites, once, at install | optional — see §4 |
+| Coordinator server (agh1, `/home/jeff/deploy_odelia_allsites/dl3.tud.de`) | all sites | restart it on the new image between runs (§5) |
+
+A kit does **not** need re-issuing for an image release. Jobs carry their own code
+(`byoc`), the shared training code travels in the image, and kits only change on
+re-provisioning (new site, new ports, new certificates).
+
+Kits provisioned before `image.conf` existed (the **1.5.0** kits) pin the exact image they
+were built with and never see `:current`. Such a site needs either a new kit or
+`./docker.sh --image jefftud/odelia:<version> --start_client`.
+
+## 1. Preconditions
+
+- `main` is green **and** the four MVP deploy tests passed on real kits
+  (`scripts/deploy/run_deploy_test.sh`, server on dl3, clients dl0+dl2).
+- Working tree clean — `buildDockerImageAndStartupKits.sh` refuses local changes.
+- `odelia_image.version` bumped and `CHANGELOG.md` written (this is shared with DECADE;
+  bumping it moves both consortia's next build).
+- dl0 and dl2 idle: publishing the GitHub release fires `odelia-deploy-test.yml`
+  (`release: [published]`), which runs there.
+
+## 2. Build
+
+```bash
+export MEDISWARM_IMAGE_VERSION=1.8.0          # without this the tag is <v>-dev.<date>.<sha>
+./scripts/build/buildDockerImageAndStartupKits.sh -p application/provision/project_Odelia_allsites.yml
+```
+
+- Never pass `--num-rounds` or `--min-clients` on a release build (1.6.0 shipped with
+  `num_rounds = 3` that way).
+- Kits land in `workspace/odelia_allsites/prod_NN` (next free number) with
+  `kit_manifest.csv` and one `.zip` per participant. The **root CA is reused** from
+  `workspace/odelia_allsites/state/cert.json` — NVFlare's `CertBuilder` loads it when
+  present — so new kits interoperate with the running server and with every earlier
+  generation (prod_00 … prod_03 all share the CA the production server uses). Do not
+  delete `state/` and do not provision the production project into a fresh workspace:
+  that mints a new CA and every site would need a new kit before it could connect (F9).
+
+## 3. Publish the image
+
+```bash
+docker push jefftud/odelia:1.8.0
+docker tag  jefftud/odelia:1.8.0 jefftud/odelia:current
+docker push jefftud/odelia:current
+```
+
+Verify on Docker Hub that the two tags carry the same digest:
+
+```bash
+for t in 1.8.0 current; do curl -s "https://hub.docker.com/v2/repositories/jefftud/odelia/tags/$t" | python3 -c "import json,sys; d=json.load(sys.stdin); print('$t', d['digest'])"; done
+```
+
+The sites' heartbeats (`/srv/mediswarm/live/<SITE>/…/heartbeat.json`, field `image_id`)
+show when each site has actually moved.
+
+## 4. Tag and release
+
+```bash
+git tag -a v1.8.0 -m "ODELIA/MediSwarm v1.8.0"
+git push origin v1.8.0
+gh release create v1.8.0 --title "v1.8.0 — …" --notes-file <notes.md>
+```
+
+A plain tag triggers nothing. **Publishing** the release runs the ODELIA deploy test on
+dl0/dl2.
+
+## 5. Move the coordinator server
+
+The production server's kit also has `image.conf → :current`, so a restart pulls the new
+image. Do this between runs, with the owner's go-ahead — clients reconnect on their own
+using their stored tokens, but a running job would be lost.
+
+```bash
+cd /home/jeff/deploy_odelia_allsites/dl3.tud.de/startup
+docker stop odelia_swarm_server_flserver_a19be57 && docker rm odelia_swarm_server_flserver_a19be57
+rm -f ../daemon_pid.fl            # a stale lock makes start.sh refuse to launch
+./docker.sh --start_server
+```
+
+## 6. Kits and the announcement
+
+- Upload `prod_NN/*.zip` + `SHA256SUMS.txt` the way 1.6.0 was delivered
+  (`workspace/UPLOAD_odelia_kits_v1.6.0/` is the template).
+- Sites on a 1.6.0 kit: restart once, nothing else. Sites on a 1.5.0 kit: new kit or
+  `--image`. The email template for 1.8.0 is `docs/EMAIL_release_1.8.0.md`.
+- Update the kit registry / run schedule with the exact tag.
+
+## 7. After the release
+
+- Watch the live monitor's version-skew view: a site still on the old `image_id` after
+  its next restart has a pinned `image.conf` or a 1.5.0 kit.
+- Record the release in `CHANGELOG.md` with the final date.