From f0cb74e1c8ae93389530811a984b0780c0e36d49 Mon Sep 17 00:00:00 2001 From: Jeff Zhu Date: Sun, 13 Sep 2026 00:13:44 +0200 Subject: [PATCH 1/5] docs: catalogue F11 (unlabelled wrong-architecture mirror) and F12 (pin-memory shm race) Both found by the 1.8.0 MVP deploy test and fixed in this PR; both read as something else at first (F11 as an aggregation bug, F12 as F7). Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 3e951cd0e8678b515c0045a088e8ce8bca42a2e4) --- docs/SWARM_FAILURE_MODES.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/SWARM_FAILURE_MODES.md b/docs/SWARM_FAILURE_MODES.md index 1e015e0f..1a6da30f 100644 --- a/docs/SWARM_FAILURE_MODES.md +++ b/docs/SWARM_FAILURE_MODES.md @@ -22,6 +22,8 @@ Most of these are now caught automatically by the pre-run checks in the startup- | F8 | Run trains on a **subset**: `clients [...] did not configure within timeout but min_clients=N allows proceeding` | Controller stops waiting once `configure_min_clients` answer; slower sites lose the key exchange | Set `configure_min_clients` **= number of participating sites** | | F9 | Site never appears in `check_status server`, container reports `(healthy)` for weeks | Startup kit older than the server's provisioning generation → `ClientConnectorCertificateError` | Re-issue the current startup kit to that site | | F10 | `cross_val_results.json` is `{}` although the server logged `Published metrics for N site(s)` | Two components act on the same `END_RUN`; `ValidationJsonGenerator` writes the file before the later-listed collector publishes into it | Publish on `ABOUT_TO_END_RUN`, which is fired strictly earlier (already fixed in `per_site_metrics.py`) | +| F11 | Aggregator fails with `None of the N incoming model parameter(s) matched the local model's M`; a client logged `missing keys` when loading the global | An **unlabelled** warm-start mirror from another architecture was auto-loaded (`warm_start_mode=auto`, no provenance sidecar) | Guard now intersects parameter names and refuses on zero overlap (#575); deploy test wipes mirrors first (#573) | +| F12 | `Pin memory thread exited unexpectedly` / `unable to open shared memory object ` at an epoch end — **not** F7 | `file_system` sharing-strategy cleanup race, aggravated by many DataLoader workers on a tiny training set | `cap_loader_workers` (#575, #574): ≥4 samples per worker; never binds on a real site | --- @@ -175,6 +177,33 @@ anything fails. The server log simply stops advancing. the workflow never installed NVFlare (cf. #416/#423). A skipped test file is not a passing one. - **Observed 2026-09-04:** job `7c6e72c6` reported all eight sites and still wrote `{}`. +## F11 — A wrong-architecture warm-start mirror that nobody labelled + +- **Symptom (aggregating client, `startup/nohup.out`):** + ``` + FaultTolerantSwarmClientController - ERROR - exception ending gatherer: + ValueError: None of the 187 incoming model parameter(s) matched the local model's 450 parameter(s). + ``` + Earlier on the same client: `WarmStart: /scratch/mediswarm_latest_global.pt carries no provenance; cannot confirm it was produced by 'MST'. … Proceeding.` and then + `FLCallback - WARNING - There were missing keys when loading the global state_dict`. +- **Root cause:** every ODELIA job mirrors its latest global to one path, `/scratch/mediswarm_latest_global.pt`, and `warm_start_mode = "auto"` loads whatever is there. The #545 guard refused a mirror whose *sidecar* named another model, but a mirror **without** a sidecar — every one written before provenance existed, and anything copied by hand — was accepted with a warning. On 2026-09-11 a 722 MB 1DivideAndConquer mirror left by a failed run warm-started an MST client. The two key sets were disjoint from the first byte. +- **Detection:** the `carries no provenance` warning followed by `missing keys` on load is the tell. `checkpoint_keys()` in `warm_continue.py` lists a mirror's parameter names (and the model its own `train_conf` records) without loading it. +- **Fix:** #575 — the guard intersects the checkpoint's parameter names with the model's whether or not a sidecar exists; an empty intersection is refused with `WARM_START_MODEL_MISMATCH`, and a refusal now returns `None` from `load_model` instead of loading anyway. The deploy test wipes mirrors before each model (#573, `clear_stale_mirrors`; `DEPLOY_TEST_KEEP_MIRROR=1` to opt out). +- **Prevention:** never leave a mirror beside a run that did not write it. This is E2 (`docs/EVALUATION_PITFALLS.md`) arriving through the training path instead of the evaluation path — same lesson: the file's contents, not its name or location, say what model it is. + +## F12 — Pin-memory thread dies at an epoch boundary (reads as F7; is not) + +- **Symptom (client `startup/nohup.out`):** + ``` + Epoch 8: 100%|██████████| 38/38 … Exception in thread Thread-10 (_pin_memory_loop): + RuntimeError: unable to open shared memory object in read-write mode: No such file or directory (2) + threedcnn_ptl - ERROR - Error in main function: Pin memory thread exited unexpectedly + ``` + No OOM in the kernel log; `/dev/shm` large; `--shm-size=16g --ipc=host` set. That rules out F7. +- **Root cause:** the `file_system` sharing strategy (chosen in `8ac8f85` to stop file-descriptor exhaustion — do not switch it back) hands tensors between DataLoader workers and the pin-memory thread by shm *filename*, and the segment can be cleaned before the reader opens it. Sixteen workers on a 38-volume set is close to the worst case: with batch size 1 the whole epoch is in shm at once. +- **Fix:** `cap_loader_workers` (#575, #574) — each worker gets at least four samples per epoch, so 16 → 9 on 38 volumes; on a real site the cap never binds. The fault-tolerant controller retries the round, but on a two-client test with `min_clients=2` a retry is a whole round. +- **Observed 2026-09-11:** `TEST_A_1` (dl0), 77 s into round 0; trained normally on the retry. + ## Operator diagnostic playbook **Drive the live server** via the admin startup kit (run `./fl_admin.sh` in the odelia image with `--net=host`, username line first): From e848aaed2fedb0a3ab2f06a86c9db5857f5d2d9c Mon Sep 17 00:00:00 2001 From: Jeff Zhu Date: Sun, 13 Sep 2026 00:32:41 +0200 Subject: [PATCH 2/5] =?UTF-8?q?release:=201.8.0=20=E2=80=94=20per-site=20m?= =?UTF-8?q?etrics,=20per-case=20predictions,=20warm-start=20guard,=20deplo?= =?UTF-8?q?y=20test=20on=20real=20kits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Version bump and changelog only. The image and the :current re-tag are pushed manually after the four MVP deploy tests pass on real kits. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 86 ++++++++++++++++++++++++++++++++++++++++++++ odelia_image.version | 2 +- 2 files changed, 87 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e8f4e680..bb9b43cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,92 @@ All notable changes to MediSwarm are documented in this file. +## [1.8.0] - UNRELEASED + +Per-site evaluation and deploy-test release. A completed swarm run now reports what +every site measured, can return per-case predictions from sites that opt in, and +refuses to warm-start from another architecture's weights. The two-node deploy test +runs end to end on real startup kits again, and it is where most of the fixes below +were found. Every training-path change was confirmed on a real 2-node run (dl0 + dl2, +server on dl3), not by unit tests alone. + +### Added + +- **Per-site metrics reach the coordinator (#534; #525, #441)** — each site's validation + metrics, including per-class support counts, are published to the server and returned + in `cross_val_results.json`. Previously the file was `{}` although the server logged + `Published metrics for N site(s)` (F10). Verified on real kits: both test sites present + with per-class support. +- **Per-case predictions, opt-in per site (#556, #557; #526, #527, #528)** — a site that + starts its client with `ODELIA_RETURN_PER_CASE=1` returns per-case class probabilities + (no identifiers) with its metrics. Off by default; nothing changes for a site that does + not set it. Needed for regional fine-tuning comparison (D2.5), active learning (D3.2) + and the testing node (D3.3). +- **Warm-start provenance and a structural guard (#545, #575; #535)** — the mirrored + global checkpoint carries a `.provenance.json` sidecar naming the model that wrote it. + A mirror from another architecture is refused by label and, since #575, also by + comparing parameter names, so an unlabelled mirror is caught too. A refusal now returns + no model instead of falling through and loading the file anyway (F11). +- **DataLoader worker cap (#575; #574)** — a client never runs more loader workers than its + training set can feed (at least four samples per worker). Removes the pin-memory + shared-memory race on tiny sets (F12); no effect at consortium data sizes. +- **Strict swarm runs are exact-client by default (#514)** — large-model result references + stay alive for the whole run budget, only missing deliveries are retried, repeated + delivery of a round is idempotent, and strict mode rejects timeout-driven partial + aggregation. +- **Active learning (D3.2): sample-selection strategies (#555) and the acquisition + experiment (#563, corrected in #568)** — entropy and margin sampling against a random + control over the real per-case predictions of the eight-site run. The first result was + computed on predictions from the wrong architecture and was retracted; the analysis now + refuses predictions whose malignant AUROC is below 0.60 (E2 guard) and computes its + verdict instead of stating it. +- **Robust aggregation and a poisoning simulation (#566; D3.4 #529)** — norm-clipped + aggregation measured against a scaled-update attacker. The size-weighted mean every job + uses has breakdown point zero; the smallest site can move it arbitrarily. +- **White-hat site-inference probe (#571; MS6 #531)** — the part of the white-hat attack + TUD can run without partner scheduling: whether the three class probabilities of a case + reveal which hospital it came from. +- **Privacy accounting MVP (#563; T3.3 phase 2)**. +- **Frozen run history (#560, #573)** — `scripts/analysis/extract_run_history.py` and the + committed dataset under `workspace/run_history/`. Records are attributed by the + heartbeat's hostname, not the site directory name, so TU Dresden test machines no + longer masquerade as hospitals. +- **Supply chain (#446; #395)** — base image pinned by digest, apt left unpinned on purpose, + CVE scan in CI. +- **Docs** — consortium briefing rewritten for a non-technical audience (#564); deck index + (#558); evaluation pitfalls E1–E4 (#533, #568); failure modes F11 and F12; manuscript + evidence matrix with a mechanically enforced anonymisation rule (#569). + +### Fixed + +- **`finalize_training` in swarm mode now runs through a callback (#502; #480)** — the + post-loop call could not execute because the launcher terminates the training + subprocess at job end. +- **STAMP scheduler horizon follows the job's `num_rounds` (#520; #503)** — the client read + `STAMP_NUM_ROUNDS` from its environment while the server ran the job's `num_rounds`; + when they disagreed training died with `Tried to step 9 times`. +- **STAMP installs from a release tarball (#523)** — the git clone in the Dockerfile failed + under rate limiting and looked like a permissions error. +- **The 2-node deploy test can run at all (#544)** — its project sat on the productive + server's ports and name, so test clients reached the production server and died on + certificates from another provisioning generation; `min_clients` and the gitignore for + site configs fixed alongside. +- **The 2-node deploy test runs on real kits (#573)** — clients are `TEST_A_1`/`TEST_B_1` + instead of hospital names (which polluted the live monitor and a published chart), the + data folder is decoupled from the FL identity, stale warm-start mirrors are wiped before + each model, staged jobs can be submitted by absolute path, and server-side artifacts are + saved before cleanup deletes them. +- **Weekly all-models preflight (#552, #559, #572)** — builds the kits it needs first, covers + the ResNet variants, verifies that preflight succeeded rather than started, and prints + readable output. + +### Changed + +- **`jefftud/odelia:current` is re-tagged to 1.8.0 with this release.** (It was not + re-tagged for 1.7.0, so sites have been running the 1.6.0-era image since July.) +- Dependencies: actions/checkout 7, actions/setup-python 7, trivy-action 0.36 (#548, #551, + #549). + ## [1.7.0] - 2026-07-29 Swarm evaluation release: a completed swarm run is now scientifically usable. diff --git a/odelia_image.version b/odelia_image.version index 8ff97ebd..378feb68 100644 --- a/odelia_image.version +++ b/odelia_image.version @@ -1,2 +1,2 @@ # version of the ODELIA Docker image, read by different scripts -1.7.0 +1.8.0 From 0fe7236c9e2a18ccb6e9957658a887cdd5c8b5c7 Mon Sep 17 00:00:00 2001 From: Jeff Zhu Date: Sun, 13 Sep 2026 00:35:22 +0200 Subject: [PATCH 3/5] =?UTF-8?q?docs:=20release=20runbook=20=E2=80=94=20the?= =?UTF-8?q?=20manual=20image/kit=20release=20procedure,=20verified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents what was previously only in the operator's head: MEDISWARM_IMAGE_VERSION must be exported for a clean tag, :current is the only update channel sites have (skipped for 1.7.0), provisioning the production project reuses the root CA in workspace/odelia_allsites/state so new kits are drop-in, and a published GitHub release fires the deploy test on the runners. Co-Authored-By: Claude Fable 5.1 --- docs/RELEASE_RUNBOOK.md | 104 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 docs/RELEASE_RUNBOOK.md diff --git a/docs/RELEASE_RUNBOOK.md b/docs/RELEASE_RUNBOOK.md new file mode 100644 index 00000000..ef8c493d --- /dev/null +++ b/docs/RELEASE_RUNBOOK.md @@ -0,0 +1,104 @@ +# Release runbook (ODELIA image + startup kits) + +Everything here is manual; nothing in CI builds, pushes or re-tags a release. Each step +was verified on 2026-09-13 against the repository and the live consortium, not assumed. +Written for the 1.8.0 release; the procedure is version-independent. + +## 0. What a release changes, and what it does not + +| Artefact | Who consumes it | Changes on release? | +|---|---|---| +| `jefftud/odelia:` | nobody directly | yes, built and pushed | +| `jefftud/odelia:current` | every site whose kit ships `startup/image.conf` (1.6.0 kits and later): `docker.sh` sources it and **pulls on every start** | **yes — this is the only update channel sites have.** It was skipped for 1.7.0, so sites ran the 1.6.0 image until 1.8.0 | +| Startup kits (`workspace/odelia_allsites/prod_NN`) | sites, once, at install | optional — see §4 | +| Coordinator server (agh1, `/home/jeff/deploy_odelia_allsites/dl3.tud.de`) | all sites | restart it on the new image between runs (§5) | + +A kit does **not** need re-issuing for an image release. Jobs carry their own code +(`byoc`), the shared training code travels in the image, and kits only change on +re-provisioning (new site, new ports, new certificates). + +Kits provisioned before `image.conf` existed (the **1.5.0** kits) pin the exact image they +were built with and never see `:current`. Such a site needs either a new kit or +`./docker.sh --image jefftud/odelia: --start_client`. + +## 1. Preconditions + +- `main` is green **and** the four MVP deploy tests passed on real kits + (`scripts/deploy/run_deploy_test.sh`, server on dl3, clients dl0+dl2). +- Working tree clean — `buildDockerImageAndStartupKits.sh` refuses local changes. +- `odelia_image.version` bumped and `CHANGELOG.md` written (this is shared with DECADE; + bumping it moves both consortia's next build). +- dl0 and dl2 idle: publishing the GitHub release fires `odelia-deploy-test.yml` + (`release: [published]`), which runs there. + +## 2. Build + +```bash +export MEDISWARM_IMAGE_VERSION=1.8.0 # without this the tag is -dev.. +./scripts/build/buildDockerImageAndStartupKits.sh -p application/provision/project_Odelia_allsites.yml +``` + +- Never pass `--num-rounds` or `--min-clients` on a release build (1.6.0 shipped with + `num_rounds = 3` that way). +- Kits land in `workspace/odelia_allsites/prod_NN` (next free number) with + `kit_manifest.csv` and one `.zip` per participant. The **root CA is reused** from + `workspace/odelia_allsites/state/cert.json` — NVFlare's `CertBuilder` loads it when + present — so new kits interoperate with the running server and with every earlier + generation (prod_00 … prod_03 all share the CA the production server uses). Do not + delete `state/` and do not provision the production project into a fresh workspace: + that mints a new CA and every site would need a new kit before it could connect (F9). + +## 3. Publish the image + +```bash +docker push jefftud/odelia:1.8.0 +docker tag jefftud/odelia:1.8.0 jefftud/odelia:current +docker push jefftud/odelia:current +``` + +Verify on Docker Hub that the two tags carry the same digest: + +```bash +for t in 1.8.0 current; do curl -s "https://hub.docker.com/v2/repositories/jefftud/odelia/tags/$t" | python3 -c "import json,sys; d=json.load(sys.stdin); print('$t', d['digest'])"; done +``` + +The sites' heartbeats (`/srv/mediswarm/live//…/heartbeat.json`, field `image_id`) +show when each site has actually moved. + +## 4. Tag and release + +```bash +git tag -a v1.8.0 -m "ODELIA/MediSwarm v1.8.0" +git push origin v1.8.0 +gh release create v1.8.0 --title "v1.8.0 — …" --notes-file +``` + +A plain tag triggers nothing. **Publishing** the release runs the ODELIA deploy test on +dl0/dl2. + +## 5. Move the coordinator server + +The production server's kit also has `image.conf → :current`, so a restart pulls the new +image. Do this between runs, with the owner's go-ahead — clients reconnect on their own +using their stored tokens, but a running job would be lost. + +```bash +cd /home/jeff/deploy_odelia_allsites/dl3.tud.de/startup +docker stop odelia_swarm_server_flserver_a19be57 && docker rm odelia_swarm_server_flserver_a19be57 +rm -f ../daemon_pid.fl # a stale lock makes start.sh refuse to launch +./docker.sh --start_server +``` + +## 6. Kits and the announcement + +- Upload `prod_NN/*.zip` + `SHA256SUMS.txt` the way 1.6.0 was delivered + (`workspace/UPLOAD_odelia_kits_v1.6.0/` is the template). +- Sites on a 1.6.0 kit: restart once, nothing else. Sites on a 1.5.0 kit: new kit or + `--image`. The email template for 1.8.0 is `docs/EMAIL_release_1.8.0.md`. +- Update the kit registry / run schedule with the exact tag. + +## 7. After the release + +- Watch the live monitor's version-skew view: a site still on the old `image_id` after + its next restart has a pinned `image.conf` or a 1.5.0 kit. +- Record the release in `CHANGELOG.md` with the final date. From cdde0b79a18d1ac726ac2105d165b34eaa03bc9a Mon Sep 17 00:00:00 2001 From: Jeff Zhu Date: Sun, 13 Sep 2026 00:36:38 +0200 Subject: [PATCH 4/5] docs: 1.8.0 announcement email draft with verified per-site actions One restart for the 1.6.0-kit sites, a drop-in kit for the two 1.5.0-kit sites, and a pre-send checklist tied to the runbook. Co-Authored-By: Claude Fable 5.1 --- docs/EMAIL_release_1.8.0.md | 74 +++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/EMAIL_release_1.8.0.md diff --git a/docs/EMAIL_release_1.8.0.md b/docs/EMAIL_release_1.8.0.md new file mode 100644 index 00000000..2df23ba8 --- /dev/null +++ b/docs/EMAIL_release_1.8.0.md @@ -0,0 +1,74 @@ +# Email draft — ODELIA/MediSwarm 1.8.0: what changes at your site (nothing to install for most) + +**To:** all ODELIA site contacts (CAM, MHA, RSH, RUMC, UKA, UMCU, USZ, VHIO) +**Subject:** ODELIA software 1.8.0 — one client restart at your convenience; new kits only for RSH and USZ + +--- + +Hi all, + +We have released version 1.8.0 of the ODELIA swarm software. It was tested end to end on +real startup kits on our own three-machine test bed before publishing, and this is what it +means for you. + +**What is new** + +1. **Your site's results now reach the coordinator by themselves.** Each site's validation + metrics (AUROC, accuracy, and how many cases of each class they were computed on) are + returned with the run. Until now we collected them by logging into each site by hand, + which is why the results reports came late. + +2. **Optional: per-case predictions.** For the regional fine-tuning comparison and the + active-learning work we will need, per validation case, the model's three class + probabilities and the label. This is **off by default**. When a run needs it we will ask + you first, and switching it on is one environment variable at start + (`ODELIA_RETURN_PER_CASE=1`). What leaves the site is a row number, the label and three + numbers — no identifiers, no image data. + +3. **Safer resumes.** A run that continues from an earlier model now checks that the saved + weights really belong to the model being trained, and refuses otherwise. Previously a + leftover file from a different model could be loaded silently. + +4. Robustness fixes found on our test bed, and the groundwork for the active-learning, + robustness and privacy deliverables — none of which needs anything from you yet. + +**What you need to do** + +- **CAM, MHA, RUMC, UMCU, VHIO, UKA** (1.6.0 kits): restart your client once at a convenient + moment — stop the running client container, then start it again as usual with + `./docker.sh --start_client`. It picks up the new version on start; there is nothing to + download or install. Please do not restart while a training run is in progress; I will + announce runs before they start, as before. + +- **RSH and USZ** (1.5.0 kits): your kit does not follow the release channel, so a restart + alone would keep the old version. I will send you a 1.8.0 kit through the usual folder. + It is a drop-in: same certificates, unpack, copy your `sync.conf` over if you have one, and + start. Until then you can also start the current kit with + `./docker.sh --image jefftud/odelia:1.8.0 --start_client`. + +That is all. As before, a full twenty-round run takes about two days across the eight sites +and I will announce each one before it starts. + +Best regards, +Jeff + +--- + +## Notes before sending + +- **Do not send before `:current` is re-tagged and the coordinator server is restarted on + 1.8.0** (`docs/RELEASE_RUNBOOK.md` §3 and §5). A site that restarts before the re-tag + simply gets the old image again and will believe it has updated. +- Verified 2026-09-13 from the sites' own heartbeats: CAM, MHA, UMCU, VHIO and UKA run 1.6.0 + kits resolving `jefftud/odelia:current` (image id `b8b918541293`, which is the 1.6.0 image — + `:current` was never moved for 1.7.0). RSH's heartbeat comes from a 1.5.0 kit + (`/home/asoro/rsh_v150/RSH_1`) although a 1.6.0 RSH record was last seen on 9 Sep — RSH may + have two installs; ask which one is meant to run. USZ's monitoring feed has been silent + since 22 Jul (1.5.0 kit). UKA has been unreachable since 4 Sep. +- RUMC's line in the live monitor is currently masked by a deploy-test record from dl0 + (fixed going forward by #573); the real RUMC node is on a 1.6.0 kit per the run history. +- The 1.5.0-kit sites are the only ones that need a kit. New kits come from + `workspace/odelia_allsites/prod_NN` and reuse the production root CA, so they coexist with + everyone else's 1.6.0 kits — no re-registration, no server change. +- If UKA's kernel problem (crashes on 6.8.0-138) is still open, say so separately rather + than here; this email should stay a one-action message. From 263399f8b50ebc2411f2e5e73ac624396c84fb34 Mon Sep 17 00:00:00 2001 From: Jeff Zhu Date: Sun, 13 Sep 2026 10:30:37 +0200 Subject: [PATCH 5/5] docs(F10): say where cross_val_results.json actually is after a completed run The server deletes the run dir on completion and keeps the workspace in its job store inside the container; download_job is the retrieval path. Found when the deploy test's clean 20-round run saved nothing on 13 Sep. Co-Authored-By: Claude Fable 5.1 --- docs/SWARM_FAILURE_MODES.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/SWARM_FAILURE_MODES.md b/docs/SWARM_FAILURE_MODES.md index 1a6da30f..59f0078c 100644 --- a/docs/SWARM_FAILURE_MODES.md +++ b/docs/SWARM_FAILURE_MODES.md @@ -176,6 +176,13 @@ anything fails. The server log simply stops advancing. green throughout: `pytest.importorskip("nvflare")` skipped the collector's tests entirely because the workflow never installed NVFlare (cf. #416/#423). A skipped test file is not a passing one. - **Observed 2026-09-04:** job `7c6e72c6` reported all eight sites and still wrote `{}`. +- **Where the file is after a run:** for a *completed* job the server keeps nothing under + `//` — NVFlare packs the server workspace into its job store + (`/tmp/nvflare/jobs-storage//workspace`, a zip **inside the server container**) and + deletes the run dir. Retrieve it with `download_job ` from the admin console; it lands in + the admin kit's `transfer//workspace/cross_site_val/`. Only an *aborted* run leaves a run + dir behind — which is how the deploy test read the file on 12 Sep and found nothing after the + clean 20-round run on 13 Sep (harness fixed to read the store). ## F11 — A wrong-architecture warm-start mirror that nobody labelled