All six real ODELIA jobs read and write the same warm-start mirror:
source_ckpt_file_full_name = "/scratch/mediswarm_latest_global.pt"
latest_global_path = "/scratch/mediswarm_latest_global.pt"
challenge_1DivideAndConquer, challenge_2BCN_AIM, challenge_3agaldran, challenge_4abmil, challenge_5pimed and ODELIA_ternary_classification — five different architectures, one file. warm_continue.py records no architecture, model name or provenance alongside the checkpoint, and checks none on load.
Two consequences.
1. Any run silently replaces the previous global at every site. Verified today: before job 24cdf247 the mirror on UKA was 722,229,662 bytes dated 15 Aug — sha256 ad26080c71161e6f8abb6a279c06a41d459b7bb237699cbe2d4522ddb942270d, byte-identical to the 87c5bbee pan-European global. A one-round run overwrites it. Nothing warns, and the previous checkpoint is only recoverable if someone happened to collect it centrally beforehand.
2. A cross-architecture warm start is not guarded. Run 2BCN_AIM, then 1DC, and the 1DC run warm-starts from a 2BCN_AIM checkpoint. Whether that raises or silently loads a partial state dict, neither outcome is acceptable: this is E2 in docs/EVALUATION_PITFALLS.md — "wrong architecture loads silently" — with the pathway wired in by default.
This matters concretely for D2.5 (#526), where regional fine-tuning must start from exactly the checkpoint the pan-European baseline metrics were computed from.
Suggested fix
- Namespace the mirror by model:
/scratch/mediswarm_latest_global_{MODEL_NAME}.pt.
- Write a sidecar (model name, job id, round, sha256) and refuse to warm-start when the model name does not match, rather than trusting the tensor shapes.
- Keep the previous mirror as
.prev so one unintended run is not destructive.
Found while preparing #526. Not urgent for correctness of any completed run — 87c5bbee is preserved centrally and checksum-verified — but it is a live trap for every future one.
All six real ODELIA jobs read and write the same warm-start mirror:
challenge_1DivideAndConquer,challenge_2BCN_AIM,challenge_3agaldran,challenge_4abmil,challenge_5pimedandODELIA_ternary_classification— five different architectures, one file.warm_continue.pyrecords no architecture, model name or provenance alongside the checkpoint, and checks none on load.Two consequences.
1. Any run silently replaces the previous global at every site. Verified today: before job
24cdf247the mirror on UKA was 722,229,662 bytes dated 15 Aug —sha256 ad26080c71161e6f8abb6a279c06a41d459b7bb237699cbe2d4522ddb942270d, byte-identical to the87c5bbeepan-European global. A one-round run overwrites it. Nothing warns, and the previous checkpoint is only recoverable if someone happened to collect it centrally beforehand.2. A cross-architecture warm start is not guarded. Run 2BCN_AIM, then 1DC, and the 1DC run warm-starts from a 2BCN_AIM checkpoint. Whether that raises or silently loads a partial state dict, neither outcome is acceptable: this is E2 in
docs/EVALUATION_PITFALLS.md— "wrong architecture loads silently" — with the pathway wired in by default.This matters concretely for D2.5 (#526), where regional fine-tuning must start from exactly the checkpoint the pan-European baseline metrics were computed from.
Suggested fix
/scratch/mediswarm_latest_global_{MODEL_NAME}.pt..prevso one unintended run is not destructive.Found while preparing #526. Not urgent for correctness of any completed run —
87c5bbeeis preserved centrally and checksum-verified — but it is a live trap for every future one.