gpusnap warm/snapshot pause hook - #9
Conversation
Add Runner.gpusnap_warm_pause(), called in lifespan right after setup()+warmup and BEFORE the queue worker starts / health goes green. When GPUSNAP_WARM=<dir> is set (only while gpusnap builds a snapshot), the worker signals WARM_READY and blocks until restored + RESUMEd — so a snapshot is taken at a clean point (model warm, no job claimed) and a restored worker resumes straight into serving. Unset on a normal boot => no-op, behaviour unchanged. Works for queue mode, HTTP mode, and TorchRun multi-GPU workers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Broly Security ScanNote Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR. Warning
Note Re-scan this PR anytime with
|
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
…only on a real GPU move The gpusnap tool records whether the physical GPUs moved vs dump time and writes GPUS_CHANGED in the warm dir. The warm-pause hook reads it and exports GPUSNAP_GPUS_CHANGED, so a TorchRun/multi-GPU worker reinitialises its process group only on a cross-GPU retarget and keeps its valid communicators for an in-place restore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
What
Adds
Runner.gpusnap_warm_pause()to the worker lifecycle so a Sprocket worker can be snapshotted warm by gpusnap and restored fast on scale-up (background/queue workers included).In
lifespan, right aftersetup()+maybe_run_warmup()and before the queue worker starts //healthgoes green:GPUSNAP_WARM=<dir>is set (only while gpusnap builds a snapshot), the worker writes<dir>/WARM_READYand blocks until restored +<dir>/RESUME.Why
gpusnap must capture the worker after the model is loaded + JIT-warm but before it claims a real job, or a snapshot would swallow (and lose) a job. This hook is that clean pause point; on restore the worker resumes straight into serving. Works for queue mode, HTTP mode, and TorchRun (multi-GPU) workers.
Pairs with the gpusnap
restore.shRESUME signal and federated-applications#252 (the injecting PodSpecMutator). Full contract: gpusnapdeploy/SPROCKET.md.🤖 Generated with Claude Code