Skip to content

gpusnap warm/snapshot pause hook - #9

Open
mbrostami wants to merge 2 commits into
mainfrom
feat/gpusnap-warm-hook
Open

gpusnap warm/snapshot pause hook#9
mbrostami wants to merge 2 commits into
mainfrom
feat/gpusnap-warm-hook

Conversation

@mbrostami

Copy link
Copy Markdown
Contributor

What

Adds Runner.gpusnap_warm_pause() to the worker lifecycle so a Sprocket worker can be snapshotted warm by gpusnap and restored fast on scale-up (background/queue workers included).

In lifespan, right after setup() + maybe_run_warmup() and before the queue worker starts / /health goes green:

  • If GPUSNAP_WARM=<dir> is set (only while gpusnap builds a snapshot), the worker writes <dir>/WARM_READY and blocks until restored + <dir>/RESUME.
  • Unset on a normal boot ⇒ no-op, behaviour unchanged.

Why

gpusnap must capture the worker after the model is loaded + JIT-warm but before it claims a real job, or a snapshot would swallow (and lose) a job. This hook is that clean pause point; on restore the worker resumes straight into serving. Works for queue mode, HTTP mode, and TorchRun (multi-GPU) workers.

Pairs with the gpusnap restore.sh RESUME signal and federated-applications#252 (the injecting PodSpecMutator). Full contract: gpusnap deploy/SPROCKET.md.

🤖 Generated with Claude Code

Add Runner.gpusnap_warm_pause(), called in lifespan right after setup()+warmup and BEFORE the queue
worker starts / health goes green. When GPUSNAP_WARM=<dir> is set (only while gpusnap builds a snapshot),
the worker signals WARM_READY and blocks until restored + RESUMEd — so a snapshot is taken at a clean
point (model warm, no job claimed) and a restored worker resumes straight into serving. Unset on a normal
boot => no-op, behaviour unchanged. Works for queue mode, HTTP mode, and TorchRun multi-GPU workers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@broly-code-security-scanner

Copy link
Copy Markdown

Broly Security Scan

Note

Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR.

Warning

⚠️ Potential results — Broly is still completing analysis or verification. This comment will update automatically.
Files pending AI analysis: 1; findings pending verification: 0.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.3-Flash) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

Ask in #security-engineering   Powered by Together AI

@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

…only on a real GPU move

The gpusnap tool records whether the physical GPUs moved vs dump time and writes GPUS_CHANGED in the warm
dir. The warm-pause hook reads it and exports GPUSNAP_GPUS_CHANGED, so a TorchRun/multi-GPU worker
reinitialises its process group only on a cross-GPU retarget and keeps its valid communicators for an
in-place restore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants