refactor(pd-lm): job-side node workspaces, drop the submit-time shared-FS build#960
Merged
Merged
Conversation
…d-FS build Return the trainer launch to the repo's job-side setup pattern (the one harvest/autointerp never left): each node clones the snapshot into node-local /tmp and builds the CUDA venv at job start, instead of pd-lm materializing an immutable shared-FS workspace per run at submit. - Config delivery moves to the run dir: submit pins the (group/tags-stamped) config as runs/<id>/launch_config.yaml — the file the trainer already pins — and the job (and every requeue) reads that copy. The config no longer needs to live inside the repo, and the inline (dp=null) path no longer mutates the submitted config file in place. - Nodes clone/fetch from ~/param-decomp (durable; worktrees share its refs), not the submitting checkout, which may be an ephemeral worktree a requeue would no longer find; visibility of the snapshot ref is asserted at submit. - The srun task still execs the trainer (SIGTERM-save delivery unchanged), so cleanup is a batch-script EXIT trap sweeping /tmp workspaces, with a start-of-task sweep as the backstop for hard kills. Trainer jobs hold whole nodes, so the sweep cannot race a concurrent job. - pd-pretrain keeps the submit-time build: its 8-tasks-per-node layout would race a per-node job-side build. Smoke: p-a5769c4b (job 171732, 1 node). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm
…~/param-decomp The $HOME/param-decomp hardcoding was inherited from infra's shared fragment, where it exists only because torch-era async slow-eval generated scripts from inside a SLURM job (REPO_ROOT = node-local /tmp there). pd-lm always submits from a login-node checkout, so resolve `git rev-parse --git-common-dir` at submit instead: the main checkout's .git even under a worktree — durable across worktree deletion for requeues, carries the snapshot ref by construction, and assumes nothing about where the checkout lives. .env rides from its parent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm
…re ground truth The async-slow-eval-era local-checkout plumbing isn't blessed; origin is the source of truth for snapshot refs. create_git_snapshot's origin push becomes a hard requirement (a failed push fails the submit), and each node now git-inits an empty workspace and shallow-fetches the snapshot ref straight from the origin URL — no job dependency on any local checkout. .env (secrets, not in git) is staged into the run dir at submit alongside the pinned launch config, mode-preserved, and copied into the workspace by the node setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm
Port the cuda13 extra from #959 (jax[cuda13]==0.10.1 + the Blackwell-TMEM-fixed cuBLAS >= 13.2) and drop its --cuda-extra flag: now that the venv is built job-side on the compute node, the driver IS observable at build time, so each node picks cuda13 vs cuda off its own driver major (>= 580 -> cuda13; and-gmi's r570 falls back to cu12). #959's stated reason for keeping cu12 the default was only the and-gmi driver floor — this gives every driver-capable cluster the corruption-fixed cuBLAS automatically. Co-Authored-By: Dan Braun <dan.braun@goodfire.ai> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm
…n push as provenance Separate the two things "remotely pushed refs are ground truth" bought: the hard origin push stays (durable provenance record for every run), but the job-side fetch source returns to local git state like main — the submitting checkout's common git dir over shared FS (file:// so --depth survives the local transport). Requeues no longer depend on GitHub reachability or SSH auth from compute nodes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm
Jobs fetch snapshots from local shared-FS git, so origin is a provenance backup — a GitHub outage or missing push perms shouldn't block submits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm
This was referenced Jul 8, 2026
claude-spd1
pushed a commit
that referenced
this pull request
Jul 8, 2026
… drop $HOME/param-decomp generate_git_snapshot_setup (harvest/autointerp/investigate SLURM jobs) cloned "$HOME/param-decomp" and did a full `git fetch <ref>:<ref>` — an unblessed hack from the torch era whose motivation (sbatch generated from inside a SLURM job) is dead on feature/jax. #960 already returned the pd-lm trainer launch to the same job-side setup pattern these jobs never left, and resolved the fetch source with `git rev-parse --path-format=absolute --git-common-dir` (worktree-safe: the main checkout's .git on shared FS even under a linked worktree). Rather than duplicate that helper, promote it to `infra.git.snapshot_source_repo()` as the single source of truth and have BOTH launchers use it: - infra.git: new `snapshot_source_repo()`. - infra.slurm.generate_git_snapshot_setup: `git init` + `git fetch --depth 1 "file://<common-dir>" <ref>` + `git checkout FETCH_HEAD`, replacing the clone of $HOME/param-decomp. file:// is required — git silently ignores --depth on the bare-path local transport. `.env` (untracked secrets) now comes from the durable main checkout (the common git dir's parent worktree), not the possibly-ephemeral submitting worktree. - experiments.lm.launch: drop its private `_snapshot_source_repo`, import the shared one. Adds param_decomp_lab/tests/test_slurm.py. Crew-Address: task/harvest-fragment-drop-home-checkout
claude-spd1
added a commit
that referenced
this pull request
Jul 8, 2026
… drop $HOME/param-decomp (#963) generate_git_snapshot_setup (harvest/autointerp/investigate SLURM jobs) cloned "$HOME/param-decomp" and did a full `git fetch <ref>:<ref>` — an unblessed hack from the torch era whose motivation (sbatch generated from inside a SLURM job) is dead on feature/jax. #960 already returned the pd-lm trainer launch to the same job-side setup pattern these jobs never left, and resolved the fetch source with `git rev-parse --path-format=absolute --git-common-dir` (worktree-safe: the main checkout's .git on shared FS even under a linked worktree). Rather than duplicate that helper, promote it to `infra.git.snapshot_source_repo()` as the single source of truth and have BOTH launchers use it: - infra.git: new `snapshot_source_repo()`. - infra.slurm.generate_git_snapshot_setup: `git init` + `git fetch --depth 1 "file://<common-dir>" <ref>` + `git checkout FETCH_HEAD`, replacing the clone of $HOME/param-decomp. file:// is required — git silently ignores --depth on the bare-path local transport. `.env` (untracked secrets) now comes from the durable main checkout (the common git dir's parent worktree), not the possibly-ephemeral submitting worktree. - experiments.lm.launch: drop its private `_snapshot_source_repo`, import the shared one. Adds param_decomp_lab/tests/test_slurm.py. Crew-Address: task/harvest-fragment-drop-home-checkout Co-authored-by: PD User (shared) <pd-user@goodfire.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Return
pd-lmto the job-side workspace pattern main used (and harvest/autointerp still use): each node builds its workspace + venv in node-local/tmpat job start, instead of the submit-time shared-FS workspace the JAX launcher introduced. Three updates to the old pattern:file://so--depth 1survives the local transport) — durable under worktree deletion, and requeues never depend on GitHub. The snapshot's origin push stays best-effort (a provenance backup, as on main), and is not the job's fetch source.runs/<id>/launch_config.yamlat submit (plus a staged.env), and the job reads that copy — so configs can live anywhere, and the inline path no longer mutates the user's config file.cuda13vscudaoff its own driver major (>= 580 → cuda13, i.e. the Blackwell-TMEM-fixed cuBLAS >= 13.2; and-gmi's r570 falls back to cu12). feat(launch): opt-in cuda13 workspace venv (pd-lm --cuda-extra cuda13) #959's--cuda-extraflag existed only because the venv was built on a GPU-less login node.The srun task still
execs the trainer (SIGTERM-save unchanged);/tmpcleanup = batch EXIT trap + start-of-task sweep.pd-pretrainkeeps the submit-time build (8 tasks/node would race a per-node build).Motivation and Context
The submit-time build bought little over main's pattern while costing complexity (hardlink uv-cache machinery, workspace branching) and a multi-GB shared-FS workspace leaked per run. ~1–2 min job-start latency is a fine price. Discussion: Oli, 2026-07-07.
How Has This Been Tested?
Four 1-node smokes (cw-east B200), run dirs deleted after:
scontrol requeuemid-save → rebuild → resumed from 400 → COMPLETED.cuda13(driver 595), the cuBLAS TMEM warning present in the cu12 smokes is gone, COMPLETED.file://shallow fetch + cuda13 gate), COMPLETED.make checkclean,uv lock --checkpasses, launcher tests updated.Does this PR introduce a breaking change?
--run-idresubmit (no pinned launch config); relaunch from their config instead.🤖 Generated with Claude Code
https://claude.ai/code/session_01PvoYnzxByyYP4dmQtXWSmm