refactor(slurm): fetch snapshots from the git common dir, drop $HOME/param-decomp#962
Closed
claude-spd1 wants to merge 1 commit into
Closed
refactor(slurm): fetch snapshots from the git common dir, drop $HOME/param-decomp#962claude-spd1 wants to merge 1 commit into
claude-spd1 wants to merge 1 commit into
Conversation
…param-decomp `generate_git_snapshot_setup` (harvest/autointerp/investigate jobs) cloned a hard-coded `$HOME/param-decomp` at shell-time — a hack from 24190c2 whose motivation (torch-era async slow-eval sbatching from inside a SLURM job, where `REPO_ROOT` was a node-local /tmp workspace) is dead on feature/jax. Align it with pd-lm's node-workspace pattern (`experiments/lm/launch.py`): resolve the submitting checkout's COMMON git dir at submit via `git rev-parse --path-format=absolute --git-common-dir` (worktree-safe; main-era REPO_ROOT semantics), then in-job `git init` + shallow `git fetch --depth 1 "file://<common dir>" <ref>` + checkout FETCH_HEAD. `file://` (not the bare path) so git honours `--depth`. `.env` (secrets, not in git) now comes from the durable main checkout that owns the common dir, not the possibly-ephemeral submitting worktree. Shared FS throughout, so neither submit nor requeue depends on GitHub reachability. Adds param_decomp_lab/tests/test_slurm.py. Crew-Address: task/harvest-fragment-drop-home-checkout
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up from #960.
generate_git_snapshot_setupinparam_decomp_lab/infra/slurm.py(harvest / autointerp / investigate jobs) still cloned a hard-coded$HOME/param-decompat shell-time — a hack from 24190c2 whose motivation (torch-era async slow-eval sbatching from inside a SLURM job, whereREPO_ROOTresolved to the submitter's node-local/tmpworkspace) is dead onfeature/jax.What changed
Aligned it with pd-lm's node-workspace pattern (
experiments/lm/launch.py::_snapshot_source_repo/_node_workspace_setup):git rev-parse --path-format=absolute --git-common-dir(new_snapshot_source_repohelper). Worktree-safe: a linked worktree shares the main checkout's.git, so a snapshot survives the worktree being deleted before a requeue re-fetches.git init+ shallowgit fetch --depth 1 "file://<common dir>" <ref>+git checkout FETCH_HEAD, replacing thegit clone/git fetch refspair.file://(not the bare path) so git honours--depth— it silently ignores it on the bare-path local transport..env(secrets, not in git) now copied from the durable main checkout that owns the common dir, not the possibly-ephemeral submitting worktree.Shared FS throughout, so neither submit nor requeue depends on GitHub reachability (the origin push in
create_git_snapshotstays a best-effort provenance backup).Only caller is
_workspace_setup(viaSlurmConfig.snapshot_ref); signature unchanged, so harvest/autointerp/investigate need no edits.Tests
Adds
param_decomp_lab/tests/test_slurm.py: asserts the fragment fetches from the common dir viafile://with--depth 1, checks outFETCH_HEAD, copies the durable.env, has nogit clone/$HOME/param-decomp, and that_snapshot_source_reporesolves an absolute existing path.ruff check+ruff format --checkclean.Crew-Address: task/harvest-fragment-drop-home-checkout