Skip to content

refactor(slurm): fetch snapshots from the git common dir, drop $HOME/param-decomp#962

Closed
claude-spd1 wants to merge 1 commit into
feature/jaxfrom
bridge/task-harvest-fragment-drop-home-checkout
Closed

refactor(slurm): fetch snapshots from the git common dir, drop $HOME/param-decomp#962
claude-spd1 wants to merge 1 commit into
feature/jaxfrom
bridge/task-harvest-fragment-drop-home-checkout

Conversation

@claude-spd1

Copy link
Copy Markdown
Collaborator

Follow-up from #960. generate_git_snapshot_setup in param_decomp_lab/infra/slurm.py (harvest / autointerp / investigate jobs) still cloned a hard-coded $HOME/param-decomp at shell-time — a hack from 24190c2 whose motivation (torch-era async slow-eval sbatching from inside a SLURM job, where REPO_ROOT resolved to the submitter's node-local /tmp workspace) is dead on feature/jax.

What changed

Aligned it with pd-lm's node-workspace pattern (experiments/lm/launch.py::_snapshot_source_repo / _node_workspace_setup):

  • Resolve the submitting checkout's common git dir at submit via git rev-parse --path-format=absolute --git-common-dir (new _snapshot_source_repo helper). Worktree-safe: a linked worktree shares the main checkout's .git, so a snapshot survives the worktree being deleted before a requeue re-fetches.
  • In-job: git init + shallow git fetch --depth 1 "file://<common dir>" <ref> + git checkout FETCH_HEAD, replacing the git clone/git fetch refs pair. file:// (not the bare path) so git honours --depth — it silently ignores it on the bare-path local transport.
  • .env (secrets, not in git) now copied from the durable main checkout that owns the common dir, not the possibly-ephemeral submitting worktree.

Shared FS throughout, so neither submit nor requeue depends on GitHub reachability (the origin push in create_git_snapshot stays a best-effort provenance backup).

Only caller is _workspace_setup (via SlurmConfig.snapshot_ref); signature unchanged, so harvest/autointerp/investigate need no edits.

Tests

Adds param_decomp_lab/tests/test_slurm.py: asserts the fragment fetches from the common dir via file:// with --depth 1, checks out FETCH_HEAD, copies the durable .env, has no git clone / $HOME/param-decomp, and that _snapshot_source_repo resolves an absolute existing path. ruff check + ruff format --check clean.

Crew-Address: task/harvest-fragment-drop-home-checkout

…param-decomp

`generate_git_snapshot_setup` (harvest/autointerp/investigate jobs) cloned a
hard-coded `$HOME/param-decomp` at shell-time — a hack from 24190c2 whose
motivation (torch-era async slow-eval sbatching from inside a SLURM job, where
`REPO_ROOT` was a node-local /tmp workspace) is dead on feature/jax.

Align it with pd-lm's node-workspace pattern (`experiments/lm/launch.py`):
resolve the submitting checkout's COMMON git dir at submit via
`git rev-parse --path-format=absolute --git-common-dir` (worktree-safe;
main-era REPO_ROOT semantics), then in-job `git init` + shallow
`git fetch --depth 1 "file://<common dir>" <ref>` + checkout FETCH_HEAD.
`file://` (not the bare path) so git honours `--depth`. `.env` (secrets, not in
git) now comes from the durable main checkout that owns the common dir, not the
possibly-ephemeral submitting worktree.

Shared FS throughout, so neither submit nor requeue depends on GitHub
reachability. Adds param_decomp_lab/tests/test_slurm.py.

Crew-Address: task/harvest-fragment-drop-home-checkout
@claude-spd1

Copy link
Copy Markdown
Collaborator Author

Duplicate of #963 (two parallel crew bodies opened equivalent PRs ~30s apart; identical code, minor docstring/test-text differences only). Consolidating on #963. Crew-Address: task/harvest-fragment-drop-home-checkout

@claude-spd1 claude-spd1 closed this Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant