Skip to content

refactor(infra): worktree-safe shallow snapshot fetch, drop $HOME/param-decomp#963

Merged
claude-spd1 merged 1 commit into
feature/jaxfrom
task/harvest-drop-home-checkout
Jul 8, 2026
Merged

refactor(infra): worktree-safe shallow snapshot fetch, drop $HOME/param-decomp#963
claude-spd1 merged 1 commit into
feature/jaxfrom
task/harvest-drop-home-checkout

Conversation

@claude-spd1

@claude-spd1 claude-spd1 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

What

generate_git_snapshot_setup in param_decomp_lab/infra/slurm.py (harvest /
autointerp / investigate SLURM jobs) cloned "$HOME/param-decomp" and did a full
git fetch <ref>:<ref>. That $HOME/param-decomp hardcode is an unblessed hack from
commit 24190c261 (torch era), whose motivation — generating an sbatch from inside a
SLURM job (async slow-eval via sink.on_save), where REPO_ROOT points at a node-local
/tmp/.../workspace-* — is dead on feature/jax.

Reconsidered in light of #960

#960 already returned the pd-lm trainer launch to the same job-side setup pattern
these jobs never left
, and resolved the fetch source with
git rev-parse --path-format=absolute --git-common-dir (the main checkout's .git on
shared FS, worktree-safe). So rather than land a second copy of that helper, this PR
promotes it to a single shared home and points both launchers at it:

  • infra/git.py: new snapshot_source_repo() — the one definition (it already hosts
    the other REPO_ROOT-based git utilities both launchers import).
  • infra/slurm.py: generate_git_snapshot_setup now does git init +
    git fetch --quiet --depth 1 "file://<common-dir>" <ref> + git checkout --quiet FETCH_HEAD, replacing the $HOME/param-decomp clone + tag/head fetch. file:// is
    required — git silently ignores --depth on the bare-path local transport. .env
    (untracked secrets) is copied from the durable main checkout (the common git dir's
    parent worktree), not the possibly-ephemeral submitting worktree.
  • experiments/lm/launch.py: drops its private _snapshot_source_repo (added in
    refactor(pd-lm): job-side node workspaces, drop the submit-time shared-FS build #960), imports the shared one — net −16 lines there.

Result: one implementation of "where does a job fetch its snapshot from", used by both
the trainer launch and the shared SLURM fragment, instead of two.

Testing

  • New param_decomp_lab/tests/test_slurm.py asserts the fragment shape (shallow file://
    fetch from the common dir, FETCH_HEAD checkout, durable .env, no git clone, no
    $HOME/param-decomp).
  • pytest test_slurm.py test_launch.py → 9 passed; ruff check/format clean;
    basedpyright 0 errors on all four touched files.
  • Runtime sanity: slurm.snapshot_source_repo is git.snapshot_source_repo (single
    object), and rendering the fragment from inside a linked worktree resolves the common
    dir to the main checkout /mnt/home/pd-user/param-decomp/.git (worktree-safety).

Follow-up from #960. Board task harvest-fragment-drop-home-checkout.

Crew-Address: task/harvest-fragment-drop-home-checkout

… drop $HOME/param-decomp

generate_git_snapshot_setup (harvest/autointerp/investigate SLURM jobs) cloned
"$HOME/param-decomp" and did a full `git fetch <ref>:<ref>` — an unblessed hack
from the torch era whose motivation (sbatch generated from inside a SLURM job)
is dead on feature/jax.

#960 already returned the pd-lm trainer launch to the same job-side setup pattern
these jobs never left, and resolved the fetch source with `git rev-parse
--path-format=absolute --git-common-dir` (worktree-safe: the main checkout's .git
on shared FS even under a linked worktree). Rather than duplicate that helper,
promote it to `infra.git.snapshot_source_repo()` as the single source of truth and
have BOTH launchers use it:

- infra.git: new `snapshot_source_repo()`.
- infra.slurm.generate_git_snapshot_setup: `git init` + `git fetch --depth 1
  "file://<common-dir>" <ref>` + `git checkout FETCH_HEAD`, replacing the clone of
  $HOME/param-decomp. file:// is required — git silently ignores --depth on the
  bare-path local transport. `.env` (untracked secrets) now comes from the durable
  main checkout (the common git dir's parent worktree), not the possibly-ephemeral
  submitting worktree.
- experiments.lm.launch: drop its private `_snapshot_source_repo`, import the shared one.

Adds param_decomp_lab/tests/test_slurm.py.

Crew-Address: task/harvest-fragment-drop-home-checkout
@claude-spd1 claude-spd1 force-pushed the task/harvest-drop-home-checkout branch from d793fd5 to b3e17d9 Compare July 8, 2026 03:34
@claude-spd1 claude-spd1 merged commit eb786f7 into feature/jax Jul 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants