Regression harness comparisons against `local` are confounded by untracked Manifest.toml

[d-burg]

Summary

Manifest.toml is not tracked by git. The regression harness git worktree adds each ref and runs
Pkg.instantiate() there, which — with no committed manifest — resolves fresh to the newest
compatible package versions. The local ref, by contrast, runs in the working tree against whatever
is already pinned in the developer's untracked Manifest.toml.

So --refs <ref>,local silently compares baseline code + today's packages against
your code + your (possibly months-old) packages. The harness attributes the entire difference to
the code under test. This makes it unsound as a PR gate, which is exactly what CLAUDE.md mandates it for.

Evidence (measured 2026-08-11, on #238)

--cases diiid_n1 --refs develop,local reported 14 changed, 34 unchanged, including:

Quantity	Reported diff
resonant area-weighted field b^r	9.80%
Chirikov parameter	6.90%
island half-widths	2.21%
‖resonant area-weighted field‖	1.91%
ODE steps (total)	1.33% (2033 → 2006)
Mercier D_I, D_R, ballooning Δ′ checksums	all flipped

Re-running the same two code states as two git refs — --refs develop,HEAD, so both sides
instantiate identically — gives:

Ref 1: develop  @ 167ada76      Ref 2: HEAD  @ 1d27fea7
Summary: 48 unchanged

Every quantity bit-identical at 0.0e+00, ODE steps 2033 both sides. The PR changes nothing
numerically; all 14 "regressions" were package drift.

Diffing the two manifests showed 153 of 388 packages differed, including:

Package	fresh worktree	working tree
OrdinaryDiffEqCore	3.33.1	3.29.0
FastInterpolations	0.4.18	0.4.8
DiffEqBase	6.218.0	6.214.1
SciMLBase	2.155.2	2.154.0
LinearSolve	3.87.0	3.72.0
HDF5_jll	2.1.2+0	1.14.2+1

A different adaptive-stepping implementation and a spline package ten minor versions apart fully
account for a 1.3% step-count shift and the resonant-layer quantities that cascade from it.

Related symptom

Running an older ref also produces spurious hard failures: develop at 41cfbcb3 (three weeks old)
died with MethodError: no method matching Float64(::Vector{Float64}) and reported every quantity as
FAILED. That is the harness's extractor meeting re-resolved packages whose APIs have moved, not a bug
at that commit.

Proposed fixes

1. Commit Manifest.toml (or a dedicated harness manifest) so every ref resolves identically.
   Most direct fix; also makes CI reproducible.
2. Or have the harness copy the working-tree Manifest.toml into each worktree before
   Pkg.instantiate(), so package versions are held fixed while only code varies.
3. Warn loudly when comparing against local, since that ref can never be environment-matched
   to a worktree ref under the current design.
4. Warn when the local develop baseline is stale. The harness resolves develop to whatever
   the local branch points at; it was 120 commits behind origin/develop on 2026-08-11, so the
   baseline was three-week-old code and nothing surfaced that fact.

Acceptance criteria

• Two runs of identical code in different worktrees produce identical package sets
• --refs X,local either environment-matches or emits an explicit warning
• Stale-baseline warning when the local ref is behind its remote
• Re-running the Feature: add SLAYER and GGJ tearing growth rates #238 comparison via --refs develop,local reproduces the 48 unchanged result

[jhalpern30]

@d-burg is this a duplicate? See #338

[d-burg]

Duplicate of #338 — same root cause (untracked Manifest.toml → each worktree Pkg.instantiate()s fresh → package drift attributed to the code under test), independently found.

@jhalpern30 yes, thanks for the pointer. I've moved the one part that wasn't already covered there — the stale local baseline, where resolve_ref takes the local branch pointer with no check against its upstream — into #338 as #338 (comment), along with the evidence from this issue. Closing here; let's track it all in #338.