Regression harness compares results across different package environments (Manifest drift), producing spurious regressions

[jhalpern30]

Regression harness compares results across different package environments (Manifest drift), producing spurious regressions

Summary

The regression harness caches numerical baselines keyed only on (commit_hash, case_name), with no fingerprint of the package environment that produced them. Because Manifest.toml is untracked and each worktree run calls Pkg.instantiate() against a bare checkout, a baseline cached weeks ago is silently resolved against older package versions than a fresh local run. The resulting machine-epsilon (~1e-16) differences in library math (BLAS/SciML/ArrayInterface) are then amplified by the adaptive ODE step controller and by ill-conditioned near-resonant diagnostics into large apparent regressions — up to 200% — that are pure environment artifacts, not code changes.

This matters because the harness is mandated on every PR (per CLAUDE.md). A reviewer can waste significant time chasing a phantom, or — worse — "fix" correct code to mask an artifact.

Root cause (confirmed in code)

Three independent factors combine:

1. Manifest.toml is untracked. A worktree created for a cached commit has no pinned Manifest.

   • regression-harness/src/utils.jl:74-83 (create_worktree) does a bare git worktree add --detach.

2. Each run resolves the environment fresh. With no Manifest present, Pkg.instantiate() picks the newest versions compatible with Project.toml's [compat] at run time.

   • regression-harness/src/runner.jl:43-54 (RUNNER_SCRIPT_TEMPLATE) → %INSTANTIATE% expands to Pkg.instantiate() (default; only skipped with --no-instantiate).
   • Invoked at regression-harness/src/runner.jl:409-423 against --project=$worktree_path.

3. The cache has no environment key. Results are stored/looked up by (commit_hash, case_name) only.

   • Schema: regression-harness/src/database.jl:5-18 (runs table, UNIQUE(commit_hash, case_name)) — no Julia version, no Manifest hash, no package-set hash.
   • is_cached (regression-harness/src/database.jl:63-67) ignores the environment entirely.

Net effect: regress --refs develop,local compares a develop baseline produced under whatever package set was newest at cache-population time against a fresh local run under today's package set.

Evidence

Investigating a branch that only refactors FourierTransforms/Vacuum (no changes to EulerLagrange.jl/Riccati.jl/Fourfit.jl or the equilibrium metric), the harness reported for develop (cached) vs local:

Quantity	Reported "regression"
diiid ODE steps (saved)	1353 → 1325 (2.07%)
diiid ODE steps (total)	1990 → 1951 (1.96%)
diiid Chirikov parameter	11.13%
diiid resonant field	10.80%
diiid PE toroidal torque	200.47%
solovev ODE steps (total)	605 → 614 (1.49%)

Controlled re-test — same source diff, but the develop worktree pinned to the identical Manifest.toml as the working tree (source code the only variable), 4 threads throughout:

Quantity	develop	local	Result
solovev nstep / total	385 / 614	385 / 614	identical
diiid nstep / total	1325 / 1951	1325 / 1951	identical
diiid Chirikov	—	—	rel 2.8e-14
diiid resonant field	—	—	rel 6.3e-14
all other datasets	—	—	rel ≤ 1e-13

The develop worktree instantiated without pinning had resolved newer versions (e.g. 4.6.1→4.7.0, 7.25.0→7.28.1 in the SciML/ArrayInterface stack). Pinning the Manifest collapsed every reported regression to machine epsilon. Two back-to-back runs of the same environment are bit-identical, so this is not thread-schedule noise.

Mechanism: a 1e-16 change in equilibrium/EL-matrix math (from a different library version) crosses an adaptive-step acceptance boundary (384↔385 steps), which reshuffles the integration ψ-grid, and the near-resonant singular-coupling quantities (Chirikov, Δ′, torque) are ill-conditioned enough to turn that seed into double-digit-percent swings.

Proposed fixes (increasing effort)

1. Fingerprint the environment and warn on mismatch (minimum). Store a hash of the resolved Manifest.toml (or Julia version + package-set) alongside each cached run, and print a loud warning (or auto-invalidate) when a cached ref's fingerprint differs from the current run's. Cheapest change; converts a silent confound into a visible one. Requires a runs schema column + a check in is_cached/reporter.

2. Run both refs in one shared, pinned environment at compare time. Resolve a single Manifest once and reuse it for every worktree (--project pointed at a shared depot, or copy the working tree's Manifest into each worktree before instantiate). Makes source code the only variable by construction.

3. Track Manifest.toml (or a harness-specific pinned Manifest) so worktree checkouts are reproducible, and instantiate against it.

Workaround (today)

Regenerate the baseline in the current environment before trusting a report:

regress --cases diiid_n1,solovev_n1 --refs develop,local --force

--force re-runs the develop ref under the same packages as local, eliminating the cross-environment artifact.

[jhalpern30]

@matt-pharr (and @logan-nc) wanted to flag this to your attention since you're more familiar with the regression harness. I came across this issue while chasing down changes in the regression tests for wayyyyyy too long only for it to end up being a Manifest/package difference and had Claude generate this summary for me.

I'm thinking the first option seems reasonable (i.e. pass a warning if the Manifest is different), since I think its reasonable for the regression to assume all packages are the same. Any thoughts on this?

[logan-nc]

Hmmmm... I dislike the tracking of the manifest due to worries that it will pin us unnecessarily in the past when we should be following updates of dependencies as much as possible to keep up-to-date. It's better to stay up to date than to find ourselves stale and full of update conflicts 5 years down the line. So we don't want to force everyone to a lowest-common denominator of conservative package pinning.

I suppose the easiest fix is just to make the default regression test the "force" case where it runs both the new and the old in the users local env rather than using shared pinned values.

Next step I could see would be to do option (1) of hashing the manifest of the shared pin values and only forcing if there is a change. Ideally, we'd have this provide instructions to update the local env if it's behind on some package updates. We would then only update the git pins if the local env is all greater than or equal as far as the manifest goes (i.e. keep the pins pinned to whoever has the most up-to-date local env). I'd rather have a situation that is pulling everyone up to the most recent package set rather than slowing everyone down to stale ones.

Neither of the above feels like a complete / clean vision... but hopefully the general principle makes sense? Thoughts @matt-pharr ?

[matt-pharr]

Ah yup I just discovered this independently today.We clearly need to be at least resolving dependency changes better. This is the worry I expressed during the original hackathon and IIRC Brendan Lyons' response was to track the manifest.

One option is we could track the manifest and then use dependabot to open a monthly pull request of consolidated updates, on which we can run tests/regression harness, and review.

[logan-nc]

ok. that sounds good. As. long as it doesn't fail quietly and let us get stale too easily 😅

[matt-pharr]

@logan-nc I am getting a set of package updates now that flip the sign of the torque in the d3d ideal example and change b_res by 25%... So I think requiring we pay attention to dependency bumps is a good thing

[logan-nc]

Yikes!

[d-burg]

Adding one more failure mode I hit independently yesterday (originally clauded as #353, which I'm closing as a duplicate). It's adjacent to the original manifest issue, but this is about which code becomes the baseline, not which packages.

From claude:

The harness silently uses a stale local baseline.

resolve_ref (regression-harness/src/utils.jl:22-36) resolves a branch name with a plain git rev-parse --verify --quiet <ref>, i.e. the local branch pointer, falling back to origin/<ref> only when the local ref doesn't exist. Nothing compares the resolved commit to its upstream tracking branch. My local develop was 120 commits behind origin/develop, so --refs develop,local was benchmarking against three-week-old code and nothing in the report said so.

The runner does log the commit's author date (runner.jl:241, runner.jl:385), but an author date is easy to read past and doesn't answer the question that matters — is this the current baseline?

This compounds the Manifest issue rather than duplicating it: a stale baseline commit is also the one most likely to have a badly mismatched package set. In my case the two effects together produced a hard failure rather than just wrong numbers — develop at 41cfbcb3 died with MethodError: no method matching Float64(::Vector{Float64}) and reported every quantity as FAILED. That's the extractor meeting re-resolved packages whose APIs had moved, not a bug at that commit.

Suggested addition to the acceptance criteria here: warn when a resolved ref is behind its remote tracking branch (git rev-list --count <ref>..origin/<ref>), reporting the count, and say plainly that the baseline is not current.

For the record, the Manifest half of #353 reproduced this issue exactly: --cases diiid_n1 --refs develop,local reported 14 changed / 34 unchanged (resonant b^r 9.80%, Chirikov 6.90%, island half-widths 2.21%, ODE steps 2033→2006, Mercier/ballooning checksums all flipped), while re-running the same two code states as two git refs — --refs develop,HEAD, so both sides instantiate identically — gave 48 unchanged, every quantity bit-identical at 0.0e+00. 153 of 388 packages differed between the two manifests, including OrdinaryDiffEqCore 3.33.1 vs 3.29.0 and FastInterpolations 0.4.18 vs 0.4.8.