fix(toys): pd-tms/pd-resid-mlp eval crash — missing remat kwarg#928
Open
ocg-goodfire wants to merge 1 commit into
Open
fix(toys): pd-tms/pd-resid-mlp eval crash — missing remat kwarg#928ocg-goodfire wants to merge 1 commit into
ocg-goodfire wants to merge 1 commit into
Conversation
Collaborator
Author
|
Base corrected main → feature/jax: the remat kwarg this fixes exists only on feature/jax (main's lab has no remat call sites), so the bug doesn't exist on main. Branch still carries foreign commit f423233 (checkpoint fix, delivered by #927); a rebase dropping it is staged at bridge/task-fix-bridge-pr-base-branches/clean-928 pending force-push approval. |
CIFn.__call__ grew a keyword-only `remat: bool` but the toy run.py eval closures were never updated, so both pd-tms and pd-resid-mlp crashed at the first eval step (TypeError: missing required keyword-only argument 'remat'). The closures were near-duplicates of each domain's model.single_feature_ci helper — the tests exercised the helper, so the duplicate rotted unseen. Deduplicate instead of patching: the helper now returns the full CI bundle (run.py needs `upper` for the UV figure) and run.py jit-wraps it. Verified: both SMOKE configs run end-to-end with identical eval metrics to the pre-dedup kwarg-only fix; toy suites pass incl. slow (--runslow). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5e120e1 to
1b192fc
Compare
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
CIFn.__call__grew a keyword-onlyremat: bool, but the toy composition roots' eval closures were never updated — bothpd-tmsandpd-resid-mlpcrashed at the first eval step:Reproduced with stock
tms_5-5_SMOKE.yamlonf423233fb. (Found by bridge crew-try-muon while using the TMS smoke as a muon testbed.)Why the tests didn't catch it
Each
run.pycarried a near-duplicate of its domain'smodel.single_feature_cihelper. The tests exercise the helper (which passesremat=Falsecorrectly); the run.py duplicate rotted unseen.Fix
Deduplicate rather than patch:
model.single_feature_cinow returns the fullCIbundle (run.py needsupperfor the UV figure, tests take.lower), and each run.py jit-wraps the helper instead of re-implementing it. Theremat=Falsedecision now lives in exactly one place per domain, on the code path the tests cover.rematis a no-op for the MLP CI fns, soremat=Falseis semantics-neutral.Verification
pd-tms --config tms_5-5_SMOKE.yamlandpd-resid-mlp --config resid_mlp_1l_SMOKE.yamlrun end-to-end on CPU;eval/identity_ci_error/*metrics identical between the minimal kwarg-only fix and the dedup version--runslow)Remaining gap (not addressed here)
The toy composition roots (
run.py: YAML→BuiltRun→engine + eval_fn wiring) still have no automated coverage — the SMOKE configs are run manually. A ~10s--runslowtest invoking the SMOKE config with a tmpdirPARAM_DECOMP_OUT_DIRwould close this class of rot entirely; left out to keep this PR minimal.🤖 Generated with Claude Code