feat(eval): per-site faithfulness observability (rel/abs Frobenius)#934
Open
ocg-goodfire wants to merge 1 commit into
Open
feat(eval): per-site faithfulness observability (rel/abs Frobenius)#934ocg-goodfire wants to merge 1 commit into
ocg-goodfire wants to merge 1 commit into
Conversation
The global FaithfulnessLoss (Σ_s‖Δ_s‖²/Σ_s numel) hides catastrophically
broken small-norm sites (lore 2026-07-03--unmasked-kl-vs-faith-no-eval-bug-4-culprit-sites:
4 of 224 sites carried ~all of the 6.6 unmasked KL while faith sat at ~1e-8).
Add per_site_faith_eval.make_per_site_faith_step: a jitted (model, components) ->
{site: ‖Δ_s‖²_F} reusing model.weight_deltas — the SAME per-site deltas the global
FaithfulnessLoss reduces, so Σ_s/Σ_s numel == FaithfulnessLoss exactly. ‖W_s‖²_F (the
constant relative denominator) comes from the same step on a zeroed vu (Δ=W−0), cached
once in the LM eval_fn closure.
The LM in-loop eval logs, at eval cadence (always-on, cheap — one faith-loss-sized
materialization per eval, which already runs every train step):
- eval/faith/rel_frob_top{1..8} — 8 largest rel-Frobenius, RANK-indexed (stable line keys)
- eval/faith/abs_frob_max
- eval/faith/rel_frob_bar_chart — all-site, site-labeled (rank 0)
Observability only — relative Frobenius is per-site WEIGHT-space error, NOT behavioral
sensitivity (the culprit sites above were behaviorally inert on rel-Frobenius). The
KL-under-delta-ablation probe is the diagnostic that ranks behavioral sensitivity;
deferred pending a compute/design decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds per-site faithfulness observability to the LM in-loop eval. Bridge task
per-site-faith-observability; follow-up to the investigation in lore2026-07-03--unmasked-kl-vs-faith-no-eval-bug-4-culprit-sites.The global
FaithfulnessLoss(Σ_s‖Δ_s‖²/Σ_s numel) is an aggregate that hides catastrophically broken small-norm sites: on p-8e2380e2, 4 of 224 sites carried nearly all of the 6.6 unmasked KL while faith sat at ~1e-8. This surfaces the per-site breakdown.How
param_decomp/per_site_faith_eval.py:make_per_site_faith_step(lm, co)→ jitted(model, components) -> {site: ‖Δ_s‖²_F}, reusingmodel.weight_deltas— the same per-site deltas the global loss reduces, soΣ_s result / Σ_s numel == FaithfulnessLossexactly (asserted in tests).modelis the jit arg (HLO-baking rule); each Δ is reduced to a scalar so XLA frees it → peak mem ≈ one delta.‖W_s‖²_F(the constant relative denominator) is obtained from the same step on a zeroed vu (Δ = W − 0), cached once in the eval closure — no new protocol surface, fully generic.per_site_faith_scalars(...)→ the rank-indexed top-k scalar helper._make_lm_eval_fn(param_decomp_lab/experiments/lm/run.py). At eval cadence (always-on — cheap: one faith-loss-sized materialization per eval, already done every train step) it logs:eval/faith/rel_frob_top{1..8}— the 8 largest‖Δ_s‖_F/‖W_s‖_F, rank-indexed so line-plot keys stay stable as the ranking reshuffles (top1 == max)eval/faith/abs_frob_maxeval/faith/rel_frob_bar_chart— all-site, site-labeled (rank-0, mirrorseval/l0/bar_chart)Resolves the task's design question on wandb key volume (224 sites): rank-indexed scalars for stable trends + one bar chart for per-site identity, instead of 224 line keys.
Scope / caveat
Observability, not a culprit detector. Relative Frobenius is per-site weight-space error, which ≠ behavioral sensitivity — per the investigation, the worst rel-Frobenius sites (l0 o/v_proj) were behaviorally inert; only KL-under-delta-ablation ranked the real culprits. That probe (proposal (b)) and per-site-relative faith-loss normalization (proposal (c), an algorithm change needing a SPEC amendment) are deferred pending a decision — see the bridge task log.
Tests
param_decomp/tests/test_per_site_faith_eval.py: per-site ==weight_deltas,Σ/Σnumel==FaithfulnessLoss, zeroed-vu ==‖W_s‖², top-k ranking + keys. Fullmake checkclean;test_runtime_standalone+test_evalgreen (44 tests).🤖 Generated with Claude Code