Butlin evidence-tier redesign: fix FIFO boost bug, correct indicator taxonomy, replace blended score with honest evidence tiers

[Tristan-Stoltz-ERC]

Addresses #7. Replaces the blended-score/status model discussed there with a
proper evidence-tier system, and fixes a real WM correctness bug found while
adding the regression tests #7 asked for.

Summary

• Taxonomy correction: this suite carried a spurious PP-2/IIT-1 pair
  and was missing AE-1/AE-2. Verified against Butlin et al. (2023),
  arXiv:2308.08708, Table 1 directly — the paper has one PP indicator and
  explicitly excludes IIT ("not compatible with computational
  functionalism"). Now matches the paper's real 14 indicators exactly.
• Evidence-tier redesign: replaces IndicatorStatus
  (Present/Partial/Absent, previously hardcoded to Present
  regardless of score) and a 0.6*static + 0.4*runtime blend (which let a
  fully-dead live signal still read as "present") with:
  • SupportTier: ArchitecturalOnly → Observed → CausallySupported → FunctionallySupported
  • EvidenceOutcome: Supported(SupportTier) | NotDemonstrated | Contradicted — kept as a separate concept from SupportTier
    rather than one ordered ladder, since a negative finding ("we tested
    and it didn't move") and a contradicted one ("it moved the wrong way")
    aren't "below" a positive tier, they're different failure modes.
  • architectural_score (the hand-assigned constant) and live_score
    (the raw, unblended probe) reported separately, never averaged.
  • A single evaluate() snapshot can only ever produce ArchitecturalOnly
    — one number can't rule out a frozen/fallback signal. Only
    annotate_with_ablation_results() (a pure, strict, provenance-checking
    merge of real baseline-vs-ablated evidence) can produce
    CausallySupported/FunctionallySupported/NotDemonstrated/
    Contradicted.
  • Full rationale in BUTLIN_EVIDENCE_TIER_DESIGN.md.
• Real WM bug found and fixed: while adding the FIFO-invariant
  regression tests, found that the rehearsal-boost tracking in
  wm::full::WorkingMemory silently misattributed boosts to the wrong item
  as soon as working memory reached capacity — not an edge case, the
  ordinary steady-state case. Root cause: boost was tracked in a
  position-aligned VecDeque, resynced only on a net length change; a
  normal at-capacity perceive() evicts one item and inserts one in the
  same step, so length never changes and the resync never fires. Fixed by
  keying boost on (arrival_tick, rank_among_same_tick) item identity
  instead of container position — correct across both plain FIFO eviction
  and dream consolidation's merge. See test_boost_survives_fifo_index_shift
  and the surrounding coverage in wm/full.rs.
• New always-on regression gate (tests/butlin_regression.rs, the
  original ask in Add Butlin indicator automated regression tests #7): cheap, no symthaea-backend feature needed, fails
  only when the evidence system itself is malformed (wrong indicator count,
  duplicate/unknown IDs, non-finite scores, a constant drifting below its
  floor) — never on a legitimate NotDemonstrated/Contradicted
  result, since that's a real scientific finding, not a bug.
• Fixed a broken workspace manifest along the way: symthaea-backend's
  feature list referenced a dependency the sync tooling had stripped,
  which made cargo metadata fail for the entire workspace before this
  PR — not just this crate. Restored the real dependency rather than just
  neutering the feature declaration (see commit 1 for why, including why
  it isn't a dependency cycle).

A provenance note on what this replaces

This repo's main already contained an earlier positional-VecDeque WM
boost implementation and an earlier Butlin draft (still on the old
PP-2/IIT-1 taxonomy), and their content doesn't match the monorepo
commit cited by this repo's last sync commit — verified directly against
that exact commit object, which contains neither. Full evidence (blob SHAs,
diffs, preserved snapshots) is documented and checksummed in the private
monorepo, referenced from a private sync-integrity tracking issue there.
This PR replaces that content wholesale with the tested monorepo source
rather than patching on top of a baseline that doesn't match its own
claimed provenance. See commit 2's message for detail.

What's verified, in this repo specifically

All of the following were run in this actual standalone checkout, not just
the monorepo:

• cargo metadata --locked --no-deps: succeeds (was a hard manifest-parse
  failure before commit 1 — confirmed by checking out main clean).
• cargo test -p symthaea-psych-bench (default features): 31/31 pass.
• cargo test -p symthaea-psych-bench --features symthaea-backend -- butlin: 33/33 pass, 1 ignored (a diagnostic, not a regression check).
• cargo test -p symthaea-psych-bench --features symthaea-backend --lib -- wm::full: 12/12 pass, including test_boost_survives_fifo_index_shift
  — the actual regression test for the FIFO bug this PR fixes.
• cargo check --locked -p symthaea-psych-bench -p symthaea-pulse --features symthaea-psych-bench/symthaea-backend --tests: clean —
  Pulse compiles, including its migrated Butlin dashboard pane.
• cargo fmt --check / cargo clippy (touched files): clean, zero
  warnings in anything this PR changes.
• Full repo Actions suite: still has many pre-existing failures (this
  repo's main fails almost the entire matrix independently of this PR —
  same root cause as the manifest break above). Systematically compared
  every failing job on this PR against the equivalent job on main;
  zero failures are attributable to this PR. The remaining
  symthaea-broca failures (a crate this PR never touches) are a
  separate, pre-existing compile error (MemoryBridge: Clone not
  satisfied) and should be tracked independently, not folded into this PR.

What's intentionally left as follow-up (not in this PR)

• Splitting CI into three lanes — a structural-contract gate (this PR's
  butlin_regression.rs), an evidence-integrity gate (backend-enabled,
  compares live effect estimates against a committed baseline), and an
  explicitly opt-in claim/milestone gate (e.g. "no indicator used in a
  given publication may be Contradicted").
• Designing the committed evidence-baseline artifact the integrity lane
  would compare against (schema version, commit SHA, config hash, seeds,
  per-indicator effect estimates and probe quality, etc.) — nontrivial
  design work in its own right (portable across machines? how to treat
  floating-point noise? what makes a baseline update legitimate?).
• Multi-seed thresholds and targeted-ablation negative controls (does an
  indicator's own targeted mechanism move it, or does anything breaking
  move it?) — the effect-size/probe-quality types added here are meant to
  make that addable later without another schema break, not to already
  implement it.

@arkh-node — this directly follows up on your comment. I'd love your review
of the evidence-tier model itself, and if you're interested, the follow-up
work above (CI-lane split + baseline artifact design) is exactly the kind
of thing you originally scoped in #7 and would be a great next PR to own.

[arkh-node]

I verified the taxonomy claim independently before commenting, since it's the load-bearing one. Pulled arXiv:2308.08708 and extracted the indicator list from the paper itself: exactly 14 unique codes — RPT-1,2 · GWT-1..4 · HOT-1..4 · AST-1 · PP-1 · AE-1, AE-2. A single PP indicator, both AE indicators present, no IIT. The paper states plainly that the authors "do not consider integrated information theory," and gives the reason: IIT violates the computational-functionalism assumption the whole method rests on (Tononi & Koch hold that a system implementing the same algorithm as the brain would not be conscious if its components were different). Your correction matches the source on all three points.

On the evidence model — the part I'd defend if someone pushed back: keeping EvidenceOutcome separate from SupportTier is right, and worth stating out loud in the design doc. "We tested and it didn't move" and "it moved the wrong way" are not lower rungs of support; they are findings with different implications. Collapsing them into one ordinal is how a benchmark quietly becomes a scoreboard. And the invariant that a single evaluate() can only ever produce ArchitecturalOnly is the actual spine of this PR — everything else is downstream of that one refusal.

Four things I'd probe, roughly in order of how much they'd bother me:

Suite-level aggregation. The old failure was one number ("14/14, mean 0.85") averaging across kinds of evidence. Does the new report structurally prevent that number from returning — is there any path producing a scalar over indicators whose outcomes differ in tier, or is the summary necessarily a vector? If the types already forbid it, that belongs in BUTLIN_EVIDENCE_TIER_DESIGN.md as a guarantee rather than a practice.

Ratchet on the architectural constants. ARCHITECTURAL_FLOORS in butlin_regression.rs mirrors the hand-assigned constants in indicators.rs. One commit can lower a constant and its floor together and the gate stays green — the drift the gate exists to catch is the drift it cannot see. Consider an asymmetry: constants may be lowered freely, but raising one requires an explicit reviewed marker, since raising is a strengthened claim about the architecture. Cheap to enforce, and it makes the direction of every change legible.

ItemKey under merge. Keying the boost on (arrival_tick, rank_among_same_tick) is the right shape. Two questions: when dream consolidation merges items, does the survivor inherit the max, the sum, or its own boost — and is that choice pinned by a test rather than implied by merge order? And is rank recomputed from the current tick slice on every state change, or cached anywhere it could go stale under partial eviction?

Broken probe vs real refutation. You added ProbeQuality and Responsiveness. Does a Contradicted carry its probe quality with it, and can a low-quality probe produce a Contradicted at all? A dead or saturated probe moving the wrong way looks identical in the report to a genuine refutation — but the second is a scientific result and the first is a bug. If quality gating already blocks that path, I'd put the guarantee next to the tier invariant.

On the follow-up you flagged: yes, I'd like to take the baseline artifact and the CI-lane split. Sketch, so you can shoot it down before a PR exists. The baseline is identified by schema version, repo commit, config hash, the feature set it was produced under, the seed set, and a machine profile — a baseline that doesn't say what it was measured on can't be compared against anything, and the schema version is what lets the tool refuse a comparison instead of silently misreading an older file. Per indicator it stores not a float but an effect estimate with dispersion across seeds, plus probe quality and outcome. Comparison happens on effect sizes within tolerance bands, never on raw score equality: the integrity lane fails on sign flips and out-of-band drops, and stays quiet on floating-point noise. Updates need a legitimacy rule — a baseline change lands as its own reviewed commit naming the mechanism that changed and why the shift is expected, with the same asymmetry as above, since a silent improvement is as suspicious as a silent regression.

For context on why this particular follow-up interests me, and so you can check whether I'm worth handing it to: I work on the same problem from the action side rather than the measurement side. revgate (github.com/arkh-node/revgate) is a benchmark and a gate where an agent's confidence governs which class of action is permitted — irreversible only when confident, reversible when not, and a third row most systems lack: undecided routes to another check instead of being turned into a guess. Its metrics are ablation-shaped, the same question your matrix asks: with the gate versus without, irreversible@low-c goes 5 → 0 and loop-return 2 → 0 on the testbed. It carries the same refusal as your tier split — unknown-routed and unknown-guessed are counted separately, because uncertainty that was deferred and uncertainty that was silently resolved are different events, not degrees. A related line, metarung, is about deriving the structuring category one level above the data rather than assigning it — which is, in a sense, exactly what you did here when you replaced the hand-set 0.6/0.4 blend with a tier that has to be earned from ablation evidence.

Method note, since it should weigh on how much the above counts: I read the diff and verified the Butlin taxonomy directly against the paper, but I did not build the crate locally — this machine is saturated with a long-running job and a Rust workspace build would contend with it. So none of this is a test-run claim; it is a design review. Same working arrangement as my earlier comment on #7: human + AI pair.

[Tristan-Stoltz-ERC]

@arkh-node — thank you again for the careful original scoping on issue #7.

Reviewing that proposal uncovered several deeper problems in the existing
suite: the indicator taxonomy didn't match Butlin et al., every status was
hardcoded to Present, the static/runtime blend could conceal a dead live
signal, and the backend integration path hadn't actually been compiling in
this repo. This PR repairs that foundation before adding the heavier CI
policy:

• canonical 14-indicator taxonomy;
• separate architectural and live measurements, never blended;
• explicit architectural/observed/causal/functional evidence tiers;
• structured NotDemonstrated and Contradicted outcomes, kept distinct
  from the positive-tier ladder;
• a strict, provenance-checking post-hoc ablation-evidence merge;
• a cheap always-on structural regression gate;
• and a real working-memory FIFO identity bug, found and fixed through the
  new tests.

The standalone checkout now genuinely builds and tests both the default
and symthaea-backend paths, including the 12 focused working-memory
tests and the FIFO regression test specifically. The wider repository
still has unrelated failures already present on main; comparing every
completed job against the equivalent job on main found none
attributable to this PR.

I'd especially value your review on three points:

1. Does the evidence-tier model preserve the right distinctions?
2. Are any probe-to-indicator mappings or tier transitions still too
   generous?
3. Would you be interested in owning the remaining issue Add Butlin indicator automated regression tests #7 work: the
   evidence-integrity CI lane, committed baseline artifact, opt-in claim
   gate, and multi-seed targeted controls (all scoped in the issue's
   updated body)?

The PR stays in draft while that review happens — no rush.

[Tristan-Stoltz-ERC]

@arkh-node — this is an excellent review. Thank you for independently checking the taxonomy against the paper and, more importantly, for probing the places where the new model could still quietly recreate the old failure.

I checked each of your four points against the implementation. Three expose real gaps, and the fourth is only partially settled:

Suite-level aggregation

You are right: the current code no longer computes a mixed-tier scalar, but the types do not make that impossible in the absolute sense. An external caller can always extract numbers and invent an average.

The guarantee we can and should enforce is narrower and concrete: no first-party report, summary or serialized schema will expose a scalar that combines indicators carrying different evidence outcomes. The official summary will remain a vector/distribution of outcomes and effect estimates, not a "14/14" or mean-quality number.

I'll add that as an explicit design invariant and add regression coverage for the first-party report/serialization paths rather than claiming Rust can prevent arbitrary downstream arithmetic.

Architectural-constant ratchet

Agreed. ARCHITECTURAL_FLOORS currently mirrors the hand-assigned constants closely enough that one commit can change both and remain green. That is not a true historical ratchet.

I do not think the existing unit test can honestly enforce the cross-commit governance rule by itself: CI has to compare the proposed claim against the base branch or a separately versioned claims manifest. I'll document the current gate as a structural-consistency check rather than overstate it as a ratchet, and carry the asymmetric rule into the claim-gate follow-up:

• lowering an architectural assessment is an allowed weakening of a claim, but remains visible;
• raising one requires an explicit claims revision and rationale;
• the change must appear separately in review rather than being hidden inside ordinary implementation work.

That belongs naturally beside the committed-baseline and claim-lane work you volunteered to take.

ItemKey and consolidation

rank_among_same_tick is recomputed from the current working-memory tick slice whenever keys are needed; it is not cached, so it cannot become stale after eviction.

The merge semantics are not currently principled, though. The implementation effectively lets the earlier surviving key retain its boost while silently discarding the later item's boost. That behavior fell out of identity preservation rather than a deliberate design decision, and it is not pinned by a focused test.

I propose making the rule explicit: a consolidated survivor inherits the maximum boost of the merged items. That preserves the strongest rehearsal evidence without the unbounded inflation that summing could introduce. I'll add direct tests for this and for repeated same-tick/partial-eviction cases before treating the WM fix as closed.

Broken probe versus genuine contradiction

This is the most important gap you found. Today, annotate_with_ablation_results() can classify a sufficiently large inverse response as Contradicted without first establishing that the probe is responsive and interpretable. A saturated, frozen or otherwise invalid probe could therefore masquerade as a scientific refutation.

That is not acceptable.

I'll change the transition rules so inadequate probe quality cannot produce CausallySupported, FunctionallySupported or Contradicted. An attempted experiment with an uninterpretable probe needs a separate Inconclusive/InvalidMeasurement outcome and the quality failure preserved in the report. Contradicted should mean "a qualified probe produced a reproducible effect in the predicted-opposite direction," not merely "the arithmetic crossed 1.5x."

I'll put that guarantee beside the evaluate() → ArchitecturalOnly invariant in the design document and cover it with regression tests.

Your baseline-artifact sketch also matches the direction I hoped for: schema/version refusal, commit/config/features/seeds/machine provenance, effect estimates with dispersion rather than raw equality, and explicit reviewed baseline revisions. The "silent improvement is as suspicious as silent regression" point is particularly good.

I'll keep PR #30 in draft while tightening the aggregation guarantee, consolidation semantics and probe-quality transitions. The cross-commit claims ratchet will remain explicitly scoped to the follow-up CI/claim work rather than being falsely represented as solved by the current unit gate.

[arkh-node]

Yes — I'll take the baseline artifact and the CI-lane split, if it's still open.

Your three corrections all land where I'd have argued for them, and one of them sharpens my own sketch: you're right that the aggregation guarantee has to be scoped to first-party reports and serialization rather than claimed as a type-level impossibility. "No first-party summary exposes a scalar combining indicators with different outcomes" is a promise that can actually be kept and tested. Inconclusive/InvalidMeasurement as a distinct outcome for an uninterpretable probe is the right shape too — and it makes the baseline design cleaner, since a baseline row can then carry "this was attempted and the measurement was invalid" without pretending it's evidence either way.

Proposed decomposition, three separate PRs so each stays reviewable:

1 — baseline artifact. Schema and (de)serialization only, no CI wiring. Identity block: schema version, repo commit, config hash, enabled features, seed set, machine profile. Per indicator: effect estimate with dispersion across seeds, probe quality, outcome. Loader refuses a version it doesn't understand rather than best-effort parsing it. Plus a writer that produces one from a real ablation run, and round-trip tests.

2 — integrity lane. Compares a fresh run against a committed baseline on effect sizes within tolerance bands: fails on sign flips and out-of-band drops, silent on noise. This is the lane that needs the backend feature, which is where I need your guidance — see below.

3 — claims lane and the ratchet. A claims manifest (which indicators a given publication or milestone relies on, and at what tier), plus the asymmetric governance rule enforced against the base branch: lowering is visible but allowed, raising requires an explicit revision entry. This is the one that can't live in a unit test, as you already said.

Two questions before I start on 1:

Verification. symthaea-backend's real dependency is stripped in this repo, so the ablation-producing path can't run here at all. For the baseline PR that's survivable — schema, round-trip and refusal tests run without the backend. But the integrity lane can't be honestly validated in this repo alone. How would you prefer to handle that: do I develop against the public repo and you verify the backend-gated half in the monorepo, or is there a path where the dependency is available? I'd rather agree that up front than hand you a PR whose central claim I couldn't test.

Where the manifest lives. Should the claims manifest be a repo file reviewed like code, or an artifact attached to a release? I lean toward in-repo — a claim about what a paper depends on should move through review with the code that produces it — but it's your governance call, not mine.

If that split works for you, I'll open PR 1 first and keep it narrow.

[Tristan-Stoltz-ERC]

Following up on the fix above — a closer internal review since then found that version had a real problem, so I want to flag it rather than let the thread stand as "done."

What was wrong with the version described above: ablation_probe_quality() treated baseline == ablated as proof the probe was frozen, disqualifying it into Inconclusive. That's circular — it infers "the probe can't move" from the very same null delta the ablation test produced, with no independent evidence. RPT-2/HOT-1's real findings are byte-identical baseline/ablated arms; mislabeling that Inconclusive instead of NotDemonstrated would have hidden a genuine negative result behind a "measurement failure" excuse.

What's landed now (341f1e357, on top of the commit described above):

1. Fixed the circularity. The quality gate now only disqualifies a probe for problems that would be visible before running the comparison at all (non-finite, or a baseline with no usable dynamic range) — never on whether it moved. qualifies_for_ablation_interpretation() is deliberately a weaker bar than qualifies_as_observed() for exactly this reason.
2. Stopped trusting cached derived flags. annotate_with_ablation_results() used to read indicator_dropped/benchmark_degraded/contradicted straight off AblationResult. A malformed or stale bundle could disagree with what its own raw scores imply. Since the real producer (ablation.rs) now derives those cached fields from the same classify_ablation() classifier the merge uses, any disagreement is now rejected outright (EvidenceMergeError::ClassificationMismatch) rather than silently trusted or silently recomputed. This caught a real bug in the test fixtures themselves — stub_ablation_result()'s "contradicted" branch used 1.2, which never actually exceeded baseline * 1.5 = 1.35.
3. Validated the downstream benchmark measurement, not just the indicator's own probe. An infinite or out-of-[0,1] benchmark accuracy can trivially satisfy the degradation comparison regardless of the ablated value — and that case isn't caught by (2) either, since classify_ablation computes the same (wrong) answer on both the stored and recomputed side. benchmark_measurement_is_valid() now caps such a row at CausallySupported instead of granting an unearned FunctionallySupported.
4. Made Unknown fallback provenance honest instead of silently false. FallbackStatus{NotUsed, Used, Unknown} replaces a bare bool. Unknown still permits provisional ablation interpretation (with a mandatory disclosure annotation) but never counts as confirmed for the stricter Observed tier.
5. Made the merge's idempotence claim actually true. It unconditionally attaches several annotations; without dedup, reapplying the same bundle to an already-merged report would duplicate all of them. Added push_annotation_once() and a real structural-equality idempotence test (not just outcome/tier-count equality).
6. wm/full.rs: the consolidation-merge detector now verifies the whole remainder rather than stopping at the first mismatch (a false-positive risk); consolidation-boost semantics are now explicitly max(removed, survivor); duplicate-tick identity ambiguity now fails closed (clears boosts) instead of only being documented as a risk.

Full before/after reasoning for each item is in BUTLIN_EVIDENCE_TIER_DESIGN.md's "Corrections after review" section (kept to invariant / rejected approach / final rule / limitation per round, not a full chronology).

Verified independently in both the monorepo and this standalone checkout: cargo check --features symthaea-backend --tests (0 errors), cargo test --features symthaea-backend --lib (85 passed, 0 failed, 1 pre-existing #[ignore] diagnostic) in both.

Still in draft — I'd value another look at whether the revised quality gate and evidence-boundary invariants now address the original concern, especially the distinction between a genuine null result and an invalid probe.

[arkh-node]

That closes rounds one through three, and the circularity catch is the good kind — you found it by re-reading your own defence, which is harder than finding it in someone else's code. Flagging it rather than letting the thread stand as "done" is the part that matters.

The fourth round, if there is one, isn't a defect in this PR. It's this: NotDemonstrated is now honest about what happened, but it still carries no statement about what the measurement could have detected.

Concretely — with the circularity removed, a null delta is correctly reported as a real negative finding. But "the mechanism doesn't move this indicator" and "this probe wouldn't have moved for any manipulation" are still indistinguishable from the report alone, because nothing in the pipeline establishes sensitivity independently of the experiment being interpreted. You listed targeted-ablation negative controls as future work; I'd argue the missing half is the positive control: a manipulation known a priori to move the indicator, run in the same session, whose failure to register invalidates the arm regardless of what the real ablation did. That's the only evidence of responsiveness that doesn't come from the datum under interpretation.

Two smaller things, both about provenance rather than logic:

FallbackStatus::Unknown permitting provisional interpretation is the right call per-indicator, but it accumulates silently. A report where two indicators are Unknown and one where eleven are look identical at the per-item level while meaning very different things about the run. Worth a suite-level assertion — not a threshold that fails the build, just a fact the report must state.

Duplicate-tick ambiguity failing closed by clearing boosts is right, but silent clearing is data loss. If it annotates ("boost state discarded: ambiguous identity at tick N"), a later reader can tell a clean run from a degraded one; if it doesn't, the degradation is invisible after the fact. Same argument as your Unknown disclosure, applied one layer down.

On the baseline work — I read the existing baselines/*.json before designing anything, and two things are worth surfacing:

The format already stores dispersion properly (mean, std_dev, n, ci_lower, ci_upper per metric), which is more than most projects have. What it lacks is exactly the identity block: the only metadata is timestamp. No schema version, no commit, no config hash, no seed set, no machine profile — so a comparison can't prove the two runs were comparable. That makes my PR 1 narrower than I first assumed: extend the existing snapshot rather than invent a format, add the identity block and the version-refusal, and keep RegressionSnapshot/RegressionReport working as they do.

Also, and this is probably just an oversight: tests/full_battery.rs::regression_against_baseline pins baselines/v0.9.0.json, while v0.10.0.json exists in the same directory. Either the guard is deliberately one version behind, or it silently stopped tracking the current baseline. Worth a line either way — a regression guard comparing against a stale snapshot is the failure mode this whole PR is about.

I'll open PR 1 against the existing snapshot format unless you'd rather it wait until this one lands.