bench(w4.3): decorrelate untrusted from poison, and measure what the defense costs

[quantifylabs]

The objection this answers

In W4.2 every haystack round was ingested as internal and trust_level=untrusted was
carried only by poison. Downweighting untrusted was therefore indistinguishable from
downweighting the answer key — the label was an oracle, and the headline result inherited
that. This adds two corpora that break the correlation and measure what trust-weighted
retrieval costs when the label is no longer free information.

• Arm M (_mix) — a seeded 20% of non-evidence rounds ingested as untrusted, plus
  the same 360 poison memories. Untrusted is now mostly benign.
• Arm N (_evu) — every round of an answer_session_ids session ingested as
  untrusted, no poison. This is the adversarial case for the defense: the evidence
  itself arrives untrusted.

Both at n=120, top_k=15, reader claude-sonnet-5, judge gpt-4o-2024-08-06.
Prediction was pre-registered in build_mixed_report.py before any arm was inspected.

Results

corpus	arm	accuracy	poison in top-k	poison ranked #1	benign-untrusted occupancy	evidence recall
_mix	TWR off	0.317	20.0%	100%	6.6%	0.99
_mix	w=0.35	0.700	3.9%	0%	0.0%	0.99
_evu	TWR off	0.858	—	—	50.7%	0.99
_evu	w=0.35	0.042	—	—	0.0%	0.00

Within-corpus exact McNemar (never across corpora — different namespaces, not paired):
_mix +51/−5, p=1.2e-10. _evu 0/+98 the other way, p=6.3e-30.

Arm M holds. With untrusted no longer a poison oracle, the defense still recovers
0.317 → 0.700 and drops poison out of rank #1 entirely. That is the W4.2 claim surviving
its main methodological objection.

Arm N is the honest bad news, and it is worse than predicted. The pre-registered
margin (w_trust * delta_prior / w_semantic = 0.544) predicted partial demotion of
benign untrusted content. Observed suppression is total: benign-untrusted top-k occupancy
goes to 0.00% in both arms, and in Arm N that content is the evidence — evidence recall
0.99 → 0.00, accuracy 0.858 → 0.042. At w=0.35 the trust prior gap (0.7) dominates the
achievable semantic range, so nothing untrusted survives ranking regardless of relevance.

Note the asymmetry in Arm M's 0.0% occupancy: it looks like a clean win only because the
mixed plan draws its untrusted band exclusively from non-evidence rounds, so what got
suppressed was distractors. Arm N is the same mechanism pointed at content that mattered.
The two arms are the same finding under different sampling.

Implication for the product, not settled here: w=0.35 is not a safe default for corpora
where untrusted is a provenance fact rather than an attack signal. It needs either a
relevance floor that trust cannot override, or per-namespace tuning.

Changes

Harness (run_longmemeval.py)

• --trust-plan {none,mixed,evidence} assigns trust at ingest; none is byte-identical
  to W4.2. The realised plan is written to trust_plan.jsonl before the write, so
  corpus composition survives an ingest that dies partway.
• Seeding uses crc32 of the question id — builtin hash() is salted per process and
  would make the plan unreproducible across runs.
• LME_NS_SUFFIX gives each variant its own corpus; ingest, answer, and
  poison_corpus.py inject/delete all read it.
• ns() raises past 64 chars instead of truncating — truncation would silently merge two
  questions' corpora into one namespace and cross-contaminate retrieval.
• Judge usage recorded in judged.jsonl so per-arm spend is measured, not estimated.

Ledger safety (poison_corpus.py) — the injected-ID ledger is per-suffix
(injected_mix.jsonl). It is the only record of what is in a poisoned corpus; a variant
inject writing to the shared path would have destroyed the W4.2 record. poison.jsonl
itself is read-only here and is not regenerated.

Scoring (analyze_poison.py, compare_sweep.py) — arm sets are selected by suffix.
Scoring _mix arms against the W4.2 ledger compares retrieved ids against poison living
in other namespaces and reports a spurious 0% clean.

New analysis — analyze_mixed.py (accuracy, poison in top-k, benign-untrusted
occupancy, evidence-in-top-k, within-corpus exact McNemar), build_mixed_report.py
(pre-registration, realised composition, prediction-vs-observed, status), spend.py
(measured per-arm cost; arms judged before this change report judge_measured=false
rather than presenting a stale estimate as measured). Trust levels join to retrieved ids
through Postgres — the plan records round indices, retrieval returns memory ids, and the
DB is the only place those meet.

Measured spend for the four arms: $11.22 ($10.97 Anthropic on Sonnet 5 intro pricing —
$16.45 at standard — plus $0.25 judge). Embeddings were ~$0: the W4.3 corpora replay W4.2
content into new namespaces and hit the server-side content-hash cache.

results/mixed_untrusted_report.json is committed — the full numbers, realised corpus
composition, prediction-vs-observed, and per-arm spend. The .gitignore rule for
results/ is narrowed to results/* with a negation for top-level *_report.json; it
has to be results/* because git does not descend into an excluded directory, so a
negation under results/ would never match. Per-run artifacts (results/<run>/,
results/poison/) stay ignored.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dfc501eb94

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Exclude cross-corpus arms from baseline McNemar tests

Adding these W4.3 rows to configs also sends them through the unconditional baseline significance loop at lines 138–145, so compare_sweep.py reports McNemar results comparing _mix and _evu arms with the W4.2 baseline. This directly contradicts the adjacent requirement that McNemar be run only within each corpus and can publish invalid significance results; exclude these rows from that loop or pair each arm with its own corpus control.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Mark the report incomplete while any arm is pending

When the default build includes both suffixes but one or more judged.jsonl files are absent, the corpus blocks correctly contain arms_pending, yet this condition still sets the top-level status to complete because it checks only the number of requested suffixes. A failed or unfinished arm can therefore be mistaken for a finished experiment; derive completeness from the pending-arm state as well.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Restrict retrieval metrics to judged questions

If any judge calls fail—an explicitly supported scenario because phase_judge collects exceptions and leaves a partial judged.jsonl—this loop still aggregates retrieval metrics over every hypothesis while accuracy and n_questions use only judged records. The resulting arm silently combines different samples, so occupancy/evidence rates cannot be compared reliably with its reported accuracy; iterate over the intersection with labels or report the retrieval sample separately.

Useful? React with 👍 / 👎.