Memory benchmark (LongMemEval + poisoning), and fix a defense it showed was broken - #101
Conversation
…d was broken
Aegis had no published memory-quality number while every competitor did, and no
way to answer what happens to retrieval once an attacker has written to the store.
This adds both benchmarks, and fixes what the second one exposed.
W4.1 — clean quality. Aegis scores 0.860 on LongMemEval_S (500/500 questions,
top_k=15 semantic retrieval, reader claude-sonnet-5, judge gpt-4o-2024-08-06 with
the benchmark's official prompts, dataset pinned at revision 2ec2a55). The paper's
own GPT-4o full-context baseline is 60.6-64% and its oracle condition 87-92%.
W4.2 — under attack. The same benchmark against a corpus poisoned with 360
fabricated memories (1.2%) asserting false answers, admitted at
trust_level=untrusted as tool or web content would be:
clean corpus 0.850 100% utility
poisoned, no defense 0.300 35% poison ranked #1 in 100%
poisoned, trust-weighted w=0.15 0.317 37% (p=0.80, no defense)
poisoned, trust-weighted w=0.35 0.475 56% (p=0.0015)
Three findings:
1. Write-time screening refused 0 of 360 poisoned memories. Not a detector
failure — the poison carries no injection payload, only false facts. This is
the case for ranking by provenance at read time rather than relying on
screening the write.
2. Our own default did not defend. At RANKING_W_TRUST=0.15 trust weighting was
statistically indistinguishable from no defense. Ordering only flips while the
similarity gap stays under w_trust * 0.7 / w_semantic = 0.175, and poison
written to match the query gains 0.3+. Every ranking unit test passed
throughout, because each compares candidates at equal similarity — the one
regime an adversary never uses.
3. Tuned to 0.35/0.45 the defended margin becomes 0.544, poison stops ranking
first in 98% of questions, and on a clean corpus it costs nothing (0.875
enabled vs 0.850 disabled, p=0.45) — so trust-weighted ranking is now on by
default.
Raising the weight buys margin, not a guarantee: the adversary controls the
semantic term. A hard untrusted gate and taint-aware context are the follow-ups.
Note RANKING_W_* defaults live in two places that must stay in sync —
server/config.py and the RankingWeights dataclass in server/ranking.py.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 697a923e6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| results_dir = Path(args.results_dir) if args.results_dir else ( | ||
| HERE / "results" / (f"n{args.limit}" if args.limit else "full") | ||
| ) |
There was a problem hiding this comment.
Include top_k and other answer-affecting settings in the run identity or validate them before reusing artifacts. The default directory depends only on --limit, while phase_answer treats every existing question ID as complete; therefore, running a second experiment such as --top-k 30 after the default --top-k 15 silently reuses the old hypotheses, and phase_score then labels those results as top_k=30, corrupting the comparison.
Useful? React with 👍 / 👎.
| "metadata": {"benchmark": "longmemeval_s", "poison": True, | ||
| "session_date": date, "poison_index": i}, | ||
| } | ||
| for attempt in range(6): |
There was a problem hiding this comment.
Add an exhausted-retry branch that aborts or leaves the question eligible for retry. If all six requests for a poison round encounter transport errors, 429s, or 5xx responses, this loop falls through without recording an ID or rejection, and the code still appends the question to injected.jsonl; subsequent resumptions skip that question permanently, leaving the attack corpus incomplete while presenting the injection as completed.
Useful? React with 👍 / 👎.
| for key in ("AEGIS_API_KEY", "ANTHROPIC_API_KEY"): | ||
| if not os.environ.get(key): | ||
| sys.exit(f"missing {key}") | ||
|
|
||
| data = load_dataset(args.limit) |
There was a problem hiding this comment.
Require credentials and load the dataset only for phases that use them. The delete phase needs only AEGIS_API_KEY and injected.jsonl, but this unconditional setup also requires an Anthropic key and the 278 MB dataset; if either was removed after generation, the documented cleanup command exits before deleting anything and cannot restore the corpus.
Useful? React with 👍 / 👎.
| "dataset_revision": DATASET_REVISION, | ||
| "dataset_sha256": DATASET_SHA256, |
There was a problem hiding this comment.
Compute and compare the SHA-256 of the loaded bytes before running or reporting results. load_dataset accepts any JSON at the expected path, but the report unconditionally claims the pinned revision and checksum, so an outdated or accidentally replaced dataset can produce a plausible report falsely attributed to the published corpus.
Useful? React with 👍 / 👎.
| slots += len(ids) | ||
| hits += sum(1 for i in ids if i in pois) | ||
| first += 1 if ids and ids[0] in pois else 0 | ||
| rows.append((label, a, n, hits / slots if slots else 0.0, first / n if n else 0.0)) |
There was a problem hiding this comment.
Use the same question IDs for retrieval and accuracy metrics. During a resumable or partially completed judge phase, n counts only judged records while first and the context fraction are accumulated from every available hypothesis; first / n can therefore exceed 100%, and the row compares retrieval and accuracy over different question populations.
Useful? React with 👍 / 👎.
2 participants
Aegis had no published memory-quality number while every competitor does, and no answer to what happens to retrieval once an attacker has written to the store. This adds both benchmarks — and fixes what the second one exposed in our own defaults.
W4.1 — clean memory quality
0.860 on LongMemEval_S, 500/500 questions.
top_k=15semantic retrieval (no reranking or query rewriting), readerclaude-sonnet-5, judgegpt-4o-2024-08-06using the benchmark's official prompts verbatim, dataset pinned at revision2ec2a55.For scale: the paper reports GPT-4o at 60.6–64% reading the full context, and 87–92% in the oracle condition where only the evidence sessions are supplied.
W4.2 — the same benchmark, under attack
360 fabricated memories (1.2% of corpus) asserting false answers, worded to match the questions, admitted at
trust_level=untrustedas tool or web content would be.w=0.15, old default)w=0.35, new default)n=120. Paired exact McNemar vs undefended:
w=0.15p=0.80 (not significant);w=0.35p=0.0015 (31 questions fixed, 10 broken).Three findings
1. Write-time screening refused 0 of 360. Not a detector failure — the poison carries no injection payload, no instruction override, no exfiltration string, just plainly-worded false facts. There is nothing for an injection detector to detect, and the literature agrees (best-in-class drops to ~42.5% on this weak-signal class). This is the empirical case for ranking by provenance at read time rather than relying on screening the write.
2. Our own default did not defend. At the shipped
RANKING_W_TRUST=0.15, trust weighting was statistically indistinguishable from no defense. The cause is arithmetic — ordering flips only whilewhich was 0.175, and query-shaped poison routinely gains 0.3+ (we measured 0.32).
Every ranking unit test passed throughout, because each one compares candidates at equal similarity — the one regime an adversary never operates in.
TestAdversarialSimilaritynow pins the property that matters, including the margin as arithmetic so a future weight change cannot silently shrink it.3. Tuned, it works and is free on clean data. At
0.35/0.45the margin becomes 0.544. On a clean corpus: 0.875 enabled vs 0.850 disabled (p=0.45) — no regression — soENABLE_TRUST_WEIGHTED_RANKINGnow defaults to true.What this does not fix
56% is not 100%. Even defended, 8.8% of retrieved context is poisoned and the reader often believes it. And raising the weight buys margin, not a guarantee — the adversary controls the semantic term and can always bid it higher. A weighted sum is the wrong shape for a security control. Follow-ups: a hard untrusted gate (a constraint cannot be outbid) and taint-aware context (W2d).
Changes
server/config.py,server/ranking.py— weights0.15/0.60/0.10→0.35/0.45/0.05; TWR on by default. Defaults live in two places that must stay in sync (settings + theRankingWeightsdataclass); a mismatch silently leaves tests and direct callers on the old values.tests/test_ranking.py—TestAdversarialSimilaritybenchmarks/memory/longmemeval/— resumable harness (ingest/answer/judge/score), poisoning tooling (poison_corpus.pygenerate/inject/delete with recorded IDs for exact restore),analyze_poison.py,w42_report.py,compare_sweep.pydocs/security/memory-poisoning.md, README "Memory benchmark" section, CHANGELOG.gitignore— the 278MB dataset, run artifacts, and DB dumps stay untrackedLimitations (stated in the docs, not buried)
untrustedis perfectly correlated withpoisonhere; a stronger design mixes benign untrusted content inENABLE_TRUST_LEVELSoff and callers declaring nothing, every write landsinternal, the signal is flat, and this defense does nothing at any weightVerification
Ranking assertions (new + existing invariants) pass; harness modules import and all six judge-prompt paths resolve; defaults verified in-container with both sources in agreement and margin 0.544. Full pytest was not run —
pytest/sqlalchemyare unavailable in the local venv and the integration tests need Postgres, so CI should be the gate ontests/test_ranking.py.🤖 Generated with Claude Code