Phase 2: provenance-native memory (W2a/W2b/W2c) + security hardening - #100
Merged
Merged
Conversation
…round 3)
The W1 authorization sweeps keyed on the string "MemoryRepository", so the
entire ACERepository read/write surface was invisible to them. Four ACE routes
had holes:
- POST /memories/ace/reflection: no AuthContext at all — agent_id from the
request body, no content scan, no integrity hash, no trust level, defaulting
to global scope (the ACE experience->procedure channel, the scope every agent
reads). Now runs the same gate sequence as /memories/add.
- POST /memories/ace/playbook and /ace/playbook/agent: passed
requesting_agent_id=body.agent_id to the scope ACL — the spoofed-read hole W1
closed on the query routes. Now derive the identity from the API key and apply
the principal-trust scope restriction.
- POST /memories/ace/vote/{id}: any project key could vote as any agent on any
memory, including memories the voter cannot read — a ranking-poisoning channel.
Now pins the voter to the key and requires read access to the target.
- POST /memories/ace/curate and /consolidate: memory content behind a
body-supplied agent identity; now agent-authorized (bound key = own agent,
unbound key = application), matching the handoff/decay posture.
ACERepository.create_reflection gains trust_level/content_flags/integrity_hash
so reflections are screened and signed like every other write; both playbook
queries gain a scope_filter for the principal-trust restriction.
The sweeps are re-keyed so this class of miss fails CI:
- the write sweep keys on every write sink (adds ACERepository.create_reflection)
- the read sweep additionally matches memory-touching ACERepository.* methods,
derived from the repository source (any method referencing the Memory model)
- new test_direct_memory_constructors_are_pinned: the set of files constructing
Memory(...) must equal {ace_repository, memory_repository, playbook_loader},
so a new constructor bypassing the repository fails immediately
Adversarial HTTP-layer tests validated to fail against pre-fix main (4e96df1)
in a scratch worktree (15 fail before, all pass after); the affected
integration suites stay green (unbound keys are unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pleteness, backfill
The v1 integrity hash (HMAC over "{project_id}:{agent_id}:{content}") covered
neither scope nor trust_level, so a direct DB edit flipping a memory from
agent-private to global — or relabelling its trust — verified clean. It was also
un-delimited, so a variable-length field could absorb the next.
v2 (stored with a "v2:" prefix) fixes both: the message is
aegis-mem-v2 \x1f project_id \x1f agent_id \x1f scope \x1f trust_level \x1f content
delimited by the unit separator and opened with a domain constant so a memory MAC
cannot be replayed as another Aegis MAC. verify_integrity dispatches on the prefix,
so legacy v1 rows and Context Hub (prompt/skill/subagent) rows still verify.
Signing completeness — prerequisites for safe verify-on-read (W2b), where an
unsigned or stale-hash row must not be mistaken for tampering:
- add_batch now signs every item; it was the one write path that never computed a
hash, leaving every batch-added memory unsigned.
- consolidation re-signs the keeper after rewriting its content; the stored HMAC
was going stale, so POST /security/verify would flag legitimately merged memories.
- PATCH re-signs whenever content OR trust_level changes (v2 covers trust_level).
- all memory writers (add, add_batch, ace_delta, typed, reflection) emit v2.
scripts/backfill_integrity.py upgrades unsigned/v1 rows to v2 (idempotent, batched,
--dry-run, --project-id) so a deployment can move to drop + INTEGRITY_REQUIRE_SIGNED.
Migration 0010 widens memories.integrity_hash 64 -> 80 ("v2:" + 64 hex = 67);
round-trips cleanly. New unit tests pin scope-flip / trust-relabel detection,
delimiter and domain separation, and v1 back-compat (including the documented
v1 scope-flip residual that the backfill closes); integration tests cover backfill
idempotence and the consolidation re-sign.
Stacked on the ACE-route gating branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signing on write did nothing on the way back out — a row edited directly in the
database was returned as if trusted. This adds the read-side gate: every retrieval
path recomputes the HMAC and, per INTEGRITY_READ_MODE, drops tamper-detected rows
and records a security event.
server/read_integrity.py:
- classify(memory) -> verified | unsigned | mismatched
- apply_read_integrity(...) — pure partition into (kept, dropped) by the decision
table; annotates integrity_verified on kept rows
- emit_integrity_events(...) — one INTEGRITY_FAILED event per drop, on a DEDICATED
write session (never the caller's): hybrid_query/get_memory/export run on
get_read_db, which never commits and may be a replica, so an event on that
session would be lost. Best-effort; never blocks a read.
- screen_read(...) — the entry point retrieval paths call.
Modes (INTEGRITY_READ_MODE, default "drop"; INTEGRITY_REQUIRE_SIGNED default false):
- off = return everything
- warn = annotate + emit, return everything
- drop = exclude MISMATCHED rows + emit; unsigned/legacy rows kept unless
require_signed, so drop is safe before scripts/backfill_integrity.py runs.
Hardened end state after backfill: drop + require_signed.
Wired into semantic_search (before decay/telemetry so counts reflect drops),
hybrid_search, get_agent_memories_for_handoff (tampered content would otherwise be
delivered as key_facts), the typed timeline/entity reads, the get_memory route
(not get_by_id — that also serves mutation and /security/verify, which must still
fetch a tampered row), and the context bundle. Export annotates integrity_status
per row and never drops (a backup should show tampering, not hide it). MemoryOut
gains integrity_verified; SecurityConfigResponse surfaces both new flags.
Tests: unit decision table + integration (real Postgres, raw UPDATE tamper) proving
drop excludes tampered rows from query/hybrid/handoff/get_memory, the event is
committed even on the hybrid read-session route, warn annotates, and unsigned rows
survive drop until require_signed. 13 new + broad regression green.
Local-mode (aegis_memory/local) verify-on-read deferred: single-user offline threat
model is weaker; server is the security-critical path.
Stacked on the integrity-signing-v2 branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Injection detections are only flagged by default, but a memory written to global scope is readable by every agent in the project — the exact reach a poisoning payload wants. New CONTENT_POLICY_INJECTION_GLOBAL_SCOPE (default "reject", alt "inherit") escalates a flagged injection to a hard reject when scope=="global". scan() has no scope parameter, so the escalation lives in scan_async, placed ahead of the Stage-4 classifier early-return so it applies even with the LLM classifier off (the default). Applied byte-identically to both content_security.py copies (server/ and aegis_memory/security/); the no-drift guard passes. The base injection policy is unchanged, so agent-private/shared writes still flag. The /security/scan preview reports at global scope, so it now shows this worst-case verdict — documented as intended. Tests: unit coverage of flag-at-agent / reject-at-global / inherit / clean, and an end-to-end pair (injection reflection -> 422 since reflections default to global; agent-private injection still flags). The reject-at-global test validated to fail against the pre-fix tip (8879bc1) in a scratch worktree. Stacked on the verify-integrity-on-read branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Origin was scattered across metadata_json, derived_from_agents, trust_level and the
event log, and the full content-security verdict was discarded after its flags were
copied onto the memory. This promotes provenance to a first-class object.
New memory_provenance table (immutable, 1:1 with memories, migration 0011): origin
channel + kind (the C1-C4 mapping), producing/acting agent and principal, source
run/interaction/trajectory, immediate parents + derivation depth, the taint set
(trust label + detections/confidences), the full policy verdict, a policy_version
hash, the admitted trust/scope, and a domain-separated HMAC over the record.
server/provenance.py: ProvenanceInput (frozen), the HMAC (distinct "aegis-provenance-v1:"
prefix so it can't be replayed as a content MAC), verify_provenance, and
compute_policy_version (sha256 over policy settings + SCANNER_RULES_VERSION, cached).
Choke point: MemoryRepository.add takes a ProvenanceInput and writes the signed row in
the same flush; a missing one synthesizes an "unattributed" record + metric so a memory
is never left with no origin. Every write channel supplies one — memories add/add_batch,
ace_delta, typed_memory, ace_reflection and the genesis seed (the last two build Memory
directly, so they persist the row themselves). A source-derived sweep test fails CI if a
new write router omits ProvenanceInput.
Promotion history is events, not record mutation: new SCOPE_CHANGED / TRUST_CHANGED /
PROMOTED / DEDUPLICATED event types. A trust relabel now records its old value (UPDATED
only listed the field name); a dedup emits DEDUPLICATED with the colliding write's origin
so a second, differently-trusted write stays auditable.
New GET /memories/{id}/provenance returns the record, its HMAC verification status, and
promotion history. policy_version is surfaced in GET /security/config. Riders: fixed the
TypedMemoryOut divergence (it lacked trust_level/content_flags) and the export
provenance-stripping round trip (added trust_level/content_flags/integrity_hash).
Tests: HMAC sign/verify + tamper, policy_version stability, per-channel origin recording,
the provenance endpoint, a tampered record reporting unverified, the dedup event, and a
trust relabel recorded as a promotion event. Migration 0011 round-trips. Both
content_security copies stay byte-identical (drift guards green).
Stacked on the injection-reject-global-scope branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retrieval ordered purely by vector distance; the votes, trust, decay, and provenance
Aegis records never influenced what came back — the ACE loop's final arrow was
disconnected. This fuses them, closing the loop with the same signals that make memory
safer: one mechanism, both benefits.
server/ranking.py fuses, gated by ENABLE_TRUST_WEIGHTED_RANKING (default off, so
ordering is unchanged until opted in):
score = w_sem·similarity + w_trust·trust + w_eff·effectiveness + w_decay·decay
+ w_prov·provenance (weights sum-validated at startup)
- ranks CONTENT trust (Memory.trust_level), distinct from ENABLE_TRUST_LEVELS which
gates principal-trust authorization
- neutral priors (0.5) for unproven signals: an unvoted memory or one with no
provenance record is neutral, never bad, so the un-voted corpus is never buried.
Effectiveness uses a Beta(1,1) mean instead of get_effectiveness_score's 0.0-for-unvoted
- provenance depth (from the W2a record) discounts derived memories: 1/(1+depth)
semantic_search over-fetches a candidate pool (top_k·multiplier, clamped) when the flag
is on — LIMIT top_k in SQL left nothing to promote — then fuses, sorts, truncates.
score stays cosine; the fused value is surfaced as relevance_score (replacing the old
query-independent recompute) and also fixes the decay-collapse: the decay-adjusted score
was computed then discarded, so order changed but the surfaced number didn't.
hybrid_search: same fusion when on. Bugfix even when off — its decay branch multiplied
by effectiveness×decay, which zeroed every unvoted memory; now score×decay, matching
semantic_search.
rerank_with_decay keeps its 3-tuple shape (local-lib parity, test pins intact); the
server routes through ranking.py.
Tests: unit priors + fusion invariants (all-neutral preserves semantic order; untrusted
below unknown at equal similarity; a helpful vote strictly raises the fused score — the
named regression). Integration (identical embeddings): trust reorders when on, votes flip
order at equal trust, and order is stable/semantic when off. 13 new + 278-test regression
green (temporal-decay 3-tuple pins, ACL grep, and search canaries all intact).
Stacked on the memory-provenance branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
aegis-memory/server/memory_repository.py
Line 455 in 7c66932
Over-fetch hybrid candidates before trust reranking
When trust-weighted ranking is enabled, HybridRetriever.search still hydrates only top_k rows here, so the later reranker merely permutes the existing result window. A trusted or highly effective candidate ranked just below the initial hybrid top-k can never be promoted, unlike the semantic path's explicit over-fetching; request a larger candidate result set and truncate only after trust reranking.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…y, warn events, hybrid fusion Five findings from the PR bot, all valid: - P1 Screen ACE playbook rows: /ace/playbook and /ace/playbook/agent returned rows straight from ACERepository (bypassing screen_read), so a tampered playbook entry reached the prompt under INTEGRITY_READ_MODE=drop. Both repository methods now screen their results. - P1 Verify consolidation inputs: consolidate_pair re-signed the merged keeper without checking the inputs, so a database-tampered keeper got a fresh valid HMAC — laundering the tampering past verify-on-read. It now classifies both inputs and refuses a mismatched pair (skipped_integrity_mismatch), leaving the row unsigned-as-tampered. - P2 Warn-mode events: apply_read_integrity tracked only drops, so warn mode annotated integrity_verified but emitted no INTEGRITY_FAILED event. It now returns failures independently of the keep/drop decision, so warn emits audit telemetry while still returning the row. - P2 Over-fetch hybrid candidates: hybrid_search hydrated only top_k, so trust reranking merely permuted that window. It now over-fetches a candidate pool (like semantic_search) and truncates to top_k after reranking. - P2 Normalize hybrid RRF scores: the hybrid channel's scores are raw RRF (~0.01-0.03), not ~[0,1] cosine, so the 0.60 semantic weight was swamped by trust. They are min-max normalized to [0,1] before fusion; the tuple still surfaces the original hybrid score. Also fixes a latent bug the new playbook test surfaced: query_playbook's QUERIED event omitted the required memory_id kwarg, so /ace/playbook would 500 on every call. New tests: playbook drops tampered rows, warn emits an event, consolidation refuses a tampered input, plus updated warn-mode unit coverage. Full suite green (838 passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cut the [Unreleased] changelog section to [2.7.0] and add the Phase 2 entries (provenance, verify-on-read, trust-weighted ranking, v2 integrity + backfill, ACE-route gating, injection@global, consolidation anti-laundering). Bump pyproject to 2.7.0 and update the README "what's shipped" table. The release also carries the W1 authorization work that had been sitting in [Unreleased] since v2.6.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_version_bumped asserts the package __version__ matches pyproject; the v2.7.0 bump updated pyproject but not the package literal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the v3 roadmap: make provenance a first-class citizen so security becomes a property of memory, not just a gate in front of it. Six self-contained commits, built and tested in order; combined into one PR to keep it to a single review pass.
Recommend merging with a merge commit (not squash) so the six commits stay distinct in history — each is a coherent, separately-validated unit.
Commits (in order)
934c7c4security: gate the ACE memory routes — an unplanned but live W1-class hole found during exploration. TheACERepositorysurface (/ace/reflectionwrote unscanned, unsigned memories straight into global scope with a body-suppliedagent_id; plus/ace/playbook,/ace/vote,/ace/curate,/ace/consolidate,/ace/playbook/agent) was invisible to the W1 sweeps because they keyed on the stringMemoryRepositoryand these go throughACERepository. Re-keyed the sweeps on memory creation/access; added an allowlist test pinning the files allowed to constructMemory(...)directly.8cc9ac2v2 integrity hash + signing completeness + backfill — v2 HMAC covers scope and trust_level (v1 covered only content, so a DB-level scope flip verified clean), delimited + domain-separated. Fixesadd_batchnever signing, consolidation leaving a stale hash, PATCH not re-signing on trust relabel. Migration 0010 (64→80);scripts/backfill_integrity.py.8879bc1W2b: verify integrity on read — every retrieval path recomputes the HMAC;INTEGRITY_READ_MODE(defaultdrop) excludes tamper-detected rows before they reach a prompt and emitsINTEGRITY_FAILEDon a dedicated write session (replica-safe). Unsigned/legacy rows kept untilINTEGRITY_REQUIRE_SIGNED, sodropis safe before the backfill runs.973b078W3.2: injection rejects at global scope —CONTENT_POLICY_INJECTION_GLOBAL_SCOPE=rejectescalates a flagged injection to a hard reject for global-scope writes.3a93fd5W2a: provenance as a first-class record — immutable, HMAC-signedmemory_provenance(migration 0011): origin channel/kind, taint set, admitting policy verdict + version, producing run/interaction, derivation depth. One choke point (MemoryRepository.add), all write channels wired, source-derived sweep test. Promotions are events (SCOPE_CHANGED/TRUST_CHANGED/PROMOTED/DEDUPLICATED). NewGET /memories/{id}/provenance.7c66932W2c: trust-weighted ranking (keystone) —server/ranking.pyfuses similarity with content trust, effectiveness votes, decay, and provenance depth, behindENABLE_TRUST_WEIGHTED_RANKING(default off, so ordering is unchanged until opted in).relevance_scoreis now the real ranking score. Closes the ACE loop: a vote actually moves retrieval order. Fixes the decay-collapse and the hybrid-search unvoted-zeroing bugs.Testing
Full suite green (~835 passed, 2 skipped, 1 xfailed) on Postgres. Security-relevant commits (0, 2, 3) carry adversarial tests validated to fail against the pre-fix commit in a scratch worktree, then pass. Migrations 0010 and 0011 round-trip.
🤖 Generated with Claude Code