Epic: Memory quality, temporal truth, and lifecycle authority

[dcellison]

Outcome

Make Kai's semantic memory trustworthy over time. Facts should be worth retaining, traceable to the exact extraction decision that created them, resistant to duplication and fragmentation, and explicitly superseded or retracted when reality changes. Episodes should remain accurate historical records while later evidence can revise their conclusions without rewriting history.

Why this precedes other work

Memory is injected into future agent context. Low-value, duplicated, or stale memories therefore affect every agent and adapter. The current extractor runs after every successful conversational run, uses inexpensive role models by default, sees only a bounded semantic candidate set, and has no first-class active/superseded/retracted/expired fact lifecycle. Memory reflection #518 must not promote this evidence until its quality and temporal authority are trustworthy.

Product and authority principles

• Precision is more important than maximizing extracted-memory count.
• Model changes are selected through replayable evaluation against operator-reviewed production examples, not intuition.
• Every extraction decision records the actual backend, provider, model, prompt version, input provenance, candidate set, and outcome without copying private conversation text into diagnostics.
• A fact that describes current reality is different from an episode that records historical reality.
• Supersession preserves evidence and revision history; it does not silently delete the earlier record.
• Normal retrieval and agent context use the current active truth projection by default.
• Conflicts, ambiguous scope, and incomplete lifecycle state fail closed.
• Existing memories are audited through a read-only report before any bulk mutation.
• Principal and project scopes remain isolated throughout extraction, lifecycle mutation, retrieval, and remediation.

Delivery order

The native sub-epics are authoritative and sequential:

1. Evidence-backed extraction quality and model selection.
2. Temporal fact and episode lifecycle authority.

Completion criteria

• A reviewed corpus establishes measurable baselines for usefulness, factual precision, redundancy, fragmentation, scope, and episode quality.
• Fact and episode models are chosen using reproducible evidence and can be changed independently without altering a user's conversational model.
• Extraction cadence and candidate retrieval are deliberate, bounded, and observable.
• Facts have canonical active/superseded/retracted/expired lifecycle semantics with durable evidence and revision history.
• Episodes remain immutable historical events and can link to later follow-up evidence or superseded conclusions.
• Stale and contradictory records cannot silently enter ordinary retrieval or agent context.
• Existing memories can be reviewed and reconciled safely without unreviewed destructive mutation.
• Installation diagnostics and installed multi-principal qualification prove model provenance, isolation, replay safety, restart continuity, and current-truth retrieval.

Non-goals

• Automatic memory reflection or self-modification; Epic: Memory reflection and self-improvement #518 remains downstream.
• Treating every conversational statement as a fact.
• Replacing principal preferences with inferred facts.
• Rewriting historical episodes to match current conditions.
• Changing a principal's selected conversational backend or model.

[lyr-ai]

I maintain TypedMem, a Python memory library, and ran into almost exactly this design question. Your principles match what we converged on almost line for line
("supersession preserves evidence", "conflicts fail closed", "current
active truth projection by default"). One detail that made "fail closed"
precise for us, in case it's useful for sub-epic 2:

• A later fact supersedes only if it starts later and its source is
  at least as authoritative as the one it replaces.
• A fact that starts earlier (a late-arriving record about the past)
  goes into history, even if it was extracted last.
• Everything else (same start, or a weaker source starting later against
  an open-ended stronger fact) is a conflict: kept, cross-linked, and
  excluded from the current-truth projection until stronger evidence
  arrives. We specifically don't let a later start close an earlier
  open-ended fact on a weaker source's word.

We wrote it down as an invariant (a conflict set only contains values that
are still simultaneously plausible) and measured it against an existing
benchmark before trusting it: history questions went from 0/11 to 11/11,
and it scored lower where the benchmark assumed per-type vetoes. In TypedMem it's
set/get/history over named states, which may or may not fit Kai's
extraction model:
https://github.com/lyr-ai/typedmem/blob/main/design/0002-changing-facts.md

How are you planning to represent "retracted" relative to "superseded":
as a lifecycle state on the fact, or as a separate evidence record?

[dcellison]

Thanks for writing this up, and for the numbers. It's good to hear the same principles held up against a benchmark.

To answer your question: both are lifecycle states on the fact. Each fact is a claim with a chain of revisions, and every revision has one state: active, superseded, retracted, expired, or unresolved_conflict. Each change is an immutable event carrying its reason and evidence, so history is the event log itself. The difference is only in what happens to current truth:

• Superseded means a newer revision of the same claim replaced it. The claim still has a current value.
• Retracted means the claim no longer has a current value, with no replacement (someone asked Kai to forget it). Expired is the same, except it records that the fact stopped being true rather than that it was wrong or unwanted.

Nothing is deleted. A retracted or expired fact can be restored later, which adds a new active revision pointing back at the old one rather than reviving it. Only active revisions reach recall. The other states stay in the owner's history view.

Our supersession rule is based on source and confidence rather than on when a fact starts. A person's edit supersedes directly. An extracted update supersedes only if it's confident (0.9 or higher) and cites evidence. Otherwise it opens a conflict: both revisions are kept and marked unresolved_conflict, neither is recalled, and the owner picks one. Conflicts are settled by the owner today, not by stronger evidence arriving later.

Your valid-time rule covers a case we don't handle yet. A late-arriving record about the past currently gets the time it was extracted, so it can't be placed in history on its own. We keep "what happened" in a separate episode store, where later evidence links to an episode instead of rewriting it, but facts don't get that treatment. "Starts earlier goes to history" and "a weaker source can't close an open-ended stronger fact" are precise enough to be worth building and measuring against our quality corpus once it exists, so I've filed #1790 to do that, crediting your comment. Thanks for the link to the design note.

[lyr-ai]

Thanks — glad the valid-time distinction was useful. I have a small set of edge cases/tests around that rule, including late-arriving historical values, same-time conflicts, and weaker later sources overlapping stronger open-ended claims. Happy to share those if useful when you run this against the quality corpus.