Build an operator-reviewed production memory quality corpus and scoring rubric

[dcellison]

Outcome

Create an operator-reviewed, reproducible corpus that measures whether Kai's generated facts and episodes are genuinely useful and accurate.

Requirements

• Sample authorized production exchanges and their extraction outputs without placing private corpus material in the public repository.
• Define labels for useful, incorrect, unsupported, transient, stale-on-arrival, redundant, fragmented, wrongly scoped, wrong-speaker, missed update, and episode false positive/false negative.
• Separate fact extraction, episode classification, episode generation, consolidation intent, and scope quality.
• Include changed-value, negation, preference reversal, renamed resource, completed workflow, routine acknowledgment, multi-agent, and project/global-scope cases.
• Preserve immutable corpus inputs and reviewer decisions with provenance.
• Support blind comparison of model outputs.

Acceptance

• Baseline precision, useful-memory rate, duplication rate, fragmentation rate, update-detection rate, and episode quality are reported for the current production pipeline.
• Corpus storage remains private and principal-authorized.
• The corpus is large and varied enough to prevent selecting a model from anecdotal examples.

[dcellison]

Reopened during epic reconciliation because the implementation is installed, but this issue's operational acceptance criteria have not yet been met.

Installed diagnostics currently report baselines=0 and qualified=0. The private corpus tooling and scoring rubric exist, but no operator-reviewed production baseline has been completed. This issue remains open until a sufficiently varied private corpus is reviewed and the required baseline precision, usefulness, duplication, fragmentation, update-detection, and episode-quality metrics are recorded.

[dcellison]

Current installed readiness evidence (2026-09-21):

• receipt-backed cases available: 10
• fact-extraction receipts: 10
• episode-generation receipts: 0
• stored/replaced outputs: 0
• reviewed baselines: 0
• qualified baselines: 0

The installed scorer requires at least 30 reviewed cases, both extraction roles, all eleven required scenario tags, at least one reviewed output, and measurable update, episode, scope, and consolidation decisions before qualification_ready can be true.

An immutable snapshot and review made now would necessarily remain unqualified and could not satisfy this issue's acceptance criteria. The next prerequisite is to accumulate enough genuine receipt-backed production exchanges—especially episode-worthy, update/refinement, scope, consolidation, and other required scenario cases—before creating the immutable snapshot. Review labels must then be supplied by a human reviewer rather than inferred or fabricated automatically.

[dcellison]

Readiness note: stored and replaced outputs cannot become nonzero on the current code. The extraction fact path emits a lifecycle evidence kind that validation rejects, so the first extraction that reaches storage will fail (#1765). Accumulating more traffic won't unblock this issue until #1765 lands.