Skip to content

Commit e334a7d

Browse files
NetDevAutomateNetDevAutomate
committed
feat(learning-memory): Stage B — canonical event store with provenance invariants
New package packages/learning-memory implementing ADR-0011's store: typed events with UNIQUE(session_id, content_hash); evidence written in the same transaction (OBSERVED native bytes or REPORTED prose for the archive); lineage; prose-only FTS5 (porter unicode61 default, unicode61 alternative, refused on mismatch); claims with a code-point bound-proof trigger on insert AND update, immutability, supersedes; claim_relations; review_items; exchanges/concept_tags/recurrence tables ready for the Stage D derivation pass. 67 tests, hypothesis-backed, exercise the invariants adversarially: re-ingest is a no-op over many passes; a session cannot exist without evidence; altered body, stale offsets, cross-session evidence, grapheme-splitting offsets, ambiguous repeats, empty and zero-width quotes are all rejected with no partial write; UPDATE on claims raises; tool text never reaches the prose index. Accepted deviations from the ADR text (now recorded in the ADR): deferred lineage for not-yet-ingested parents; position-free content hashes; native-byte body hashes; zero-width and rebind hardening; explicit dedupe instead of INSERT OR IGNORE, which had been swallowing CHECK violations. Verified independently by the orchestrator: 67 passed, ruff clean, pyright clean (after restoring the worktree env with `uv sync --all-packages` — the package-local sync had dropped studyloop from it), lane confined to the package plus uv.lock (root members glob already includes packages/*).
1 parent 753dca1 commit e334a7d

16 files changed

Lines changed: 2731 additions & 0 deletions

‎docs/adr/0011-claim-centric-learning-memory.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,21 @@ alongside recall@5 on every receipt. Every arm keeps the same session ids as `se
106106
- Retention becomes an explicit contract: harnesses rotate transcripts within weeks, so the sweep
107107
cadence and a doctor check on "age of last capture" are load-bearing.
108108

109+
## Implementation notes accepted from Stage B (2026-09-10)
110+
111+
- `lineage` edges whose parent is not yet ingested are **deferred** (`IngestResult.lineage_deferred`)
112+
and land on the child's re-ingest; a stub parent would violate the no-session-without-evidence
113+
invariant.
114+
- `content_hash` covers text and kind, **not** position, so re-import is a no-op and exact
115+
duplicates collapse — the two properties the archive's 6,591 duplicates require together.
116+
- `body_sha256` is taken over native bytes for `OBSERVED` evidence and over the UTF-8 prose for
117+
`REPORTED`; the row is a capture receipt of what was actually read.
118+
- Hardening beyond the ADR text: `claim_citations` CHECKs `length(quote) > 0` and `end > start`
119+
(a zero-width extent would bind vacuously), and a BEFORE UPDATE twin of the bound-proof trigger
120+
so citations cannot be rebound after the fact.
121+
- `INSERT OR IGNORE` was rejected for events because it swallows CHECK and FK violations; the
122+
dedupe conflict is handled explicitly and any other violation fails the whole ingest.
123+
109124
## Open questions (to be settled by measurement, not debate)
110125

111126
Tokenizer for `prose_fts`/claims (porter vs unicode61); whether embeddings on claims clear G4;

‎packages/learning-memory/README.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# learning-memory
2+
3+
The ADR-0011 PoC store: **capture is lossless and dumb; usefulness is derived at
4+
capture time and bound to provenance the database itself can prove.**
5+
6+
Own SQLite file, stdlib only. The live `~/.config/studyloop/sessions.db` is never
7+
opened by this package.
8+
9+
## What is in here
10+
11+
| Module | What it owns |
12+
| --- | --- |
13+
| `model.py` | `Session`, `Event`, `ParsedSession`, `SourceRef`, the `HarnessAdapter` protocol, and the event content hash |
14+
| `schema.py` | the whole DDL, including the two triggers that make the invariants non-negotiable |
15+
| `store.py` | `Store.connect` / `install` / `ingest` / `add_claim` / `visible_evidence` |
16+
17+
```python
18+
from learning_memory import Event, ParsedSession, Session, Store
19+
20+
store = Store.connect("poc.db") # foreign_keys=ON, WAL
21+
store.install()
22+
23+
store.ingest(
24+
ParsedSession(
25+
session=Session(id="kiro-2026-09-10-1", harness="kiro", project="studyloop"),
26+
events=[Event(turn_id=0, seq=0, kind="user", text="why did the gate fail?")],
27+
native_source=raw_transcript_bytes, # OBSERVED: we still hold the original
28+
)
29+
)
30+
31+
claim = store.add_claim(
32+
"kiro-2026-09-10-1",
33+
"Finding",
34+
"Recall was the failing layer",
35+
"The keyword path scored 0.107 macro recall@5 on gold v2.",
36+
("retrieval", "gold-v2"),
37+
0.9,
38+
"distiller/model-pass",
39+
citations=[{"evidence_id": ..., "quote": "0.107 macro recall@5"}],
40+
)
41+
```
42+
43+
## The four invariants (all property-tested)
44+
45+
1. **Re-import is a no-op.** Events are content-addressed over
46+
`(kind, actor, tool_name, text)` with `UNIQUE(session_id, content_hash)`, so
47+
running the export sweep twice — or twice over overlapping windows — adds no
48+
rows, and a message repeated inside one transcript collapses to one row.
49+
`IngestResult` reports what was skipped.
50+
2. **No session row without at least one evidence row.** `OBSERVED` needs the
51+
harness's native bytes; `REPORTED` (the archive path, where the original has
52+
been rotated away) synthesises evidence from the prose already held, labelled
53+
`origin='archive'`. If neither exists the whole transaction rolls back, so a
54+
refused ingest leaves nothing behind.
55+
3. **A claim cannot exist with a citation that does not bind.** `add_claim`
56+
resolves each quote to **code-point** offsets with `str.find`, refusing quotes
57+
that are missing, empty, or ambiguous (more than one occurrence, so the offsets
58+
would be a guess) — and then `claim_citation_bound_proof` re-proves it in SQL:
59+
`substr(evidence.body, start+1, end-start) = quote`. Byte or UTF-16 offset
60+
arithmetic desynchronises on any astral character and is refused. Claim plus
61+
citations are one transaction: a good citation never survives a bad sibling.
62+
4. **Claims never change.** `claims_immutable` aborts every `UPDATE`. A correction
63+
is a new claim whose `supersedes` names the old one. Claim ids are content
64+
addresses, so a re-run cannot fork the same assertion.
65+
66+
Plus: **`prose_fts` indexes prose only.** It is an FTS5 external-content index over
67+
`events` whose only writers are triggers gated on `kind IN ('user',
68+
'assistant_prose')`, so tool output is stored but never searchable. The tokenizer is
69+
a constructor parameter (`porter unicode61` default, `unicode61` the alternative)
70+
because ADR-0011 leaves that choice to measurement; a store records which one built
71+
it and refuses to be reopened under the other.
72+
73+
## Running the gates
74+
75+
From this directory:
76+
77+
```bash
78+
uv sync
79+
uv run --group dev pytest -q
80+
uv run ruff check
81+
uv run ruff format --check
82+
uv run --group dev pyright src
83+
```
84+
85+
## Not implemented here (deliberately)
86+
87+
The derivation pass — exchanges, concept tags, recurrence, review items — has its
88+
tables and constraints in `schema.py` but no writer yet: ADR-0011 puts it in the
89+
export sweep, which is a later stage. Same for the adapters: `HarnessAdapter` is
90+
the contract they will satisfy, and the shared base owning dedupe, evidence and
91+
lineage is `Store.ingest`.

‎packages/learning-memory/pyproject.toml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
[project]
2+
name = "learning-memory"
3+
version = "0.1.0"
4+
description = "ADR-0011 PoC: claim-centric learning memory — typed events, native evidence, quote-bound claims"
5+
authors = [
6+
{name = "Andy Taylor"}
7+
]
8+
requires-python = ">=3.12"
9+
readme = "README.md"
10+
license = "MIT"
11+
keywords = ["sqlite", "fts5", "provenance", "claims", "sessions"]
12+
classifiers = [
13+
"Development Status :: 3 - Alpha",
14+
"Intended Audience :: Developers",
15+
"License :: OSI Approved :: MIT License",
16+
"Programming Language :: Python :: 3",
17+
"Programming Language :: Python :: 3.12",
18+
"Programming Language :: Python :: 3.13",
19+
"Topic :: Database",
20+
"Typing :: Typed",
21+
]
22+
# The store is stdlib-only on purpose: it is the layer every gate in ADR-0011's
23+
# evaluation binding runs through, so it must not be able to fail for a reason
24+
# that lives in someone else's release.
25+
dependencies = []
26+
27+
[build-system]
28+
requires = ["hatchling"]
29+
build-backend = "hatchling.build"
30+
31+
[tool.hatch.build.targets.wheel]
32+
packages = ["src/learning_memory"]
33+
34+
[tool.hatch.build.targets.sdist]
35+
include = [
36+
"/src",
37+
"/tests",
38+
"/README.md",
39+
]
40+
41+
[dependency-groups]
42+
dev = [
43+
"pytest>=8.0",
44+
"hypothesis>=6.100",
45+
"ruff>=0.8",
46+
"pyright>=1.1",
47+
]
48+
49+
[tool.ruff]
50+
target-version = "py312"
51+
line-length = 100
52+
53+
[tool.ruff.lint]
54+
select = ["E", "F", "W", "I", "N", "UP", "B", "A", "C4", "SIM", "TCH", "RUF"]
55+
56+
[tool.ruff.lint.isort]
57+
known-first-party = ["learning_memory"]
58+
59+
[tool.pyright]
60+
include = ["src", "tests"]
61+
exclude = ["**/__pycache__"]
62+
pythonVersion = "3.12"
63+
# Stricter than the workspace root's "basic": this package's whole value is that
64+
# its invariants hold, and an unannotated function is an invariant nobody checked.
65+
typeCheckingMode = "standard"
66+
reportMissingImports = true
67+
reportMissingTypeStubs = false
68+
reportUnknownParameterType = "error"
69+
reportMissingParameterType = "error"
70+
reportUntypedFunctionDecorator = "error"
71+
reportImplicitStringConcatenation = "none"
72+
extraPaths = ["src"]
73+
74+
[tool.pytest.ini_options]
75+
testpaths = ["tests"]
76+
pythonpath = ["src"]
77+
# Duplicated rather than inherited: pytest picks its configfile from the rootdir
78+
# it derives from the arguments, so a package-scoped run never reads the
79+
# workspace-root settings (same reasoning as packages/agent-session-tools).
80+
addopts = "--tb=short"

‎packages/learning-memory/src/learning_memory/__init__.py‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
"""Claim-centric learning memory (ADR-0011 PoC).
2+
3+
Capture is lossless and typed; usefulness is derived at capture time and bound to
4+
provenance the database itself can prove.
5+
"""
6+
7+
from __future__ import annotations
8+
9+
from learning_memory.model import (
10+
CLAIM_KINDS,
11+
EVENT_KINDS,
12+
PROSE_KINDS,
13+
ClaimKind,
14+
ClaimRelationKind,
15+
ConceptSource,
16+
Event,
17+
EventKind,
18+
EvidenceBasis,
19+
HarnessAdapter,
20+
ParsedSession,
21+
ReviewItemKind,
22+
Session,
23+
SourceRef,
24+
event_content_hash,
25+
)
26+
from learning_memory.schema import (
27+
DEFAULT_TOKENIZER,
28+
PRAGMAS,
29+
SCHEMA_VERSION,
30+
TOKENIZERS,
31+
Tokenizer,
32+
ddl,
33+
)
34+
from learning_memory.store import (
35+
CitationError,
36+
CitationProblem,
37+
ClaimValidationError,
38+
DuplicateClaimError,
39+
IngestResult,
40+
LearningMemoryError,
41+
NoEvidenceError,
42+
SchemaError,
43+
Store,
44+
claim_id,
45+
evidence_id,
46+
)
47+
48+
__version__ = "0.1.0"
49+
50+
__all__ = [
51+
"CLAIM_KINDS",
52+
"DEFAULT_TOKENIZER",
53+
"EVENT_KINDS",
54+
"PRAGMAS",
55+
"PROSE_KINDS",
56+
"SCHEMA_VERSION",
57+
"TOKENIZERS",
58+
"CitationError",
59+
"CitationProblem",
60+
"ClaimKind",
61+
"ClaimRelationKind",
62+
"ClaimValidationError",
63+
"ConceptSource",
64+
"DuplicateClaimError",
65+
"Event",
66+
"EventKind",
67+
"EvidenceBasis",
68+
"HarnessAdapter",
69+
"IngestResult",
70+
"LearningMemoryError",
71+
"NoEvidenceError",
72+
"ParsedSession",
73+
"ReviewItemKind",
74+
"SchemaError",
75+
"Session",
76+
"SourceRef",
77+
"Store",
78+
"Tokenizer",
79+
"__version__",
80+
"claim_id",
81+
"ddl",
82+
"event_content_hash",
83+
"evidence_id",
84+
]

0 commit comments

Comments
 (0)