diff --git a/docs/coding-agent/lessons.md b/docs/coding-agent/lessons.md index 4075d068..51662500 100644 --- a/docs/coding-agent/lessons.md +++ b/docs/coding-agent/lessons.md @@ -454,3 +454,69 @@ Prevention: ## Purge note (2026-07-23) Eleven entries purged per the user-directed low-value/invalid sweep (Codex purge map, agmsg 2026-07-23T12:28Z): ten PURGE-LOW-VALUE (restatements of now-mandatory harness/rule content — plan-format task records, PR monitoring, canonical-byte verification, compatibility policy, module layout, evidenced-scope rulebook default, parallel dispatch — plus two cheaply rediscovered one-off quirks and one unstructured batch-notes bundle) and one PURGE-INVALID (the phase-bounded v0.1 compatibility ruling, superseded by the repo-wide Compatibility Policy). Full entries recoverable from git history at 4997bdc. + +## 2026-09-03 — Reconcile The Companion Pin Before Filing Cross-Repository Breakage [tags: review, scope, cross-repo, assumptions] + +Context: +- Plan: `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` +- Task/Wave: Task_3 / Wave 2 +- Roles involved: Reviewer | Orchestrator + +Symptom: +- The reviewer filed a preliminary HIGH because a new variant in the closed error vocabulary would not compile in the evaluation repository's exhaustive match once that repository re-pins. + +Root cause: +- The finding treated the companion repository's current main as if it were already pinned to the change under review, although the companion stays pinned at an older library commit until its own migration plan runs; ADR-I-0023's impact section names that conversion as the companion's re-pin prerequisite. + +Fix applied: +- The orchestrator ruled the item out of scope; the reviewer recorded it as the already-tracked re-pin obligation and completed the review on the in-PR acceptance bullets. + +Prevention: +- Before filing cross-repository breakage, reconcile the companion's pin and the plan that owns its migration; when the companion intentionally remains pinned until its own migration, record an obligation, not an in-PR defect. + +Evidence: +- Reviewer messages of 2026-09-03 (preliminary HIGH, ruling acknowledgement, final REVIEW3_DONE) on Task_3 at 773d65e. + +## 2026-09-04 — PR Watchers Must Not Depend On Tools Absent From The Monitor Shell [tags: workflow, monitoring, tooling, orchestrator] + +Context: +- Plan: `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` +- Task/Wave: Wave 1–2 PR monitoring +- Roles involved: Orchestrator + +Symptom: +- Copilot reviews on the wave PRs and on the evaluation repository's PR arrived without any watcher notification; the decider noticed the review before the orchestrator did. + +Root cause: +- The watcher scripts piped GitHub API output through `jq`, which is not on the PATH of the background monitor shell; every poll failed on stderr, which the monitor does not surface, so the watchers stayed silent while appearing armed. + +Fix applied: +- Watchers re-armed using only `gh api --jq` (no external `jq`), one loop covering the whole stack, printing on any change in review count, review-comment count, or merge state. + +Prevention: +- A watcher script uses only tools proven available in the monitor shell (`gh --jq`, POSIX sh); before trusting a new watcher, read its output file once to confirm it produced a first sample rather than errors. + +Evidence: +- Monitor output file for the PR #77 watcher on 2026-09-04: eighteen consecutive `jq: command not found` lines and no events, while Copilot's "approval recommended" review was already posted. + +## 2026-09-04 — Edit Only Through Absolute Worktree Paths, And Run Broad Suites Only After The Live Window Is Granted [tags: workflow, worktrees, live-mutex, worker] + +Context: +- Plan: `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` +- Task/Wave: Task_4 / Wave 3 +- Roles involved: Worker | Orchestrator + +Symptom: +- A relative-path patch briefly modified files in the shared main checkout instead of the task worktree before being reverted; and a full test suite started before the exclusive service window was confirmed, so service-backed tests could have collided with another agent's run. + +Root cause: +- Relative paths resolve against whatever the current directory happens to be, and several checkouts of the same repository share identical relative paths; the live-window protocol grants exclusivity only on the explicit `WINDOW_YOURS` reply, not on sending `LIVE_START`. + +Fix applied: +- The stray edits were reverted and verified clean; the suite was re-run inside the granted window. + +Prevention: +- Every edit and every git command names an absolute path inside the task worktree, and the current directory is checked before each edit; a suite that may reach a shared service starts only after `WINDOW_YOURS` is observed, otherwise run the service-free command. + +Evidence: +- Worker Task_4 report of 2026-09-03 (commit d232830) and the orchestrator's clean `git status` check on the main checkout. diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md new file mode 100644 index 00000000..a6e5df1c --- /dev/null +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -0,0 +1,270 @@ +# Plan: v0.1.6 Embedded Vector Candidate Recall + +- status: approved +- generated: 2026-09-02 +- last_updated: 2026-09-03 +- work_type: mixed + +## Goal +- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0027: a redesigned vector port contract, a five-field vector record, an embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) as the default vector mode with the service adapter retained as the explicit service mode, and a shared contract suite over both adapters; the evaluation repository's move of its vector-only baseline onto the retrieval trace is planned there and consumed here as closeout evidence. + +## Definition of Done +- Every acceptance criterion in the phase document's "Acceptance criteria" section holds with recorded evidence. +- Every row of the phase document's deferral-reconfirmation checklist has its evidence produced and cited in the Progress Log. +- Every deletion listed under "Deletions that are deliverables" is gone, with a zero-hit census. +- Both repositories' service-gated suites execute (not skip) under the service-backed CI job. +- One PR per wave in this repository, merged by the decider; the evaluation repository's cross-mode comparison is available as consumed evidence at closeout. + +## Scope / Non-goals +- Scope: the phase document's deliverables and deletions, all in this repository. +- Non-goals: the phase document's non-goals (no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode for non-empty scopes and non-degenerate queries; ADR-I-0024's empty-scope change, zero candidates for an empty scope and boundary rejection of an empty configured scope, is an intended change and in scope). + +## Context (workspace) +- Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. +- As-built port: `src/ports/vector_candidate.rs`, `src/models/vector/candidate_record.rs`, `src/models/vector/record.rs`, `src/adapters/qdrant/{store,payload}.rs`, `src/policy/embedding_surface.rs`, `src/usecases/retrieve.rs`, `src/api/types/retrieval.rs`, `src/composition.rs`, `src/config/app_settings.rs`, `src/test_support.rs`. +- Prerequisite in this repository, landed: the toolchain pin moved to the embedded engine's minimum (Rust 1.97.0) in its own change, merged 2026-09-02 as a88c117. +- Prerequisite tracked in the evaluation repository: its evidence-integrity fixes must be merged before this phase cites any harness measurement. +- Repo reference docs consulted: the five ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. + +## Open Questions (max 3) +- none (the draft's five open questions were ruled by the decider on 2026-09-02 and are recorded in the phase document and the ADRs). + +## Assumptions +- A1: The embedded engine is `qdrant-edge` pinned exactly at 0.8.0 (beta); its API is guarded by a contract canary, and the pin is bumped only with a re-run of the canary and the parity suite. +- A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison consumed as closeout evidence and as a revisit trigger for the embedded default. + +## Tasks + +### Task_1: Live-gate hardening in the library test suite +- type: test +- owns: + - tests/support/base.rs + - tests/write_planning_tests.rs + - tests/initialization_tests.rs + - tests/public_facade_tests.rs + - tests/retrieval_guardrails_tests.rs + - .github/workflows/*.yaml +- depends_on: [] +- description: | + Add one environment switch honored by the shared test support that turns every service-unavailable skip into a panic, set it in the CI job that provisions the vector service, and delete the prose-matched timeout skip (`is_qdrant_timeout_signature`) or replace it with a typed match on the existing transport classification. +- acceptance: + - No test passes by skipping when the switch is set; the CI service-backed job sets it. + - No test gates on error prose. +- validation: + - kind: command + required: true + owner: worker + detail: "service-up cargo test with the switch set: every former skip site executes (census the skip sites before and after; the count is not fixed); service-down with the switch set: the suites fail, not pass" + - kind: review + required: true + owner: reviewer + detail: "Diff review; confirm the CI job sets the switch" + +### Task_2: Port contract: completeness envelope and scope-only query (ADR-I-0024) +- type: impl +- owns: + - src/ports/vector_candidate.rs + - src/models/vector/candidate_record.rs + - src/api/types/retrieval.rs + - src/adapters/qdrant.rs + - src/adapters/qdrant/store.rs + - src/adapters/qdrant/tie_closure.rs + - src/usecases/retrieve.rs + - src/usecases/remember.rs + - src/usecases/correct_forget.rs + - src/memory.rs + - src/test_support.rs + - src/adapters/oxigraph/tests.rs +- depends_on: [] +- description: | + Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; extract the service adapter's private tie-closure loop (fetch decision, fetch bound, cohort closure, canonical construction) into `src/adapters/qdrant/tie_closure.rs` as crate-visible shared logic that takes an engine-neutral fetch callback, so the embedded adapter (Task_4) calls it rather than re-implementing it; make the service adapter map the shared fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. +- acceptance: + - The envelope and verdict express the four situations in ADR-I-0024's Decision section (its appendix shape is a non-binding reference); the canonical-candidates newtype is unchanged. + - Query-side zero-norm rule (ADR-I-0024) implemented in the service adapter: a zero-norm query scores every candidate zero and returns a truthful verdict, with a unit test and a parity fixture that Task_4 inherits. + - Telemetry carries the verdict for every retrieval; a retrieval test asserts each variant. + - The tie-closure loop lives in `src/adapters/qdrant/tie_closure.rs`, the service adapter calls it, and its existing unit tests (fetch decision, all-tied cohort at the bound) move with it; nothing in `store.rs` closes a cohort on its own. + - Fetch-decision unit tests assert closed and open verdicts including the all-tied cohort at the bound. + - Zero-hit census: no match-or-unknown condition, no filter type beyond object-type scope. +- validation: + - kind: command + required: true + owner: worker + detail: "cargo fmt --check; cargo clippy --all-targets -- -D warnings; service-up cargo test; ignored qdrant_ lib tests" + - kind: review + required: true + owner: reviewer + detail: "Diff review vs ADR-I-0024; confirm no pipeline path inspects the verdict for control flow" + +### Task_3: Vector record read contract (ADR-I-0025) +- type: impl +- owns: + - src/models/vector/record.rs + - src/adapters/qdrant/payload.rs + - src/adapters/qdrant/store.rs + - src/policy/embedding_surface.rs + - src/domain.rs + - src/usecases/retrieve.rs + - src/usecases/vector_indexing.rs + - src/errors.rs + - src/models/vector.rs + - src/models/vector/candidate_record.rs + - src/api/types/retrieval.rs + - src/api/** + - src/lib.rs + - docs/design/database/vector_payload_design.md + - docs/design/database/schema_cheat_sheet.md + - docs/design/database/README.md + - docs/design/database/graph_schema_design.md +- depends_on: [Task_2] +- description: | + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Consolidating the surface enum to one domain definition touches its internal definition and re-export and the public copy, all owned here. Publish the maximum number of embedding surfaces per object kind as public policy next to the surface policy that defines it (ADR-I-0026), with a test that the published value matches the builders. Update the database documentation that advertises the old record: the schema cheat sheet, the database README, and the graph schema design note's cross-store section lose the hint fields, the graph URI, the lifecycle-hint drift diagnostics, and the readable text column, and point at the five-field contract. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. +- acceptance: + - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. + - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository removes its reader under its own plan; its zero-hit census is consumed as closeout evidence, not ordered here). + - One token mapping per enum; census shows no copy in adapters or use cases. + - A zero-norm record embedding yields a typed per-record indexing failure and never reaches an adapter (ADR-I-0024); unit test present. +- validation: + - kind: command + required: true + owner: worker + detail: "cargo fmt --check; cargo clippy --all-targets -- -D warnings; service-up cargo test; ignored qdrant_ lib tests; census commands recorded" + - kind: review + required: true + owner: reviewer + detail: "Diff review vs ADR-I-0025; verify the payload design note's supersession note matches what landed" + +### Task_4: Embedded Qdrant Edge vector candidate store, settings, and parity suite (ADR-I-0023) +- type: impl +- owns: + - src/adapters/qdrant_edge/** + - src/adapters.rs + - src/composition.rs + - src/config/app_settings.rs + - src/errors.rs + - Cargo.toml + - Cargo.lock + - tests/vector_port_contract_tests.rs + - tests/support/** + - .env.example + - README.md + - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +- depends_on: [Task_3] +- description: | + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the shared tie-closure loop Task_2 extracts into `src/adapters/qdrant/tie_closure.rs` and the canonical constructor, with the verdict mapping (Exhaustive when the shard is unindexed per the adapter's own threshold configuration and the loop closed the cutoff cohort, with `scanned` from a filtered count of the scope, never from returned rows; BoundaryTieClosed when an indexed shard's cutoff cohort closed, describing the index's returned prefix and never global recall; BoundaryTieOpen whenever the bound is reached with the cohort open, on an exact scan too; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). +- acceptance: + - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. + - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). + - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. + - ADR-I-0027 holds: every engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count, index build, shutdown) runs on the adapter's dedicated blocking owner, which opens the shard itself; a write is acknowledged only after the engine's flush; the facade drop only signals the owner; a constructor meeting a locked directory waits with a bounded backoff; the close-then-reopen test, the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write), and the responsiveness benchmark (construction and reopen with a lock-backoff wait, a concurrent scan, a write burst, a build, a close) pass; the contract canary pins the three engine facts ADR-I-0027 rests on: no persistence before flush (a probe that writes, skips flush and drop, and reopens empty), no log replay on load, and the directory lock. + - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. + - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. + - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. + - The dependency-weight report records unstripped and stripped release deltas and the result of feature trimming; the latency guidance is measured and documented with the single-process expectation and the rebuild-from-graph-authority path. +- validation: + - kind: command + required: true + owner: worker + detail: "cargo test with no service (parity suite + embedded tests + canary execute); service-up cargo test with the live switch set; benchmark and weight numbers recorded in the report" + - kind: review + required: true + owner: reviewer + detail: "Diff review vs ADR-I-0023; independent service-free and service-up runs; tie-closure reuse verified by reading, not by test names" + +### Task_7: Fake retirement, closeout docs, and reconfirmation evidence +- type: chore +- owns: + - src/test_support.rs + - src/**/tests (fake stores only) + - docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md + - docs/coding-agent/lessons.md + - docs/roadmap/development_roadmap.md + - Cargo.toml + - Cargo.lock +- depends_on: [Task_4] +- description: | + Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened on a temporary shard directory, as the phase document specifies, so tests exercise the persistence path (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; bump the package version to 0.1.6 in the manifest and lockfile as prior milestone closeouts did; mark the roadmap row finished; move the plan to completed. +- acceptance: + - Zero-hit census for the retired fake and record type. + - All five checklist rows cite evidence in the Progress Log. +- validation: + - kind: command + required: true + owner: worker + detail: "full cargo test with no service and service-up with the live switch set" + - kind: review + required: true + owner: reviewer + detail: "Closeout review; Definition of Done census" + +### Task_8: Design-value audit at the pre-merge milestone gate +- type: review +- owns: + - docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +- depends_on: [Task_4] +- description: | + Altitude review (Claude) against philosophy and roadmap: nothing designed twice across the two adapters, no hint field re-entered without its predicate, the evaluation repository holds no store-private knowledge, the ADR boundaries respected. +- acceptance: + - Audit verdict recorded in the Decision Log with any EARNS-ITS-PLACE / OVERSIZED / DELETE findings dispositioned. +- validation: + - kind: review + required: true + owner: orchestrator + detail: "Audit report consumed; dispositions recorded" + +## Task Waves (explicit parallel dispatch sets) + +- Wave 1 (parallel): [Task_1, Task_2] +- Wave 2 (parallel): [Task_3] +- Wave 3 (parallel): [Task_4] +- Wave 4 (parallel): [Task_8] +- Wave 5 (parallel): [Task_7] + +Task identifiers 5 and 6 were evaluation-repository work and moved to that repository's own plan; identifiers are not reused. + +Each wave ends with reviewer approval and a PR stacked on the previous wave's PR (GitHub stack on the planning PR); the next wave branches from the approved tip without waiting for a merge, and the decider merges the entire stack in one go at phase end (Decision Log, 2026-09-03 and 2026-09-04). +Notification duty: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository before merge; how and when that repository adopts the change is planned there, and this plan only consumes the resulting compatibility evidence at closeout. + +## Rollback / Safety +- Embedded mode is the default and the default-mode construction test asserts it; the service mode's behavior is unchanged except for the reported verdict, the shrunken record, and the intended empty-scope change (an empty object-type scope selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary), all covered by the parity suite and the retrieval tests. +- Stored service-mode payloads with dropped fields remain readable (extra fields tolerated unread); rebuild from graph authority is the recovery path. +- Each wave is a separately revertible PR. + +## Progress Log (append-only) + +Append-only editing rule (applies to both logs below): when appending an entry, anchor the edit on the previous entry and reproduce it (or anchor on the section's tail marker) so the edit inserts rather than replaces, and verify afterward that the log grew. + +- 2026-09-02 Planning wave completed: five parallel inputs (design consult, two altitude audits, two forensic censuses) consolidated; decider ruled the five design questions; ADR-I-0023 through ADR-I-0026, the rewritten phase document, and the roadmap section authored on branch `plan/v0-1-6-embedded-vector-recall`. Plan awaits approval. + +- 2026-09-04 Wave 1 done: Task_1 (a633b2e, PR #74) and Task_2 (5b30856, PR #75) approved by the Tier D reviewer; Task_2 needed one revision (checked backend-limit conversion with a BoundaryTieOpen regression at the u32 cap; public re-exports) and carries one post-review fix (dimension check before the zero-norm scroll). PRs stacked on the planning PR as stack #76; merges pending. Wave 2 (Task_3) dispatched on a branch stacked on Task_2. +- 2026-09-04 Wave 2 done: Task_3 (36ef8c8, PR #77) approved by the Tier D reviewer after two revisions on the zero-norm rejected-record fixture (final shape: one definition behind the non-default `test-fixtures` feature, enabled for integration tests by a self dev-dependency, doc-hidden). The evaluation repository's exhaustive error-vocabulary conversion is recorded as its re-pin obligation (ADR-I-0023), not a finding. PR appended to stack #76. +- 2026-09-04 — Merge shape refined: the stack (planning PR at the bottom, one PR per wave above it) is merged in one go at phase end; GitHub's bottom-up stack order blocks interim merges of wave PRs while the planning PR is a draft, which is the intended shape. Waves proceed by branching from the previous wave's approved tip. + +## Decision Log (append-only; re-plans and major discoveries) + +- 2026-09-02 Decision: the draft's port description was rewritten as an intentional new port contract. + - Trigger / new insight: the draft (2026-07-20) described filter, diagnostics, and reconciliation capabilities that the structured-verdict phase deleted; thirty of thirty-three payload fields were write-only; the readable text column's only reader was the evaluation repository's direct store access. + - Plan delta: contract-first waves (envelope and query, then record, then adapter); evaluation baseline moved to the trace; deletions promoted to deliverables. + - Tradeoffs considered: recorded in the ADRs' rejected alternatives; the forward-looking keep case for hint fields (immutable time window) is recorded as a re-entry path rather than kept. + - User approval: rulings on all five questions given 2026-09-02; plan approval pending. +- 2026-09-02 Decision: evidence-integrity defects in the evaluation repository are fixed before this phase, outside this plan. + - Trigger / new insight: batch ingest produced phantom repair attempts and the evaluated rank was harness-invented; both would corrupt the parity and baseline evidence this phase cites. + - Plan delta: none inside this plan; recorded as a prerequisite in Context. + - User approval: yes, 2026-09-02. +- 2026-09-02 Decision: the embedded vector store is the in-process edition of the service engine (Qdrant Edge), not an in-house exact scan and not a SQLite vector extension. + - Trigger / new insight: the decider restated the objective as a portable memory that plugs into any foundation model and tracks years to decades of continuous character development; the draft's in-house exact scan violated the library-over-in-house principle; two feasibility spikes on the same probe set measured the two library-backed candidates (numbers in ADR-I-0023). + - Plan delta: Task_4 rewritten for the in-process engine (shard directory per collection, threshold-based exactness, shared tie-closure loop, contract canary, dependency-weight report); the toolchain pin moved to 1.97.0 as a prerequisite; approximate indexing leaves the non-goals; named-vector coexistence and shard-to-server sync are recorded as available but not exercised this phase. + - Tradeoffs considered: recorded in ADR-I-0023's rejected alternatives (in-house scan; sqlite-vec, lighter and deterministic but exhaustive-only in its stable release with a single-maintainer approximate-index future; LanceDB; in-memory only; default flip now); a scale probe was offered and declined as not worth the effort, so the interactive-latency assumption at decade scale is a recorded revisit trigger rather than a measurement. + - User approval: yes, 2026-09-02. +- 2026-09-02 — Durability rule for the embedded adapter (review round 22). + - Finding: the engine persists a write only on flush and does not replay its log on load; process-level probes that skipped the shard drop reopened with zero of two hundred points. + - Ruling: the blocking owner flushes after every write and acknowledges only then; the signal-only facade drop stays; no port or facade method is added; a hard-exit test joins Task_4's acceptance. + - Tradeoff: one synchronous disk sync per write, measured by the write burst in the benchmark; batching behind an acknowledgement is the named upgrade. +- 2026-09-03 — Decision records restructured on the decider's review. + - ADR-I-0023 now decides the default: embedded is the default vector mode from this phase, licensed by the phase's own parity suite and service-free integration path (defaults-match-evidence, ADR-I-0021); ADR-I-0003's vector default is partially superseded; the evaluation repository's cross-mode run becomes a revisit trigger, not a gate. Task_4's default-mode test asserts embedded and its README and phase-document deliverables lead with the local path. + - The blocking-owner and per-write-flush rules moved out of ADR-I-0023 into ADR-I-0027 as their own decision. + - ADR-I-0024 narrowed to the completeness verdict and the prefilter rule (unknown never matches); the scope-only query is a current state, the type shape is a non-binding appendix, and the two predicate paths are notes binding on no phase. +- 2026-09-03 — Merge shape: the records and plan are not merged ahead of the implementation, because ADR-I-0023's embedded default rests on evidence the implementation produces. Implementation waves land as PRs against the planning branch (`plan/v0-1-6-embedded-vector-recall`), which stays the readable home of the records; the phase merges to main as one change once the decision is solidified by its evidence. + +## Notes +- Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the latency guidance and the stripped dependency weight must be measured, not assumed; the engine is beta, so its pin is exact and its bump is gated by the canary. +- Edge cases: an empty object-type scope or `limit == 0` issues no search and reports the not-requested verdict in both adapters; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality across adapters (both engines normalise cosine internally) is asserted, not assumed. diff --git a/docs/coding-agent/rules/common.md b/docs/coding-agent/rules/common.md index 62aedd08..04e9c337 100644 --- a/docs/coding-agent/rules/common.md +++ b/docs/coding-agent/rules/common.md @@ -2,7 +2,7 @@ rule_schema_version: 2 suite_id: "rules-cm-20260719" rule_file: "common" -last_updated: "2026-07-24" +last_updated: "2026-09-02" --- # Common Repository Rules @@ -15,6 +15,7 @@ last_updated: "2026-07-24" - Committed artifacts in this repository must not contain machine-local absolute paths (for example user-profile paths); refer to sibling repositories by name and relative relationship instead. - When mentioning the `CharacterMemoryEvals` repository in committed docs, describe it as the public companion evaluation repository and state that evaluation tooling is a development aid, not core library functionality. Do not describe it as private or inaccessible (it was made public 2026-07-19); historical records (completed plans, dated ADR bodies) that reflect the earlier private status stay unchanged. +- Records in this repository (plans, decision records, phase documents) state what the evaluation harness's measurements allow this library to decide and when those measurements are used; the harness's own work is planned and tracked in the evaluation repository and is never mixed into library plans (ruled 2026-09-02). - Do not hard-wrap prose in committed documents: never insert line breaks mid-sentence to fit a column width. Write each sentence/paragraph/list item as one line and let editors soft-wrap. Structural line breaks (list items, headings, YAML keys, code) are fine. - ADR frontmatter `consulted` entries record model names only (for example "Claude Fable 5", "GPT-5.5 Pro") — no role, platform, or product designations such as "(orchestrator)" or "Codex" (user-directed 2026-07-18). diff --git a/docs/coding-agent/rules/orchestrator.md b/docs/coding-agent/rules/orchestrator.md index fc2dd857..a9681518 100644 --- a/docs/coding-agent/rules/orchestrator.md +++ b/docs/coding-agent/rules/orchestrator.md @@ -9,6 +9,8 @@ last_updated: "2026-07-23" ## Repo-Specific Orchestrator Policies - When creating or updating a PR, follow the format specified in `.github/pull_request_template.md`. +- PR titles state what the change achieves, not a list of its contents; the contents go in the body. Never use bare version numbers or milestone labels as titles. +- When a phase merges as one change, wave PRs target the planning branch and are registered as a GitHub stack on the planning PR with `gh stack link ...` (bottom to top); later waves are appended with `gh stack link ...`. The stack decides merge order. - Layer-boundary reorganizations must include a `use crate::` dependency-direction audit as required Reviewer evidence per ADR-I-0018 (ports/policy/models never import usecases, and import api only for the ADR's one named exception: the `api::types::retrieval` trace/telemetry vocabulary; errors/domain import no upper layer); file-placement conformance alone does not catch inverted edges hidden behind re-export shims. - Scope the ADR-I-0018 dependency-direction audit to the diff under review (e.g. `git diff | grep '^+.*use crate::'`) when reviewing incremental changes: pre-existing ports/policy/models imports of domain types via `crate::api::types` are grandfathered debt awaiting a one-time sweep to `crate::domain`, and a blanket grep forces per-line disambiguation between old and newly introduced edges. - PR feedback monitors must include terminal merged/closed state (harness pr-review-monitoring owns arming for reviews/comments but does not cover terminal-state watch) (user-directed 2026-07-19). diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 23c7e8ee..58480e2a 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -28,5 +28,12 @@ Separate numbering per track; IDs are never reused. - Partial supersession stays in place: the record remains authoritative for its surviving clauses, with `supersession_scope` and reciprocal frontmatter links recording the split. - In records predating the current template, a blank warrant means it was not recorded at decision time, not an authoring omission. Fill blank newer frontmatter fields only when the record is substantively revisited. +## Authoring rules +- One decision per record. A choice of component and a choice of how that component is run or made durable are separate decisions with separate records. +- A record encodes a locked-in decision. A deciding factor is deferred only when it is out of reach because of external factors, never when the same change produces the evidence. +- The decision body pins the system's design and, at most, the behaviour of a narrow part of the implementation. Code shapes go in a non-binding appendix, never in the decision. +- Every protected clause is checked against the project philosophy: it stays only if a philosophy goal (continuity, provenance, inspectable recall, correction) is what it protects. Current state is recorded under "Not covered", not as an invariant. +- Records read the same at any time. No wording that hinges on when the record was written ("this phase", "later decision", "at decision time", "once", "already", change verbs such as "gains" or "moves"); history is anchored to named records, versions, and absolute dates, and Implementation Impact describes the resulting state. + ## Status values `accepted`, `rejected`, `superseded`, `deprecated`. Records capture decisions, not undecided proposals. diff --git a/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md b/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md index b0fc26d0..d305620a 100644 --- a/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md +++ b/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0025-vector-record-is-a-read-contract.md +supersession_scope: partial --- # ADR-I-0001: Use stable cross-store IDs and deterministic graph IRIs diff --git a/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md b/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md index e4ee4be3..53ea5ebe 100644 --- a/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md +++ b/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0025-vector-record-is-a-read-contract.md +supersession_scope: partial --- # ADR-I-0002: Embed natural-language semantic surfaces, not structured metadata templates diff --git a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md index 706b0aa8..a611f6ce 100644 --- a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md +++ b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +supersession_scope: partial # the vector-backend default only (embedded Qdrant Edge is the default); backend roles remain authoritative --- # ADR-I-0003: Use Qdrant and Oxigraph as default storage backends diff --git a/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md b/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md index a84f1977..b238f423 100644 --- a/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md +++ b/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0025-vector-record-is-a-read-contract.md +supersession_scope: partial --- # ADR-I-0005: Keep Qdrant metadata filterable while graph relationships remain authoritative diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md new file mode 100644 index 00000000..9e8ae070 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -0,0 +1,138 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely replace the embedded engine with a lighter exhaustive-scan store the first time the dependency weight is questioned, or treat the embedded store as a test convenience whose semantics may drift from the service adapter, because both are the shortest path at small corpus sizes and both were measured as viable" + detected_signals: "cross-boundary contract shape with tempting alternatives; rejected alternatives likely to be re-proposed (the two spiked candidates); costly reversal (an engine switch rebuilds every embedded store); premises likely to expire (the engine is beta; the weight measurement is unstripped); deliberately bounded scope (single process, index tuning deferred)" + cost_of_violation: "an engine switch after embedded stores exist in the field rebuilds every character's recall index from graph authority and re-embeds it; two adapters with different admission semantics produce different continuity packs from the same memory, which evaluation evidence would attribute to retrieval regressions" + cost_of_wrong_preservation: "if the engine's beta API breaks or its footprint proves unacceptable on a target platform and this record is preserved as settled, local deployments carry a dependency that no longer earns its place" + cost_of_over_extension: "treating the embedded mode as validated for multi-process access misrepresents what the library has validated; treating the index knobs as tuned while they sit at their untuned exact-scan setting would ship approximate recall nobody measured" +depends_on: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] +implements: [] +supersedes: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md] +superseded_by: null +supersession_scope: partial # ADR-I-0003's vector-backend default only; its backend roles and its embedded graph description stay authoritative +--- + +# ADR-I-0023: Embedded Qdrant Edge is the default vector candidate store; the service adapter remains the service mode + +## Context and Problem Statement + +With the graph store embedded by default (ADR-I-0021) and retrieval statistics file-backed (ADR-I-0009), the vector candidate store was the one component that required an external service. +That conflicts with the intended deployment shapes: desktop companions and game or simulation characters run on end-user machines where a container runtime cannot be assumed, and it keeps a service dependency in the default test path. +ADR-I-0003's own revisit clause, "operating two stores becomes too heavy for target users", was recorded as triggered in the v0.1.5 closeout report. +The vector layer is candidate recall only: the vector store suggests, retrieval statistics guide fanout, and graph authority decides final inclusion, so an embedded adapter has a low correctness bar for any single query — it must prefilter and rank candidates well, never be authoritative for anything. +The bar that matters is longevity: a character's memory is expected to accumulate continuously for years to decades and to outlive several generations of embedding model, so the embedded recall index must be chosen for where a memory ends up, not for where it starts. +Two feasibility spikes were run on 2026-09-02 against the same probe set (build weight, lifecycle and reopen, exactness control, a thirty-way identical-vector tie cohort, five filtered queries against the live service, API shape): the in-process build of the service backend (Qdrant Edge 0.8.0, beta) and a SQLite vector extension (sqlite-vec 0.1.9, stable). + +## Decision Drivers + +- The decade standard: at continuous-accumulation scale the recall index plausibly reaches hundreds of thousands to a million vectors, where an exhaustive scan costs seconds per query, so approximate indexing, quantization, and memory-mapped read-only segments are the baseline for an interactive character, not an escalation. +- Embedding models will change several times over a memory's life; named vectors that let two embedding spaces coexist during lazy re-embedding are a required capability, not a nicety. +- The store is a rebuildable cache over graph authority, which bounds the cost of an engine switch but does not eliminate it: every embedded store in the field is rebuilt and re-embedded. +- Library over in-house: the library does not own a vector engine; it owns the port contract, the tie-closure loop, the canonical ordering, the verdict mapping, and the error classification, and it holds any engine to those. +- The embedded adapter must satisfy the same port contract as the service adapter, proven by a shared parity suite, or the evaluation suite stops being a regression instrument. +- Defaults must match validation evidence (ADR-I-0021's rule): when the default test path runs on the embedded adapter, a service default would repeat the asymmetry that record corrected, so the default and its evidence land in the same change. +- Portability across deployment shapes: an engine that can synchronise with the service backend keeps a path from a local character to a hosted one. + +## Decision + +Add an embedded vector candidate store mode behind the existing vector candidate port, implemented on the in-process build of the service backend (Qdrant Edge), selected by a dedicated store-mode setting. +Embedded mode is the default vector mode. +The service adapter remains fully supported as the service and cloud mode, selected explicitly; this decision adds a mode and deprecates nothing. +The evidence that licenses the default is produced by the same change that introduces the mode: the shared parity suite proves both adapters identical below their indexing thresholds, including identical-vector tie cohorts, and the library's integration suite runs on the embedded adapter without a service, so the shipped default is the validated path (ADR-I-0021's rule). +The companion evaluation repository's cross-mode run is closeout evidence and a revisit trigger, not a gate: a difference between modes on its continuity suite reopens this record. +The embedded store is single-process, matching the embedded graph store's expectation. + +Exactness is a threshold property, not a promise: below the configured indexing threshold a shard answers by exhaustive scan, above it the index answers, and in both cases the completeness verdict (ADR-I-0024) reports the boundary state of the returned top-K. +The threshold ships at its exact-scan setting; index construction, quantization, and memory-mapped segments are available capabilities whose defaults are set by a separate measured decision on real corpora, never silently. + +The adapter is held to the port contract by the shared library logic it may not re-implement: the tie-closure loop and the canonical constructor (the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards), the filter and payload conventions, the verdict mapping, and the error classification. +The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific. +The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). +A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. +How the adapter runs the engine inside an async host and what makes a write durable is a separate decision (ADR-I-0027). + +Configuration follows the one-key-per-backend pattern the graph and statistics stores use: a mode setting (`embedded` or `service`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. +The path names a directory; each collection is one engine shard directory inside it, named by the collection name the public constructor takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has shard directories). +Embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name and the record schema version in an adapter-owned marker inside the shard directory, and requires the path setting to be present, with a missing path a configuration error rather than an implicit location, exactly as the embedded graph store's path. +A reopened shard is validated against the configured embedding model (vector size and distance from the shard's own configuration) and the supported record schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires. + +## Implementation Impact + +- An adapter module implements the vector candidate port on the embedded engine; the composition root has a mode switch mirroring the statistics-store switch, defaulting to embedded. +- The library's toolchain pin is the minimum the pinned engine compiles on (Rust 1.97 for engine 0.8.0, which does not build on 1.95). +- The settings type carries the mode and path keys; the service connection string is optional and validated as present only in service mode. +- The vector database error vocabulary includes an engine-error kind for the embedded backend beside the filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion, planned in that repository, is its prerequisite for re-pinning to a library version that carries the embedded backend, not part of the library change. +- The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured; the embedded adapter opened on a temporary directory replaces the deterministic test fake, so the default test path has no vector-service dependency. +- Dependency weight is a recorded deliverable: the unstripped release delta, the stripped delta, and the effect of feature trimming are measured and recorded with the adapter. +- Documentation states the embedded default, the single-process expectation, the threshold semantics, the measured latency guidance, and rebuild-from-graph-authority as the path between modes; the README's backend setup leads with the local path and presents the service as the explicit alternative. + +## Considered Options + +1. The in-process build of the service backend (Qdrant Edge) as the default, service mode retained. +2. An in-house SQLite exact cosine scan on the `rusqlite` dependency the statistics store carries. +3. The SQLite vector extension (sqlite-vec). +4. A columnar embedded vector database (LanceDB). +5. An in-memory-only embedded store. +6. Ship embedded as an opt-in mode and decide the default separately. + +## Decision Outcome + +Chosen option: **Option 1**. +Measured on the shared probe set: lifecycle and reopen identical; the zero indexing threshold keeps a shard on a plain exhaustive scan while a threshold of one plus an optimise call builds the index; five filtered queries against the live service returned identical id sets and order with a maximum score delta of 0.0; cost of 489 additional dependency-tree lines and about 30.5 MB of unstripped release binary (stripped size unverified, feature trimming untested). +It is the only candidate that offers, in one engine, the capabilities the decade standard makes baseline — approximate indexing, quantization, named vectors, datetime payload indexes, memory-mapped read-only segments — plus synchronisation to the service backend, and it is the same engine family the service adapter targets, so payload and filter conventions are shared rather than translated. + +### Rejected Alternatives + +Option 2 (in-house exact scan) violates the library-over-in-house driver: the library would own distance computation, blob encoding, and scan scheduling, and every capability the decade standard needs (index, quantization, named vectors) would have to be written or migrated to later; it is rejected outright, not deferred. +Option 3 (sqlite-vec 0.1.9) measured well — builds on Rust 1.95 and 1.97, 19 additional tree lines, about 6.2 MB unstripped, deterministic ties across fresh files and processes, parity five of five with score delta at or below 1.6e-7 — but its stable release is exhaustive-only with approximate indexing existing only in a pre-release, it carries limits on dimensions, result count, and metadata columns, and it is a pre-1.0 binding with a single maintainer; at decade scale its future is a second migration, so it is rejected for this role and reopened only if the chosen engine fails its Revisit When triggers and the extension has shipped a stable approximate index. +Option 4 (LanceDB) was rejected on dependency weight relative to the chosen engine without a spike, since the chosen engine covers its capabilities; it is reopened only alongside Option 3's reopening. +Option 5 fails restart safety, which the persistent-graph-authority phase made a requirement for every store that survives a process; rejected outright. +Option 6 defers a decision whose deciding evidence the same change produces: the parity suite and the service-free integration path are its acceptance, so with that change merged the validated path is embedded and a service default would be the unvalidated one, the exact asymmetry ADR-I-0021 corrected; deferral would also leave every consumer-facing document describing a default the evidence does not support. Rejected outright; the default is reopened only by the triggers under Revisit When. + +## Consequences + +- Positive: a fully self-contained local deployment exists; the default test path needs no running service; both adapters are held to one contract by one suite; the embedded store can grow into indexed, quantized, and memory-mapped operation without an engine switch. +- Positive: shared engine family means payload and filter conventions are written once and the service parity result (score delta 0.0) is structural, not coincidental. +- Negative / tradeoffs: the engine is beta and its API may change; the canary test and the pinned version turn that into a build-time failure rather than a runtime one. +- Negative / tradeoffs: about 30.5 MB of unstripped binary and a higher toolchain floor; the weight deliverable exists to establish the real number. +- Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. +- Negative / tradeoffs: consumers who followed the service-first setup must set the store path or select service mode explicitly; with no external consumers (Compatibility Policy) no migration hint is carried. + +## Decision Boundary + +Invariant: embedded is the default vector mode and the embedded adapter implements the same port contract as the service adapter, proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string; tie closure and canonical ordering come from the shared library loop and constructor, never from engine ordering; the indexing threshold ships at its exact-scan setting until a measured decision changes it. + +Not covered: the index, quantization, and memory-map tuning values (calibrated through a separate measured decision) and the latency guidance numbers (measured and revised through documentation). + +## Validation + +- Embedded mode constructs without any running service and survives process restart with identical search results. +- The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop; above the threshold a recall comparison against the exhaustive setting is recorded. +- A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. +- The contract canary passes on the pinned engine version and is re-run on every engine bump. +- The default-mode construction test asserts embedded mode; service mode requires the explicit mode value and a connection string. + +## Revisit When + +- The engine leaves beta or changes its API — re-pin, re-run the canary, and re-measure parity before adopting the new version. +- A stripped-footprint measurement or feature trimming changes the weight picture materially in either direction — revisit the weight tradeoff recorded above and, if the footprint is unacceptable on a target platform, reopen Option 3. +- A corpus benchmark contradicts the interactive-latency assumption behind the decade standard (exhaustive scan acceptable far beyond the assumed scale, or the index insufficient at it) — revisit the threshold default and the tuning decision. +- The companion evaluation repository's cross-mode run shows a difference between embedded and service mode on the continuity suite, or a corpus at the guidance size misses the interactive-latency assumption in embedded mode — reopen the default. +- A multi-replica deployment shape is designed (the remote graph-authority phase ADR-I-0021 anticipates) — the single-process expectation is reconsidered together with the graph and statistics stores, never alone. + +## Consultation impact + +Question asked: which embedded engine, on the two spikes' evidence; the consult's initial recommendation of an in-house exact scan was overruled by the decider on the decade-scale portability standard, the settings shape was adopted as recommended, and the consult's proposal to defer the default was overruled because the deciding evidence is produced by the same phase. + +## More Information + +- ADR-I-0003 remains authoritative for the backend roles (vectors in the service backend family, graph authority in Oxigraph); this record supersedes only its vector-backend default, answering its own revisit clause; its embedded graph description is untouched (ADR-I-0021 made persistent embedded storage the validated graph default without superseding it). +- ADR-I-0024 (port contract this adapter implements, including the verdict rule), ADR-I-0025 (the record it stores), and ADR-I-0027 (how the adapter runs the engine and makes writes durable). +- The two spike reports (2026-09-02) are transient working artifacts; the numbers above are their record. +- The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md new file mode 100644 index 00000000..e7e4df52 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md @@ -0,0 +1,125 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely add a vector-layer predicate as a three-valued hint filter that matches unknown values, or let an adapter truncate an unclosed equal-score cohort without saying so, because both are the natural first implementation and both have already happened in this repository" + detected_signals: "cross-boundary contract shape (port postcondition) with tempting alternatives; rejected alternative likely to be re-proposed; premises likely to expire (no vector-layer predicate has a caller, and stored values are not guaranteed present)" + cost_of_violation: "a prefilter that matches unknown values admits stale candidates that graph verification then silently discards, and an unreported open cohort makes top-K membership vary between runs — both surface as unexplained retrieval nondeterminism in evaluation evidence long after the cause is forgotten" + cost_of_wrong_preservation: "if the unknown-never-matches rule is preserved after every stored value is guaranteed present and synchronised, adapters carry a defensive arm for a case that cannot occur" + cost_of_over_extension: "treating the completeness verdict as an error condition would fail retrieval on a determinism caveat about non-authoritative candidates" +depends_on: [implementation/ADR-I-0018-responsibility-boundary-modules-with-enforced-dependency-direction.md, implementation/ADR-I-0022-retain-measured-retrieval-defaults.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0024: Vector candidate recall reports its completeness, and a vector-layer prefilter never matches an unknown value + +## Context and Problem Statement + +The vector candidate port promised deterministic admission: at most `limit` unique object-and-surface matches in canonical order, with equal-score cohorts at the cutoff closed before truncation (ADR-I-0022 records the fix). +The service adapter closes the cohort by growing its fetch up to a bound, but when the bound is hit it returns the truncated set with no signal, so a caller cannot tell "this top-K is determinate" from "membership may vary between runs", and evaluation evidence later attributes the variation to retrieval. +Separately, the port once carried currentness predicates implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake, so the filter admitted records under a rationale ("current") that was not true of them; and because the field was written only at upsert, a record whose value had since changed in graph authority was filtered on a stale value, admitted when it should not have been or excluded when it should have been returned. +Those filters were deleted as speculative when no caller used them. +A second adapter (ADR-I-0023) makes both gaps matter: below its indexing threshold an embedded shard scans exhaustively and needs a way to say so, and two adapters must agree on what a prefilter may do. + +## Decision Drivers + +- Retrieval rationale must be inspectable: the philosophy asks that a developer can see why a memory was or was not retrieved, and an unreported open cohort is unexplained recall. +- Candidate recall is non-authoritative; graph authority verifies every candidate, so a determinism caveat must never become a retrieval failure. +- A prefilter false negative is a memory that silently never returns — a continuity loss nobody can inspect — while a false positive costs one root slot and is discarded by verification; the asymmetry decides what a vector-layer predicate may do. +- Each port owns its stated postconditions; upper layers never repair lower-layer output. + +## Decision + +The search result carries, beside the canonical candidates, a completeness verdict stated by the adapter. +The verdict distinguishes four situations: no search was issued because the limit was zero or the scope was empty; every stored record in scope was scored, so the requested top-K is determinate over the population; an index answered with a prefix whose cutoff cohort was closed, so the returned set is determinate for that index state although an approximate index may have omitted records it never surfaced; and the overfetch bound was reached with the cutoff cohort still open, so membership may vary. +Adapters state the verdict truthfully: exhaustive only when the adapter knows the shard is unindexed and the cutoff cohort was closed, with the scanned count taken from the scope rather than the rows returned; an exhaustive scan whose cohort stays open at the bound reports open. +The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. +Degenerate vectors are defined on both sides of the port so that neither adapter has an undefined path: a zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero and reports a truthful verdict. + +A vector-layer predicate may be evaluated only over a column that is fully populated for every searchable record (backfilled from graph authority before the predicate is enabled, per ADR-I-0025) and that is immutable or synchronised on every mutation; under those two conditions a missing or unknown value is a defect, not a state, and it never satisfies a positive predicate, so the rule produces no false negative on a correctly populated column and turns an incorrectly populated one into a visible failure rather than a silent widening. +A predicate that needs a stored value the write paths do not keep current, or that is not populated for every record, is not a prefilter; it is a graph-authority question. + +## Character Memory Relevance + +Recall that silently varies between runs, or that admits and excludes memories on values nobody keeps true, is the unexplained recall the philosophy forbids: a character that forgets an episode because a stale column excluded it looks like a character that never lived it, and a filter that admits on a blank gives a rationale that is false. +The verdict keeps determinism inspectable; the prefilter rule keeps a candidate stage from being the reason a memory is unreachable, and keeps every stated filter rationale true. + +## Implementation Impact + +- The port's search method returns candidates plus verdict; the pipeline copies the verdict into telemetry; test fakes report exhaustive. +- The service adapter's existing fetch decision maps onto the closed and open situations. +- The evaluation repository mirrors the telemetry field (ADR-I-0026). + +## Considered Options + +1. A completeness verdict beside the candidates, plus the prefilter rule. +2. Silent degradation at the fetch bound (as built). +3. A boolean complete flag. +4. Fail with an error when the cohort is open at the bound. +5. Resurrect the deleted hint filters for the embedded adapter, which can evaluate them exactly. + +## Decision Outcome + +Chosen option: **Option 1**. +It makes the postcondition expressible by the layer that owns it, distinguishes the exhaustive case an unindexed shard can report from the closed-cohort case an index can promise, and keeps every consumer a field access away from unchanged code. + +### Rejected Alternatives + +Option 2 hides a determinism caveat that evaluation evidence later attributes to retrieval; rejected outright. +Option 3 loses the exhaustive-versus-closed distinction that tells a caller whether population-level determinacy was achieved; rejected outright. +Option 4 fails retrieval on a caveat about non-authoritative candidates that graph authority verifies anyway; rejected outright. +Option 5 recreates a prefilter over values that only the upsert path wrote, which the rule above forbids; a predicate is admitted when its stored value is kept in sync or is immutable. + +## Consequences + +- Positive: top-K determinism is observable per retrieval and per adapter. +- Positive: any future prefilter has one admission test — is the column populated for every record and always current — instead of a case-by-case argument. +- Negative / tradeoffs: callers wanting a scoped or time-bounded semantic search wait for a synchronised or immutable column rather than filtering on what happens to be stored. + +## Decision Boundary + +Invariant: the search result carries a completeness verdict stated truthfully by the adapter, and the pipeline never repairs, retries, or fails on it; a vector-layer predicate reads only a fully populated column that is immutable or synchronised, and an unknown value never satisfies a positive predicate. + +Not covered: the current query shape (an embedding, a limit, and an object-type scope, with an empty scope selecting zero — a current state, not a rule; the zero-norm rules above are the one query-shape behaviour this record fixes), the verdict's type and wire shape (the appendix is a reference, not a contract), the telemetry field name, and the service adapter's overfetch bound. + +## Validation + +- Unit tests on the service adapter's fetch decision assert the closed and open verdicts, including the all-tied cohort at the bound. +- A retrieval test asserts the telemetry verdict for each situation using the fakes. +- The parity suite asserts exhaustive for the embedded adapter below its indexing threshold and closed for the service adapter on the identical-vector tie fixture. +- A census of the vector adapters shows no match-or-unknown condition. +- Parity fixtures cover the zero-norm query and the rejected zero-norm record in both adapters. + +## Revisit When + +- An adapter appears that cannot classify its own cutoff (a remote index without a fetch count) — the verdict may need an "unknown" situation, which must still never be treated as an error. +- Every stored value a predicate could read is guaranteed present and synchronised — the unknown arm becomes unreachable and may be removed. + +## Consultation impact + +Question asked: whether the deleted hint filters should return for the embedded adapter; ruling adopted the prefilter rule instead. Revised 2026-09-03 on the decider's review: the type shape is an appendix and the scope-only query is recorded as current state, not as a rule. + +## More Information + +- ADR-I-0022 (tie-cohort closure and canonical ordering, the postcondition this record makes expressible); ADR-I-0023 (the embedded adapter); ADR-I-0025 (the stored record a predicate would extend); ADR-I-0026 (the evaluation reader of the verdict). +- Candidate predicates that satisfy the rule, noted for whichever phase needs them and binding on none: a scope id written at upsert and kept in sync by the link and reflection write paths (scoped continuity); an immutable time window over creation and observation time (a time-bounded retrieval route). + +## Appendix: reference shape (non-binding) + +```rust +pub struct VectorCandidateRecall { pub candidates: CanonicalCandidates, pub completeness: VectorRecallCompleteness } + +pub enum VectorRecallCompleteness { + NotRequested, + Exhaustive { scanned: usize }, + BoundaryTieClosed { fetched: usize }, + BoundaryTieOpen { fetched: usize, fetch_bound: usize }, +} +``` diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md new file mode 100644 index 00000000..0ade46f9 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -0,0 +1,114 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely re-add relationship, lifecycle, time, or readable-text columns to the vector record because the earlier payload design lists them as intended, or delete the embedded-text column as unread once surfaces become generated" + detected_signals: "cross-boundary contract shape (two adapters mirror one record); rejected alternative likely to be re-proposed; premises likely to expire (a retrieval route may need a prefilter); cross-repository obligation (the evaluation baseline read a payload column)" + cost_of_violation: "every column that returns without a reader is mirrored across two adapters, indexed at every collection initialisation, and carried stale by write paths that never update it; a column deleted as unread would erase the only record of what a generated vector embedded" + cost_of_wrong_preservation: "if a retrieval route needs a prefilter and the five-column rule is preserved as prohibition rather than current state, the predicate is blocked instead of landing through the named re-entry path" + cost_of_over_extension: "extending the rule to the graph store would strip graph authority of denormalised fields it legitimately owns" +depends_on: [implementation/ADR-I-0007-schema-versioning.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md] +implements: [] +supersedes: [implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md, implementation/ADR-I-0002-natural-language-embedding-surfaces.md, implementation/ADR-I-0001-stable-cross-store-ids.md] +superseded_by: null +supersession_scope: partial +--- + +# ADR-I-0025: The vector record is a read contract — identity, surface, schema version, embedded text + +## Context and Problem Statement + +ADR-I-0005 decided that the vector payload stores filterable metadata and graph pointers, and the payload design note enumerated thirty-four fields with thirty-one of them indexed; the implemented manifest carried thirty-three with thirty indexed after the record-type field was dropped in the structured-verdict phase. +When the embedded adapter (ADR-I-0023) was designed, the library read back exactly three of those fields — object id, object type, surface — and the only external reader was the companion evaluation repository's vector-only baseline reading the readable text column. +The relationship hints were frozen at upsert and never updated by the link write path; the lifecycle hints described vectors the correction and forgetting paths delete; the readable text column duplicated graph text with a prefix removed; and the embedded adapter would have mirrored every field into a second physical schema. +ADR-I-0002's implementation note said to "persist both `embedding_text` and `content_text` where useful", which left the two text columns' meanings undefined. +The forward-looking case for each family was analysed against the planned phases (scoped continuity, factual rigor and temporal validity, retrieval observability, associative recall, assisted remember, multimodal) before deciding. + +## Decision Drivers + +- A column earns its place when a reader exists; carrying it unread costs two adapter mappings, index creation per collection, a parity fixture, and the sync discipline ADR-I-0005 named in its own tradeoffs. +- Prefilter hints are only safe on immutable or synchronised data; the relationship, lifecycle, ranking, and mutable time hints were none of those. +- Re-adding an immutable column is a backfill from graph authority, not a re-index. +- When embedding surfaces are generated or caller-supplied (the assisted-remember phase; the write plan carries a caller-supplied surface), the text a vector embeds is not re-derivable from graph authority, so it is provenance in the philosophy's sense. +- Read-out text is graph authority's job; the vector layer suggests, it does not describe. + +## Decision + +Both adapters persist exactly these fields per vector record: object id, object type, surface, schema version (ADR-I-0007), and `embedding_text`. + +Three sentences govern the text columns: +Read-out text lives in graph authority. +The vector record stores only the embedded surface, as provenance of what was ranked. +Consumers needing candidate content hydrate by object id. + +There is no readable-text column (`content_text`). +The relationship refs (episode, observation, thread, entity, participant, speaker, supersedes), the lifecycle and currentness flags, the time hints, the ranking and salience hints, the object-specific hints, the graph URI, and the raw source reference are not part of the vector record. +Dropping the graph URI partially supersedes ADR-I-0001's clause that every vector payload carries it: the stable object id remains the cross-store identity and the graph URI is derived from it by graph authority, so the pointer was a redundant copy of the id; ADR-I-0001's stable-id decision itself is unchanged. +The typed field manifest is the single source of both adapters' column sets and holds exactly the five entries. +ADR-I-0024 rules that a predicate reads only synchronised or immutable values and notes the two candidate predicates (a synchronised scope id; an immutable time window over `created_at` and `observed_at` backfilled from graph authority), so a returning column arrives with its predicate and its reader. + +## Implementation Impact + +- The vector record type and the surface builders carry no hint fields; the payload map serialises five fields for both adapters, which share the engine family's payload conventions. +- The service adapter creates payload indexes only for object id and object type. +- The companion evaluation repository's vector-only baseline sources item text from its own ingest records, not from a payload column (ADR-I-0026). +- The payload design note's field categories and indexing policy are superseded by this record and carry a supersession note saying so. +- No migration: under the Compatibility Policy, existing stores are rebuilt from graph authority. + +## Considered Options + +1. Five-column read contract; keep `embedding_text` only; no hint families, with the candidate predicates noted in ADR-I-0024. +2. Keep both text columns. +3. Drop both text columns. +4. Keep the unread hints for the planned phases. +5. Keep only the two immutable timestamp columns as a hedge against top-K starvation. + +## Decision Outcome + +Chosen option: **Option 1**. +It stores what is read, keeps the one column that ceases to be re-derivable when surfaces are generated, and prices a returning column honestly. + +### Rejected Alternatives + +Option 2: `content_text` is a deterministic function of graph object fields at every surface builder, so it never carries information graph authority lacks, and its one reader has its own ingest records; rejected outright. +Option 3: `embedding_text` is cheap and is the only record of what a generated or caller-supplied vector embeds; rejected outright. +Option 4: no planned phase names a vector-layer predicate the existing fields could serve without new synchronisation work — scoped retrieval needs a scope id kept in sync by linking, temporal validity is a ranking property of new claim objects, salience evolves by reinforcement, and lifecycle hints describe vectors the write path deletes; reopened only under ADR-I-0024's prefilter rule. +Option 5 is the only subset with a forward-looking case that survives the synchronisation test; it was declined because no phase document asks for the predicate and the columns backfill cheaply when one does. + +## Consequences + +- Positive: both adapters mirror one five-field manifest; the embedded shard stores those fields as payload beside the vector with keyword indexes on object id (the delete selector) and object type (the scope predicate); ADR-I-0023 owns the physical layout. +- Positive: the embedded surface is preserved as vector provenance, which matters once surfaces are generated rather than derived. +- Negative / tradeoffs: a scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the candidate predicates noted in ADR-I-0024 make that step predictable. + +## Decision Boundary + +Invariant: the vector record carries only fields a reader consumes, plus the embedded surface as provenance and the schema version ADR-I-0007 requires; readable content is hydrated from graph authority by object id; a returning hint arrives with its predicate and parity fixture under ADR-I-0024's prefilter rule. + +Not covered: the physical encoding of each column per adapter, and graph authority's own denormalised fields. + +## Validation + +- A census of both repositories shows no reader of the dropped fields and no reader of `content_text`. +- The manifest test asserts the five entries; the parity suite serialises and reads them through both adapters. +- The evaluation baseline reproduces its results with text sourced from ingest records. + +## Revisit When + +- A retrieval route needs a scoped or time-bounded semantic search — add the column under ADR-I-0024's prefilter rule; this record's invariant is satisfied by a column that arrives with its reader. +- The assisted-remember phase makes the embedding surface a graph-authoritative provenance artifact — the vector copy becomes a cache and this record's provenance argument moves to the graph. +- A re-indexing workflow appears that cannot rebuild from graph authority — the readable-text question reopens with that workflow as its reader. + +## Consultation impact + +Question asked: whether the unread hint families and the readable text column should be kept for planned phases; ruling adopted the five-column contract with the three governing sentences and the noted candidate predicates. + +## More Information + +- ADR-I-0005 remains authoritative for graph authority over relationships; this record supersedes its payload field list and its "payload metadata as candidate filter" implementation guidance. +- ADR-I-0002 remains authoritative for natural-language embedding surfaces; this record supersedes only its note to persist both text columns. +- ADR-I-0024 (completeness verdict and prefilter rule), ADR-I-0023 (embedded shard layout), ADR-I-0026 (evaluation baseline reader). diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md new file mode 100644 index 00000000..cedd0578 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -0,0 +1,111 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely add a public raw vector search to the facade for the evaluation baseline, or let evaluation tooling read a store's physical schema directly again, because each is the shortest path to a number" + detected_signals: "externally observable contract shape with a tempting alternative; rejected alternative likely to be re-proposed; cross-repository obligation; deliberately bounded scope (no product use case for raw recall exists)" + cost_of_violation: "a raw-recall facade makes the library a vector-database abstraction and exposes unverified candidates as if they were memory; a schema-reading baseline breaks silently the moment a second vector adapter ships a different physical schema, and it reimplements canonical ordering the library already owns" + cost_of_wrong_preservation: "if a product use case for candidate-level recall arrives and this record is preserved as a blanket prohibition, the diagnostic surface the observability phase plans would be blocked instead of designed" + cost_of_over_extension: "reading this record as forbidding evaluation tooling from using the trace at all would leave the baseline with no honest data source" +depends_on: [implementation/ADR-I-0020-restart-identity-via-caller-supplied-ids-not-a-lookup-surface.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0026: Raw vector baselines read the retrieval trace; the library exposes no candidate-search surface + +## Context and Problem Statement + +The companion evaluation repository (a development aid, not core library functionality) runs a vector-only baseline: ingest through the library, then rank by plain vector similarity to measure what hybrid retrieval adds. +Before this record, that baseline held its own client to the vector service, ran one filtered search per object kind against the library's collection, read three payload fields by hard-coded name, re-implemented best-score-per-object deduplication and score ordering, and took item text from a payload column. +That is a hidden capability: the baseline depended on an adapter-private schema, duplicated ordering the library owns, and could not run at all against an embedded store (ADR-I-0023). +The question is what capability the library must expose so the baseline stops reaching into a store. + +## Decision Drivers + +- Evaluation tooling must not grow library surface that no product use case has demanded (ADR-I-0020's driver). +- The library is not a vector-database abstraction, and vector-only candidates must never become behavior-influencing memory without graph verification (project philosophy; the persistent-graph-authority phase's acceptance criteria). +- The retrieval trace is the raw vector recall: the canonical, pre-verification top-K with object reference, surface, score, and rank, scoped by the configured object types and sized by the candidate limit, and ADR-I-0024 adds the completeness verdict, which says either that the scoped population was scored exhaustively (top-K determinate over the population) or, for an indexed answer, only whether the cutoff cohort within the index's returned prefix was closed. +- Every store's physical schema is adapter-private; two adapters must not create two baseline implementations. + +## Decision + +The library exposes no raw candidate-search surface and no facade change. +The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. +The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant the library publishes as public policy per object kind, alongside the surface policy that defines it, so an ordinary caller never duplicates private policy and a new surface changes the published value in the same change; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. +That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore an exhaustive surface-level verdict at that limit makes the object-level top-budget determinate over the scoped population, while a closed boundary verdict makes it determinate only relative to the index's returned prefix (an approximate index may have omitted records before tie closure), and the baseline records which of the two it obtained; the contract is pinned by a fixture whose objects carry every surface. +Item text comes from the evaluation repository's own ingest records, keyed by the external identity it reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). + +What this record asks of the evaluation repository, recorded as the library-facing contract and nothing more: + +- The evaluation repository consumes the retrieval trace and the completeness telemetry as an ordinary caller; the library adds no candidate-search surface for it, and the only public addition made for this reading is the published maximum-surfaces-per-object-kind policy value, which is a policy constant, not a query surface. +- A raw-vector baseline that wants per-kind top-K uses one singleton-scoped traced retrieval per kind with the multiplied limit and object-level deduplication described above; any other reading of the trace is not covered by the parity claim. +- How the evaluation repository migrates its baseline, mirrors telemetry, labels its rows, or guards its cleanup is planned and tracked in that repository. + + +## Character Memory Relevance + +Retrieval that bypasses graph authority is exactly the "generic RAG wrapper" and "unexplained recall" the philosophy warns against; the trace exists so that every candidate a developer sees is one the library can explain, whether it was admitted or not. +Keeping the baseline inside the traced retrieval path means the measurement of "what does the graph add" is taken from the same recall the character actually experiences, not from a parallel search that may drift from it. + +## Implementation Impact + +- Library: ADR-I-0024's telemetry field plus one published policy value, the maximum number of embedding surfaces per object kind, exported beside the surface policy that defines it; no candidate-search facade, so the acceptance criterion "no public facade change beyond the telemetry field and this policy value" holds. +- Evaluation repository: its baseline reads the trace under its own plan; nothing in this repository depends on how. + +## Considered Options + +1. The baseline consumes the retrieval trace; no library surface. +2. A public candidate-recall method on the facade returning references, surfaces, and scores. +3. A retrieval mode that skips graph verification. +4. Publish the payload manifest so evaluation tooling can keep reading the store. + +## Decision Outcome + +Chosen option: **Option 1**. +It holds the no-facade-change line, covers every vector adapter automatically, deletes a duplicate implementation, and takes the measurement from the recall the character actually experiences. + +### Rejected Alternatives + +Option 2 is an evaluation-driven surface with no product use case and a vector-database-abstraction shape; it is reopened only by a product use case for candidate-level recall, at which point it lands as a designed diagnostic surface in the retrieval-observability phase, not as a search method. +Option 3 contradicts the acceptance criterion that candidates whose graph objects are missing are rejected from normal retrieval; rejected outright. +Option 4 leaves two implementations of one capability and breaks with the first adapter whose physical schema differs; rejected outright. + +## Consequences + +- Positive: one retrieval entry point; both adapters covered; the baseline reports the completeness of the top-K it measured. +- Negative / tradeoffs: the baseline pays for graph expansion it discards, an evaluation-run cost accepted in exchange for not inventing a second retrieval path. +- Negative / tradeoffs: the baseline issues one traced retrieval per measured kind instead of one search, so its cost scales with the number of kinds; if a kind's budget cannot be closed within the service adapter's fetch bound, that retrieval's completeness verdict says so and the run records it. + +## Decision Boundary + +Invariant: no public raw candidate-search surface; evaluation baselines consume the retrieval trace and telemetry; store schemas are adapter-private; candidate content is hydrated by object id from the consumer's own records or graph authority. + +Not covered: any headroom the baseline adds to a kind's limit, the shape of the evaluation repository's ingest record store, and any future diagnostic surface the observability phase designs on product demand. + +## Validation + +- An A/B run of the vector-only configuration (direct search versus trace-derived) shows identical item identities and ranks per question before the direct path is deleted. +- The evaluation adapter contains no search call against the vector service and no payload field constant. +- The baseline runs unchanged in embedded mode. + +## Revisit When + +- A product use case demands candidate-level recall — design a diagnostic surface in the retrieval-observability phase and supersede this record's prohibition for that surface only. +- One retrieval per kind proves too costly on a real dataset, or a kind's budget cannot be closed within the service adapter's fetch bound — reopen whether the port needs per-type limits in a single query. + +## Consultation impact + +Question asked: trace-derived baseline versus a public candidate-recall method; ruling adopted the trace as recommended. + +## More Information + +- ADR-I-0020 (the precedent: evaluation needs met without a lookup surface). +- ADR-I-0024 (the completeness verdict the baseline reads), ADR-I-0025 (why item text is not a payload column). +- The companion evaluation repository's vector-only baseline plan (historical record of the direct-search implementation this record replaces). diff --git a/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md new file mode 100644 index 00000000..ea7c8dbd --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md @@ -0,0 +1,102 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-03 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely call the embedded engine directly from the async retrieval path (it is a plain synchronous API and the first implementation compiles), and would rely on the shard's final drop for persistence because the engine's write call returns success before anything is durable; both were the natural first draft of this phase" + detected_signals: "cross-boundary contract shape (an engine with synchronous, non-durable writes inside an async host); premises likely to expire (the engine is beta and its persistence model may change); costly to detect (lost writes surface as a character forgetting after a crash, long after the cause)" + cost_of_violation: "an engine call on an executor thread stalls every other retrieval in the process for the duration of a scan, build, or flush; a write acknowledged before its flush is lost on any exit that skips the shard's drop, and the loss is silent — the store reopens cleanly and simply lacks the memories" + cost_of_wrong_preservation: "if the engine starts replaying its log on load or persisting on write and this record is preserved, every write keeps paying a synchronous disk sync it no longer needs" +depends_on: [implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0027: The embedded vector engine runs on a dedicated blocking owner that flushes every write before acknowledging it + +## Context and Problem Statement + +The embedded vector engine (ADR-I-0023) exposes a synchronous API: opening a shard, creating payload indexes, upserting and deleting, searching, counting, building an index, and dropping a shard (which flushes) all block the calling thread. +The library's retrieval and write paths are async, and the composition entry point is itself async, so the natural first implementation calls the engine on an executor thread and stalls every other task in the process. +Separately, the engine's write call returns success before anything is durable. +Measured on the pinned version on 2026-09-02: a writer that skipped the shard's drop and a writer that exited the process after successful writes both reopened cleanly with zero of two hundred points, while the normal-drop control reopened with all of them; the engine persists only when its flush runs, and a load does not replay the write-ahead log. +An adapter that relies on the final drop for persistence therefore loses every unflushed memory on any exit that pre-empts that drop, and the loss is invisible at reopen. + +## Decision Drivers + +- No engine call may occupy an async executor thread; the library's other embedded stores hold this line. +- A write the library has acknowledged must survive a process exit that skips orderly shutdown; a character that forgets after a crash violates continuity silently. +- No port or facade method is added for shutdown; the existing facade drop remains the only close path. +- The rule must be pinned to measured engine behaviour so a change in the engine reopens it rather than silently voiding it. + +## Decision + +The adapter creates a dedicated blocking owner at construction: one blocking worker that opens the shard itself, holds it, and serialises every engine call — shard open and load including the lock backoff, payload index creation, upsert and delete, search, the filtered scope count behind the exhaustive verdict, index build, and the final drop. +The async composition entry point never touches the engine; it only hands work to the owner and awaits the result. +The owner acknowledges an upsert or delete only after the engine's flush has completed, so every acknowledged write is durable independently of the shard's final drop. +Dropping the adapter through the existing facade drop only signals the owner; the shard's final drop happens on the owner's thread, and a process exit that pre-empts it loses nothing acknowledged. +A shard directory stays locked while an owner holds it; a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. +The contract canary (ADR-I-0023) additionally pins the three engine facts this record rests on: the engine does not persist a write until its flush runs, a load does not replay the log, and a shard directory held by one owner refuses a second open until it is released. + +## Implementation Impact + +- The adapter owns a blocking thread and a request channel; every port method becomes a message to the owner. +- Each write costs one synchronous disk sync on the owner's thread; the in-phase benchmark's write burst measures it. +- The close-then-reopen test, the hard-exit test, and the responsiveness benchmark (construction and reopen including a lock-backoff wait, a scan, a write burst, a build, and a close) are the phase's evidence. + +## Considered Options + +1. A dedicated blocking owner that flushes after every write and acknowledges only then; signal-only facade drop. +2. Call the engine directly from the async paths and rely on the shard's drop for persistence. +3. A blocking owner with a signal-only drop and no per-write flush, documenting a weaker crash guarantee. +4. An explicit awaited close on the port or facade that flushes before returning. + +## Decision Outcome + +Chosen option: **Option 1**. +It is the only option that keeps executor threads free, makes every acknowledged write durable, and adds no API surface. + +### Rejected Alternatives + +Option 2 stalls the process on every scan, build, and flush, and was measured to lose every unflushed write on exit; rejected outright. +Option 3 makes the library's write acknowledgement a lie under crash or exit, and a character's lost memories are the cost; rejected outright. +Option 4 adds a close method every consumer must remember to call and still loses writes on any exit that skips it; the per-write flush makes it unnecessary; it is reopened only if the write-burst measurement shows the per-write flush dominating ingestion cost, in which case batched flushes behind an explicit acknowledgement are the shape, not a weaker guarantee. + +## Consequences + +- Positive: executor responsiveness is independent of corpus size and engine activity; acknowledged writes survive crashes and hard exits; no new API. +- Negative / tradeoffs: a synchronous disk sync per write; a serialised engine (one call at a time per adapter), which the candidate-recall role tolerates. + +## Decision Boundary + +Invariant: every engine call runs on the adapter's blocking owner; a write is acknowledged only after it is durable; the facade drop stays signal-only; a constructor waits for a locked directory rather than opening a second handle; the three engine facts are pinned by the canary. + +Not covered: the channel and thread mechanics, the backoff bound, and the batching of flushes behind an acknowledgement if measurement calls for it. + +## Validation + +- The hard-exit test writes, exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write. +- The close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write. +- The benchmark shows no engine call occupying an async executor thread and observes the shard's final drop on the owner's thread. +- The contract canary fails if the pinned engine starts replaying its log on load, persisting on write, or admitting a second open of a held shard directory. + +## Revisit When + +- The engine persists on write or replays its log on load (the canary fails in that direction) — the per-write flush becomes optional and this record is revised. +- The write-burst measurement shows the per-write flush dominating ingestion cost — batch flushes behind an explicit acknowledgement rather than weakening the durability rule. +- The engine's directory lock changes semantics (the canary fails in that direction) — the constructor's wait-for-release rule is re-derived before the pin moves. +- A multi-process deployment shape is designed — the single-owner lock discipline is reconsidered with the graph and statistics stores, never alone. + +## Consultation impact + +Question asked in review: whether the shard's drop-time flush is persistence or compaction; a process-level probe settled it as persistence, and the decider's rule "await the flush or document the weaker guarantee" was met by flushing per write. + +## More Information + +- ADR-I-0023 (the engine and the canary this record extends). +- The durability probe report of 2026-09-02 is a transient working artifact; the numbers above are its record. diff --git a/docs/design/database/vector_payload_design.md b/docs/design/database/vector_payload_design.md index 84b29d38..9c00ba95 100644 --- a/docs/design/database/vector_payload_design.md +++ b/docs/design/database/vector_payload_design.md @@ -1,5 +1,7 @@ # Vector Database Payload Design +> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 rules what any returning prefilter column must satisfy (synchronised or immutable, unknown never matches). The Record Shape, Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were (Record Shape still lists the graph URI, which the read contract dropped); the Design Goal (except its third rule, which named relationship and lifecycle hints the read contract dropped), Why Natural-Language Surfaces, and Consistency Model sections remain current. + This document describes the Qdrant payload design for Character Memory. It is intentionally a design note, not a field-by-field copy of the Rust mapping code. Qdrant is the semantic candidate index. It is not the memory database of record. The authoritative memory state lives in the graph store. A Qdrant hit means "this object may be relevant"; it does not mean "this object is current, related, or safe to include." Retrieval must verify candidates through the graph authority before returning them in a continuity context pack. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md new file mode 100644 index 00000000..cac763fa --- /dev/null +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -0,0 +1,208 @@ +# v0.1.6 Design: Embedded Vector Candidate Recall + +Status: decided 2026-09-02, revised 2026-09-03 (ADR-I-0023 through ADR-I-0027); supersedes the 2026-07 draft of this document; the embedded engine ruling (Qdrant Edge over an in-house scan) was taken the same day on two feasibility spikes. + +## Version intent + +Complete the zero-infrastructure local deployment story by adding an embedded vector candidate store mode behind the existing vector candidate port, and settle the port contract that both adapters must satisfy before a second adapter exists. +With graph authority defaulting to embedded persistent storage (ADR-I-0021) and retrieval statistics already file-backed (ADR-I-0009), the vector candidate store is the only component that still requires an external service. +That conflicts with the desktop-companion and game or simulation use cases, where end users cannot be expected to operate containers, and it keeps a service dependency in the default test path. +The embedded engine is chosen for where a character's memory ends up, not where it starts: a memory expected to accumulate for years to decades across several generations of embedding model needs approximate indexing, quantization, memory-mapped segments, and coexisting embedding spaces as baseline capabilities, so the in-process build of the service backend is adopted rather than an exhaustive scan the library would own (ADR-I-0023). + +Sequencing: this phase runs before scoped continuity, so the vector record is mirrored across two adapters while it is five fields, and so the scoped-continuity evaluation fixtures know which vector backend they validate against. + +## Why this is safe to do before scoped continuity + +The vector layer is candidate recall only: the vector store suggests, statistics guide fanout, and graph authority decides final inclusion. +An embedded adapter therefore has a low correctness bar for any single query: it must prefilter and rank candidates well, never be authoritative for anything. +The port is small (upsert, scoped search, delete), provider-neutral, and already exercised by deterministic fakes and a live smoke surface. +The write path already removes the vectors of superseded and suppressed objects, so the live vector population is the active population by construction, and stale residue from failed maintenance is caught by graph verification; the embedded adapter inherits both guarantees without new code. +The embedded engine is the same family as the service backend, so payload and filter conventions are shared rather than translated, and the spike measured identical id sets, identical order, and a score delta of 0.0 on five filtered queries against the live service. + +## Design direction + +### The port contract (ADR-I-0024, ADR-I-0025) + +This phase fixes the port contract deliberately, because two adapters cannot be held to an implicit one. + +Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. +A vector-layer predicate reads only immutable or synchronised stored values, and an unknown or missing value never satisfies a positive predicate (ADR-I-0024); a future predicate lands in both adapters with a parity fixture, as every port change does. + +Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (every scoped record was scored, so the requested top-K is determinate), boundary tie closed (an index returned a prefix and the cutoff cohort within that prefix was verified closed), or boundary tie open (the overfetch bound was reached with the cohort within the prefix still open). +The boundary verdicts describe the index's answer, not global recall: an approximate index may omit records it never surfaced, so only the exhaustive verdict asserts determinate membership over the scoped population, while the boundary verdicts assert that the returned set is deterministic for a given index state and whether its cutoff cohort was closed. +Both adapters answer through the shared tie-closure loop and the canonical constructor; before this phase the service adapter returned the truncated set silently at its bound, and the port's bare list type could not say whether even the returned prefix's cutoff was closed. +The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. +The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. + +Record: both adapters persist exactly five fields — object id, object type, surface, schema version, and the embedded text. +Read-out text lives in graph authority; the vector record stores only the embedded surface, as provenance of what was ranked; consumers needing candidate content hydrate by object id. +The relationship, lifecycle, time, ranking, object-specific, graph-URI, and raw-reference hints leave the write path: the library read none of them, the relationship hints were frozen at upsert and never updated by linking, the lifecycle hints described vectors the write path deletes, and the readable text column duplicated graph text. + +Two candidate predicates that satisfy that rule are noted so the scope-only query and five-field record are read as current state, not prohibition (binding on no phase): + +1. A synchronised scope predicate, owned by the scoped-continuity phase: a scope-id column written at upsert and kept in sync by the link and reflection write paths. +2. An immutable time-window predicate over `created_at` and `observed_at`, owned by whichever phase first ships a time-bounded retrieval route; immutability makes a write-time column correct without a sync path, and the columns are backfilled from graph authority if ever needed. + +### The embedded adapter (ADR-I-0023) + +The adapter runs the in-process build of the service backend (the `qdrant-edge` crate, pinned exactly; beta at adoption) as one engine shard per collection. + +Exactness is a threshold property: a shard below its configured indexing threshold answers by exhaustive scan, a shard above it answers from its index, and the completeness verdict reports the boundary state either way. +This phase ships the threshold at its exact-scan setting (the spike confirmed that a zero threshold leaves a shard unindexed while a threshold of one plus an optimise call builds the index); index construction, quantization, and memory-mapped segments are available capabilities whose defaults are tuned in a later measured decision, never silently. + +Shard: cosine distance at the configured vector size; the five-field payload with keyword indexes on object id (the delete selector removes every surface of an object by it) and on object type (the scope predicate); the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). +Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. +Verdict mapping: exhaustive when the shard is unindexed (known from the adapter's own threshold configuration) and the shared loop closed the cutoff cohort, with the scanned count taken from a filtered count of the scope rather than from the rows returned, so every exact scan with a closed cutoff reports exhaustive regardless of population size; boundary tie closed when an indexed shard's cutoff cohort closed (a statement about the index's returned prefix, not about global recall); boundary tie open whenever the bound is reached with the cohort still open, on an exact scan too; not requested at limit zero or empty scope. +Delete: remove every surface of each object id, matching the service adapter's selector. +Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. +Blocking discipline and durability (ADR-I-0027): every engine call — shard open and load with the lock backoff, payload index creation, upsert and delete, search, the filtered scope count behind the exhaustive verdict, index build, and the final drop — runs on a dedicated blocking owner the adapter creates at construction and that opens the shard itself, so the async composition entry point never touches the engine; the owner acknowledges a write only after the engine's flush, because the pinned engine persists only on flush and does not replay its log on load (writers that skipped the drop reopened with zero of two hundred points); the facade drop only signals the owner and no port or facade method is added. +A shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff; the two engine facts are pinned by the contract canary and proven by the close-then-reopen test and the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write). +The in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. +Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. +Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance, and the three facts ADR-I-0027 rests on: no persistence before flush, no log replay on load, and the directory lock — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. +Score parity across adapters was measured at 0.0 delta on the spike; the parity suite still asserts it with non-unit query and record vectors rather than assuming it. + +### Settings and composition (ADR-I-0023) + +Follow the one-key-per-backend pattern the graph and statistics stores already use rather than overloading the service connection string. + +```text +VECTOR_STORE_MODE embedded | service (default: embedded) +VECTOR_STORE_PATH directory, required in embedded mode (missing is a configuration error, never an implicit default), ignored in service mode +QDRANT_CONNECTION_STRING required only in service mode +``` + +`VECTOR_STORE_PATH` is a directory; each collection is one engine shard directory inside it named by the collection name the public constructor already takes, so `collection_name` is the backend-neutral namespace key in both modes. +Because the collection name becomes a directory name under the configured path, embedded mode validates it at construction with a contract owned here: lowercase ASCII letters, digits, underscore, and hyphen only (lowercase so that names stay unique on the case-insensitive filesystems of desktop targets), first character a letter or digit, at most 128 characters, not a reserved device name on Windows (con, prn, aux, nul, com1 to com9, lpt1 to lpt9), no path separators, dots, or empty name; anything else is rejected with the configuration error before any directory is touched, the name and the record schema version are recorded in an adapter-owned marker inside the shard directory, and a path-confinement test proves that separator and parent-directory inputs cannot escape the directory. +The engine requires the shard root directory to exist before opening; the adapter creates it, as the statistics store creates its parent directory. +The composition root gains a vector-store mode switch mirroring the statistics-store switch; the vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds. +The toolchain pin moves to the engine's minimum (Rust 1.97 at adoption; the previous pin did not compile it), in its own change before the adapter lands. + +### Parity suite placement (ADR-I-0023) + +Library: a port-conformance suite in the integration tests — scope filtering, empty scope selects zero, canonical order, identical-vector tie cohort closed through the shared loop, best-score-per-object-and-surface deduplication, delete removes all surfaces, restart reopen, completeness verdict per adapter — run against the embedded adapter unconditionally and against the service adapter when a service connection is configured. +Parity acceptance is identical admitted sets and orderings while both adapters are below their indexing thresholds (the suite's fixtures are small enough that both are), and a recorded recall comparison of the embedded adapter above its threshold against its exhaustive setting, informational this phase. +This follows the precedent that port conformance is enforced by contract tests run against every adapter, not by a runtime wrapper. +Evaluation repository: no second contract suite; it adds an embedded-mode configuration to the continuity scenarios and requires identical scenario results between modes, the behaviour-level regression instrument. + +## Deliverables + +```text +port contract: completeness envelope, scope-only query with empty-scope-selects-zero, retrieval telemetry completeness field +vector record read contract: five-field manifest shared by both adapters +QdrantEdgeVectorCandidateStore adapter: shard per collection at the exact-scan threshold, upsert/delete, scoped search through the shared tie-closure loop, verdict mapping, restart validation, blocking discipline +engine contract canary test, re-run on every engine bump +VectorStoreMode and VectorStorePath settings; composition mode switch; service connection string required only in service mode; toolchain pin at the engine minimum +port-conformance parity suite in the library integration tests, run against both adapters; above-threshold recall comparison recorded +restart-safety test for the embedded store; pipeline test over the embedded adapter with a deleted graph object +dependency-weight report: unstripped and stripped release deltas, effect of feature trimming +latency benchmark: exhaustive scan across corpus sizes at the configured dimension; executor responsiveness under a concurrent scan +documentation: settings, single-process expectation, threshold semantics, measured latency guidance, rebuild-from-graph-authority as the path between modes +five implementation ADRs (ADR-I-0023 through ADR-I-0027) with reciprocal partial-supersession frontmatter on ADR-I-0001, ADR-I-0002, ADR-I-0003, and ADR-I-0005 +``` + +Deletions that are deliverables, not side effects: + +```text +the hint carriers on the vector record type and the surface builders' hint population +the readable text column and the per-field payload index creation for dropped fields +the test-only payload field constants and the prose-assertion note constant +the service adapter's private enum token mappers, replaced by one Display/FromStr per enum in the domain (the embedded adapter must not add another copy) +the deterministic vector fake and its embedding-bearing record type, replaced by the embedded adapter opened on a temporary directory (failure-injecting and recording fakes stay) +the port doc comment's "documented bounded-overfetch degradation policy" clause, now expressed by the type +``` + +## Non-goals + +```text +changing the authority split, or any retrieval semantics for non-empty scopes and non-degenerate queries (the empty-scope and zero-norm rules in ADR-I-0024 are intended and in scope) +deprecating the service adapter, or altering it beyond what the shared port and record contracts require +tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold, tuned by a later measured decision) +named-vector coexistence of two embedding spaces (an engine capability this decision was taken for; its use lands with the first embedding-model migration) +migration tooling between modes or between record shapes (rebuild-from-graph-authority is the documented path) +multi-process access to the embedded store (same single-process expectation as embedded graph storage) +synchronisation between an embedded shard and a service collection (an engine capability; not exercised this phase) +any vector-layer predicate beyond the object-type scope (the candidate predicates noted in ADR-I-0024 belong to later phases) +any new public facade method (the evaluation baseline consumes the retrieval trace) +reconciliation diagnostics (the reconciliation slice was deleted in the structured-verdict phase; graph verification is the guard) +``` + +## Technology posture + +- The in-process build of the service backend (Qdrant Edge, pinned exactly at 0.8.0, beta) is the embedded engine: measured on the spike as lifecycle-and-reopen identical, exactness controllable by the indexing threshold, five-of-five service parity at score delta 0.0, at a cost of 489 additional dependency-tree lines and about 30.5 MB of unstripped release binary (stripped size and feature trimming are measured in-phase), requiring Rust 1.97 or later. +- The decade standard behind the choice: at continuous-accumulation scale the recall index plausibly reaches hundreds of thousands to a million vectors, where exhaustive scan is seconds per query, so approximate indexing, quantization, and memory-mapped segments are the baseline for an interactive character; embedding models will change several times, so named vectors that let two spaces coexist during lazy re-embedding matter; the store is a rebuildable cache over graph authority, which bounds but does not eliminate the cost of an engine switch. +- Library over in-house: the library owns the port contract, the tie-closure loop, the canonical ordering, the verdict mapping, and the error classification, and holds any engine to them; it does not own distance computation, storage layout, or index construction. +- Rejected on the same probe set: an in-house SQLite exact scan (violates library-over-in-house and would re-implement every capability above later); the SQLite vector extension sqlite-vec 0.1.9 (builds on both toolchains, 19 tree lines, about 6.2 MB unstripped, deterministic ties across fresh files, parity five of five at delta at or below 1.6e-7, but exhaustive-only in its stable release with approximate indexing only in a pre-release, a pre-1.0 binding, and a single maintainer); LanceDB (weight, without a spike, since the chosen engine covers its capabilities); an in-memory-only store (no restart safety). +- Beta risk is handled structurally: exact pin, contract canary, parity re-run on every bump; the engine leaving beta or changing its API, a stripped-footprint measurement that changes the weight picture, or a corpus benchmark contradicting the interactive-latency assumption each reopen ADR-I-0023. +- Deployments that outgrow the embedded mode are exactly the deployments that should use the service mode; the engine's synchronisation to a service collection is the recorded portability path from a local character to a hosted one. + +## Acceptance criteria + +```text +Embedded mode is configurable and constructs without any running service. +The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop. +A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded. +Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). +Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. +Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. +ADR-I-0027 holds: no engine call occupies an async executor thread and every acknowledged write survives a hard exit, shown by the hard-exit test, the close-then-reopen test, and the responsiveness benchmark during construction and reopen (including a lock-backoff wait), a concurrent scan, a write burst, a build, and a close. +A zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero with a truthful verdict, both proven in both adapters by parity fixtures. +The engine contract canary passes on the pinned version. +The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. +The default test path requires no vector service; service-gated suites continue to pass unchanged. +Both adapters persist exactly the five-field read contract; a census of both repositories shows no reader of a dropped field. +Documentation states the single-process expectation, the threshold semantics, the latency guidance, and the rebuild-from-authority path. +No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behaviour change in service mode for non-empty scopes and non-degenerate queries (an empty object-type scope now selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary, both intended). +``` + +## What the evaluation repository provides and when it is used (ADR-I-0026) + +The companion evaluation repository is a development aid; its own work is planned and tracked there, and this document records only what its measurements let this phase decide. + +- The library exposes, through an ordinary traced retrieval, everything a raw-vector baseline needs: the vector candidates with scores and the completeness verdict in telemetry; the honest way to use them is one singleton-scoped traced retrieval per measured object kind with a limit of the section budget multiplied by the maximum surfaces per object, deduplicated by object. +- The cross-mode comparison (service mode against embedded mode on the continuity suite, identical baselines expected under the parity contract) is closeout evidence and a revisit trigger for the embedded default recorded in ADR-I-0023 (a difference between modes reopens it); it is consumed at the closeout task, not produced by this plan. +- No candidate-search facade or configuration surface is added for the evaluation repository; the one public addition made for its trace reading is the published maximum-surfaces-per-object-kind policy value (ADR-I-0026), and if its measurements ever require more, that is a library decision taken on its own record. + +## Evaluation tie-in + +The evaluation repository is expected to run its continuity suite in both vector modes; identical scenario results are what the parity contract predicts, and a difference between modes is a revisit trigger for the embedded default (ADR-I-0023). +How that configuration is built and run is planned in the evaluation repository; this phase consumes the comparison at closeout and cites nothing else from it. + +## Deferral-reconfirmation checklist + +Each item was parked on this phase by the structured-verdict phase; each row states the parked claim, what was re-verified at design time, and the evidence the implementation must produce. + +1. Canonical-candidates newtype survival. + Parked claim: the newtype survives the port redesign or is absorbed into its result envelope. + Re-verified: every consumer is a slice read (the pipeline's count, telemetry, trace, and root selection sites, plus the test fakes); none relies on the newtype being the whole return value. + Evidence: after the change a census shows only the envelope field, the constructor, and the fakes' exhaustive wrapping; the existing deduplication-and-ordering test is unchanged. +2. Dual text columns. + Parked claim: the text columns' fate depends on the port's read contract. + Re-verified: the readable text column had exactly one reader (the evaluation baseline) and the embedded text column none; the evaluation repository can source item text from its own ingest. + Evidence: zero-hit census for the readable text column across both repositories; a vector-only run before and after produces identical item identities and text. +3. Search completeness. + Parked claim: the port cannot express whether the top-K was determinate. + Re-verified: the only degradation site is the service adapter's fetch bound; no pipeline path inspects or retries on it; the embedded engine's own tie order is not stable across fresh shards, which makes the shared loop a requirement for both adapters rather than a service-only workaround. + Evidence: the fetch-decision unit test asserts the open verdict at the bound; a retrieval test asserts the telemetry field per variant; the live boundary test asserts the closed verdict; the parity tie fixture asserts exhaustive versus closed through the same loop. +4. Hint filter semantics. + Parked claim: query-side hint semantics belong to the port contract. + Re-verified: the filter type and both match-or-unknown implementations were deleted in the structured-verdict phase, and no consumer asks for a vector-layer predicate (the evaluation surface policy carries object types and budgets only). + Evidence: zero-hit census for the filter type and for empty-or-null match conditions in the service adapter; the prefilter rule and candidate predicates are recorded in ADR-I-0024. +5. Evaluation baseline capability. + Parked claim: the baseline re-implements a hidden raw-vector capability against the payload schema. + Re-verified: one singleton-scoped traced retrieval per measured kind reproduces the direct per-kind search exactly, which a sliced mixed-kind top-K would not; each retrieval's completeness verdict reports whether that kind's top-K was determinate over the scoped population (exhaustive) or only closed within the index's returned prefix (boundary verdicts), and the baseline records which; the evaluation adapter can hold item text from ingest. + Evidence: the A/B run with row-level diff of item identities and ranks; after the switch, zero-hit census for vector-service search calls and payload constants in the evaluation adapter. + +## Decisions (the draft's open questions, resolved 2026-09-02) + +- Embedded engine: the in-process build of the service backend (Qdrant Edge), on the decade-scale portability standard and the two spikes' measurements; the in-house exact scan and the SQLite vector extension are rejected alternatives (ADR-I-0023). +- Exactness: a threshold property shipped at the exact-scan setting; index, quantization, and memory-map tuning is a later measured decision (ADR-I-0023). +- Default mode: embedded from this phase, licensed by the phase's own parity suite and service-free integration path under the defaults-match-evidence rule (ADR-I-0021); the evaluation repository's cross-mode run is a revisit trigger, not a gate (ADR-I-0023). +- Engine discipline: every engine call on a dedicated blocking owner; a write acknowledged only after the engine's flush; signal-only facade drop (ADR-I-0027). +- Latency guidance: measured in-phase by a benchmark over a synthetic corpus at the configured dimension, published in documentation, revised through documentation. +- Dependency weight: the unstripped delta is recorded; the stripped delta and feature trimming are measured in-phase, and a material change reopens ADR-I-0023. +- Parity suite placement: contract parity in the library, behaviour parity in the evaluation repository (above). +- Settings shape: separate mode and path keys with `collection_name` as the backend-neutral namespace key naming one shard directory per collection, not a connection string interpreted by mode (ADR-I-0023). +- Hint families: all dropped from the vector record, with the two candidate predicates noted (ADR-I-0024, ADR-I-0025). +- Text columns: readable text dropped, embedded text kept as provenance, governed by the three sentences in ADR-I-0025. +- Evaluation baseline: trace-sourced, no facade change (ADR-I-0026). diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index 4badbc91..00525f49 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -259,6 +259,7 @@ Assisted remember workflows may accept raw or semi-raw input as transient proces | v0.1.3 | Remember intake interfaces and deterministic write planning | Finished. Generation-ready write path with `RememberWritePlan`, memory candidates, validation, deterministic helpers, prepare/validate/commit flow, and shared manual/future-generated commit machinery. | | v0.1.4 | Continuity evaluation harness | Finished. Deterministic long-horizon evaluation harness implemented in the public companion `CharacterMemoryEvals` repository as a development aid, not core library functionality: synthetic interaction fixtures, a minimal example assistant loop, continuity-oriented retrieval-quality metrics, selectivity/fanout measurement, and hub-entity stress scenarios. | | v0.1.5 | Eval-driven v0.1 family closeout | Finished. Ran the evaluation harness across the v0.1 family, dispositioned eleven findings (none critical, none open), fixed deterministic vector admission and write-path warning diagnostics in the library, retained the measured defaults with a recorded basis (ADR-I-0022), adopted embedded persistent Oxigraph as the validated default (ADR-I-0021), and expanded the evaluation suite to 33 scenarios including benchmark-adapted and real-embedding fixtures. Closeout report: [`v0_1_5_closeout_report.md`](v0_1_5_closeout_report.md). | +| v0.1.6 | Embedded vector candidate recall | Planned. An embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) behind the vector port as the default vector mode, shipped at its exact-scan indexing threshold, so zero-infrastructure local deployments are the default and the default test path needs no external service, with the service adapter retained as the explicit service mode; a redesigned port contract that reports recall completeness, forbids prefilters that match unknown values, and stores a five-field record (identity, surface, schema version, and the embedded text kept as provenance); the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0027. | | v0.2 | Scoped continuity and reflection | `ContinuityScope`, scoped reflection, relationship state between arbitrary entities, character signals for continuing entities, open-loop/commitment lifecycle, and current continuity views. | | v0.3 | Factual rigor, temporal validity, and entity evolution | Assertions, claims, evidence links, belief assessments, source assessment, temporal validity, entity drift handling, and current-belief views. | | v0.4 | Retrieval observability and governance | Retrieval traces, context subgraphs, validation rules, graph health reports, policy diagnostics, rejected expansion traces, cluster/activation diagnostics, and retention assessment. | @@ -1101,6 +1102,7 @@ The intended sequence is: v0.1.3 completes the generation-ready write path v0.1.4 builds the harness that exercises the full write and retrieve paths v0.1.5 runs the harness, fixes what it reveals, and closes the v0.1 family +v0.1.6 makes zero-infrastructure local deployment possible and removes the service dependency from the default test path then v0.2 builds scoped continuity on a measured substrate ``` @@ -1194,7 +1196,56 @@ v0.2 entry is explicitly confirmed against the closed v0.1 family. --- -# 12. v0.2: scoped continuity and reflection +# 12. v0.1.6: embedded vector candidate recall + +Detailed draft: [`v0_1_6_embedded_vector_candidate_recall.md`](../design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md) + +Decisions: ADR-I-0023 (embedded Qdrant Edge is the default vector candidate store), ADR-I-0024 (vector candidate recall reports completeness and prefilters never match unknown values), ADR-I-0025 (the vector record is a read contract), ADR-I-0026 (raw vector baselines read the retrieval trace), ADR-I-0027 (the embedded engine runs on a blocking owner that flushes every write). + +## Intent + +Complete the zero-infrastructure local deployment story. +Graph authority already defaults to embedded persistent storage and retrieval statistics are file-backed; the vector candidate store is the only component that still requires an external service, which conflicts with desktop-companion and game deployments and keeps a service dependency in the default test path. +Because a second adapter must implement the vector port, this phase also settles the port contract that the structured-verdict work deferred: recall completeness is reported instead of silently degraded, the query is scope-only, and the stored payload is exactly what a reader consumes. + +## Goals + +```text +add an embedded vector candidate store on the in-process build of the service backend behind the existing vector port, selected by a store-mode setting with its own path setting, shipped at its exact-scan indexing threshold, and made the default vector mode on the phase's own parity evidence +make the port result carry a typed completeness verdict that the retrieval telemetry records and never repairs +reduce the vector payload to its read contract: identity, surface, schema version, and the embedded text as provenance of what was ranked +run one shared contract suite against both adapters, with the embedded adapter exercised unconditionally so the default test path needs no service +move the evaluation repository's vector-only baseline onto the retrieval trace so no consumer depends on a store's private schema +rule that a vector-layer predicate reads only synchronized or immutable values and never matches an unknown one, noting the candidate predicates a later phase may need +``` + +## Non-goals + +```text +changing the authority split, or any retrieval semantics in the service mode for non-empty scopes and non-degenerate queries (the intended empty-scope and zero-norm rules are in scope) +deprecating or altering the service-mode adapter beyond the shared port contract +tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold this phase) +migration tooling between modes; rebuild from graph authority is the path +multi-process access to the embedded store +a public candidate-search facade +``` + +## Acceptance criteria + +```text +Embedded mode constructs and serves retrieval without any running service. +The shared contract suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds; above them a recall comparison is recorded. +Deterministic admission holds in embedded mode; repeated runs are byte-identical. +Embedded state survives process restart. +Retrieval telemetry reports the completeness verdict for every retrieval in both modes. +The default test path requires no vector service; service-gated suites still execute under the service-backed CI job and cannot pass by skipping. +The evaluation repository's vector-only baseline produces its rows from the retrieval trace in both modes. +No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behavior change in service mode for non-empty scopes and non-degenerate queries (the intended empty-scope change: zero candidates, and boundary rejection of an empty configured scope). +``` + +--- + +# 13. v0.2: scoped continuity and reflection Detailed draft: [`v0_2_scoped_continuity_reflection.md`](../design/roadmap-phases/v0_2_scoped_continuity_reflection.md) @@ -1242,7 +1293,7 @@ Open loops and commitments can be retrieved by scope without assuming who the ma --- -# 13. v0.3: factual rigor, temporal validity, and entity evolution +# 14. v0.3: factual rigor, temporal validity, and entity evolution Detailed draft: [`v0_3_factual_rigor_temporal_validity_entity_evolution.md`](../design/roadmap-phases/v0_3_factual_rigor_temporal_validity_entity_evolution.md) @@ -1273,7 +1324,7 @@ This is important, but it should not block the starter because Character Memory' --- -# 14. v0.4: retrieval observability and governance +# 15. v0.4: retrieval observability and governance Detailed draft: [`v0_4_retrieval_observability_governance.md`](../design/roadmap-phases/v0_4_retrieval_observability_governance.md) @@ -1355,7 +1406,7 @@ The default intent is `Continuity`. --- -# 15. v0.5: controlled associative recall and clustering +# 16. v0.5: controlled associative recall and clustering Detailed draft: [`v0_5_controlled_associative_recall_clustering.md`](../design/roadmap-phases/v0_5_controlled_associative_recall_clustering.md) @@ -1417,7 +1468,7 @@ Durable graph truth is the associative unit, membership lifecycle, and support e --- -# 16. v0.6: assisted remember workflow and memory candidate generation +# 17. v0.6: assisted remember workflow and memory candidate generation Detailed draft: [`v0_6_assisted_remember_workflow_memory_candidate_generation.md`](../design/roadmap-phases/v0_6_assisted_remember_workflow_memory_candidate_generation.md) @@ -1511,7 +1562,7 @@ Generated candidates use the same validation and commit path as manual candidate --- -# 17. v1.0+: multimodal and embodied expansion +# 18. v1.0+: multimodal and embodied expansion Detailed draft: [`v1_0_multimodal_embodied_expansion.md`](../design/roadmap-phases/v1_0_multimodal_embodied_expansion.md) @@ -1539,7 +1590,7 @@ This is a future path, not starter scope. --- -# 18. Public API evolution +# 19. Public API evolution ## v0.1 API @@ -1687,7 +1738,7 @@ Generated processors should produce `MemoryCandidate` and `RememberWritePlan` va --- -# 19. YAGNI rules +# 20. YAGNI rules Do not implement in v0.1 / v0.1.2: