From 92856c37a117dfbf1359028da162ad6dbdc5f33d Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 13:52:05 +0900 Subject: [PATCH 01/44] =?UTF-8?q?=F0=9F=93=9D=20Carry=20the=20v0.1.6=20emb?= =?UTF-8?q?edded=20vector=20recall=20draft=20onto=20the=20planning=20branc?= =?UTF-8?q?h?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- ...v0_1_6_embedded_vector_candidate_recall.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md new file mode 100644 index 00000000..2747a87b --- /dev/null +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -0,0 +1,78 @@ +# v0.1.6 Design Draft: Embedded Vector Candidate Recall + +## Version intent + +Complete the zero-infrastructure local deployment story by adding an embedded vector candidate store mode behind the existing vector port. +With graph authority defaulting to embedded persistent storage and retrieval statistics already file-backed, the vector candidate store is the only component that still requires an external service. +That conflicts with the desktop-companion and game/simulation use cases, where end users cannot be expected to operate containers, and it keeps a service dependency in the default test path. + +Sequencing: this phase runs before v0.2, so the payload surface is mirrored across two adapters while it is still small, and so v0.2 fixture work knows which vector backend it validates against. + +## Why this is safe to do now + +The vector layer is candidate recall only: Qdrant suggests, statistics guide fanout, and graph authority decides final inclusion. +An embedded adapter therefore has a low correctness bar — it must prefilter and rank candidates well, not be authoritative for anything. +The port is small (upsert, filtered search, diagnostics listing, delete), provider-neutral, and already exercised by deterministic fakes and a live parity surface. + +## Design direction + +- Add a `VectorStoreMode` setting (`service` | `embedded`) mirroring the graph store mode pattern, with the vector connection string interpreted as URL or local path accordingly. +- First embedded implementation: a SQLite-backed exact-scan adapter. + The port's filter contract (object types, retention states, currentness, entity/thread/episode ID lists, time ranges) maps natively onto SQL predicates with junction tables for the ID lists; after prefiltering, exact cosine scan over the survivors. + At character-memory scale (tens of thousands of vectors), exact scan is honest, fast enough, deterministic, and strictly better recall than approximate search. +- The Qdrant adapter remains fully supported as the service/cloud mode; this phase adds a mode, it does not deprecate one. +- The canonical candidate ordering contract (score, object type rank, object ID, surface rank) established for deterministic admission applies identically to the embedded adapter. +- Parity is the acceptance instrument: one shared filter-contract fixture suite runs against both adapters and must produce identical admitted sets; the embedded adapter runs it unconditionally (no service gating), which also removes the vector-service dependency from the default test path. + +## Deliverables + +```text +VectorStoreMode setting and configuration interpretation +SqliteVectorCandidateStore adapter (schema, upsert/delete, filtered exact-scan search, diagnostics) +composition wiring and mode selection +shared filter-contract parity suite exercised by both adapters +restart-safety and reconciliation coverage for embedded mode +documentation: payload mapping addendum, setup, corpus-size guidance +an implementation ADR recording the technology selection and its revisit triggers +``` + +## Non-goals + +```text +changing the authority split or any retrieval semantics +deprecating or altering the Qdrant adapter +approximate-nearest-neighbor indexing (LanceDB is the recorded escalation path if embedded ANN ever becomes necessary) +migration tooling between modes (rebuild-from-graph-authority is the documented path) +changing the default vector mode in this phase (embedded ships opt-in first; flipping the default is a separate decision once parity evidence exists) +multi-process access to the embedded store (same single-process expectation as embedded graph storage) +``` + +## Technology posture (from the v0.1.5 closeout analysis) + +- SQLite exact-scan first: zero heavyweight dependencies (`rusqlite` direction already exists via the statistics store), exact filter semantics, deterministic, restart-safe. +- LanceDB recorded as the embedded-ANN escalation path if corpora outgrow exact scan. +- The in-process edge build of the current vector backend is a revisit candidate once it stabilizes; it would maximize payload-convention reuse and add a cloud-sync story. +- Deployments that outgrow the embedded mode are exactly the deployments that should use the service mode; document a corpus-size guidance number rather than engineering for it. + +## Acceptance criteria + +```text +Embedded mode is configurable and constructs without any running service. +The shared parity suite produces identical admitted candidate sets from both adapters across the full filter contract. +Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical). +Embedded state survives process restart; reconciliation diagnostics work against the embedded store. +The default test path requires no vector service; Qdrant-gated suites continue to pass unchanged. +Documentation states the single-process expectation, the corpus-size guidance, and the rebuild-from-authority migration path. +No public facade change; no retrieval behavior change in service mode. +``` + +## Evaluation tie-in + +The continuity evaluation suite gains an embedded-mode configuration so the confirmation scenarios (including restart) run against the embedded vector store; the frozen-embedding infrastructure applies unchanged. +Scenario baselines are expected to be identical between modes under the parity contract; any divergence is a finding, which makes the eval suite the cross-adapter regression instrument. + +## Open questions + +- Should embedded become the default vector mode once parity evidence exists, matching the embedded-default graph decision, or stay opt-in until a full release cycle passes? +- What corpus-size number goes in the guidance (the closeout analysis suggested exact-scan comfort up to low hundreds of thousands of vectors; measure rather than assume)? +- Does the parity suite live in the library's integration tests, the evaluation repository, or both (recommendation: shared fixtures in the library, evaluation reuse where cheap)? From 9828e03af933a77daf67641aec37360e62644e54 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 17:34:10 +0900 Subject: [PATCH 02/44] =?UTF-8?q?=F0=9F=93=9D=20v0.1.6=20embedded=20vector?= =?UTF-8?q?=20recall:=20decisions=20ADR-I-0023..0026,=20phase=20document?= =?UTF-8?q?=20rewrite,=20roadmap=20section,=20harness=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four implementation ADRs (embedded exact-scan vector store as the opt-in local mode; recall reports completeness and takes a scope-only query; the vector record is a five-field read contract; raw vector baselines read the retrieval trace) with reciprocal partial-supersession frontmatter on ADR-I-0002/0003/0005; the phase document rewritten against the as-built port; roadmap version row, sequencing line, and section 12 with later sections renumbered; the execution plan in draft awaiting approval. Co-Authored-By: Claude Fable 5.1 --- .../v0-1-6-embedded-vector-recall-plan.md | 273 ++++++++++++++++++ ...002-natural-language-embedding-surfaces.md | 4 +- .../ADR-I-0003-qdrant-oxigraph-defaults.md | 4 +- ...-0005-qdrant-payload-vs-graph-authority.md | 4 +- ...qlite-exact-scan-vector-candidate-store.md | 115 ++++++++ ...mpleteness-and-takes-a-scope-only-query.md | 124 ++++++++ ...I-0025-vector-record-is-a-read-contract.md | 113 ++++++++ ...ctor-baselines-read-the-retrieval-trace.md | 111 +++++++ docs/design/database/vector_payload_design.md | 2 + ...v0_1_6_embedded_vector_candidate_recall.md | 195 ++++++++++--- docs/roadmap/development_roadmap.md | 68 ++++- 11 files changed, 955 insertions(+), 58 deletions(-) create mode 100644 docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md create mode 100644 docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md create mode 100644 docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md create mode 100644 docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md create mode 100644 docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md new file mode 100644 index 00000000..f1a910de --- /dev/null +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -0,0 +1,273 @@ +# Plan: v0.1.6 Embedded Vector Candidate Recall + +- status: draft +- generated: 2026-09-02 +- last_updated: 2026-09-02 +- work_type: mixed + +## Goal +- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0026: a redesigned vector port contract, a five-field vector record, an embedded SQLite exact-scan vector candidate store as the opt-in local mode, a shared contract suite over both adapters, and the evaluation repository's vector-only baseline moved onto the retrieval trace. + +## Definition of Done +- Every acceptance criterion in the phase document's "Acceptance criteria" section holds with recorded evidence. +- Every row of the phase document's deferral-reconfirmation checklist has its evidence produced and cited in the Progress Log. +- Every deletion listed under "Deletions that are deliverables" is gone, with a zero-hit census. +- Both repositories' service-gated suites execute (not skip) under the service-backed CI job. +- One PR per repository per wave, merged by the decider; the evaluation repository's obligations in ADR-I-0026 are all landed. + +## Scope / Non-goals +- Scope: the phase document's deliverables and deletions; the evaluation repository obligations in ADR-I-0026. +- Non-goals: the phase document's non-goals (no default flip, no approximate index, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode). + +## Context (workspace) +- Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. +- As-built port: `src/ports/vector_candidate.rs`, `src/models/vector/candidate_record.rs`, `src/models/vector/record.rs`, `src/adapters/qdrant/{store,payload}.rs`, `src/policy/embedding_surface.rs`, `src/usecases/retrieve.rs`, `src/api/types/retrieval.rs`, `src/composition.rs`, `src/config/app_settings.rs`, `src/test_support.rs`. +- Prerequisite landed separately: the evaluation repository's evidence-integrity light-delta (batch outcome duplication, harness-invented rank, unhonored manifest/hash knobs, shared graph-path fallback, live-skip panic switch), branch `chore/evidence-integrity-pre-v016`. +- Repo reference docs consulted: the four ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. + +## Open Questions (max 3) +- none (the draft's five open questions were ruled by the decider on 2026-09-02 and are recorded in the phase document and the ADRs). + +## Assumptions +- A1: `rusqlite` (already a dependency for the statistics store) is sufficient for the embedded adapter; no vector extension is added. +- A2: The evaluation repository's row/summary schema move for the typed backend identity follows that repository's normal clean-schema procedure and is owned by its wave task. + +## Tasks + +### Task_1: Live-gate hardening in the library test suite +- type: test +- owns: + - tests/support/base.rs + - tests/write_planning_tests.rs + - tests/initialization_tests.rs + - tests/public_facade_tests.rs + - tests/retrieval_guardrails_tests.rs + - .github/workflows/*.yml +- depends_on: [] +- description: | + Add one environment switch honored by the shared test support that turns every service-unavailable skip into a panic, set it in the CI job that provisions the vector service, and delete the prose-matched timeout skip (`is_qdrant_timeout_signature`) or replace it with a typed match on the existing transport classification. +- acceptance: + - No test passes by skipping when the switch is set; the CI service-backed job sets it. + - No test gates on error prose. +- validation: + - kind: command + required: true + owner: worker + detail: "service-up cargo test with the switch set: all nine former skip sites execute; service-down with the switch set: the suites fail, not pass" + - kind: review + required: true + owner: reviewer + detail: "Diff review; confirm the CI job sets the switch" + +### Task_2: Port contract: completeness envelope and scope-only query (ADR-I-0024) +- type: impl +- owns: + - src/ports/vector_candidate.rs + - src/models/vector/candidate_record.rs + - src/api/types/retrieval.rs + - src/adapters/qdrant/store.rs + - src/usecases/retrieve.rs + - src/usecases/remember.rs + - src/usecases/correct_forget.rs + - src/memory.rs + - src/test_support.rs + - src/adapters/oxigraph/tests.rs +- depends_on: [] +- description: | + Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; make the service adapter map its fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. +- acceptance: + - The envelope and enum match ADR-I-0024's Decision section; the canonical-candidates newtype is unchanged. + - Telemetry carries the verdict for every retrieval; a retrieval test asserts each variant. + - Fetch-decision unit tests assert closed and open verdicts including the all-tied cohort at the bound. + - Zero-hit census: no match-or-unknown condition, no filter type beyond object-type scope. +- validation: + - kind: command + required: true + owner: worker + detail: "cargo fmt --check; cargo clippy --all-targets -- -D warnings; service-up cargo test; ignored qdrant_ lib tests" + - kind: review + required: true + owner: reviewer + detail: "Diff review vs ADR-I-0024; confirm no pipeline path inspects the verdict for control flow" + +### Task_3: Vector record read contract (ADR-I-0025) +- type: impl +- owns: + - src/models/vector/record.rs + - src/adapters/qdrant/payload.rs + - src/adapters/qdrant/store.rs + - src/policy/embedding_surface.rs + - src/domain.rs + - src/usecases/retrieve.rs + - src/usecases/vector_indexing.rs + - docs/design/database/vector_payload_design.md +- depends_on: [Task_2] +- description: | + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Existing stored payloads with extra fields are tolerated unread (schema version unchanged unless the reader contract changes). +- acceptance: + - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. + - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository's reader is removed by Task_5). + - One token mapping per enum; census shows no copy in adapters or use cases. +- validation: + - kind: command + required: true + owner: worker + detail: "cargo fmt --check; clippy -D warnings; service-up cargo test; ignored qdrant_ lib tests; census commands recorded" + - kind: review + required: true + owner: reviewer + detail: "Diff review vs ADR-I-0025; verify the payload design note's supersession note matches what landed" + +### Task_4: Embedded SQLite vector candidate store, settings, and parity suite (ADR-I-0023) +- type: impl +- owns: + - src/adapters/sqlite_vector/** + - src/adapters.rs + - src/composition.rs + - src/config/app_settings.rs + - src/errors.rs + - tests/vector_port_contract_tests.rs + - tests/support/** + - .env.example + - README.md + - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +- depends_on: [Task_3] +- description: | + Implement the embedded adapter per the phase document (schema keyed on object id and surface, normalised vector blobs, object-type scope predicate, exact scan returning Exhaustive, restart safety), the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), composition mode switch with `collection_name` as the backend-neutral namespace key, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Measure and document corpus-size guidance from an in-phase benchmark. +- acceptance: + - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets on the shared fixtures; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service). + - Restart test passes; repeated runs are byte-identical. + - Settings docs, single-process expectation, corpus-size guidance, and rebuild-from-graph-authority path are documented. +- validation: + - kind: command + required: true + owner: worker + detail: "cargo test with no service (parity suite + embedded tests execute); service-up cargo test with the live switch set; benchmark numbers recorded in the report" + - kind: review + required: true + owner: reviewer + detail: "Diff review vs ADR-I-0023; independent service-free and service-up runs" + +### Task_5: Evaluation repository: trace-sourced vector-only baseline and telemetry mirror (ADR-I-0026) +- type: impl +- owns: + - crates/cmem-eval-adapter-cmem/src/lib.rs + - crates/cmem-eval-adapter-cmem/Cargo.toml + - crates/cmem-eval-core/src/{config,runtime,results,verdict,metrics}.rs + - crates/cmem-eval-runner/src/pipeline.rs + - configs/** + - docs/** +- depends_on: [Task_2] +- description: | + Replace the direct vector-service search in the vector-only baseline with the retrieval trace (overfetch-and-slice per kind; item text from the evaluation repository's own ingest records); mirror the completeness telemetry field; add the typed backend identity to result rows through the repository's clean-schema procedure; make the cleanup guard backend-neutral; drop the payload constants, the second vector client's search path, and the second embeddings client's divergent dimension handling. Perform the A/B run with a row-level diff of item identities and ranks against the pre-switch baseline before deleting the old path. +- acceptance: + - Zero-hit census for vector-service search calls and payload constants in the evaluation adapter. + - A/B evidence recorded; vector-only rows carry the completeness verdict. + - Cleanup and namespace guards work for both vector modes. +- validation: + - kind: command + required: true + owner: evals-worker + detail: "fmt; workspace clippy -D warnings; service-up cargo test --workspace with the live switch set; A/B run artifacts under .agent-work with the diff" + - kind: review + required: true + owner: evals-reviewer + detail: "Diff review vs ADR-I-0026; verify the A/B diff and that no sealed evidence changed" + +### Task_6: Evaluation repository: embedded-mode configuration and cross-mode baselines +- type: impl +- owns: + - crates/cmem-eval-core/src/config.rs + - crates/cmem-eval-adapter-cmem/src/lib.rs + - configs/** + - docs/** +- depends_on: [Task_4, Task_5] +- description: | + Add the embedded vector mode to the evaluation backend configuration, run the continuity suite in embedded mode, and record that scenario baselines are identical to service mode under the parity contract (any divergence is a finding). +- acceptance: + - An embedded-mode configuration exists and runs without a vector service. + - Cross-mode baseline comparison recorded with zero unexplained divergence. +- validation: + - kind: command + required: true + owner: evals-worker + detail: "continuity suite in both modes; comparison artifact recorded" + - kind: review + required: true + owner: evals-reviewer + detail: "Verify the comparison and the register entries" + +### Task_7: Fake retirement, closeout docs, and reconfirmation evidence +- type: chore +- owns: + - src/test_support.rs + - src/**/tests (fake stores only) + - docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md + - docs/coding-agent/lessons.md + - docs/roadmap/development_roadmap.md +- depends_on: [Task_4, Task_6] +- description: | + Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened in memory (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; mark the roadmap row finished; move the plan to completed. +- acceptance: + - Zero-hit census for the retired fake and record type. + - All five checklist rows cite evidence in the Progress Log. +- validation: + - kind: command + required: true + owner: worker + detail: "full cargo test with no service and service-up with the live switch set" + - kind: review + required: true + owner: reviewer + detail: "Closeout review; Definition of Done census" + +### Task_8: Design-value audit at the pre-merge milestone gate +- type: review +- owns: [] +- depends_on: [Task_4, Task_5] +- description: | + Altitude review (Claude) against philosophy and roadmap: nothing designed twice across the two adapters, no hint field re-entered without its predicate, the evaluation repository holds no store-private knowledge, the ADR boundaries respected. +- acceptance: + - Audit verdict recorded in the Decision Log with any EARNS-ITS-PLACE / OVERSIZED / DELETE findings dispositioned. +- validation: + - kind: review + required: true + owner: orchestrator + detail: "Audit report consumed; dispositions recorded" + +## Task Waves (explicit parallel dispatch sets) + +- Wave 1 (parallel): [Task_1, Task_2] +- Wave 2 (parallel): [Task_3, Task_5] +- Wave 3 (parallel): [Task_4] +- Wave 4 (parallel): [Task_6, Task_8] +- Wave 5 (parallel): [Task_7] + +Each wave ends with reviewer approval and a PR per touched repository, merged by the decider before the next wave starts; the evaluation repository's sibling checkout is re-pinned to the merged library commit at every wave boundary. + +## Rollback / Safety +- Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict and the shrunken record, both covered by the parity suite. +- Stored service-mode payloads with dropped fields remain readable (extra fields tolerated unread); rebuild from graph authority is the recovery path. +- Each wave is a separately revertible PR pair. + +## Progress Log (append-only) + +Append-only editing rule (applies to both logs below): when appending an entry, anchor the edit on the previous entry and reproduce it (or anchor on the section's tail marker) so the edit inserts rather than replaces, and verify afterward that the log grew. + +- 2026-09-02 Planning wave completed: five parallel inputs (design consult, two altitude audits, two forensic censuses) consolidated; decider ruled the five design questions; ADR-I-0023 through ADR-I-0026, the rewritten phase document, and the roadmap section authored on branch `plan/v0-1-6-embedded-vector-recall`. Plan awaits approval. + +## Decision Log (append-only; re-plans and major discoveries) + +- 2026-09-02 Decision: the draft's port description was rewritten as an intentional new port contract. + - Trigger / new insight: the draft (2026-07-20) described filter, diagnostics, and reconciliation capabilities that the structured-verdict phase deleted; thirty of thirty-three payload fields were write-only; the readable text column's only reader was the evaluation repository's direct store access. + - Plan delta: contract-first waves (envelope and query, then record, then adapter); evaluation baseline moved to the trace; deletions promoted to deliverables. + - Tradeoffs considered: recorded in the four ADRs' rejected alternatives; the forward-looking keep case for hint fields (immutable time window) is recorded as a re-entry path rather than kept. + - User approval: rulings on all five questions given 2026-09-02; plan approval pending. +- 2026-09-02 Decision: evidence-integrity defects in the evaluation repository are fixed before this phase, outside this plan. + - Trigger / new insight: batch ingest produced phantom repair attempts and the evaluated rank was harness-invented; both would corrupt the parity and baseline evidence this phase cites. + - Plan delta: none inside this plan; recorded as a prerequisite in Context. + - User approval: yes, 2026-09-02. + +## Notes +- Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the exact-scan corpus-size guidance must be measured, not assumed. +- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` returns an empty exhaustive result; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior. diff --git a/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md b/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md index e4ee4be3..53ea5ebe 100644 --- a/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md +++ b/docs/decisions/implementation/ADR-I-0002-natural-language-embedding-surfaces.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0025-vector-record-is-a-read-contract.md +supersession_scope: partial --- # ADR-I-0002: Embed natural-language semantic surfaces, not structured metadata templates diff --git a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md index 706b0aa8..032c9fa2 100644 --- a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md +++ b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +supersession_scope: partial --- # ADR-I-0003: Use Qdrant and Oxigraph as default storage backends diff --git a/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md b/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md index a84f1977..b238f423 100644 --- a/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md +++ b/docs/decisions/implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0025-vector-record-is-a-read-contract.md +supersession_scope: partial --- # ADR-I-0005: Keep Qdrant metadata filterable while graph relationships remain authoritative diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md new file mode 100644 index 00000000..8460e18f --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md @@ -0,0 +1,115 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely either treat the embedded vector store as a test convenience and let it drift from the service adapter's contract, or reach for an embedded approximate-nearest-neighbour library the moment a corpus feels large, discarding the exact-scan determinism the parity contract depends on" + detected_signals: "cross-boundary contract shape with a tempting alternative; rejected alternative likely to be re-proposed; premises likely to expire (corpus scale, in-process build of the service backend); deliberately bounded scope (single process, opt-in default)" + cost_of_violation: "two vector adapters with different admission semantics silently produce different continuity packs from the same memory, which the evaluation suite would attribute to retrieval regressions rather than backend divergence" + cost_of_wrong_preservation: "once corpora exceed the exact-scan guidance or the service backend ships a stable in-process build, keeping the exact scan as the only embedded option would make local deployments slow for no contractual reason" + cost_of_over_extension: "applying the single-process expectation to multi-replica deployments, or treating the embedded mode as the validated default before parity evidence exists, misrepresents what the library has validated" +depends_on: [implementation/ADR-I-0009-use-sqlite-as-default-retrieval-stats-store.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] +implements: [] +supersedes: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md] +superseded_by: null +supersession_scope: partial +--- + +# ADR-I-0023: Embedded SQLite exact-scan vector candidate store as the opt-in local mode + +## Context and Problem Statement + +After the embedded persistent graph store became the validated default (ADR-I-0021) and retrieval statistics were already file-backed (ADR-I-0009), the vector candidate store was the only component that still required an external service. +That conflicts with the intended deployment shapes: desktop companions and game or simulation characters run on end-user machines where a container runtime cannot be assumed, and it keeps a service dependency in the default test path. +ADR-I-0003's own revisit clause, "operating two stores becomes too heavy for target users", was recorded as triggered at the close of the eval-driven family closeout. +The vector layer is candidate recall only: the vector store suggests, retrieval statistics guide fanout, and graph authority decides final inclusion, so an embedded adapter has a low correctness bar — it must prefilter and rank candidates well, never be authoritative for anything. + +## Decision Drivers + +- Zero-infrastructure local deployment is a product requirement, not a convenience. +- The embedded adapter must satisfy the same port contract as the service adapter, proven by a shared parity suite; anything less makes the evaluation suite an unreliable regression instrument. +- No heavyweight dependency for a first implementation; `rusqlite` with the bundled engine is already a dependency through the statistics store. +- Exact scan at character-memory scale (tens of thousands of vectors) is honest, deterministic, and strictly better recall than approximate search. +- Defaults must match validation evidence (ADR-I-0021's rule); flipping the default before parity evidence exists would repeat the mistake that ADR-I-0021 corrected. + +## Decision + +Add an embedded vector candidate store mode behind the existing vector candidate port, implemented as a SQLite-backed exact cosine scan, selected by a dedicated store-mode setting. +The service adapter remains fully supported as the service and cloud mode; this decision adds a mode and deprecates nothing. +The default mode stays service until the parity suite and the evaluation suite have produced identical results across modes; flipping the default is a separate, evidence-gated decision. +The embedded store is single-process, matching the embedded graph store's expectation. + +Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. +The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files). + +Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and schema version so a reopened file is validated against the configured embedding model with the same compatibility error the service adapter raises for a mismatched collection. +Search is a scan of the scoped rows scored by dot product in a fixed order, canonicalised by the shared constructor the port requires, and truncated to the requested limit; the result always reports exhaustive completeness (ADR-I-0024). + +## Implementation Impact + +- A new adapter module implementing the vector candidate port; the composition root gains a mode switch mirroring the statistics-store switch. +- The settings type gains the mode and path keys; the service connection string becomes optional and is validated as present only in service mode. +- The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion is updated in the same wave. +- The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured. +- The deterministic test fake that reimplements cosine scoring and scope filtering is retired in favour of the embedded adapter opened in memory, which removes the vector-service dependency from the default test path. +- Documentation states the single-process expectation, the measured corpus-size guidance, and rebuild-from-graph-authority as the path between modes. + +## Considered Options + +1. SQLite exact cosine scan behind the existing port, opt-in, service mode retained. +2. An embedded approximate-nearest-neighbour library as the first embedded implementation. +3. The in-process build of the service backend. +4. An in-memory-only embedded store. +5. Flip the default to embedded in the same change. + +## Decision Outcome + +Chosen option: **Option 1**. +It reuses an existing dependency, gives exact filter and ranking semantics that make the parity contract checkable by set equality, is deterministic by construction, and is restart-safe through an ordinary file. + +### Rejected Alternatives + +Option 2 adds a heavyweight dependency and approximate membership semantics before any corpus has demonstrated that exact scan is on the critical path; it is the recorded escalation path, reopened by a measured corpus exceeding the published exact-scan guidance or a benchmark showing the scan dominating retrieval latency. +Option 3 was not stable at decision time; it is a revisit candidate once it ships a stable release, because it would maximise reuse of the service adapter's conventions. +Option 4 fails restart safety, which the persistent-graph-authority phase made a requirement for every store that survives a process. +Option 5 contradicts the defaults-match-evidence rule; it is reopened by the evidence named under Revisit When. + +## Consequences + +- Positive: a fully self-contained local deployment exists; the default test path needs no running service; both adapters are held to one contract by one suite. +- Positive: the embedded adapter is the honest reference implementation for the port's semantics because it computes them exactly. +- Negative / tradeoffs: exact scan is linear in corpus size and reads every scoped embedding per query; the guidance number is measured, not engineered around. +- Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. + +## Decision Boundary + +Invariant: the embedded adapter implements the same port contract as the service adapter and is proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string. + +Not covered: the corpus-size guidance number (measured and revised through documentation), the choice to add an in-memory embedding cache in front of the scan (an implementation optimisation), and the default mode (a separate evidence-gated decision). + +## Validation + +- Embedded mode constructs without any running service and survives process restart with identical search results. +- The parity suite produces identical admitted candidate sets and orderings from both adapters across the full contract, including identical-vector tie cohorts. +- A reopened embedded store with a different vector size fails with the collection-compatibility error. +- The default-mode construction test asserts service mode. + +## Revisit When + +- A measured corpus exceeds the published exact-scan guidance, or a benchmark shows the scan on the retrieval critical path — reopen the approximate-index escalation. +- The service backend's in-process build reaches a stable release — reopen the choice of embedded engine. +- The evaluation suite has run every dataset in embedded mode with results identical to service mode and one corpus at the guidance size — reopen the default mode. +- A multi-replica deployment shape is designed (the remote graph-authority phase ADR-I-0021 anticipates) — the single-process expectation is reconsidered together with the graph and statistics stores, never alone. + +## Consultation impact + +Question asked: whether the embedded store should overload the service connection string or take its own settings, and whether to flip the default now; ruling adopted the separate settings and the opt-in default as recommended. + +## More Information + +- ADR-I-0003 remains authoritative for the service-mode backends; this record supersedes only its claim that those backends are the sole defaults. +- ADR-I-0024 (port contract this adapter implements) and ADR-I-0025 (the record it stores). +- The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md new file mode 100644 index 00000000..e6352b67 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -0,0 +1,124 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely add a vector-layer predicate as a three-valued hint filter that matches unknown values, or let an adapter truncate an unclosed equal-score cohort without saying so, because both are the natural first implementation and both have already happened in this repository" + detected_signals: "cross-boundary contract shape (port postcondition) with tempting alternatives; rejected alternative likely to be re-proposed; premises likely to expire (no vector-layer predicate is needed yet)" + cost_of_violation: "a prefilter that matches unknown values admits stale candidates that graph verification then silently discards, and an unreported open cohort makes top-K membership vary between runs — both surface as unexplained retrieval nondeterminism in evaluation evidence long after the cause is forgotten" + cost_of_wrong_preservation: "if a retrieval route needs a scoped or time-bounded semantic search and the scope-only query is preserved as a rule rather than a current state, retrieval will starve at scale (an unfiltered top-K contains only the in-scope fraction) and callers will overfetch instead of adding the predicate" + cost_of_over_extension: "treating the completeness verdict as an error condition would fail retrieval on a determinism caveat about non-authoritative candidates" +depends_on: [implementation/ADR-I-0018-responsibility-boundary-modules-with-enforced-dependency-direction.md, implementation/ADR-I-0022-retain-measured-retrieval-defaults.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0024: Vector candidate recall reports completeness and takes a scope-only query + +## Context and Problem Statement + +The vector candidate port promised deterministic admission: at most `limit` unique object-and-surface matches in canonical order, with equal-score cohorts at the cutoff closed before truncation (ADR-I-0022 records the fix). +The service adapter closes the cohort by growing its fetch up to a bound, but when the bound is hit it returns the truncated set with no signal, and the port's result type — a bare candidate list — cannot carry the difference between "this top-K is determinate" and "membership may vary between runs". +Separately, the port once carried a filter type whose currentness predicates were `Option` values implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake. +Those filters were deleted as speculative in the structured-verdict phase because no caller used them; the query is now an embedding, a limit, and an object-type scope. +An embedded adapter (ADR-I-0023) makes the gap visible: an exact scan is exhaustive by construction and needs a way to say so, and a second adapter needs a query contract that cannot drift. + +## Decision Drivers + +- Each port owns its stated postconditions; upper layers never repair lower-layer output (the structured-verdict contract's ruling), so completeness must be stated by the adapter, not inferred by the pipeline. +- Candidate recall is non-authoritative; graph authority verifies every candidate, so a determinism caveat must never become a retrieval failure. +- Prefilter false negatives are unrecoverable while false positives cost one root slot, so a vector-layer predicate is only safe on data that is immutable or synchronised on every mutation. +- Two adapters must be held to one query contract with one parity suite. +- The retrieval telemetry and trace vocabulary is the one API surface ports may import (ADR-I-0018), and it is where callers already read the returned candidate count. + +## Decision + +`search_candidates` returns a result envelope: the canonical candidates (the constructor-owned canonical newtype survives as the field type) together with a typed completeness verdict. + +```rust +pub enum VectorRecallCompleteness { + Exhaustive { scanned: usize }, // every stored record in scope was scored + BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort was closed + BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort open +} +``` + +Adapters own canonicalisation and must state the verdict truthfully: exhaustive only when the whole scoped population was scored, closed only when the cutoff cohort was verified closed or the index returned fewer rows than asked, open at the bound. +The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. +The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. + +The query is the embedding, the limit, and an object-type scope, and nothing else. +An empty scope selects zero candidates; wildcard-on-empty is prohibited, matching the graph query rule, and the retrieval context rejects an empty configured object-type set at the boundary. +Three-valued hint predicates are prohibited. +Any future vector-layer predicate arrives as an explicit enum whose unknown arm is spelled out, an unknown or missing stored value never satisfies a positive predicate, and the predicate lands with its mapping in both adapters and a parity fixture in the same change. + +Two re-entry paths are named now so the scope-only query is read as a current state, not a rule: + +1. A synchronised scope predicate, owned by the scoped-continuity phase: a scope-id column written at upsert and kept in sync by the link and reflection write paths, because the existing relationship hints were frozen at upsert and never updated by linking, which made them unusable as a prefilter. +2. An immutable time-window predicate over `created_at` and `observed_at`, owned by whichever phase first ships a time-bounded retrieval route: immutability makes a write-time column correct without a sync path, and the columns are backfilled from graph authority if they are ever needed (ADR-I-0025). + +## Implementation Impact + +- The port trait's search method changes its return type; the pipeline reads the candidates field and copies the verdict into telemetry; the test fakes wrap their existing value in the exhaustive variant. +- The service adapter's fetch-decision enum maps one-to-one onto the closed and open variants. +- Retrieval telemetry gains a completeness field with a manual default; the companion evaluation repository mirrors the field in its telemetry record (ADR-I-0026 records the obligation). +- The port doc comment stops describing a "documented bounded-overfetch degradation policy" because the type now says it. + +## Considered Options + +1. A typed completeness verdict in a result envelope; scope-only query with the predicate rule and named re-entry paths. +2. Silent degradation at the fetch bound (as built). +3. A boolean `complete` flag on the result. +4. Fail closed with an error when the cohort is open at the bound. +5. Resurrect the deleted hint filters for the embedded adapter, which can evaluate them exactly. + +## Decision Outcome + +Chosen option: **Option 1**. +It makes the postcondition expressible by the type that owns it, distinguishes the exhaustive case the embedded adapter introduces from the closed-cohort case the service adapter can promise, and keeps every consumer a field access away from unchanged code. + +### Rejected Alternatives + +Option 2 hides a determinism caveat that evaluation evidence later attributes to retrieval; rejected outright. +Option 3 loses the exhaustive-versus-closed distinction and the fetch counts that explain overfetch cost; rejected outright. +Option 4 fails retrieval on a caveat about non-authoritative candidates that graph authority verifies anyway; rejected outright. +Option 5 recreates a prefilter over hints that no write path other than upsert keeps in sync; it is reopened only through the named re-entry paths, each of which brings its own synchronisation obligation. + +## Consequences + +- Positive: top-K determinism is observable per retrieval; the parity suite can assert the verdict per adapter. +- Positive: the query contract is small enough to hold two adapters to by set equality. +- Negative / tradeoffs: callers that need scoped or time-bounded semantic recall must wait for the named predicate rather than overfetching; the re-entry paths exist to make that wait short and the shape predictable. + +## Decision Boundary + +Invariant: the search result carries a typed completeness verdict stated by the adapter; the pipeline never repairs or fails on it; the query carries no three-valued predicate; a new predicate lands in both adapters with a parity fixture. + +Not covered: the service adapter's overfetch bound constants (calibrated values), the exact telemetry field name, and the internal fetch-decision mechanics. + +## Validation + +- Unit tests on the service adapter's fetch decision assert the mapping to closed and open verdicts, including the all-tied cohort at the bound. +- A retrieval test asserts the telemetry verdict for each variant using the fakes. +- The parity suite asserts exhaustive for the embedded adapter and closed for the service adapter on the identical-vector tie fixture. +- A census of the vector adapters shows no match-or-unknown condition and no filter type beyond the object-type scope. + +## Revisit When + +- A retrieval route needs a scoped or time-bounded semantic search — take the matching re-entry path above rather than reopening the predicate rule. +- An adapter appears that cannot classify its own cutoff (for example a remote index without a fetch count) — the verdict vocabulary may need a variant for "unknown", which must still never be treated as an error. + +## Consultation impact + +Question asked: whether the deleted hint filters should return for the embedded adapter; ruling adopted the scope-only query with the two named re-entry paths as recommended. + +## More Information + +- ADR-I-0022 (tie-cohort closure and canonical ordering at the adapter boundary, the postcondition this record makes expressible). +- ADR-I-0025 (the stored record whose columns the re-entry paths would extend). +- ADR-I-0023 (the embedded adapter that always reports exhaustive completeness). diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md new file mode 100644 index 00000000..b4da592c --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -0,0 +1,113 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely re-add relationship, lifecycle, time, or readable-text columns to the vector record because the earlier payload design lists them as intended, or delete the embedded-text column as unread once surfaces become generated" + detected_signals: "cross-boundary contract shape (two adapters mirror one record); rejected alternative likely to be re-proposed; premises likely to expire (a retrieval route may need a prefilter); cross-repository obligation (the evaluation baseline read a payload column)" + cost_of_violation: "every column that returns without a reader is mirrored across two adapters, indexed at every collection initialisation, and carried stale by write paths that never update it; a column deleted as unread would erase the only record of what a generated vector embedded" + cost_of_wrong_preservation: "if a retrieval route needs a prefilter and the five-column rule is preserved as prohibition rather than current state, the predicate is blocked instead of landing through the named re-entry path" + cost_of_over_extension: "extending the rule to the graph store would strip graph authority of denormalised fields it legitimately owns" +depends_on: [implementation/ADR-I-0007-schema-versioning.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md] +implements: [] +supersedes: [implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md, implementation/ADR-I-0002-natural-language-embedding-surfaces.md] +superseded_by: null +supersession_scope: partial +--- + +# ADR-I-0025: The vector record is a read contract — identity, surface, schema version, embedded text + +## Context and Problem Statement + +ADR-I-0005 decided that the vector payload stores filterable metadata and graph pointers, and the payload design note enumerated thirty-three fields with thirty of them indexed. +By the time the embedded adapter (ADR-I-0023) was designed, the library read back exactly three of those fields — object id, object type, surface — and the only external reader was the companion evaluation repository's vector-only baseline reading the readable text column. +The relationship hints were frozen at upsert and never updated by the link write path; the lifecycle hints described vectors the correction and forgetting paths delete; the readable text column duplicated graph text with a prefix removed; and every field was about to be mirrored into a second physical schema. +ADR-I-0002's implementation note said to "persist both `embedding_text` and `content_text` where useful", which left the two text columns' meanings undefined. +The forward-looking case for each family was analysed against the planned phases (scoped continuity, factual rigor and temporal validity, retrieval observability, associative recall, assisted remember, multimodal) before deciding. + +## Decision Drivers + +- A column earns its place when a reader exists; carrying it unread costs two adapter mappings, index creation per collection, a parity fixture, and the sync discipline ADR-I-0005 named in its own tradeoffs. +- Prefilter hints are only safe on immutable or synchronised data; the relationship, lifecycle, ranking, and mutable time hints were none of those. +- Re-adding an immutable column later is a backfill from graph authority, not a re-index. +- Once embedding surfaces are generated or caller-supplied (the assisted-remember phase; the write plan already carries a caller-supplied surface), the text a vector embeds is no longer re-derivable from graph authority, so it is provenance in the philosophy's sense. +- Read-out text is graph authority's job; the vector layer suggests, it does not describe. + +## Decision + +Both adapters persist exactly these fields per vector record: object id, object type, surface, schema version (ADR-I-0007), and `embedding_text`. + +Three sentences govern the text columns: +Read-out text lives in graph authority. +The vector record stores only the embedded surface, as provenance of what was ranked. +Consumers needing candidate content hydrate by object id. + +`content_text` is dropped. +The relationship refs (episode, observation, thread, entity, participant, speaker, supersedes), the lifecycle and currentness flags, the time hints, the ranking and salience hints, the object-specific hints, the graph URI, and the raw source reference leave the vector write path. +The typed field manifest introduced in the structured-verdict phase remains the single source of both adapters' column sets and shrinks to the five entries. +ADR-I-0024 names the two re-entry paths (a synchronised scope predicate; an immutable time-window predicate over `created_at` and `observed_at` backfilled from graph authority) so a returning column arrives with its predicate, its adapter mappings, and a parity fixture. + +## Implementation Impact + +- The vector record type and the surface builders lose the hint carriers; the payload map and the embedded schema serialise five fields. +- The service adapter stops creating per-field payload indexes for dropped fields. +- The companion evaluation repository's vector-only baseline stops reading the readable text column and sources item text from its own ingest records (ADR-I-0026). +- The payload design note's field categories and indexing policy are superseded by this record and carry a supersession note. +- No migration: under the Compatibility Policy, existing stores are rebuilt from graph authority. + +## Considered Options + +1. Five-column read contract; keep `embedding_text` only; drop the hint families with named re-entry paths. +2. Keep both text columns. +3. Drop both text columns. +4. Keep the unread hints for the planned phases. +5. Keep only the two immutable timestamp columns as a hedge against top-K starvation. + +## Decision Outcome + +Chosen option: **Option 1**. +It stores what is read, keeps the one column that becomes non-re-derivable, and prices re-entry honestly. + +### Rejected Alternatives + +Option 2: `content_text` is a deterministic function of graph object fields at every surface builder, so it never carries information graph authority lacks, and its one reader moves to its own ingest records; rejected outright. +Option 3: `embedding_text` is cheap and becomes the only record of what a generated or caller-supplied vector embeds; rejected outright. +Option 4: no planned phase names a vector-layer predicate the existing fields could serve without new synchronisation work — scoped retrieval needs a scope id kept in sync by linking, temporal validity is a ranking property of new claim objects, salience evolves by reinforcement, and lifecycle hints describe vectors the write path deletes; reopened only through the re-entry paths. +Option 5 is the only subset with a forward-looking case that survives the synchronisation test; it was declined because no phase document asks for the predicate and the columns backfill cheaply when one does. + +## Consequences + +- Positive: the embedded schema is five columns and one index; both adapters mirror one small manifest. +- Positive: the embedded surface is preserved as vector provenance before surfaces become generated. +- Negative / tradeoffs: a future scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the re-entry paths make that step predictable. + +## Decision Boundary + +Invariant: the vector record carries only fields a reader consumes, plus the embedded surface as provenance; readable content is hydrated from graph authority by object id; a returning hint arrives with its predicate and parity fixture through ADR-I-0024's re-entry paths. + +Not covered: the physical encoding of each column per adapter, and graph authority's own denormalised fields. + +## Validation + +- A census of both repositories shows no reader of the dropped fields and no reader of `content_text`. +- The manifest test asserts the five entries; the parity suite serialises and reads them through both adapters. +- The evaluation baseline reproduces its results with text sourced from ingest records. + +## Revisit When + +- A retrieval route needs a scoped or time-bounded semantic search — take the ADR-I-0024 re-entry path; this record's invariant is satisfied by a column that arrives with its reader. +- The assisted-remember phase makes the embedding surface a graph-authoritative provenance artifact — the vector copy becomes a cache and this record's provenance argument moves to the graph. +- A re-indexing workflow appears that cannot rebuild from graph authority — the readable-text question reopens with that workflow as its reader. + +## Consultation impact + +Question asked: whether the unread hint families and the readable text column should be kept for planned phases; ruling adopted the five-column contract with the three governing sentences and the named re-entry paths. + +## More Information + +- ADR-I-0005 remains authoritative for graph authority over relationships; this record supersedes its payload field list and its "payload metadata as candidate filter" implementation guidance. +- ADR-I-0002 remains authoritative for natural-language embedding surfaces; this record supersedes only its note to persist both text columns. +- ADR-I-0024 (query contract and re-entry paths), ADR-I-0023 (embedded schema), ADR-I-0026 (evaluation baseline reader). diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md new file mode 100644 index 00000000..24308528 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -0,0 +1,111 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely add a public raw vector search to the facade for the evaluation baseline, or let evaluation tooling read a store's physical schema directly again, because each is the shortest path to a number" + detected_signals: "externally observable contract shape with a tempting alternative; rejected alternative likely to be re-proposed; cross-repository obligation; deliberately bounded scope (no product use case for raw recall exists)" + cost_of_violation: "a raw-recall facade makes the library a vector-database abstraction and exposes unverified candidates as if they were memory; a schema-reading baseline breaks silently the moment a second vector adapter ships a different physical schema, and it reimplements canonical ordering the library already owns" + cost_of_wrong_preservation: "if a product use case for candidate-level recall arrives and this record is preserved as a blanket prohibition, the diagnostic surface the observability phase plans would be blocked instead of designed" + cost_of_over_extension: "reading this record as forbidding evaluation tooling from using the trace at all would leave the baseline with no honest data source" +depends_on: [implementation/ADR-I-0020-restart-identity-via-caller-supplied-ids-not-a-lookup-surface.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0026: Raw vector baselines read the retrieval trace; the library exposes no candidate-search surface + +## Context and Problem Statement + +The companion evaluation repository (a development aid, not core library functionality) runs a vector-only baseline: ingest through the library, then rank by plain vector similarity to measure what hybrid retrieval adds. +As built, that baseline held its own client to the vector service, ran one filtered search per object kind against the library's collection, read three payload fields by hard-coded name, re-implemented best-score-per-object deduplication and score ordering, and took item text from a payload column. +That is a hidden capability: the baseline depended on an adapter-private schema, duplicated ordering the library owns, and could not run at all against an embedded store (ADR-I-0023). +The question is what capability the library must expose so the baseline stops reaching into a store. + +## Decision Drivers + +- Evaluation tooling must not grow library surface that no product use case has demanded (ADR-I-0020's driver). +- The library is not a vector-database abstraction, and vector-only candidates must never become behavior-influencing memory without graph verification (project philosophy; the persistent-graph-authority phase's acceptance criteria). +- The retrieval trace already is the raw vector recall: the canonical, pre-verification top-K with object reference, surface, score, and rank, scoped by the configured object types and sized by the candidate limit, and ADR-I-0024 adds the completeness verdict that says whether that top-K was determinate. +- Every store's physical schema is adapter-private; two adapters must not create two baseline implementations. + +## Decision + +The library exposes no raw candidate-search surface and no facade change. +The evaluation repository's vector-only baseline issues an ordinary `retrieve` with tracing enabled, the object types it measures, and a generous candidate limit, then slices the trace's vector candidates per object kind to its per-section budgets and reads the completeness verdict from telemetry. +Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). +The evaluation repository's vector-service client shrinks to collection lifecycle operations (existence and deletion), which the embedded mode replaces with file operations through the durable-store path list the adapter already maintains. + +Cross-repository obligations recorded here: + +- The evaluation telemetry record mirrors the completeness field. +- Result rows carry a typed vector-backend identity (service or embedded) so cross-mode comparisons are attributable. +- The namespace cleanup guard is backend-neutral: it protects an embedded store file by the same prefix rule that protects a service collection. +- The vector-only surface-policy validator keeps its object-type and budget rules and derives the overfetch from the per-section budgets. +- The baseline is re-verified by an A/B run against the direct-search implementation before that implementation is deleted. + +## Character Memory Relevance + +Retrieval that bypasses graph authority is exactly the "generic RAG wrapper" and "unexplained recall" the philosophy warns against; the trace exists so that every candidate a developer sees is one the library can explain, whether it was admitted or not. +Keeping the baseline inside the traced retrieval path means the measurement of "what does the graph add" is taken from the same recall the character actually experiences, not from a parallel search that may drift from it. + +## Implementation Impact + +- Library: none beyond ADR-I-0024's telemetry field; the acceptance criterion "no public facade change" holds. +- Evaluation repository: delete the direct search path, payload field constants, and hit mapping; add trace-derived candidate slicing and ingest-record text lookup; add the backend identity to result rows; generalise cleanup to the embedded store file. + +## Considered Options + +1. The baseline consumes the retrieval trace; no library surface. +2. A public candidate-recall method on the facade returning references, surfaces, and scores. +3. A retrieval mode that skips graph verification. +4. Publish the payload manifest so evaluation tooling can keep reading the store. + +## Decision Outcome + +Chosen option: **Option 1**. +It holds the no-facade-change line, covers every vector adapter automatically, deletes a duplicate implementation, and takes the measurement from the recall the character actually experiences. + +### Rejected Alternatives + +Option 2 is an evaluation-driven surface with no product use case and a vector-database-abstraction shape; it is reopened only by a product use case for candidate-level recall, at which point it lands as a designed diagnostic surface in the retrieval-observability phase, not as a search method. +Option 3 contradicts the acceptance criterion that candidates whose graph objects are missing are rejected from normal retrieval; rejected outright. +Option 4 leaves two implementations of one capability and breaks with the first adapter whose physical schema differs; rejected outright. + +## Consequences + +- Positive: one retrieval entry point; both adapters covered; the baseline reports the completeness of the top-K it measured. +- Negative / tradeoffs: the baseline pays for graph expansion it discards, an evaluation-run cost accepted in exchange for not inventing a second retrieval path. +- Negative / tradeoffs: per-kind quotas are satisfied by overfetch-and-slice; if a quota cannot be filled within the service adapter's fetch bound, the completeness verdict says so and the run records it. + +## Decision Boundary + +Invariant: no public raw candidate-search surface; evaluation baselines consume the retrieval trace and telemetry; store schemas are adapter-private; candidate content is hydrated by object id from the consumer's own records or graph authority. + +Not covered: the overfetch multiplier the baseline uses, the shape of the evaluation repository's ingest record store, and any future diagnostic surface the observability phase designs on product demand. + +## Validation + +- An A/B run of the vector-only configuration (direct search versus trace-derived) shows identical item identities and ranks per question before the direct path is deleted. +- After the switch, the evaluation adapter contains no search call against the vector service and no payload field constant. +- The baseline runs unchanged in embedded mode. + +## Revisit When + +- A product use case demands candidate-level recall — design a diagnostic surface in the retrieval-observability phase and supersede this record's prohibition for that surface only. +- Per-kind quotas cannot be satisfied by overfetch within the service adapter's fetch bound on a real dataset — reopen whether the trace needs per-type limits. + +## Consultation impact + +Question asked: trace-derived baseline versus a public candidate-recall method; ruling adopted the trace as recommended. + +## More Information + +- ADR-I-0020 (the precedent: evaluation needs met without a lookup surface). +- ADR-I-0024 (the completeness verdict the baseline reads), ADR-I-0025 (why item text is not a payload column). +- The companion evaluation repository's vector-only baseline plan (historical record of the direct-search implementation this record replaces). diff --git a/docs/design/database/vector_payload_design.md b/docs/design/database/vector_payload_design.md index 84b29d38..4d2b9ed8 100644 --- a/docs/design/database/vector_payload_design.md +++ b/docs/design/database/vector_payload_design.md @@ -1,5 +1,7 @@ # Vector Database Payload Design +> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 records the re-entry paths for any returning prefilter column. The Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were; the Design Goal, Record Shape, Why Natural-Language Surfaces, and Consistency Model sections remain current. + This document describes the Qdrant payload design for Character Memory. It is intentionally a design note, not a field-by-field copy of the Rust mapping code. Qdrant is the semantic candidate index. It is not the memory database of record. The authoritative memory state lives in the graph store. A Qdrant hit means "this object may be relevant"; it does not mean "this object is current, related, or safe to include." Retrieval must verify candidates through the graph authority before returning them in a continuity context pack. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 2747a87b..716f8d82 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -1,78 +1,185 @@ -# v0.1.6 Design Draft: Embedded Vector Candidate Recall +# v0.1.6 Design: Embedded Vector Candidate Recall + +Status: decided 2026-09-02 (ADR-I-0023 through ADR-I-0026); supersedes the 2026-07 draft of this document. ## Version intent -Complete the zero-infrastructure local deployment story by adding an embedded vector candidate store mode behind the existing vector port. -With graph authority defaulting to embedded persistent storage and retrieval statistics already file-backed, the vector candidate store is the only component that still requires an external service. -That conflicts with the desktop-companion and game/simulation use cases, where end users cannot be expected to operate containers, and it keeps a service dependency in the default test path. +Complete the zero-infrastructure local deployment story by adding an embedded vector candidate store mode behind the existing vector candidate port, and settle the port contract that both adapters must satisfy before a second adapter exists. +With graph authority defaulting to embedded persistent storage (ADR-I-0021) and retrieval statistics already file-backed (ADR-I-0009), the vector candidate store is the only component that still requires an external service. +That conflicts with the desktop-companion and game or simulation use cases, where end users cannot be expected to operate containers, and it keeps a service dependency in the default test path. -Sequencing: this phase runs before v0.2, so the payload surface is mirrored across two adapters while it is still small, and so v0.2 fixture work knows which vector backend it validates against. +Sequencing: this phase runs before scoped continuity, so the vector record is mirrored across two adapters while it is five fields, and so the scoped-continuity evaluation fixtures know which vector backend they validate against. -## Why this is safe to do now +## Why this is safe to do before scoped continuity -The vector layer is candidate recall only: Qdrant suggests, statistics guide fanout, and graph authority decides final inclusion. -An embedded adapter therefore has a low correctness bar — it must prefilter and rank candidates well, not be authoritative for anything. -The port is small (upsert, filtered search, diagnostics listing, delete), provider-neutral, and already exercised by deterministic fakes and a live parity surface. +The vector layer is candidate recall only: the vector store suggests, statistics guide fanout, and graph authority decides final inclusion. +An embedded adapter therefore has a low correctness bar: it must prefilter and rank candidates well, never be authoritative for anything. +The port is small (upsert, scoped search, delete), provider-neutral, and already exercised by deterministic fakes and a live smoke surface. +The write path already removes the vectors of superseded and suppressed objects, so the live vector population is the active population by construction, and stale residue from failed maintenance is caught by graph verification; the embedded adapter inherits both guarantees without new code. ## Design direction -- Add a `VectorStoreMode` setting (`service` | `embedded`) mirroring the graph store mode pattern, with the vector connection string interpreted as URL or local path accordingly. -- First embedded implementation: a SQLite-backed exact-scan adapter. - The port's filter contract (object types, retention states, currentness, entity/thread/episode ID lists, time ranges) maps natively onto SQL predicates with junction tables for the ID lists; after prefiltering, exact cosine scan over the survivors. - At character-memory scale (tens of thousands of vectors), exact scan is honest, fast enough, deterministic, and strictly better recall than approximate search. -- The Qdrant adapter remains fully supported as the service/cloud mode; this phase adds a mode, it does not deprecate one. -- The canonical candidate ordering contract (score, object type rank, object ID, surface rank) established for deterministic admission applies identically to the embedded adapter. -- Parity is the acceptance instrument: one shared filter-contract fixture suite runs against both adapters and must produce identical admitted sets; the embedded adapter runs it unconditionally (no service gating), which also removes the vector-service dependency from the default test path. +### The port contract (ADR-I-0024, ADR-I-0025) + +This phase fixes the port contract deliberately, because two adapters cannot be held to an implicit one. + +Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. +Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. + +Result: a completeness envelope, the canonical candidates plus a typed verdict — exhaustive (every scoped record was scored), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). +The service adapter's existing tie-cohort loop maps onto the last two verdicts; before this phase it returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. +The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. +The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. + +Record: both adapters persist exactly five fields — object id, object type, surface, schema version, and the embedded text. +Read-out text lives in graph authority; the vector record stores only the embedded surface, as provenance of what was ranked; consumers needing candidate content hydrate by object id. +The relationship, lifecycle, time, ranking, object-specific, graph-URI, and raw-reference hints leave the write path: the library read none of them, the relationship hints were frozen at upsert and never updated by linking, the lifecycle hints described vectors the write path deletes, and the readable text column duplicated graph text. + +Two re-entry paths are named so the scope-only query and five-field record are read as current state, not prohibition: + +1. A synchronised scope predicate, owned by the scoped-continuity phase: a scope-id column written at upsert and kept in sync by the link and reflection write paths. +2. An immutable time-window predicate over `created_at` and `observed_at`, owned by whichever phase first ships a time-bounded retrieval route; immutability makes a write-time column correct without a sync path, and the columns are backfilled from graph authority if ever needed. + +### The embedded adapter (ADR-I-0023) + +A SQLite-backed exact cosine scan, using the `rusqlite` dependency the statistics store already carries, with the same single-process, mutex-guarded connection model. + +Schema: one table keyed by object id and surface with a column per contract field and the embedding as a fixed-width little-endian floating-point blob normalised at write; an index on object type for the scope predicate; a metadata table recording vector size, distance, and schema version. +Search: select the scoped rows, score by dot product in a fixed order, canonicalise through the shared constructor, truncate to the limit, and report exhaustive completeness with the scanned count. +Delete: remove every surface of each object id, matching the service adapter's selector. +Restart safety: opening an existing file validates the recorded vector size and distance against the configured embedding model and raises the same collection-compatibility error the service adapter raises for a mismatched collection. +Determinism: same inputs, same scores, same total sort; equal-score cohorts are ordered by the shared comparator, so the embedded adapter satisfies deterministic admission by construction and never needs an overfetch loop. +Score parity across adapters is not bitwise (the service computes cosine on its own normalised copy), so parity compares membership and order with a small score tolerance, and tie fixtures use identical vectors. + +### Settings and composition (ADR-I-0023) + +Follow the one-key-per-backend pattern the graph and statistics stores already use rather than overloading the service connection string. + +```text +VECTOR_STORE_MODE service | embedded (default: service) +VECTOR_STORE_PATH directory, read only in embedded mode +QDRANT_CONNECTION_STRING required only in service mode +``` + +`VECTOR_STORE_PATH` is a directory; each collection is one SQLite file inside it named by the collection name the public constructor already takes, so `collection_name` is the backend-neutral namespace key in both modes. +Collection names in embedded mode are validated to the same character set the evaluation repository already sanitises to. +The composition root gains a vector-store mode switch mirroring the statistics-store switch; the vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds. + +### Parity suite placement (ADR-I-0023) + +Library: a port-conformance suite in the integration tests — scope filtering, empty scope selects zero, canonical order, identical-vector tie cohort, best-score-per-object-and-surface deduplication, delete removes all surfaces, restart reopen, completeness verdict per adapter — run against the embedded adapter unconditionally and against the service adapter when a service connection is configured. +This follows the precedent that port conformance is enforced by contract tests run against every adapter, not by a runtime wrapper. +Evaluation repository: no second contract suite; it adds an embedded-mode configuration to the continuity scenarios and requires identical scenario results between modes, the behaviour-level regression instrument. ## Deliverables ```text -VectorStoreMode setting and configuration interpretation -SqliteVectorCandidateStore adapter (schema, upsert/delete, filtered exact-scan search, diagnostics) -composition wiring and mode selection -shared filter-contract parity suite exercised by both adapters -restart-safety and reconciliation coverage for embedded mode -documentation: payload mapping addendum, setup, corpus-size guidance -an implementation ADR recording the technology selection and its revisit triggers +port contract: completeness envelope, scope-only query with empty-scope-selects-zero, retrieval telemetry completeness field +vector record read contract: five-field manifest shared by both adapters +SqliteVectorCandidateStore adapter: schema, upsert/delete, scoped exact-scan search, restart validation +VectorStoreMode and VectorStorePath settings; composition mode switch; service connection string required only in service mode +port-conformance parity suite in the library integration tests, run against both adapters +restart-safety test for the embedded store; pipeline test over the embedded adapter with a deleted graph object +measured corpus-size guidance from an in-phase benchmark +documentation: settings, single-process expectation, corpus-size guidance, rebuild-from-graph-authority as the path between modes +four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0003, ADR-I-0005, ADR-I-0002 +``` + +Deletions that are deliverables, not side effects: + +```text +the hint carriers on the vector record type and the surface builders' hint population +the readable text column and the per-field payload index creation for dropped fields +the test-only payload field constants and the prose-assertion note constant +the service adapter's private enum token mappers, replaced by one Display/FromStr per enum in the domain (the embedded adapter must not add another copy) +the deterministic vector fake and its embedding-bearing record type, replaced by the embedded adapter opened in memory (failure-injecting and recording fakes stay) +the port doc comment's "documented bounded-overfetch degradation policy" clause, now expressed by the type ``` ## Non-goals ```text changing the authority split or any retrieval semantics -deprecating or altering the Qdrant adapter -approximate-nearest-neighbor indexing (LanceDB is the recorded escalation path if embedded ANN ever becomes necessary) -migration tooling between modes (rebuild-from-graph-authority is the documented path) -changing the default vector mode in this phase (embedded ships opt-in first; flipping the default is a separate decision once parity evidence exists) +deprecating or altering the service adapter +approximate-nearest-neighbour indexing (the recorded escalation path if embedded ANN ever becomes necessary) +migration tooling between modes or between record shapes (rebuild-from-graph-authority is the documented path) +changing the default vector mode in this phase (embedded ships opt-in; flipping the default is a separate evidence-gated decision) multi-process access to the embedded store (same single-process expectation as embedded graph storage) +any vector-layer predicate beyond the object-type scope (the two named re-entry paths belong to later phases) +any new public facade method (the evaluation baseline consumes the retrieval trace) +reconciliation diagnostics (the reconciliation slice was deleted in the structured-verdict phase; graph verification is the guard) ``` -## Technology posture (from the v0.1.5 closeout analysis) +## Technology posture -- SQLite exact-scan first: zero heavyweight dependencies (`rusqlite` direction already exists via the statistics store), exact filter semantics, deterministic, restart-safe. -- LanceDB recorded as the embedded-ANN escalation path if corpora outgrow exact scan. -- The in-process edge build of the current vector backend is a revisit candidate once it stabilizes; it would maximize payload-convention reuse and add a cloud-sync story. -- Deployments that outgrow the embedded mode are exactly the deployments that should use the service mode; document a corpus-size guidance number rather than engineering for it. +- SQLite exact scan first: zero new dependencies, exact filter semantics, deterministic, restart-safe. +- The first escalation inside the embedded mode is an in-memory normalised matrix loaded at open with write-through, an implementation optimisation that changes no contract. +- An embedded approximate-nearest-neighbour library is the recorded escalation path if corpora outgrow exact scan; a measured corpus exceeding the guidance or a benchmark showing the scan on the critical path reopens it. +- The in-process build of the service backend is a revisit candidate once it ships a stable release; it would maximise reuse of the service adapter's conventions. +- Deployments that outgrow the embedded mode are exactly the deployments that should use the service mode; publish a measured corpus-size guidance number rather than engineering for it (at a 3072-dimension model each ten thousand vectors is about 120 MB of embedding data read per query, which bounds the honest number). ## Acceptance criteria ```text Embedded mode is configurable and constructs without any running service. -The shared parity suite produces identical admitted candidate sets from both adapters across the full filter contract. +The parity suite produces identical admitted candidate sets and orderings from both adapters across the full contract, including identical-vector tie cohorts. Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical). -Embedded state survives process restart; reconciliation diagnostics work against the embedded store. -The default test path requires no vector service; Qdrant-gated suites continue to pass unchanged. -Documentation states the single-process expectation, the corpus-size guidance, and the rebuild-from-authority migration path. -No public facade change; no retrieval behavior change in service mode. +Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive, the service adapter reports closed on the tie fixture. +Embedded state survives process restart; a reopened store with a different vector size fails with the collection-compatibility error. +The default test path requires no vector service; service-gated suites continue to pass unchanged. +Both adapters persist exactly the five-field read contract; a census of both repositories shows no reader of a dropped field. +Documentation states the single-process expectation, the corpus-size guidance, and the rebuild-from-authority path. +No public facade change; no retrieval behaviour change in service mode beyond the added telemetry field. ``` -## Evaluation tie-in +## Cross-repository obligations (ADR-I-0026) + +The companion evaluation repository is a development aid, not core library functionality; these obligations land in the same wave as the library change. -The continuity evaluation suite gains an embedded-mode configuration so the confirmation scenarios (including restart) run against the embedded vector store; the frozen-embedding infrastructure applies unchanged. -Scenario baselines are expected to be identical between modes under the parity contract; any divergence is a finding, which makes the eval suite the cross-adapter regression instrument. +- Trace-sourced baseline: the vector-only baseline issues an ordinary traced `retrieve` with the measured object types and a generous candidate limit, slices the trace's vector candidates per object kind to its per-section budgets, and reads the completeness verdict from telemetry; the direct vector-service search, its payload field constants, and its hit mapping are deleted after an A/B run proves identical item identities and ranks. +- Item text: sourced from the evaluation repository's own ingest records keyed by external identity, never from a store payload. +- Telemetry mirror: the evaluation telemetry record gains the completeness field. +- Typed backend identity: result rows carry which vector backend (service or embedded) produced them, so cross-mode comparisons are attributable. +- Backend-neutral cleanup guard: the namespace cleanup guard protects an embedded store file by the same prefix rule that protects a service collection, and cleanup removes the SQLite file with its write-ahead-log sidecars alongside the statistics store. +- Configuration: a `vector_store_path` backend setting beside the graph and statistics paths; when set, the adapter selects embedded mode and derives a per-namespace file the way it derives the statistics path. +- Error vocabulary: the exhaustive conversion of the vector database error kinds gains the embedded engine kind. -## Open questions +## Evaluation tie-in -- Should embedded become the default vector mode once parity evidence exists, matching the embedded-default graph decision, or stay opt-in until a full release cycle passes? -- What corpus-size number goes in the guidance (the closeout analysis suggested exact-scan comfort up to low hundreds of thousands of vectors; measure rather than assume)? -- Does the parity suite live in the library's integration tests, the evaluation repository, or both (recommendation: shared fixtures in the library, evaluation reuse where cheap)? +The continuity evaluation suite gains an embedded-mode configuration so the confirmation scenarios, including restart, run against the embedded vector store; the frozen-embedding infrastructure applies unchanged. +Scenario results are expected to be identical between modes under the parity contract; any divergence is a finding, which makes the evaluation suite the cross-adapter regression instrument. + +## Deferral-reconfirmation checklist + +Each item was parked on this phase by the structured-verdict phase; each row states the parked claim, what was re-verified at design time, and the evidence the implementation must produce. + +1. Canonical-candidates newtype survival. + Parked claim: the newtype survives the port redesign or is absorbed into its result envelope. + Re-verified: every consumer is a slice read (the pipeline's count, telemetry, trace, and root selection sites, plus the test fakes); none relies on the newtype being the whole return value. + Evidence: after the change a census shows only the envelope field, the constructor, and the fakes' exhaustive wrapping; the existing deduplication-and-ordering test is unchanged. +2. Dual text columns. + Parked claim: the text columns' fate depends on the port's read contract. + Re-verified: the readable text column had exactly one reader (the evaluation baseline) and the embedded text column none; the evaluation repository can source item text from its own ingest. + Evidence: zero-hit census for the readable text column across both repositories; a vector-only run before and after produces identical item identities and text. +3. Search completeness. + Parked claim: the port cannot express whether the top-K was determinate. + Re-verified: the only degradation site is the service adapter's fetch bound; no pipeline path inspects or retries on it. + Evidence: the fetch-decision unit test asserts the open verdict at the bound; a retrieval test asserts the telemetry field per variant; the live boundary test asserts the closed verdict. +4. Hint filter semantics. + Parked claim: query-side hint semantics belong to the port contract. + Re-verified: the filter type and both match-or-unknown implementations were deleted in the structured-verdict phase, and no consumer asks for a vector-layer predicate (the evaluation surface policy carries object types and budgets only). + Evidence: zero-hit census for the filter type and for empty-or-null match conditions in the service adapter; the prohibition and re-entry paths are recorded in ADR-I-0024. +5. Evaluation baseline capability. + Parked claim: the baseline re-implements a hidden raw-vector capability against the payload schema. + Re-verified: trace-derived candidates with overfetch-and-slice can reproduce the direct-search baseline; the completeness verdict reports whether each question's top-K was determinate; the evaluation adapter can hold item text from ingest. + Evidence: the A/B run with row-level diff of item identities and ranks; after the switch, zero-hit census for vector-service search calls and payload constants in the evaluation adapter. + +## Decisions (the draft's open questions, resolved 2026-09-02) + +- Default mode: stays opt-in this phase; the flip is reopened by the evaluation suite running every dataset in embedded mode with identical results and one corpus at the guidance size (ADR-I-0023, Revisit When). +- Corpus-size guidance: measured in-phase by a benchmark over a synthetic corpus at the configured dimension, published in documentation, revised through documentation. +- Parity suite placement: contract parity in the library, behaviour parity in the evaluation repository (above). +- Settings shape: separate mode and path keys with `collection_name` as the backend-neutral namespace key, not a connection string interpreted by mode (ADR-I-0023). +- Hint families: all dropped from the vector record, with the two named re-entry paths (ADR-I-0024, ADR-I-0025). +- Text columns: readable text dropped, embedded text kept as provenance, governed by the three sentences in ADR-I-0025. +- Evaluation baseline: trace-sourced, no facade change (ADR-I-0026). diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index 4badbc91..45b65f86 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -259,6 +259,7 @@ Assisted remember workflows may accept raw or semi-raw input as transient proces | v0.1.3 | Remember intake interfaces and deterministic write planning | Finished. Generation-ready write path with `RememberWritePlan`, memory candidates, validation, deterministic helpers, prepare/validate/commit flow, and shared manual/future-generated commit machinery. | | v0.1.4 | Continuity evaluation harness | Finished. Deterministic long-horizon evaluation harness implemented in the public companion `CharacterMemoryEvals` repository as a development aid, not core library functionality: synthetic interaction fixtures, a minimal example assistant loop, continuity-oriented retrieval-quality metrics, selectivity/fanout measurement, and hub-entity stress scenarios. | | v0.1.5 | Eval-driven v0.1 family closeout | Finished. Ran the evaluation harness across the v0.1 family, dispositioned eleven findings (none critical, none open), fixed deterministic vector admission and write-path warning diagnostics in the library, retained the measured defaults with a recorded basis (ADR-I-0022), adopted embedded persistent Oxigraph as the validated default (ADR-I-0021), and expanded the evaluation suite to 33 scenarios including benchmark-adapted and real-embedding fixtures. Closeout report: [`v0_1_5_closeout_report.md`](v0_1_5_closeout_report.md). | +| v0.1.6 | Embedded vector candidate recall | Planned. An embedded SQLite exact-scan vector candidate store behind the vector port as an opt-in local mode, so the default deployment and test path need no external service; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores only the fields a reader consumes; the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | | v0.2 | Scoped continuity and reflection | `ContinuityScope`, scoped reflection, relationship state between arbitrary entities, character signals for continuing entities, open-loop/commitment lifecycle, and current continuity views. | | v0.3 | Factual rigor, temporal validity, and entity evolution | Assertions, claims, evidence links, belief assessments, source assessment, temporal validity, entity drift handling, and current-belief views. | | v0.4 | Retrieval observability and governance | Retrieval traces, context subgraphs, validation rules, graph health reports, policy diagnostics, rejected expansion traces, cluster/activation diagnostics, and retention assessment. | @@ -1101,6 +1102,7 @@ The intended sequence is: v0.1.3 completes the generation-ready write path v0.1.4 builds the harness that exercises the full write and retrieve paths v0.1.5 runs the harness, fixes what it reveals, and closes the v0.1 family +v0.1.6 removes the last external-service dependency from the default deployment and test path then v0.2 builds scoped continuity on a measured substrate ``` @@ -1194,7 +1196,57 @@ v0.2 entry is explicitly confirmed against the closed v0.1 family. --- -# 12. v0.2: scoped continuity and reflection +# 12. v0.1.6: embedded vector candidate recall + +Detailed draft: [`v0_1_6_embedded_vector_candidate_recall.md`](../design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md) + +Decisions: ADR-I-0023 (embedded exact-scan vector store as the opt-in local mode), ADR-I-0024 (vector candidate recall reports completeness and takes a scope-only query), ADR-I-0025 (the vector record is a read contract), ADR-I-0026 (raw vector baselines read the retrieval trace). + +## Intent + +Complete the zero-infrastructure local deployment story. +Graph authority already defaults to embedded persistent storage and retrieval statistics are file-backed; the vector candidate store is the only component that still requires an external service, which conflicts with desktop-companion and game deployments and keeps a service dependency in the default test path. +Because a second adapter must implement the vector port, this phase also settles the port contract that the structured-verdict work deferred: recall completeness is reported instead of silently degraded, the query is scope-only, and the stored payload is exactly what a reader consumes. + +## Goals + +```text +add an embedded SQLite exact-scan vector candidate store behind the existing vector port, selected by a store-mode setting with its own path setting +make the port result carry a typed completeness verdict that the retrieval telemetry records and never repairs +reduce the vector payload to its read contract: identity, surface, schema version, and the embedded text as provenance of what was ranked +run one shared contract suite against both adapters, with the embedded adapter exercised unconditionally so the default test path needs no service +move the evaluation repository's vector-only baseline onto the retrieval trace so no consumer depends on a store's private schema +record the re-entry paths for vector-layer predicates a later phase may need: a synchronized scope predicate, and an immutable time-window predicate +``` + +## Non-goals + +```text +changing the authority split or any retrieval semantics in the service mode +deprecating or altering the service-mode adapter beyond the shared port contract +approximate-nearest-neighbor indexing in the embedded mode +migration tooling between modes; rebuild from graph authority is the path +flipping the default vector mode in this phase +multi-process access to the embedded store +a public candidate-search facade +``` + +## Acceptance criteria + +```text +Embedded mode constructs and serves retrieval without any running service. +The shared contract suite produces identical admitted candidate sets from both adapters across the port contract. +Deterministic admission holds in embedded mode; repeated runs are byte-identical. +Embedded state survives process restart. +Retrieval telemetry reports the completeness verdict for every retrieval in both modes. +The default test path requires no vector service; service-gated suites still execute under the service-backed CI job and cannot pass by skipping. +The evaluation repository's vector-only baseline produces its rows from the retrieval trace in both modes. +No public facade change beyond the telemetry field; no retrieval behavior change in service mode. +``` + +--- + +# 13. v0.2: scoped continuity and reflection Detailed draft: [`v0_2_scoped_continuity_reflection.md`](../design/roadmap-phases/v0_2_scoped_continuity_reflection.md) @@ -1242,7 +1294,7 @@ Open loops and commitments can be retrieved by scope without assuming who the ma --- -# 13. v0.3: factual rigor, temporal validity, and entity evolution +# 14. v0.3: factual rigor, temporal validity, and entity evolution Detailed draft: [`v0_3_factual_rigor_temporal_validity_entity_evolution.md`](../design/roadmap-phases/v0_3_factual_rigor_temporal_validity_entity_evolution.md) @@ -1273,7 +1325,7 @@ This is important, but it should not block the starter because Character Memory' --- -# 14. v0.4: retrieval observability and governance +# 15. v0.4: retrieval observability and governance Detailed draft: [`v0_4_retrieval_observability_governance.md`](../design/roadmap-phases/v0_4_retrieval_observability_governance.md) @@ -1355,7 +1407,7 @@ The default intent is `Continuity`. --- -# 15. v0.5: controlled associative recall and clustering +# 16. v0.5: controlled associative recall and clustering Detailed draft: [`v0_5_controlled_associative_recall_clustering.md`](../design/roadmap-phases/v0_5_controlled_associative_recall_clustering.md) @@ -1417,7 +1469,7 @@ Durable graph truth is the associative unit, membership lifecycle, and support e --- -# 16. v0.6: assisted remember workflow and memory candidate generation +# 17. v0.6: assisted remember workflow and memory candidate generation Detailed draft: [`v0_6_assisted_remember_workflow_memory_candidate_generation.md`](../design/roadmap-phases/v0_6_assisted_remember_workflow_memory_candidate_generation.md) @@ -1511,7 +1563,7 @@ Generated candidates use the same validation and commit path as manual candidate --- -# 17. v1.0+: multimodal and embodied expansion +# 18. v1.0+: multimodal and embodied expansion Detailed draft: [`v1_0_multimodal_embodied_expansion.md`](../design/roadmap-phases/v1_0_multimodal_embodied_expansion.md) @@ -1539,7 +1591,7 @@ This is a future path, not starter scope. --- -# 18. Public API evolution +# 19. Public API evolution ## v0.1 API @@ -1687,7 +1739,7 @@ Generated processors should produce `MemoryCandidate` and `RememberWritePlan` va --- -# 19. YAGNI rules +# 20. YAGNI rules Do not implement in v0.1 / v0.1.2: From f225b257dcfe8e5cf443f20780306db31954ba5b Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 17:40:49 +0900 Subject: [PATCH 03/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes:=20ADR-I-00?= =?UTF-8?q?23=20adds=20a=20mode=20and=20supersedes=20nothing;=20schema-ver?= =?UTF-8?q?sion=20ruling=20in=20the=20plan;=20manifest=20wording;=20dated?= =?UTF-8?q?=20Record=20Shape;=20roadmap=20wording=20on=20the=20service=20d?= =?UTF-8?q?efault?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/HANDOFF.md | 78 +++++++++++++++++++ .../v0-1-6-embedded-vector-recall-plan.md | 2 +- .../ADR-I-0003-qdrant-oxigraph-defaults.md | 4 +- ...qlite-exact-scan-vector-candidate-store.md | 6 +- ...I-0025-vector-record-is-a-read-contract.md | 2 +- docs/design/database/vector_payload_design.md | 2 +- ...v0_1_6_embedded_vector_candidate_recall.md | 2 +- docs/roadmap/development_roadmap.md | 4 +- 8 files changed, 89 insertions(+), 11 deletions(-) create mode 100644 docs/coding-agent/HANDOFF.md diff --git a/docs/coding-agent/HANDOFF.md b/docs/coding-agent/HANDOFF.md new file mode 100644 index 00000000..4f8f8d19 --- /dev/null +++ b/docs/coding-agent/HANDOFF.md @@ -0,0 +1,78 @@ +# Session Handoff — 2026-07-23 (post structured-verdict-observability closeout; updated 2026-07-29 post legacy-reader removal) + +Audience: the next orchestrator session. Untracked working state; delete once absorbed. Committed records: the archived plan (`docs/coding-agent/plans/completed/structured-verdict-observability-plan.md`, full Decision Log), the design doc + 12 Amendments (`docs/design/structured_verdict_contract.md`), both repos' rule suites (heavily extended this phase), `FOLLOWUP-SEED.md` (next-work index), and auto-memory. + +## Setup + +- You are `orchestrator` in agmsg team `CharacterMemory`. Arm the inbox watcher per the SessionStart directive (30-60s attach; never double-invoke; on resume-restarts BOTH monitors die silently — re-arm the agmsg watcher yourself if no directive fires, and any PR monitor you need). +- Team (10): codex `worker`/`worker2`/`cm-reviewer`/`cm-researcher` rooted in CM; `evals-worker`/`evals-worker2`/`evals-reviewer`/`evals-researcher` rooted in CME; `ebigunso` via app. IMPORTANT — THE CODEX THREADS PERSIST AND ARE REUSED: only the orchestrator thread was retired. Every agent retains its full session context — standing role instructions, all protocol rules (tripwire, local-first commits, live-run mutex announce discipline, artifact placement, typed-from-introduction), the phase's rulings, and their own lesson history. Do NOT re-register roles or re-teach protocols; open with a brief 'new orchestrator session, context continues' note and dispatch normally. They read at turn boundaries; silence during gate runs is normal; they self-report strict YAML. The researchers are strictly read-only forensic (censuses with file:line, auditable methods, explicit zero-hits) for pre-ruling blast-radius coverage — both decision-grade. +- Where things stand: v0.1.5 + backcompat sweep + observability phase all MERGED (CM main 0408e71→b934d76 docs; CME main ea01f8e→038afdd docs). Everything green, all queues clear, zero unresolved anything. + +## Update 2026-09-02 (final) — Qdrant teardown hardening DONE: CM PR #70 → 512427e, CME PR #21 → 45d96c7 (both user-authorized squash merges) + +- Closeout COMPLETE (user-authorized 2026-09-02): CM closeout docs PR #71 merged as CM main 536d305 (plan in `plans/completed/`, two lessons entries, seed updated with the tracked follow-ups); the 15 July orphan collections pruned with the delivered script (Qdrant now holds 0 collections); merged branches gone in both repos, local and remote (GitHub auto-deleted the heads on merge; local refs pruned; only `main` and CM `draft/v0-1-6-embedded-vector-recall` remain); review worktrees removed; transient `.agent-work` reports deleted; sibling CM review clone pinned at CM main 512427e (code-identical to 536d305, docs-only delta; re-pin per review needs). Both checkouts on main. +- No user action outstanding from this phase. +- Next work: v0.1.6 planning (item 2 below), then v0.2, then harness candidates — unchanged. Codex sandbox delivery-layer investigation: CLOSED (below). + +## v0.1.6 PLANNING WAVE + design-flaw sweep — IN FLIGHT (started 2026-09-02 ~13:50, user-directed) + +- Branch `plan/v0-1-6-embedded-vector-recall` (main 536d305 + cherry-picked draft doc 1a5db55 → `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md`). Roadmap has NO v0.1.6 row yet (goes v0.1.5 → v0.2); insert before v0.2 when the plan is authored. +- Dispatched in parallel: cm-researcher (vector-port consumer census + CM design-flaw delta census → `.agent-work/researcher/v016-census.md`, token CENSUS_READY); evals-researcher (vector_only capability census + CME design-flaw census with BEFORE/ride/after triage → `.agent-work/evals-researcher/cme-census.md`, CENSUS_READY); Claude design consult (one-port design pass: R2-03 completeness envelope, R2-05 hint semantics, R2-13 text columns, CME r2#2 capability, CanonicalCandidates survival, SQLite adapter shape, ADR list → `.agent-work/orchestrator/v016-port-design-consult.md`); Claude altitude audits of CME (`cme-design-audit.md`) and CM (`cm-design-audit.md`) in `.agent-work/orchestrator/`. +- Sequence after results: consolidate the flaw inventory → triage BEFORE-v0.1.6 items with the user → ADRs first (user ordering rule), roadmap row, harness plan for approval → implementation only after ADRs merge. +- RESULTS + RULINGS (2026-09-02 ~15:00): all five inputs consolidated in `.agent-work/orchestrator/v016-consolidated-triage.md` (9 BEFORE items, 14 ride-with, ~20 after/leave). Ebigunso ruled all five design questions: (1) drop all thirty unread payload hint fields, ADR records two re-entry paths (synchronized scope predicate owned by scoped continuity; immutable time-window predicate over created_at/observed_at backfilled from graph authority); (2) eval vector-only baseline reads the retrieval trace, no public candidate-search facade; (3) keep `embedding_text` only, drop `content_text`, ADR codifies "read-out text lives in graph authority; the vector record stores only the embedded surface as provenance of what was ranked; consumers hydrate candidate content by object id"; (4) separate VECTOR_STORE_MODE/VECTOR_STORE_PATH, `collection_name` = backend-neutral namespace key; (5) default stays service, flip trigger recorded. Forward-looking analysis in consult memo section G. +- AUTHORED + PROMOTED (2026-09-02 ~17:40): CM PR #72 (commit 01f3782 on `plan/v0-1-6-embedded-vector-recall`) = ADR-I-0023..0026 (reciprocal partial supersession on ADR-I-0002/0003/0005), phase doc rewrite, roadmap row + sequence line + section 12 (13..20 renumbered), supersession note on `docs/design/database/vector_payload_design.md`, and the execution plan `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` (status draft; 8 tasks / 5 waves, contract-first: Task_1 live-gate hardening + Task_2 port envelope → Task_3 record read contract + Task_5 CME trace baseline → Task_4 embedded adapter/settings/parity → Task_6 CME embedded config + Task_8 design-value audit → Task_7 fake retirement/closeout). Merging #72 = user approval of ADRs + plan. Monitor armed. +- CME evidence-integrity light-delta (evals-worker, branch `chore/evidence-integrity-pre-v016`): implementation complete, tripwire ruled GO (MemoryAdapter batch methods return one outcome per batch; owns expanded to memory_adapter.rs + continuity driver.rs); in the live validation window at time of writing; then evals-reviewer review → PR. + +## Update 2026-09-02 — Qdrant teardown hardening (history of the day; superseded by the final entry above) + +- Machine rebuilt in Aug 2026 (fresh Docker 29.7, LLVM 22.1.8 installed 2026-09-02 for oxrocksdb-sys bindgen — pre-install shells need `LIBCLANG_PATH`; Qdrant volume survived; server now 1.19.0 because compose pinned `latest`). See auto-memory `machine-rebuilt-2026-09`. +- Live verification 2026-09-02: CM canary + live smoke + full suite green (write_planning 12s); CME three live tests 4/4 green incl. final cleanup, zero leaks, retry macro never fired; IPv6 loopback healthy. None of the July failure catalog reproduces. +- Reshaped spec approved by ebigunso; plan at `docs/coding-agent/plans/active/qdrant-teardown-hardening-plan.md`; Wave 1 dispatched to `worker` (CM Task_1) and `evals-worker` (CME Task_2) on branch `chore/qdrant-teardown-hardening` in each repo; Wave 2 = cm-reviewer / evals-reviewer. CM half DONE and promoted: cm-worker commit 1aa3dd6 + orchestrator docs commit 7f7b855, cm-reviewer APPROVED (zero findings), pushed as CM PR #70 → Copilot fix a2e1fcc (cm-reviewer re-approved) → MERGED 2026-09-02 as CM main 512427e (user-authorized squash). CME half: evals-worker Task_2 done at local cdaec3a (workspace 305/305, live 9/9, prune script validated on a throwaway; 15 orphans preserved for ebigunso to prune); Task_4 APPROVED by evals-reviewer (zero findings) → pushed as CME PR #21 (2026-09-02, monitor armed, awaiting CI/Copilot and user merge). Sibling CM review clone re-pinned to CM main 512427e (shim intact); review worktrees `.review-worktrees/cm-reviewer` (CM) and `.review-worktrees/evals-reviewer` (CME) still exist — remove at closeout. Still local in CM working tree (uncommitted, for a closeout docs PR): lessons.md two new entries + plan Progress/Decision Log updates; CM branch `chore/qdrant-teardown-hardening` + remote still exist (delete at closeout with user OK). +- Outstanding user action: prune the 15 July orphans with the new CME script once it lands (`scripts/qdrant_prune_collections.sh cmem_eval_continuity_continuity_v1_ --delete`); the orchestrator's auto-mode classifier blocks agent-initiated bulk deletes. +- Codex team survived the rebuild but was renamed App-side: `worker`→`cm-worker`, `worker2`→`cm-worker2` (address the new names). `cm-worker` ACKed 2026-09-02; evals-worker and both reviewers had not read any message at time of writing (terminals down). +- CODEX DELIVERY LAYER — RESOLVED 2026-09-02 (both sides, environment-only, no agmsg patch): ebigunso re-ran the Codex Windows sandbox setup (DPAPI secrets regenerated, `CodexSandboxOffline/Online` recreated; sandboxed launches verified incl. sqlite3 + git inside the sandbox) and installed PowerShell 7.4.19 (MSI) at `C:\Program Files\PowerShell\7`, which fresh codex processes pick automatically and under which the plain documented `send.sh … "$AGMSG_BODY"` form delivers intact. Codex app restarted 2026-09-02 13:4x: threads now run pwsh 7.4.19 (cm-worker's plain-form multi-word body verified intact); the escaped-quote workaround is RETIRED team-wide and the plain documented send form is standard. If a one-word body ever reappears, that thread is running Windows PowerShell 5.1 (shell is resolved at process start) — restart it. winget's 7.6.x is MSIX-only (WindowsApps, invisible to Codex's hardcoded path) — the 7.4 LTS MSI is the right package. Original root-cause record follows: (1) WRITE truncation = Windows PowerShell 5.1 strips embedded double quotes from the single-quoted `-lc` payload, so `"$AGMSG_BODY"` reaches bash unquoted and word-splits (reproduced outside any sandbox; escalation irrelevant). Proven fix: escape the inner quotes as `\"$AGMSG_BODY\"` inside the PS single-quoted payload (or `set -f; IFS=; … $AGMSG_BODY`). The agmsg SKILL.md/template.md Windows guidance is wrong for multi-word bodies — upstream fix belongs in `~/GitLocal/agmsg` (best: let send.sh take the body from `$AGMSG_BODY` when arg 4 is `-`, removing the quoting hazard entirely). (2) READ/sandbox = `CryptUnprotectData 2148073483` (0x8009000B NTE_BAD_KEY_STATE): `~/.codex/.sandbox-secrets/sandbox_users.json` (DPAPI blob dated 2026-03-20, old OS) cannot be decrypted after the 2026-08-10 OS reinstall and the `CodexSandboxOffline/Online` local accounts no longer exist, so every elevated-sandbox CreateProcess is rejected; all work ran through auto-approved escalation. USER FIX: close Codex, move `sandbox_users.json` aside, relaunch, accept the UAC sandbox-setup prompt, verify the two local accounts exist and a sandboxed `inbox.sh` works. (3) Codex launcher ignores `shell=Git Bash` and substitutes `C:\WINDOWS\system32\bash.exe` (WSL shim) — always wrap via PowerShell with the explicit Git Bash path. Original observations kept below: (a) every codex→orchestrator agmsg body arrives truncated at the first space — cm-worker's invocation is PowerShell here-string → `bash.exe -lc 'send.sh … "$AGMSG_BODY"'`; (b) an agent reported all shell launch paths (PowerShell, Git Bash, cmd) failing before process creation with `CryptUnprotectData error 2148073483` (Windows DPAPI). Workaround in force: codex agents write reports to `.agent-work//` and send a single-token notification (e.g. REPORT_READY); orchestrator→codex sends work normally. DIAGNOSTIC SIGNAL (ebigunso, 2026-09-02, observed on the codex threads): the inbox-check failures come from the Codex SANDBOX environment — the same commands run fine when escalated outside the sandbox in the same thread. So the investigation starts at the Codex Windows sandbox launcher (DPAPI/CryptUnprotectData inside the sandbox), not at agmsg scripts or shell quoting; the first-space truncation of sends is likely the same sandboxed hop mangling argv. Also: system gitconfig has `core.autocrlf=true`, both clones are CRLF in the working tree (commits normalize to LF; CME `.gitattributes` protects fixtures/configs). +- Task_1 tripwire ruled GO 2026-09-02: the qdrant-client 1.19.0 bump forces the CM dev-deps tonic 0.12→0.14 and dropping the `reqwest_012` alias (store.rs tests only); dependency alignment, not a workaround. + +## Update 2026-07-29 — CME legacy 1.0.0 reader removal COMPLETE (supersedes any bounded-legacy-dispatch mentions below/elsewhere) + +- Ebigunso ruled Option C (full deletion) over migration and containment; CME PR #20 squash-merged as 9a9f84a, closeout commits 49b97fb + 58179d9 pushed direct to main; CME main head now 58179d9. +- The "exactly one bounded legacy 1.0.0 dispatch" state is GONE: CME readers are strict current-schema-only; sealed evidence is bytes-by-hash (parseability explicitly not guaranteed); resurrection pointer = CME main 9997ccd, recorded in the findings-register addendum. +- ADRs committed BEFORE implementation per ebigunso's ordering preference ("decisions set in stone first"): ADR-I-0002 (single-schema artifact contract / bytes-by-hash sealing) + ADR-I-0003 (reader strictness scoped to hash-cited evidence readers); CME rules/common.md clause rewritten to match. +- Plan archived at CME docs/coding-agent/plans/completed/legacy-1-0-0-reader-removal-plan.md; CME HANDOFF.md deleted as absorbed; two new lessons in CME lessons.md (canonical-writer round-trip expectations; positive executed-count for filtered test evidence). +- New environment facts: CME main is now push-protected (PRs only) — ebigunso added protection 2026-07-29 after noticing the direct docs pushes from this phase's closeout; those were tolerated once but BOTH repos are PR-only from here. CME now has THREE live Qdrant tests to exclude by name in service-free runs (live_frozen_write_surface_matches_continuity_runtime_normalization joined live_adapter_reattaches_with_external_ids and live_reset_preserves_sibling_namespace_durable_stores). agmsg truncates at ~8K chars; ask senders to resend in labeled parts <6K. +- Durable-docs wording rulings (apply to all future ADRs/durable docs): version-agnostic wording by default (versions only as clarifying examples or dated history); no team-local jargon like "Tier A/D" — use terms an outside reader fully understands (e.g. "design-value audit"). +- Team/worktree state: review worktree and remediation branch removed; CME sibling review clone still pinned at CM 0408e71 (code-identical to CM main b934d76, docs-only delta). +- Next-work order unchanged: Qdrant teardown hardening (would also retire the live-test waiver context), then v0.1.6 planning, then v0.2. + +## Next work (in order, user-confirmed) + +1. Qdrant teardown hardening — first item; complete spec + failure catalog in FOLLOWUP-SEED.md. Dispatch shape: evals-worker (CME test support) + CM twin; retires the standing two-test teardown-transport waiver (scoped to observability-phase validation runs — a new phase touching those tests needs its own waiver decision until hardening lands). +2. v0.1.6 embedded vector-recall planning — draft branch `draft/v0-1-6-embedded-vector-recall` still exists; the phase doc does NOT exist yet. MUST absorb: the one-port design pass (R2-03 completeness envelope + CME vector_only capability + R2-05 hint semantics + R2-13 text columns), the Tier-A deferral-reconfirmation checklist (every consumer claim parked on v0.1.6 gets re-verified when the doc is authored), CanonicalCandidates survival expectation. +3. v0.2 (inherits R2-02/R2-04 lifecycle-mode work, R2-01 idempotency ledger, correction-path divergence-rejection residue). +4. Harness candidates: SIX staged in `skill-candidates.md` awaiting promotion (truth-tables era ×3 + workaround-tripwire/design_alerts, lossless-boundary, consolidation-completeness, negative-evidence, coordination/advice split, value-audit triggers). + +## Process rules born this phase (all codified — enforce them; listed so you know they're new and why) + +Workaround Tripwire (common.md both repos); push-after-internal-approval (orchestrator.md — pushes are the promotion step, workers commit local-only, reviewers pin from the LOCAL repo); Design-Consult Threshold (contract-shape rulings get a Claude design consult BEFORE ruling; skip only with recorded blast-radius); blast-radius rulings (you own everything the change affects — grep the sibling BEFORE ruling; two failures this phase prove why); Value-Audit Triggers (design review consumer-naming, 3rd-bounce-per-seam proportionality, pre-merge milestone gate, next-phase planning); Artifact Placement (.agent-work//, delete-or-promote stated in reports); typed-from-introduction (worker.md); reader-side admission strictness (CME common.md, SCOPED to hash-cited evidence readers); reviewer two-run trigger = byte-shape intent. + +NOT codified but proven practice (codify if it recurs): +- EXIT RUBRIC for terminal review loops (user-directed): in-PR fixes only for phase-delivered evidence-integrity defects or phase-introduced regressions; all else defers to seed with recorded disposition. Ended a very long Copilot tail; apply when a loop's marginal finding severity drops below cycle cost. +- LIVE-RUN MUTEX: shared Qdrant takes ONE live suite at a time; orchestrator schedules exclusive windows; agents announce START/END; prune orphan collections BEFORE granting (both prefixes: `test_collection_*` AND `cmem_eval_*` — sweeping only one was an actual mistake); readyz-probe at grant. +- LIGHT-DELTA path for small fixes: worker local commit → worker2 spot-check → reviewer offline formality → push, one relay each. +- Reviewer worktree provisioning is ORCHESTRATOR duty on handshake: `git worktree add .review-worktrees/ --detach`, remove the stale one; sibling-clone pin flips via stash-shim dance (stash Cargo.toml, fetch origin(=local CM repo), checkout --detach, stash pop — the uncommitted [workspace] shim is expected state). + +## Environment facts (thread-only) + +- Qdrant: docker container `charactermemory-qdrant-1`. gRPC path DEGRADES over uptime/load (delete-responses lost while REST stays 100ms-healthy; deterministic when degraded, on localhost AND VM-IP routes); remedy = `docker restart` then run promptly in the fresh window. Canonical: explicit `QDRANT_CONNECTION_STRING=http://127.0.0.1:6334` (never env-fallback → IPv6-localhost stall). Serial write_planning legitimately ~353s — set 600-900s caps. Prune recipe: REST DELETE per collection, both prefixes. Delete-timeout triage: REST-check the collection immediately — 404 means committed-but-response-lost (transport), present means server-side. +- Copilot auto-reviews every push; explicit re-request via REST fallback (`gh api .../requested_reviewers -f 'reviewers[]=copilot'`); resolve threads via GraphQL `resolveReviewThread`. It reviewed superbly this phase (~35 accepted findings) — treat its comments as signal, but through the exit rubric. +- CM main is push-protected (PRs only) — closeout docs went via PR #66/#16; plan any direct-to-main docs accordingly. +- agmsg send.sh = exactly `TEAM FROM TO MESSAGE` (extra positional silently eats the body); long reviewer messages sometimes truncate/timeout — ask for resend. The Bash safety classifier had one transient outage; queue sends and retry. +- The design-consult CLAUDE SUBAGENT (resident context: design doc + amendments + all identity rulings) dies with the retired orchestrator thread — unlike the codex agents, which persist. Its replacement: spawn fresh consults reading `structured_verdict_contract.md` + the archived plan's Decision Log — those two documents ARE the resident context, deliberately. + +## Loose ends (small, none urgent) + +- Branch/worktree cleanup DONE (user-authorized 2026-07-23): all merged branches deleted local+remote in both repos (lineage-verified before deletion — squash merges make commit-count checks useless; ancestor/content checks used instead); all review worktrees removed; empty .agent-work dirs removed; no stashes anywhere. KEPT deliberately: CM `draft/v0-1-6-embedded-vector-recall` (local+remote, needed for v0.1.6 planning), the CME sibling review clone (pinned CM main 0408e71, shim as expected), HANDOFF.md + FOLLOWUP-SEED.md + .claude/ untracked in CM. +- OVERSIZED trims seeded: MutationPlan stats-projection clones (next correct_forget touch); CME mirror vocabulary is the watch-item for dead weight. +- Exit-rubric deferrals live in the seed: bm25_only surface validation (typed-family fix shape recorded); input-side Value-parse sites (fixture/enrichment/loaders/predictions). +- `.agent-work/` gitignored dirs may exist per role — agents self-clean, but verify empty at phase ends. +- Sibling review clone (`CharacterMemoryEvals/.review-worktrees/CharacterMemory`) pinned at CM main 0408e71, shim preserved — re-pin per review needs. diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index f1a910de..d97ffcfe 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -103,7 +103,7 @@ - docs/design/database/vector_payload_design.md - depends_on: [Task_2] - description: | - Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Existing stored payloads with extra fields are tolerated unread (schema version unchanged unless the reader contract changes). + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. - acceptance: - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository's reader is removed by Task_5). diff --git a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md index 032c9fa2..706b0aa8 100644 --- a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md +++ b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md -supersession_scope: partial +superseded_by: null +supersession_scope: null --- # ADR-I-0003: Use Qdrant and Oxigraph as default storage backends diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md index 8460e18f..6e751825 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md @@ -13,9 +13,9 @@ warrant: cost_of_over_extension: "applying the single-process expectation to multi-replica deployments, or treating the embedded mode as the validated default before parity evidence exists, misrepresents what the library has validated" depends_on: [implementation/ADR-I-0009-use-sqlite-as-default-retrieval-stats-store.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] implements: [] -supersedes: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md] +supersedes: [] superseded_by: null -supersession_scope: partial +supersession_scope: null --- # ADR-I-0023: Embedded SQLite exact-scan vector candidate store as the opt-in local mode @@ -110,6 +110,6 @@ Question asked: whether the embedded store should overload the service connectio ## More Information -- ADR-I-0003 remains authoritative for the service-mode backends; this record supersedes only its claim that those backends are the sole defaults. +- ADR-I-0003 remains fully authoritative for the default backends; this record adds an opt-in mode in response to its revisit clause and changes no default, so it supersedes nothing. A later, evidence-gated record that flips the default would supersede ADR-I-0003's vector-backend default. - ADR-I-0024 (port contract this adapter implements) and ADR-I-0025 (the record it stores). - The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index b4da592c..e724eb16 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -80,7 +80,7 @@ Option 5 is the only subset with a forward-looking case that survives the synchr ## Consequences -- Positive: the embedded schema is five columns and one index; both adapters mirror one small manifest. +- Positive: both adapters mirror one five-field manifest; the embedded schema stores those fields beside the vector blob with a single scope index (ADR-I-0023 owns the physical layout). - Positive: the embedded surface is preserved as vector provenance before surfaces become generated. - Negative / tradeoffs: a future scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the re-entry paths make that step predictable. diff --git a/docs/design/database/vector_payload_design.md b/docs/design/database/vector_payload_design.md index 4d2b9ed8..e7c25fcf 100644 --- a/docs/design/database/vector_payload_design.md +++ b/docs/design/database/vector_payload_design.md @@ -1,6 +1,6 @@ # Vector Database Payload Design -> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 records the re-entry paths for any returning prefilter column. The Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were; the Design Goal, Record Shape, Why Natural-Language Surfaces, and Consistency Model sections remain current. +> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 records the re-entry paths for any returning prefilter column. The Record Shape, Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were (Record Shape still lists the graph URI, which the read contract dropped); the Design Goal, Why Natural-Language Surfaces, and Consistency Model sections remain current. This document describes the Qdrant payload design for Character Memory. It is intentionally a design note, not a field-by-field copy of the Rust mapping code. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 716f8d82..d57539a5 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -82,7 +82,7 @@ port-conformance parity suite in the library integration tests, run against both restart-safety test for the embedded store; pipeline test over the embedded adapter with a deleted graph object measured corpus-size guidance from an in-phase benchmark documentation: settings, single-process expectation, corpus-size guidance, rebuild-from-graph-authority as the path between modes -four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0003, ADR-I-0005, ADR-I-0002 +four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0005 and ADR-I-0002 ``` Deletions that are deliverables, not side effects: diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index 45b65f86..8f11285e 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -259,7 +259,7 @@ Assisted remember workflows may accept raw or semi-raw input as transient proces | v0.1.3 | Remember intake interfaces and deterministic write planning | Finished. Generation-ready write path with `RememberWritePlan`, memory candidates, validation, deterministic helpers, prepare/validate/commit flow, and shared manual/future-generated commit machinery. | | v0.1.4 | Continuity evaluation harness | Finished. Deterministic long-horizon evaluation harness implemented in the public companion `CharacterMemoryEvals` repository as a development aid, not core library functionality: synthetic interaction fixtures, a minimal example assistant loop, continuity-oriented retrieval-quality metrics, selectivity/fanout measurement, and hub-entity stress scenarios. | | v0.1.5 | Eval-driven v0.1 family closeout | Finished. Ran the evaluation harness across the v0.1 family, dispositioned eleven findings (none critical, none open), fixed deterministic vector admission and write-path warning diagnostics in the library, retained the measured defaults with a recorded basis (ADR-I-0022), adopted embedded persistent Oxigraph as the validated default (ADR-I-0021), and expanded the evaluation suite to 33 scenarios including benchmark-adapted and real-embedding fixtures. Closeout report: [`v0_1_5_closeout_report.md`](v0_1_5_closeout_report.md). | -| v0.1.6 | Embedded vector candidate recall | Planned. An embedded SQLite exact-scan vector candidate store behind the vector port as an opt-in local mode, so the default deployment and test path need no external service; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores only the fields a reader consumes; the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | +| v0.1.6 | Embedded vector candidate recall | Planned. An embedded SQLite exact-scan vector candidate store behind the vector port as an opt-in local mode, so zero-infrastructure local deployments become possible and the default test path needs no external service while the service mode stays the default; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores only the fields a reader consumes; the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | | v0.2 | Scoped continuity and reflection | `ContinuityScope`, scoped reflection, relationship state between arbitrary entities, character signals for continuing entities, open-loop/commitment lifecycle, and current continuity views. | | v0.3 | Factual rigor, temporal validity, and entity evolution | Assertions, claims, evidence links, belief assessments, source assessment, temporal validity, entity drift handling, and current-belief views. | | v0.4 | Retrieval observability and governance | Retrieval traces, context subgraphs, validation rules, graph health reports, policy diagnostics, rejected expansion traces, cluster/activation diagnostics, and retention assessment. | @@ -1102,7 +1102,7 @@ The intended sequence is: v0.1.3 completes the generation-ready write path v0.1.4 builds the harness that exercises the full write and retrieve paths v0.1.5 runs the harness, fixes what it reveals, and closes the v0.1 family -v0.1.6 removes the last external-service dependency from the default deployment and test path +v0.1.6 makes zero-infrastructure local deployment possible and removes the service dependency from the default test path then v0.2 builds scoped continuity on a measured substrate ``` From 6b0b32d9029d19c1429d85a0a66eb3679c9e2b9e Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 17:46:07 +0900 Subject: [PATCH 04/44] =?UTF-8?q?=F0=9F=93=9D=20Remove=20the=20untracked?= =?UTF-8?q?=20orchestrator=20handoff=20note=20that=20a=20broad=20add=20swe?= =?UTF-8?q?pt=20into=20the=20branch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/HANDOFF.md | 78 ------------------------------------ 1 file changed, 78 deletions(-) delete mode 100644 docs/coding-agent/HANDOFF.md diff --git a/docs/coding-agent/HANDOFF.md b/docs/coding-agent/HANDOFF.md deleted file mode 100644 index 4f8f8d19..00000000 --- a/docs/coding-agent/HANDOFF.md +++ /dev/null @@ -1,78 +0,0 @@ -# Session Handoff — 2026-07-23 (post structured-verdict-observability closeout; updated 2026-07-29 post legacy-reader removal) - -Audience: the next orchestrator session. Untracked working state; delete once absorbed. Committed records: the archived plan (`docs/coding-agent/plans/completed/structured-verdict-observability-plan.md`, full Decision Log), the design doc + 12 Amendments (`docs/design/structured_verdict_contract.md`), both repos' rule suites (heavily extended this phase), `FOLLOWUP-SEED.md` (next-work index), and auto-memory. - -## Setup - -- You are `orchestrator` in agmsg team `CharacterMemory`. Arm the inbox watcher per the SessionStart directive (30-60s attach; never double-invoke; on resume-restarts BOTH monitors die silently — re-arm the agmsg watcher yourself if no directive fires, and any PR monitor you need). -- Team (10): codex `worker`/`worker2`/`cm-reviewer`/`cm-researcher` rooted in CM; `evals-worker`/`evals-worker2`/`evals-reviewer`/`evals-researcher` rooted in CME; `ebigunso` via app. IMPORTANT — THE CODEX THREADS PERSIST AND ARE REUSED: only the orchestrator thread was retired. Every agent retains its full session context — standing role instructions, all protocol rules (tripwire, local-first commits, live-run mutex announce discipline, artifact placement, typed-from-introduction), the phase's rulings, and their own lesson history. Do NOT re-register roles or re-teach protocols; open with a brief 'new orchestrator session, context continues' note and dispatch normally. They read at turn boundaries; silence during gate runs is normal; they self-report strict YAML. The researchers are strictly read-only forensic (censuses with file:line, auditable methods, explicit zero-hits) for pre-ruling blast-radius coverage — both decision-grade. -- Where things stand: v0.1.5 + backcompat sweep + observability phase all MERGED (CM main 0408e71→b934d76 docs; CME main ea01f8e→038afdd docs). Everything green, all queues clear, zero unresolved anything. - -## Update 2026-09-02 (final) — Qdrant teardown hardening DONE: CM PR #70 → 512427e, CME PR #21 → 45d96c7 (both user-authorized squash merges) - -- Closeout COMPLETE (user-authorized 2026-09-02): CM closeout docs PR #71 merged as CM main 536d305 (plan in `plans/completed/`, two lessons entries, seed updated with the tracked follow-ups); the 15 July orphan collections pruned with the delivered script (Qdrant now holds 0 collections); merged branches gone in both repos, local and remote (GitHub auto-deleted the heads on merge; local refs pruned; only `main` and CM `draft/v0-1-6-embedded-vector-recall` remain); review worktrees removed; transient `.agent-work` reports deleted; sibling CM review clone pinned at CM main 512427e (code-identical to 536d305, docs-only delta; re-pin per review needs). Both checkouts on main. -- No user action outstanding from this phase. -- Next work: v0.1.6 planning (item 2 below), then v0.2, then harness candidates — unchanged. Codex sandbox delivery-layer investigation: CLOSED (below). - -## v0.1.6 PLANNING WAVE + design-flaw sweep — IN FLIGHT (started 2026-09-02 ~13:50, user-directed) - -- Branch `plan/v0-1-6-embedded-vector-recall` (main 536d305 + cherry-picked draft doc 1a5db55 → `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md`). Roadmap has NO v0.1.6 row yet (goes v0.1.5 → v0.2); insert before v0.2 when the plan is authored. -- Dispatched in parallel: cm-researcher (vector-port consumer census + CM design-flaw delta census → `.agent-work/researcher/v016-census.md`, token CENSUS_READY); evals-researcher (vector_only capability census + CME design-flaw census with BEFORE/ride/after triage → `.agent-work/evals-researcher/cme-census.md`, CENSUS_READY); Claude design consult (one-port design pass: R2-03 completeness envelope, R2-05 hint semantics, R2-13 text columns, CME r2#2 capability, CanonicalCandidates survival, SQLite adapter shape, ADR list → `.agent-work/orchestrator/v016-port-design-consult.md`); Claude altitude audits of CME (`cme-design-audit.md`) and CM (`cm-design-audit.md`) in `.agent-work/orchestrator/`. -- Sequence after results: consolidate the flaw inventory → triage BEFORE-v0.1.6 items with the user → ADRs first (user ordering rule), roadmap row, harness plan for approval → implementation only after ADRs merge. -- RESULTS + RULINGS (2026-09-02 ~15:00): all five inputs consolidated in `.agent-work/orchestrator/v016-consolidated-triage.md` (9 BEFORE items, 14 ride-with, ~20 after/leave). Ebigunso ruled all five design questions: (1) drop all thirty unread payload hint fields, ADR records two re-entry paths (synchronized scope predicate owned by scoped continuity; immutable time-window predicate over created_at/observed_at backfilled from graph authority); (2) eval vector-only baseline reads the retrieval trace, no public candidate-search facade; (3) keep `embedding_text` only, drop `content_text`, ADR codifies "read-out text lives in graph authority; the vector record stores only the embedded surface as provenance of what was ranked; consumers hydrate candidate content by object id"; (4) separate VECTOR_STORE_MODE/VECTOR_STORE_PATH, `collection_name` = backend-neutral namespace key; (5) default stays service, flip trigger recorded. Forward-looking analysis in consult memo section G. -- AUTHORED + PROMOTED (2026-09-02 ~17:40): CM PR #72 (commit 01f3782 on `plan/v0-1-6-embedded-vector-recall`) = ADR-I-0023..0026 (reciprocal partial supersession on ADR-I-0002/0003/0005), phase doc rewrite, roadmap row + sequence line + section 12 (13..20 renumbered), supersession note on `docs/design/database/vector_payload_design.md`, and the execution plan `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` (status draft; 8 tasks / 5 waves, contract-first: Task_1 live-gate hardening + Task_2 port envelope → Task_3 record read contract + Task_5 CME trace baseline → Task_4 embedded adapter/settings/parity → Task_6 CME embedded config + Task_8 design-value audit → Task_7 fake retirement/closeout). Merging #72 = user approval of ADRs + plan. Monitor armed. -- CME evidence-integrity light-delta (evals-worker, branch `chore/evidence-integrity-pre-v016`): implementation complete, tripwire ruled GO (MemoryAdapter batch methods return one outcome per batch; owns expanded to memory_adapter.rs + continuity driver.rs); in the live validation window at time of writing; then evals-reviewer review → PR. - -## Update 2026-09-02 — Qdrant teardown hardening (history of the day; superseded by the final entry above) - -- Machine rebuilt in Aug 2026 (fresh Docker 29.7, LLVM 22.1.8 installed 2026-09-02 for oxrocksdb-sys bindgen — pre-install shells need `LIBCLANG_PATH`; Qdrant volume survived; server now 1.19.0 because compose pinned `latest`). See auto-memory `machine-rebuilt-2026-09`. -- Live verification 2026-09-02: CM canary + live smoke + full suite green (write_planning 12s); CME three live tests 4/4 green incl. final cleanup, zero leaks, retry macro never fired; IPv6 loopback healthy. None of the July failure catalog reproduces. -- Reshaped spec approved by ebigunso; plan at `docs/coding-agent/plans/active/qdrant-teardown-hardening-plan.md`; Wave 1 dispatched to `worker` (CM Task_1) and `evals-worker` (CME Task_2) on branch `chore/qdrant-teardown-hardening` in each repo; Wave 2 = cm-reviewer / evals-reviewer. CM half DONE and promoted: cm-worker commit 1aa3dd6 + orchestrator docs commit 7f7b855, cm-reviewer APPROVED (zero findings), pushed as CM PR #70 → Copilot fix a2e1fcc (cm-reviewer re-approved) → MERGED 2026-09-02 as CM main 512427e (user-authorized squash). CME half: evals-worker Task_2 done at local cdaec3a (workspace 305/305, live 9/9, prune script validated on a throwaway; 15 orphans preserved for ebigunso to prune); Task_4 APPROVED by evals-reviewer (zero findings) → pushed as CME PR #21 (2026-09-02, monitor armed, awaiting CI/Copilot and user merge). Sibling CM review clone re-pinned to CM main 512427e (shim intact); review worktrees `.review-worktrees/cm-reviewer` (CM) and `.review-worktrees/evals-reviewer` (CME) still exist — remove at closeout. Still local in CM working tree (uncommitted, for a closeout docs PR): lessons.md two new entries + plan Progress/Decision Log updates; CM branch `chore/qdrant-teardown-hardening` + remote still exist (delete at closeout with user OK). -- Outstanding user action: prune the 15 July orphans with the new CME script once it lands (`scripts/qdrant_prune_collections.sh cmem_eval_continuity_continuity_v1_ --delete`); the orchestrator's auto-mode classifier blocks agent-initiated bulk deletes. -- Codex team survived the rebuild but was renamed App-side: `worker`→`cm-worker`, `worker2`→`cm-worker2` (address the new names). `cm-worker` ACKed 2026-09-02; evals-worker and both reviewers had not read any message at time of writing (terminals down). -- CODEX DELIVERY LAYER — RESOLVED 2026-09-02 (both sides, environment-only, no agmsg patch): ebigunso re-ran the Codex Windows sandbox setup (DPAPI secrets regenerated, `CodexSandboxOffline/Online` recreated; sandboxed launches verified incl. sqlite3 + git inside the sandbox) and installed PowerShell 7.4.19 (MSI) at `C:\Program Files\PowerShell\7`, which fresh codex processes pick automatically and under which the plain documented `send.sh … "$AGMSG_BODY"` form delivers intact. Codex app restarted 2026-09-02 13:4x: threads now run pwsh 7.4.19 (cm-worker's plain-form multi-word body verified intact); the escaped-quote workaround is RETIRED team-wide and the plain documented send form is standard. If a one-word body ever reappears, that thread is running Windows PowerShell 5.1 (shell is resolved at process start) — restart it. winget's 7.6.x is MSIX-only (WindowsApps, invisible to Codex's hardcoded path) — the 7.4 LTS MSI is the right package. Original root-cause record follows: (1) WRITE truncation = Windows PowerShell 5.1 strips embedded double quotes from the single-quoted `-lc` payload, so `"$AGMSG_BODY"` reaches bash unquoted and word-splits (reproduced outside any sandbox; escalation irrelevant). Proven fix: escape the inner quotes as `\"$AGMSG_BODY\"` inside the PS single-quoted payload (or `set -f; IFS=; … $AGMSG_BODY`). The agmsg SKILL.md/template.md Windows guidance is wrong for multi-word bodies — upstream fix belongs in `~/GitLocal/agmsg` (best: let send.sh take the body from `$AGMSG_BODY` when arg 4 is `-`, removing the quoting hazard entirely). (2) READ/sandbox = `CryptUnprotectData 2148073483` (0x8009000B NTE_BAD_KEY_STATE): `~/.codex/.sandbox-secrets/sandbox_users.json` (DPAPI blob dated 2026-03-20, old OS) cannot be decrypted after the 2026-08-10 OS reinstall and the `CodexSandboxOffline/Online` local accounts no longer exist, so every elevated-sandbox CreateProcess is rejected; all work ran through auto-approved escalation. USER FIX: close Codex, move `sandbox_users.json` aside, relaunch, accept the UAC sandbox-setup prompt, verify the two local accounts exist and a sandboxed `inbox.sh` works. (3) Codex launcher ignores `shell=Git Bash` and substitutes `C:\WINDOWS\system32\bash.exe` (WSL shim) — always wrap via PowerShell with the explicit Git Bash path. Original observations kept below: (a) every codex→orchestrator agmsg body arrives truncated at the first space — cm-worker's invocation is PowerShell here-string → `bash.exe -lc 'send.sh … "$AGMSG_BODY"'`; (b) an agent reported all shell launch paths (PowerShell, Git Bash, cmd) failing before process creation with `CryptUnprotectData error 2148073483` (Windows DPAPI). Workaround in force: codex agents write reports to `.agent-work//` and send a single-token notification (e.g. REPORT_READY); orchestrator→codex sends work normally. DIAGNOSTIC SIGNAL (ebigunso, 2026-09-02, observed on the codex threads): the inbox-check failures come from the Codex SANDBOX environment — the same commands run fine when escalated outside the sandbox in the same thread. So the investigation starts at the Codex Windows sandbox launcher (DPAPI/CryptUnprotectData inside the sandbox), not at agmsg scripts or shell quoting; the first-space truncation of sends is likely the same sandboxed hop mangling argv. Also: system gitconfig has `core.autocrlf=true`, both clones are CRLF in the working tree (commits normalize to LF; CME `.gitattributes` protects fixtures/configs). -- Task_1 tripwire ruled GO 2026-09-02: the qdrant-client 1.19.0 bump forces the CM dev-deps tonic 0.12→0.14 and dropping the `reqwest_012` alias (store.rs tests only); dependency alignment, not a workaround. - -## Update 2026-07-29 — CME legacy 1.0.0 reader removal COMPLETE (supersedes any bounded-legacy-dispatch mentions below/elsewhere) - -- Ebigunso ruled Option C (full deletion) over migration and containment; CME PR #20 squash-merged as 9a9f84a, closeout commits 49b97fb + 58179d9 pushed direct to main; CME main head now 58179d9. -- The "exactly one bounded legacy 1.0.0 dispatch" state is GONE: CME readers are strict current-schema-only; sealed evidence is bytes-by-hash (parseability explicitly not guaranteed); resurrection pointer = CME main 9997ccd, recorded in the findings-register addendum. -- ADRs committed BEFORE implementation per ebigunso's ordering preference ("decisions set in stone first"): ADR-I-0002 (single-schema artifact contract / bytes-by-hash sealing) + ADR-I-0003 (reader strictness scoped to hash-cited evidence readers); CME rules/common.md clause rewritten to match. -- Plan archived at CME docs/coding-agent/plans/completed/legacy-1-0-0-reader-removal-plan.md; CME HANDOFF.md deleted as absorbed; two new lessons in CME lessons.md (canonical-writer round-trip expectations; positive executed-count for filtered test evidence). -- New environment facts: CME main is now push-protected (PRs only) — ebigunso added protection 2026-07-29 after noticing the direct docs pushes from this phase's closeout; those were tolerated once but BOTH repos are PR-only from here. CME now has THREE live Qdrant tests to exclude by name in service-free runs (live_frozen_write_surface_matches_continuity_runtime_normalization joined live_adapter_reattaches_with_external_ids and live_reset_preserves_sibling_namespace_durable_stores). agmsg truncates at ~8K chars; ask senders to resend in labeled parts <6K. -- Durable-docs wording rulings (apply to all future ADRs/durable docs): version-agnostic wording by default (versions only as clarifying examples or dated history); no team-local jargon like "Tier A/D" — use terms an outside reader fully understands (e.g. "design-value audit"). -- Team/worktree state: review worktree and remediation branch removed; CME sibling review clone still pinned at CM 0408e71 (code-identical to CM main b934d76, docs-only delta). -- Next-work order unchanged: Qdrant teardown hardening (would also retire the live-test waiver context), then v0.1.6 planning, then v0.2. - -## Next work (in order, user-confirmed) - -1. Qdrant teardown hardening — first item; complete spec + failure catalog in FOLLOWUP-SEED.md. Dispatch shape: evals-worker (CME test support) + CM twin; retires the standing two-test teardown-transport waiver (scoped to observability-phase validation runs — a new phase touching those tests needs its own waiver decision until hardening lands). -2. v0.1.6 embedded vector-recall planning — draft branch `draft/v0-1-6-embedded-vector-recall` still exists; the phase doc does NOT exist yet. MUST absorb: the one-port design pass (R2-03 completeness envelope + CME vector_only capability + R2-05 hint semantics + R2-13 text columns), the Tier-A deferral-reconfirmation checklist (every consumer claim parked on v0.1.6 gets re-verified when the doc is authored), CanonicalCandidates survival expectation. -3. v0.2 (inherits R2-02/R2-04 lifecycle-mode work, R2-01 idempotency ledger, correction-path divergence-rejection residue). -4. Harness candidates: SIX staged in `skill-candidates.md` awaiting promotion (truth-tables era ×3 + workaround-tripwire/design_alerts, lossless-boundary, consolidation-completeness, negative-evidence, coordination/advice split, value-audit triggers). - -## Process rules born this phase (all codified — enforce them; listed so you know they're new and why) - -Workaround Tripwire (common.md both repos); push-after-internal-approval (orchestrator.md — pushes are the promotion step, workers commit local-only, reviewers pin from the LOCAL repo); Design-Consult Threshold (contract-shape rulings get a Claude design consult BEFORE ruling; skip only with recorded blast-radius); blast-radius rulings (you own everything the change affects — grep the sibling BEFORE ruling; two failures this phase prove why); Value-Audit Triggers (design review consumer-naming, 3rd-bounce-per-seam proportionality, pre-merge milestone gate, next-phase planning); Artifact Placement (.agent-work//, delete-or-promote stated in reports); typed-from-introduction (worker.md); reader-side admission strictness (CME common.md, SCOPED to hash-cited evidence readers); reviewer two-run trigger = byte-shape intent. - -NOT codified but proven practice (codify if it recurs): -- EXIT RUBRIC for terminal review loops (user-directed): in-PR fixes only for phase-delivered evidence-integrity defects or phase-introduced regressions; all else defers to seed with recorded disposition. Ended a very long Copilot tail; apply when a loop's marginal finding severity drops below cycle cost. -- LIVE-RUN MUTEX: shared Qdrant takes ONE live suite at a time; orchestrator schedules exclusive windows; agents announce START/END; prune orphan collections BEFORE granting (both prefixes: `test_collection_*` AND `cmem_eval_*` — sweeping only one was an actual mistake); readyz-probe at grant. -- LIGHT-DELTA path for small fixes: worker local commit → worker2 spot-check → reviewer offline formality → push, one relay each. -- Reviewer worktree provisioning is ORCHESTRATOR duty on handshake: `git worktree add .review-worktrees/ --detach`, remove the stale one; sibling-clone pin flips via stash-shim dance (stash Cargo.toml, fetch origin(=local CM repo), checkout --detach, stash pop — the uncommitted [workspace] shim is expected state). - -## Environment facts (thread-only) - -- Qdrant: docker container `charactermemory-qdrant-1`. gRPC path DEGRADES over uptime/load (delete-responses lost while REST stays 100ms-healthy; deterministic when degraded, on localhost AND VM-IP routes); remedy = `docker restart` then run promptly in the fresh window. Canonical: explicit `QDRANT_CONNECTION_STRING=http://127.0.0.1:6334` (never env-fallback → IPv6-localhost stall). Serial write_planning legitimately ~353s — set 600-900s caps. Prune recipe: REST DELETE per collection, both prefixes. Delete-timeout triage: REST-check the collection immediately — 404 means committed-but-response-lost (transport), present means server-side. -- Copilot auto-reviews every push; explicit re-request via REST fallback (`gh api .../requested_reviewers -f 'reviewers[]=copilot'`); resolve threads via GraphQL `resolveReviewThread`. It reviewed superbly this phase (~35 accepted findings) — treat its comments as signal, but through the exit rubric. -- CM main is push-protected (PRs only) — closeout docs went via PR #66/#16; plan any direct-to-main docs accordingly. -- agmsg send.sh = exactly `TEAM FROM TO MESSAGE` (extra positional silently eats the body); long reviewer messages sometimes truncate/timeout — ask for resend. The Bash safety classifier had one transient outage; queue sends and retry. -- The design-consult CLAUDE SUBAGENT (resident context: design doc + amendments + all identity rulings) dies with the retired orchestrator thread — unlike the codex agents, which persist. Its replacement: spawn fresh consults reading `structured_verdict_contract.md` + the archived plan's Decision Log — those two documents ARE the resident context, deliberately. - -## Loose ends (small, none urgent) - -- Branch/worktree cleanup DONE (user-authorized 2026-07-23): all merged branches deleted local+remote in both repos (lineage-verified before deletion — squash merges make commit-count checks useless; ancestor/content checks used instead); all review worktrees removed; empty .agent-work dirs removed; no stashes anywhere. KEPT deliberately: CM `draft/v0-1-6-embedded-vector-recall` (local+remote, needed for v0.1.6 planning), the CME sibling review clone (pinned CM main 0408e71, shim as expected), HANDOFF.md + FOLLOWUP-SEED.md + .claude/ untracked in CM. -- OVERSIZED trims seeded: MutationPlan stats-projection clones (next correct_forget touch); CME mirror vocabulary is the watch-item for dead weight. -- Exit-rubric deferrals live in the seed: bm25_only surface validation (typed-family fix shape recorded); input-side Value-parse sites (fixture/enrichment/loaders/predictions). -- `.agent-work/` gitignored dirs may exist per role — agents self-clean, but verify empty at phase ends. -- Sibling review clone (`CharacterMemoryEvals/.review-worktrees/CharacterMemory`) pinned at CM main 0408e71, shim preserved — re-pin per review needs. From f72f5279d49494b8c3d7ff8c0fd399807eb045a6 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 17:47:34 +0900 Subject: [PATCH 05/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?:=20query=20normalised=20once=20before=20dot-product=20scoring?= =?UTF-8?q?=20with=20a=20non-unit=20parity=20fixture;=20vector-only=20base?= =?UTF-8?q?line=20issues=20one=20singleton-scoped=20traced=20retrieval=20p?= =?UTF-8?q?er=20kind?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 6 +++--- ...mbedded-sqlite-exact-scan-vector-candidate-store.md | 3 ++- ...26-raw-vector-baselines-read-the-retrieval-trace.md | 10 +++++----- .../v0_1_6_embedded_vector_candidate_recall.md | 6 +++--- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index d97ffcfe..d3fe3a5a 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -133,7 +133,7 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter per the phase document (schema keyed on object id and surface, normalised vector blobs, object-type scope predicate, exact scan returning Exhaustive, restart safety), the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), composition mode switch with `collection_name` as the backend-neutral namespace key, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Measure and document corpus-size guidance from an in-phase benchmark. + Implement the embedded adapter per the phase document (schema keyed on object id and surface, normalised vector blobs, the query normalised once before scoring with zero-norm defined, object-type scope predicate, exact dot-product scan returning Exhaustive, restart safety), the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), composition mode switch with `collection_name` as the backend-neutral namespace key, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Measure and document corpus-size guidance from an in-phase benchmark. - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets on the shared fixtures; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service). - Restart test passes; repeated runs are byte-identical. @@ -159,7 +159,7 @@ - docs/** - depends_on: [Task_2] - description: | - Replace the direct vector-service search in the vector-only baseline with the retrieval trace (overfetch-and-slice per kind; item text from the evaluation repository's own ingest records); mirror the completeness telemetry field; add the typed backend identity to result rows through the repository's clean-schema procedure; make the cleanup guard backend-neutral; drop the payload constants, the second vector client's search path, and the second embeddings client's divergent dimension handling. Perform the A/B run with a row-level diff of item identities and ranks against the pre-switch baseline before deleting the old path. + Replace the direct vector-service search in the vector-only baseline with the retrieval trace (one traced retrieval per measured kind with a singleton object-type scope and that kind's budget as the limit, never a sliced mixed-kind top-K; item text from the evaluation repository's own ingest records); mirror the completeness telemetry field; add the typed backend identity to result rows through the repository's clean-schema procedure; make the cleanup guard backend-neutral; drop the payload constants, the second vector client's search path, and the second embeddings client's divergent dimension handling. Perform the A/B run with a row-level diff of item identities and ranks against the pre-switch baseline before deleting the old path. - acceptance: - Zero-hit census for vector-service search calls and payload constants in the evaluation adapter. - A/B evidence recorded; vector-only rows carry the completeness verdict. @@ -270,4 +270,4 @@ Append-only editing rule (applies to both logs below): when appending an entry, ## Notes - Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the exact-scan corpus-size guidance must be measured, not assumed. -- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` returns an empty exhaustive result; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior. +- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` returns an empty exhaustive result; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality (query normalised once, records normalised at write) is asserted, not assumed. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md index 6e751825..e2c1ce91 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md @@ -46,7 +46,8 @@ Configuration follows the one-key-per-backend pattern the graph and statistics s The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files). Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and schema version so a reopened file is validated against the configured embedding model with the same compatibility error the service adapter raises for a mismatched collection. -Search is a scan of the scoped rows scored by dot product in a fixed order, canonicalised by the shared constructor the port requires, and truncated to the requested limit; the result always reports exhaustive completeness (ADR-I-0024). +Search normalises the query vector once before scanning (a zero-norm query scores every row zero and is reported, not rejected), scores the scoped rows by dot product in a fixed order so the score equals the cosine the service adapter reports, canonicalises through the shared constructor the port requires, and truncates to the requested limit; the result always reports exhaustive completeness (ADR-I-0024). +Score parity across adapters is asserted by a parity fixture whose query and record vectors are deliberately non-unit. ## Implementation Impact diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index 24308528..4c8fc362 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -37,7 +37,7 @@ The question is what capability the library must expose so the baseline stops re ## Decision The library exposes no raw candidate-search surface and no facade change. -The evaluation repository's vector-only baseline issues an ordinary `retrieve` with tracing enabled, the object types it measures, and a generous candidate limit, then slices the trace's vector candidates per object kind to its per-section budgets and reads the completeness verdict from telemetry. +The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope and that kind's section budget as the candidate limit, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). The evaluation repository's vector-service client shrinks to collection lifecycle operations (existence and deletion), which the embedded mode replaces with file operations through the durable-store path list the adapter already maintains. @@ -46,7 +46,7 @@ Cross-repository obligations recorded here: - The evaluation telemetry record mirrors the completeness field. - Result rows carry a typed vector-backend identity (service or embedded) so cross-mode comparisons are attributable. - The namespace cleanup guard is backend-neutral: it protects an embedded store file by the same prefix rule that protects a service collection. -- The vector-only surface-policy validator keeps its object-type and budget rules and derives the overfetch from the per-section budgets. +- The vector-only surface-policy validator keeps its object-type and budget rules; each measured kind becomes one singleton-scoped traced retrieval whose limit is that kind's section budget. - The baseline is re-verified by an A/B run against the direct-search implementation before that implementation is deleted. ## Character Memory Relevance @@ -81,13 +81,13 @@ Option 4 leaves two implementations of one capability and breaks with the first - Positive: one retrieval entry point; both adapters covered; the baseline reports the completeness of the top-K it measured. - Negative / tradeoffs: the baseline pays for graph expansion it discards, an evaluation-run cost accepted in exchange for not inventing a second retrieval path. -- Negative / tradeoffs: per-kind quotas are satisfied by overfetch-and-slice; if a quota cannot be filled within the service adapter's fetch bound, the completeness verdict says so and the run records it. +- Negative / tradeoffs: the baseline issues one traced retrieval per measured kind instead of one search, so its cost scales with the number of kinds; if a kind's budget cannot be closed within the service adapter's fetch bound, that retrieval's completeness verdict says so and the run records it. ## Decision Boundary Invariant: no public raw candidate-search surface; evaluation baselines consume the retrieval trace and telemetry; store schemas are adapter-private; candidate content is hydrated by object id from the consumer's own records or graph authority. -Not covered: the overfetch multiplier the baseline uses, the shape of the evaluation repository's ingest record store, and any future diagnostic surface the observability phase designs on product demand. +Not covered: any headroom the baseline adds to a kind's limit, the shape of the evaluation repository's ingest record store, and any future diagnostic surface the observability phase designs on product demand. ## Validation @@ -98,7 +98,7 @@ Not covered: the overfetch multiplier the baseline uses, the shape of the evalua ## Revisit When - A product use case demands candidate-level recall — design a diagnostic surface in the retrieval-observability phase and supersede this record's prohibition for that surface only. -- Per-kind quotas cannot be satisfied by overfetch within the service adapter's fetch bound on a real dataset — reopen whether the trace needs per-type limits. +- One retrieval per kind proves too costly on a real dataset, or a kind's budget cannot be closed within the service adapter's fetch bound — reopen whether the port needs per-type limits in a single query. ## Consultation impact diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index d57539a5..f4b527cf 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -45,7 +45,7 @@ Two re-entry paths are named so the scope-only query and five-field record are r A SQLite-backed exact cosine scan, using the `rusqlite` dependency the statistics store already carries, with the same single-process, mutex-guarded connection model. Schema: one table keyed by object id and surface with a column per contract field and the embedding as a fixed-width little-endian floating-point blob normalised at write; an index on object type for the scope predicate; a metadata table recording vector size, distance, and schema version. -Search: select the scoped rows, score by dot product in a fixed order, canonicalise through the shared constructor, truncate to the limit, and report exhaustive completeness with the scanned count. +Search: normalise the query once (a zero-norm query scores every row zero and is reported, not rejected), select the scoped rows, score by dot product in a fixed order so the score equals the service adapter's cosine, canonicalise through the shared constructor, truncate to the limit, and report exhaustive completeness with the scanned count; a parity fixture with non-unit query and record vectors pins score equality across adapters. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing file validates the recorded vector size and distance against the configured embedding model and raises the same collection-compatibility error the service adapter raises for a mismatched collection. Determinism: same inputs, same scores, same total sort; equal-score cohorts are ordered by the shared comparator, so the embedded adapter satisfies deterministic admission by construction and never needs an overfetch loop. @@ -136,7 +136,7 @@ No public facade change; no retrieval behaviour change in service mode beyond th The companion evaluation repository is a development aid, not core library functionality; these obligations land in the same wave as the library change. -- Trace-sourced baseline: the vector-only baseline issues an ordinary traced `retrieve` with the measured object types and a generous candidate limit, slices the trace's vector candidates per object kind to its per-section budgets, and reads the completeness verdict from telemetry; the direct vector-service search, its payload field constants, and its hit mapping are deleted after an A/B run proves identical item identities and ranks. +- Trace-sourced baseline: the vector-only baseline issues one traced `retrieve` per measured object kind, each with a singleton object-type scope and that kind's section budget as the limit, and reads each retrieval's completeness verdict from telemetry (a single mixed-kind top-K would let a global cutoff exclude an underrepresented kind without any open verdict); the direct vector-service search, its payload field constants, and its hit mapping are deleted after an A/B run proves identical item identities and ranks. - Item text: sourced from the evaluation repository's own ingest records keyed by external identity, never from a store payload. - Telemetry mirror: the evaluation telemetry record gains the completeness field. - Typed backend identity: result rows carry which vector backend (service or embedded) produced them, so cross-mode comparisons are attributable. @@ -171,7 +171,7 @@ Each item was parked on this phase by the structured-verdict phase; each row sta Evidence: zero-hit census for the filter type and for empty-or-null match conditions in the service adapter; the prohibition and re-entry paths are recorded in ADR-I-0024. 5. Evaluation baseline capability. Parked claim: the baseline re-implements a hidden raw-vector capability against the payload schema. - Re-verified: trace-derived candidates with overfetch-and-slice can reproduce the direct-search baseline; the completeness verdict reports whether each question's top-K was determinate; the evaluation adapter can hold item text from ingest. + Re-verified: one singleton-scoped traced retrieval per measured kind reproduces the direct per-kind search exactly, which a sliced mixed-kind top-K would not; each retrieval's completeness verdict reports whether that kind's top-K was determinate; the evaluation adapter can hold item text from ingest. Evidence: the A/B run with row-level diff of item identities and ranks; after the switch, zero-hit census for vector-service search calls and payload constants in the evaluation adapter. ## Decisions (the draft's open questions, resolved 2026-09-02) From 55ade1425af42e93023f8efc2b646bf86462ebf3 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 17:55:14 +0900 Subject: [PATCH 06/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=203?= =?UTF-8?q?:=20not-requested=20verdict=20for=20a=20zero=20limit;=20zero-no?= =?UTF-8?q?rm=20records=20rejected=20at=20indexing;=20schema=20version=20c?= =?UTF-8?q?hecked=20at=20reopen;=20per-kind=20baseline=20limit=20multiplie?= =?UTF-8?q?d=20by=20surfaces=20per=20object=20with=20object-level=20dedup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 ++-- ...-0023-embedded-sqlite-exact-scan-vector-candidate-store.md | 2 +- ...ecall-reports-completeness-and-takes-a-scope-only-query.md | 4 +++- ...DR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md | 4 +++- .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- 5 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index d3fe3a5a..8117dcfb 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -159,7 +159,7 @@ - docs/** - depends_on: [Task_2] - description: | - Replace the direct vector-service search in the vector-only baseline with the retrieval trace (one traced retrieval per measured kind with a singleton object-type scope and that kind's budget as the limit, never a sliced mixed-kind top-K; item text from the evaluation repository's own ingest records); mirror the completeness telemetry field; add the typed backend identity to result rows through the repository's clean-schema procedure; make the cleanup guard backend-neutral; drop the payload constants, the second vector client's search path, and the second embeddings client's divergent dimension handling. Perform the A/B run with a row-level diff of item identities and ranks against the pre-switch baseline before deleting the old path. + Replace the direct vector-service search in the vector-only baseline with the retrieval trace (one traced retrieval per measured kind with a singleton object-type scope and a limit of that kind's budget times the maximum surfaces per object, deduplicated by object keeping the best surface and truncated to the budget, never a sliced mixed-kind top-K; item text from the evaluation repository's own ingest records); mirror the completeness telemetry field; add the typed backend identity to result rows through the repository's clean-schema procedure; make the cleanup guard backend-neutral; drop the payload constants, the second vector client's search path, and the second embeddings client's divergent dimension handling. Perform the A/B run with a row-level diff of item identities and ranks against the pre-switch baseline before deleting the old path. - acceptance: - Zero-hit census for vector-service search calls and payload constants in the evaluation adapter. - A/B evidence recorded; vector-only rows carry the completeness verdict. @@ -270,4 +270,4 @@ Append-only editing rule (applies to both logs below): when appending an entry, ## Notes - Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the exact-scan corpus-size guidance must be measured, not assumed. -- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` returns an empty exhaustive result; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality (query normalised once, records normalised at write) is asserted, not assumed. +- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` issues no search and reports the not-requested verdict in both adapters; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality (query normalised once, records normalised at write) is asserted, not assumed. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md index e2c1ce91..97488b16 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md @@ -45,7 +45,7 @@ The embedded store is single-process, matching the embedded graph store's expect Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files). -Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and schema version so a reopened file is validated against the configured embedding model with the same compatibility error the service adapter raises for a mismatched collection. +Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and record schema version so a reopened file is validated against the configured embedding model and the supported schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires; a restart test covers both. Search normalises the query vector once before scanning (a zero-norm query scores every row zero and is reported, not rejected), scores the scoped rows by dot product in a fixed order so the score equals the cosine the service adapter reports, canonicalises through the shared constructor the port requires, and truncates to the requested limit; the result always reports exhaustive completeness (ADR-I-0024). Score parity across adapters is asserted by a parity fixture whose query and record vectors are deliberately non-unit. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index e6352b67..d1b909a7 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -42,18 +42,20 @@ An embedded adapter (ADR-I-0023) makes the gap visible: an exact scan is exhaust ```rust pub enum VectorRecallCompleteness { + NotRequested, // the limit was zero; no search was issued Exhaustive { scanned: usize }, // every stored record in scope was scored BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort was closed BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort open } ``` -Adapters own canonicalisation and must state the verdict truthfully: exhaustive only when the whole scoped population was scored, closed only when the cutoff cohort was verified closed or the index returned fewer rows than asked, open at the bound. +Adapters own canonicalisation and must state the verdict truthfully: not requested only when the limit is zero and no search was issued (an admitted configuration, so the verdict must be total over it), exhaustive only when the whole scoped population was scored, closed only when the cutoff cohort was verified closed or the index returned fewer rows than asked, open at the bound. The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. The query is the embedding, the limit, and an object-type scope, and nothing else. An empty scope selects zero candidates; wildcard-on-empty is prohibited, matching the graph query rule, and the retrieval context rejects an empty configured object-type set at the boundary. +Zero-norm vectors are defined on both sides of the port: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure before any adapter sees it (so adapters may normalise at write without a division-by-zero path), and a zero-norm query scores every candidate zero and returns a truthful verdict; the parity suite carries both cases. Three-valued hint predicates are prohibited. Any future vector-layer predicate arrives as an explicit enum whose unknown arm is spelled out, an unknown or missing stored value never satisfies a positive predicate, and the predicate lands with its mapping in both adapters and a parity fixture in the same change. diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index 4c8fc362..0fb9d4e3 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -37,7 +37,9 @@ The question is what capability the library must expose so the baseline stops re ## Decision The library exposes no raw candidate-search surface and no facade change. -The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope and that kind's section budget as the candidate limit, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. +The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. +The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant declared by the embedding-surface policy; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. +That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore a closed or exhaustive surface-level verdict at that limit makes the object-level top-budget determinate, and the contract is pinned by a fixture whose objects carry every surface. Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). The evaluation repository's vector-service client shrinks to collection lifecycle operations (existence and deletion), which the embedded mode replaces with file operations through the durable-store path list the adapter already maintains. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index f4b527cf..516d2e8e 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -26,7 +26,7 @@ This phase fixes the port contract deliberately, because two adapters cannot be Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. -Result: a completeness envelope, the canonical candidates plus a typed verdict — exhaustive (every scoped record was scored), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). +Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero and no search was issued), exhaustive (every scoped record was scored), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). The service adapter's existing tie-cohort loop maps onto the last two verdicts; before this phase it returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. @@ -136,7 +136,7 @@ No public facade change; no retrieval behaviour change in service mode beyond th The companion evaluation repository is a development aid, not core library functionality; these obligations land in the same wave as the library change. -- Trace-sourced baseline: the vector-only baseline issues one traced `retrieve` per measured object kind, each with a singleton object-type scope and that kind's section budget as the limit, and reads each retrieval's completeness verdict from telemetry (a single mixed-kind top-K would let a global cutoff exclude an underrepresented kind without any open verdict); the direct vector-service search, its payload field constants, and its hit mapping are deleted after an A/B run proves identical item identities and ranks. +- Trace-sourced baseline: the vector-only baseline issues one traced `retrieve` per measured object kind, each with a singleton object-type scope and a limit of that kind's section budget multiplied by the maximum surfaces per object of that kind (the trace lists object-and-surface pairs before object-level deduplication), deduplicates by object keeping the best surface, truncates to the budget, and reads each retrieval's completeness verdict from telemetry (a single mixed-kind top-K would let a global cutoff exclude an underrepresented kind without any open verdict; the multiplied limit makes the object top-budget determinate whenever the surface verdict is closed or exhaustive); the direct vector-service search, its payload field constants, and its hit mapping are deleted after an A/B run proves identical item identities and ranks. - Item text: sourced from the evaluation repository's own ingest records keyed by external identity, never from a store payload. - Telemetry mirror: the evaluation telemetry record gains the completeness field. - Typed backend identity: result rows carry which vector backend (service or embedded) produced them, so cross-mode comparisons are attributable. From 4557d9f1c9a4d1dfd8c5d37b3e2f21e024835d27 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 18:02:33 +0900 Subject: [PATCH 07/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=204?= =?UTF-8?q?:=20zero-norm=20rejection=20owned=20by=20the=20read-contract=20?= =?UTF-8?q?task;=20exhaustive=20verdict=20only=20for=20issued=20searches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 +++- ...-0023-embedded-sqlite-exact-scan-vector-candidate-store.md | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 8117dcfb..00fa8443 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -100,14 +100,16 @@ - src/domain.rs - src/usecases/retrieve.rs - src/usecases/vector_indexing.rs + - src/errors.rs - docs/design/database/vector_payload_design.md - depends_on: [Task_2] - description: | - Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. - acceptance: - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository's reader is removed by Task_5). - One token mapping per enum; census shows no copy in adapters or use cases. + - A zero-norm record embedding yields a typed per-record indexing failure and never reaches an adapter; unit test present. - validation: - kind: command required: true diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md index 97488b16..dbfa2f89 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md @@ -46,7 +46,7 @@ Configuration follows the one-key-per-backend pattern the graph and statistics s The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files). Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and record schema version so a reopened file is validated against the configured embedding model and the supported schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires; a restart test covers both. -Search normalises the query vector once before scanning (a zero-norm query scores every row zero and is reported, not rejected), scores the scoped rows by dot product in a fixed order so the score equals the cosine the service adapter reports, canonicalises through the shared constructor the port requires, and truncates to the requested limit; the result always reports exhaustive completeness (ADR-I-0024). +Search normalises the query vector once before scanning (a zero-norm query scores every row zero and is reported, not rejected), scores the scoped rows by dot product in a fixed order so the score equals the cosine the service adapter reports, canonicalises through the shared constructor the port requires, and truncates to the requested limit; every issued search reports exhaustive completeness, and a zero limit issues no search and reports not requested (ADR-I-0024). Score parity across adapters is asserted by a parity fixture whose query and record vectors are deliberately non-unit. ## Implementation Impact From e7639ef757c19ff2a4e93ca6093c9a018adb8b0d Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 18:45:05 +0900 Subject: [PATCH 08/44] =?UTF-8?q?=F0=9F=93=9D=20Evaluation-harness=20work?= =?UTF-8?q?=20leaves=20the=20library=20records:=20plan=20drops=20the=20two?= =?UTF-8?q?=20eval-side=20tasks;=20phase=20doc=20and=20ADR-I-0026=20state?= =?UTF-8?q?=20only=20the=20library-facing=20contract;=20tracking-split=20r?= =?UTF-8?q?ule=20added?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../v0-1-6-embedded-vector-recall-plan.md | 63 +++---------------- docs/coding-agent/rules/common.md | 1 + ...ctor-baselines-read-the-retrieval-trace.md | 13 ++-- ...v0_1_6_embedded_vector_candidate_recall.md | 18 +++--- 4 files changed, 22 insertions(+), 73 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 00fa8443..13074f6e 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -22,7 +22,7 @@ ## Context (workspace) - Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. - As-built port: `src/ports/vector_candidate.rs`, `src/models/vector/candidate_record.rs`, `src/models/vector/record.rs`, `src/adapters/qdrant/{store,payload}.rs`, `src/policy/embedding_surface.rs`, `src/usecases/retrieve.rs`, `src/api/types/retrieval.rs`, `src/composition.rs`, `src/config/app_settings.rs`, `src/test_support.rs`. -- Prerequisite landed separately: the evaluation repository's evidence-integrity light-delta (batch outcome duplication, harness-invented rank, unhonored manifest/hash knobs, shared graph-path fallback, live-skip panic switch), branch `chore/evidence-integrity-pre-v016`. +- Prerequisite tracked in the evaluation repository: its evidence-integrity fixes must be merged before this phase cites any harness measurement. - Repo reference docs consulted: the four ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. ## Open Questions (max 3) @@ -30,7 +30,7 @@ ## Assumptions - A1: `rusqlite` (already a dependency for the statistics store) is sufficient for the embedded adapter; no vector extension is added. -- A2: The evaluation repository's row/summary schema move for the typed backend identity follows that repository's normal clean-schema procedure and is owned by its wave task. +- A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison that gates the later default-flip decision. ## Tasks @@ -150,55 +150,6 @@ owner: reviewer detail: "Diff review vs ADR-I-0023; independent service-free and service-up runs" -### Task_5: Evaluation repository: trace-sourced vector-only baseline and telemetry mirror (ADR-I-0026) -- type: impl -- owns: - - crates/cmem-eval-adapter-cmem/src/lib.rs - - crates/cmem-eval-adapter-cmem/Cargo.toml - - crates/cmem-eval-core/src/{config,runtime,results,verdict,metrics}.rs - - crates/cmem-eval-runner/src/pipeline.rs - - configs/** - - docs/** -- depends_on: [Task_2] -- description: | - Replace the direct vector-service search in the vector-only baseline with the retrieval trace (one traced retrieval per measured kind with a singleton object-type scope and a limit of that kind's budget times the maximum surfaces per object, deduplicated by object keeping the best surface and truncated to the budget, never a sliced mixed-kind top-K; item text from the evaluation repository's own ingest records); mirror the completeness telemetry field; add the typed backend identity to result rows through the repository's clean-schema procedure; make the cleanup guard backend-neutral; drop the payload constants, the second vector client's search path, and the second embeddings client's divergent dimension handling. Perform the A/B run with a row-level diff of item identities and ranks against the pre-switch baseline before deleting the old path. -- acceptance: - - Zero-hit census for vector-service search calls and payload constants in the evaluation adapter. - - A/B evidence recorded; vector-only rows carry the completeness verdict. - - Cleanup and namespace guards work for both vector modes. -- validation: - - kind: command - required: true - owner: evals-worker - detail: "fmt; workspace clippy -D warnings; service-up cargo test --workspace with the live switch set; A/B run artifacts under .agent-work with the diff" - - kind: review - required: true - owner: evals-reviewer - detail: "Diff review vs ADR-I-0026; verify the A/B diff and that no sealed evidence changed" - -### Task_6: Evaluation repository: embedded-mode configuration and cross-mode baselines -- type: impl -- owns: - - crates/cmem-eval-core/src/config.rs - - crates/cmem-eval-adapter-cmem/src/lib.rs - - configs/** - - docs/** -- depends_on: [Task_4, Task_5] -- description: | - Add the embedded vector mode to the evaluation backend configuration, run the continuity suite in embedded mode, and record that scenario baselines are identical to service mode under the parity contract (any divergence is a finding). -- acceptance: - - An embedded-mode configuration exists and runs without a vector service. - - Cross-mode baseline comparison recorded with zero unexplained divergence. -- validation: - - kind: command - required: true - owner: evals-worker - detail: "continuity suite in both modes; comparison artifact recorded" - - kind: review - required: true - owner: evals-reviewer - detail: "Verify the comparison and the register entries" - ### Task_7: Fake retirement, closeout docs, and reconfirmation evidence - type: chore - owns: @@ -207,7 +158,7 @@ - docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md - docs/coding-agent/lessons.md - docs/roadmap/development_roadmap.md -- depends_on: [Task_4, Task_6] +- depends_on: [Task_4] - description: | Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened in memory (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; mark the roadmap row finished; move the plan to completed. - acceptance: @@ -226,7 +177,7 @@ ### Task_8: Design-value audit at the pre-merge milestone gate - type: review - owns: [] -- depends_on: [Task_4, Task_5] +- depends_on: [Task_4] - description: | Altitude review (Claude) against philosophy and roadmap: nothing designed twice across the two adapters, no hint field re-entered without its predicate, the evaluation repository holds no store-private knowledge, the ADR boundaries respected. - acceptance: @@ -240,11 +191,13 @@ ## Task Waves (explicit parallel dispatch sets) - Wave 1 (parallel): [Task_1, Task_2] -- Wave 2 (parallel): [Task_3, Task_5] +- Wave 2 (parallel): [Task_3] - Wave 3 (parallel): [Task_4] -- Wave 4 (parallel): [Task_6, Task_8] +- Wave 4 (parallel): [Task_8] - Wave 5 (parallel): [Task_7] +Task identifiers 5 and 6 were evaluation-repository work and moved to that repository's own plan; identifiers are not reused. + Each wave ends with reviewer approval and a PR per touched repository, merged by the decider before the next wave starts; the evaluation repository's sibling checkout is re-pinned to the merged library commit at every wave boundary. ## Rollback / Safety diff --git a/docs/coding-agent/rules/common.md b/docs/coding-agent/rules/common.md index 62aedd08..4af2f104 100644 --- a/docs/coding-agent/rules/common.md +++ b/docs/coding-agent/rules/common.md @@ -15,6 +15,7 @@ last_updated: "2026-07-24" - Committed artifacts in this repository must not contain machine-local absolute paths (for example user-profile paths); refer to sibling repositories by name and relative relationship instead. - When mentioning the `CharacterMemoryEvals` repository in committed docs, describe it as the public companion evaluation repository and state that evaluation tooling is a development aid, not core library functionality. Do not describe it as private or inaccessible (it was made public 2026-07-19); historical records (completed plans, dated ADR bodies) that reflect the earlier private status stay unchanged. +- Records in this repository (plans, decision records, phase documents) state what the evaluation harness's measurements allow this library to decide and when those measurements are used; the harness's own work is planned and tracked in the evaluation repository and is never mixed into library plans (ruled 2026-09-02). - Do not hard-wrap prose in committed documents: never insert line breaks mid-sentence to fit a column width. Write each sentence/paragraph/list item as one line and let editors soft-wrap. Structural line breaks (list items, headings, YAML keys, code) are fine. - ADR frontmatter `consulted` entries record model names only (for example "Claude Fable 5", "GPT-5.5 Pro") — no role, platform, or product designations such as "(orchestrator)" or "Codex" (user-directed 2026-07-18). diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index 0fb9d4e3..8203ef89 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -43,13 +43,12 @@ That limit is sufficient by construction: an object ranked within the budget by Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). The evaluation repository's vector-service client shrinks to collection lifecycle operations (existence and deletion), which the embedded mode replaces with file operations through the durable-store path list the adapter already maintains. -Cross-repository obligations recorded here: +What this record asks of the evaluation repository, recorded as the library-facing contract and nothing more: + +- The evaluation repository consumes the retrieval trace and the completeness telemetry as an ordinary caller; the library adds no surface for it. +- A raw-vector baseline that wants per-kind top-K uses one singleton-scoped traced retrieval per kind with the multiplied limit and object-level deduplication described above; any other reading of the trace is not covered by the parity claim. +- How the evaluation repository migrates its baseline, mirrors telemetry, labels its rows, or guards its cleanup is planned and tracked in that repository. -- The evaluation telemetry record mirrors the completeness field. -- Result rows carry a typed vector-backend identity (service or embedded) so cross-mode comparisons are attributable. -- The namespace cleanup guard is backend-neutral: it protects an embedded store file by the same prefix rule that protects a service collection. -- The vector-only surface-policy validator keeps its object-type and budget rules; each measured kind becomes one singleton-scoped traced retrieval whose limit is that kind's section budget. -- The baseline is re-verified by an A/B run against the direct-search implementation before that implementation is deleted. ## Character Memory Relevance @@ -59,7 +58,7 @@ Keeping the baseline inside the traced retrieval path means the measurement of " ## Implementation Impact - Library: none beyond ADR-I-0024's telemetry field; the acceptance criterion "no public facade change" holds. -- Evaluation repository: delete the direct search path, payload field constants, and hit mapping; add trace-derived candidate slicing and ingest-record text lookup; add the backend identity to result rows; generalise cleanup to the embedded store file. +- Evaluation repository: its baseline moves onto the trace under its own plan; nothing in this repository depends on how. ## Considered Options diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 516d2e8e..d082a0f1 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -132,22 +132,18 @@ Documentation states the single-process expectation, the corpus-size guidance, a No public facade change; no retrieval behaviour change in service mode beyond the added telemetry field. ``` -## Cross-repository obligations (ADR-I-0026) +## What the evaluation repository provides and when it is used (ADR-I-0026) -The companion evaluation repository is a development aid, not core library functionality; these obligations land in the same wave as the library change. +The companion evaluation repository is a development aid; its own work is planned and tracked there, and this document records only what its measurements let this phase decide. -- Trace-sourced baseline: the vector-only baseline issues one traced `retrieve` per measured object kind, each with a singleton object-type scope and a limit of that kind's section budget multiplied by the maximum surfaces per object of that kind (the trace lists object-and-surface pairs before object-level deduplication), deduplicates by object keeping the best surface, truncates to the budget, and reads each retrieval's completeness verdict from telemetry (a single mixed-kind top-K would let a global cutoff exclude an underrepresented kind without any open verdict; the multiplied limit makes the object top-budget determinate whenever the surface verdict is closed or exhaustive); the direct vector-service search, its payload field constants, and its hit mapping are deleted after an A/B run proves identical item identities and ranks. -- Item text: sourced from the evaluation repository's own ingest records keyed by external identity, never from a store payload. -- Telemetry mirror: the evaluation telemetry record gains the completeness field. -- Typed backend identity: result rows carry which vector backend (service or embedded) produced them, so cross-mode comparisons are attributable. -- Backend-neutral cleanup guard: the namespace cleanup guard protects an embedded store file by the same prefix rule that protects a service collection, and cleanup removes the SQLite file with its write-ahead-log sidecars alongside the statistics store. -- Configuration: a `vector_store_path` backend setting beside the graph and statistics paths; when set, the adapter selects embedded mode and derives a per-namespace file the way it derives the statistics path. -- Error vocabulary: the exhaustive conversion of the vector database error kinds gains the embedded engine kind. +- The library exposes, through an ordinary traced retrieval, everything a raw-vector baseline needs: the vector candidates with scores and the completeness verdict in telemetry; the honest way to use them is one singleton-scoped traced retrieval per measured object kind with a limit of the section budget multiplied by the maximum surfaces per object, deduplicated by object. +- The cross-mode comparison (service mode against embedded mode on the continuity suite, identical baselines expected under the parity contract) is the evidence that gates the default flip recorded in ADR-I-0023; it is consumed at the closeout task and by that later decision, not produced by this plan. +- No public facade or configuration surface is added for the evaluation repository; if its measurements ever require one, that is a library decision taken on its own record. ## Evaluation tie-in -The continuity evaluation suite gains an embedded-mode configuration so the confirmation scenarios, including restart, run against the embedded vector store; the frozen-embedding infrastructure applies unchanged. -Scenario results are expected to be identical between modes under the parity contract; any divergence is a finding, which makes the evaluation suite the cross-adapter regression instrument. +The evaluation repository is expected to run its continuity suite in both vector modes; identical scenario results are what the parity contract predicts, and the comparison is the evidence that gates the later default-flip decision recorded in ADR-I-0023. +How that configuration is built and run is planned in the evaluation repository; this phase consumes the comparison at closeout and cites nothing else from it. ## Deferral-reconfirmation checklist From f72fab45c7b7d33b98931e971d1f814d668c1cae Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 18:53:40 +0900 Subject: [PATCH 09/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=206?= =?UTF-8?q?:=20library-only=20scope=20and=20completion=20clauses;=20databa?= =?UTF-8?q?se=20cheat=20sheet=20and=20README=20owned=20by=20the=20read-con?= =?UTF-8?q?tract=20task;=20rules=20freshness=20date?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../active/v0-1-6-embedded-vector-recall-plan.md | 12 +++++++----- docs/coding-agent/rules/common.md | 2 +- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 13074f6e..6b1c412d 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -13,10 +13,10 @@ - Every row of the phase document's deferral-reconfirmation checklist has its evidence produced and cited in the Progress Log. - Every deletion listed under "Deletions that are deliverables" is gone, with a zero-hit census. - Both repositories' service-gated suites execute (not skip) under the service-backed CI job. -- One PR per repository per wave, merged by the decider; the evaluation repository's obligations in ADR-I-0026 are all landed. +- One PR per wave in this repository, merged by the decider; the evaluation repository's cross-mode comparison is available as consumed evidence at closeout. ## Scope / Non-goals -- Scope: the phase document's deliverables and deletions; the evaluation repository obligations in ADR-I-0026. +- Scope: the phase document's deliverables and deletions, all in this repository. - Non-goals: the phase document's non-goals (no default flip, no approximate index, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode). ## Context (workspace) @@ -102,12 +102,14 @@ - src/usecases/vector_indexing.rs - src/errors.rs - docs/design/database/vector_payload_design.md + - docs/design/database/schema_cheat_sheet.md + - docs/design/database/README.md - depends_on: [Task_2] - description: | - Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Update the database documentation that advertises the old record: the schema cheat sheet and the database README lose the hint fields, the graph URI, and the readable text column, and point at the five-field contract. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. - acceptance: - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. - - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository's reader is removed by Task_5). + - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository removes its reader under its own plan; its zero-hit census is consumed as closeout evidence, not ordered here). - One token mapping per enum; census shows no copy in adapters or use cases. - A zero-norm record embedding yields a typed per-record indexing failure and never reaches an adapter; unit test present. - validation: @@ -203,7 +205,7 @@ Each wave ends with reviewer approval and a PR per touched repository, merged by ## Rollback / Safety - Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict and the shrunken record, both covered by the parity suite. - Stored service-mode payloads with dropped fields remain readable (extra fields tolerated unread); rebuild from graph authority is the recovery path. -- Each wave is a separately revertible PR pair. +- Each wave is a separately revertible PR. ## Progress Log (append-only) diff --git a/docs/coding-agent/rules/common.md b/docs/coding-agent/rules/common.md index 4af2f104..04e9c337 100644 --- a/docs/coding-agent/rules/common.md +++ b/docs/coding-agent/rules/common.md @@ -2,7 +2,7 @@ rule_schema_version: 2 suite_id: "rules-cm-20260719" rule_file: "common" -last_updated: "2026-07-24" +last_updated: "2026-09-02" --- # Common Repository Rules From 9a29b78ab8b9dd520258fceefa5c7b872a76d288 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 19:02:44 +0900 Subject: [PATCH 10/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=207?= =?UTF-8?q?:=20collection-name=20path-confinement=20contract;=20audit=20ta?= =?UTF-8?q?sk=20owns=20the=20plan=20file;=20ADR-I-0001's=20graph=20URI=20c?= =?UTF-8?q?lause=20partially=20superseded=20with=20reciprocal=20frontmatte?= =?UTF-8?q?r?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 3 ++- .../implementation/ADR-I-0001-stable-cross-store-ids.md | 4 ++-- .../ADR-I-0025-vector-record-is-a-read-contract.md | 3 ++- .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 2 +- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 6b1c412d..82eddb26 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -178,7 +178,8 @@ ### Task_8: Design-value audit at the pre-merge milestone gate - type: review -- owns: [] +- owns: + - docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md - depends_on: [Task_4] - description: | Altitude review (Claude) against philosophy and roadmap: nothing designed twice across the two adapters, no hint field re-entered without its predicate, the evaluation repository holds no store-private knowledge, the ADR boundaries respected. diff --git a/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md b/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md index b0fc26d0..d305620a 100644 --- a/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md +++ b/docs/decisions/implementation/ADR-I-0001-stable-cross-store-ids.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0025-vector-record-is-a-read-contract.md +supersession_scope: partial --- # ADR-I-0001: Use stable cross-store IDs and deterministic graph IRIs diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index e724eb16..14600a28 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -13,7 +13,7 @@ warrant: cost_of_over_extension: "extending the rule to the graph store would strip graph authority of denormalised fields it legitimately owns" depends_on: [implementation/ADR-I-0007-schema-versioning.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md] implements: [] -supersedes: [implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md, implementation/ADR-I-0002-natural-language-embedding-surfaces.md] +supersedes: [implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md, implementation/ADR-I-0002-natural-language-embedding-surfaces.md, implementation/ADR-I-0001-stable-cross-store-ids.md] superseded_by: null supersession_scope: partial --- @@ -47,6 +47,7 @@ Consumers needing candidate content hydrate by object id. `content_text` is dropped. The relationship refs (episode, observation, thread, entity, participant, speaker, supersedes), the lifecycle and currentness flags, the time hints, the ranking and salience hints, the object-specific hints, the graph URI, and the raw source reference leave the vector write path. +Dropping the graph URI partially supersedes ADR-I-0001's clause that every vector payload carries it: the stable object id remains the cross-store identity and the graph URI is derived from it by graph authority, so the pointer was a redundant copy of the id; ADR-I-0001's stable-id decision itself is unchanged. The typed field manifest introduced in the structured-verdict phase remains the single source of both adapters' column sets and shrinks to the five entries. ADR-I-0024 names the two re-entry paths (a synchronised scope predicate; an immutable time-window predicate over `created_at` and `observed_at` backfilled from graph authority) so a returning column arrives with its predicate, its adapter mappings, and a parity fixture. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index d082a0f1..e091f9ba 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -62,7 +62,7 @@ QDRANT_CONNECTION_STRING required only in service mode ``` `VECTOR_STORE_PATH` is a directory; each collection is one SQLite file inside it named by the collection name the public constructor already takes, so `collection_name` is the backend-neutral namespace key in both modes. -Collection names in embedded mode are validated to the same character set the evaluation repository already sanitises to. +Because the collection name becomes a file name under the configured directory, embedded mode validates it at construction with a contract owned here: ASCII letters, digits, underscore, and hyphen only, first character a letter or digit, at most 128 characters, no path separators, dots, or empty name; anything else is rejected with the configuration error before any file is touched, and a path-confinement test proves that separator and parent-directory inputs cannot escape the directory. The composition root gains a vector-store mode switch mirroring the statistics-store switch; the vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds. ### Parity suite placement (ADR-I-0023) From bbde39ed6c29789c0e375ec9745c2191a6d5beda Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 19:09:42 +0900 Subject: [PATCH 11/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=208?= =?UTF-8?q?:=20path-confinement=20and=20blocking-worker=20scan=20in=20the?= =?UTF-8?q?=20adapter=20task;=20ADR-I-0001=20listed=20among=20the=20supers?= =?UTF-8?q?eded=20records?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 2 ++ .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 82eddb26..49401342 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -140,6 +140,8 @@ Implement the embedded adapter per the phase document (schema keyed on object id and surface, normalised vector blobs, the query normalised once before scoring with zero-norm defined, object-type scope predicate, exact dot-product scan returning Exhaustive, restart safety), the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), composition mode switch with `collection_name` as the backend-neutral namespace key, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Measure and document corpus-size guidance from an in-phase benchmark. - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets on the shared fixtures; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service). + - The collection name is validated to the phase document's allowlist before any file is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. + - The synchronous scan runs on a blocking worker with serialized connection access, never on an async executor thread; the benchmark records executor responsiveness under a concurrent scan. - Restart test passes; repeated runs are byte-identical. - Settings docs, single-process expectation, corpus-size guidance, and rebuild-from-graph-authority path are documented. - validation: diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index e091f9ba..7c2c5926 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -42,7 +42,7 @@ Two re-entry paths are named so the scope-only query and five-field record are r ### The embedded adapter (ADR-I-0023) -A SQLite-backed exact cosine scan, using the `rusqlite` dependency the statistics store already carries, with the same single-process, mutex-guarded connection model. +A SQLite-backed exact cosine scan, using the `rusqlite` dependency the statistics store already carries, with the same single-process, mutex-guarded connection model, except that the scan itself is a synchronous, potentially long operation behind an async port: it runs on a blocking worker (the runtime's blocking pool) with connection access still serialized, so a scan never occupies an async executor thread, and the in-phase benchmark records executor responsiveness while a scan is in progress. Schema: one table keyed by object id and surface with a column per contract field and the embedding as a fixed-width little-endian floating-point blob normalised at write; an index on object type for the scope predicate; a metadata table recording vector size, distance, and schema version. Search: normalise the query once (a zero-norm query scores every row zero and is reported, not rejected), select the scoped rows, score by dot product in a fixed order so the score equals the service adapter's cosine, canonicalise through the shared constructor, truncate to the limit, and report exhaustive completeness with the scanned count; a parity fixture with non-unit query and record vectors pins score equality across adapters. @@ -82,7 +82,7 @@ port-conformance parity suite in the library integration tests, run against both restart-safety test for the embedded store; pipeline test over the embedded adapter with a deleted graph object measured corpus-size guidance from an in-phase benchmark documentation: settings, single-process expectation, corpus-size guidance, rebuild-from-graph-authority as the path between modes -four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0005 and ADR-I-0002 +four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0001, ADR-I-0002, and ADR-I-0005 ``` Deletions that are deliverables, not side effects: From cd52fa089d018f57058f94cba6b8271dbd5b2f32 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 19:17:07 +0900 Subject: [PATCH 12/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=209?= =?UTF-8?q?:=20portable=20lowercase=20collection-name=20allowlist=20with?= =?UTF-8?q?=20reserved-name=20rejection;=20path=20setting=20required=20in?= =?UTF-8?q?=20embedded=20mode;=20unsupported-schema=20reopen=20test;=20wor?= =?UTF-8?q?kflow=20glob?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 5 +++-- ...0023-embedded-sqlite-exact-scan-vector-candidate-store.md | 2 +- .../v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 49401342..043e539e 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -42,7 +42,7 @@ - tests/initialization_tests.rs - tests/public_facade_tests.rs - tests/retrieval_guardrails_tests.rs - - .github/workflows/*.yml + - .github/workflows/*.yaml - depends_on: [] - description: | Add one environment switch honored by the shared test support that turns every service-unavailable skip into a panic, set it in the CI job that provisions the vector service, and delete the prose-matched timeout skip (`is_qdrant_timeout_signature`) or replace it with a typed match on the existing transport classification. @@ -142,7 +142,8 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets on the shared fixtures; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service). - The collection name is validated to the phase document's allowlist before any file is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - The synchronous scan runs on a blocking worker with serialized connection access, never on an async executor thread; the benchmark records executor responsiveness under a concurrent scan. - - Restart test passes; repeated runs are byte-identical. + - Restart test passes; repeated runs are byte-identical; reopening a file with a mismatched vector size or distance raises the collection-compatibility error, and reopening one with an unsupported stored schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. + - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - Settings docs, single-process expectation, corpus-size guidance, and rebuild-from-graph-authority path are documented. - validation: - kind: command diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md index dbfa2f89..fe68c124 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md @@ -43,7 +43,7 @@ The default mode stays service until the parity suite and the evaluation suite h The embedded store is single-process, matching the embedded graph store's expectation. Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. -The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files). +The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files); embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name in the file's metadata table, and requires the path setting to be present, with a missing path a configuration error rather than an implicit default. Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and record schema version so a reopened file is validated against the configured embedding model and the supported schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires; a restart test covers both. Search normalises the query vector once before scanning (a zero-norm query scores every row zero and is reported, not rejected), scores the scoped rows by dot product in a fixed order so the score equals the cosine the service adapter reports, canonicalises through the shared constructor the port requires, and truncates to the requested limit; every issued search reports exhaustive completeness, and a zero limit issues no search and reports not requested (ADR-I-0024). diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 7c2c5926..ece1b97d 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -57,12 +57,12 @@ Follow the one-key-per-backend pattern the graph and statistics stores already u ```text VECTOR_STORE_MODE service | embedded (default: service) -VECTOR_STORE_PATH directory, read only in embedded mode +VECTOR_STORE_PATH directory, required in embedded mode (missing is a configuration error, never an implicit default), ignored in service mode QDRANT_CONNECTION_STRING required only in service mode ``` `VECTOR_STORE_PATH` is a directory; each collection is one SQLite file inside it named by the collection name the public constructor already takes, so `collection_name` is the backend-neutral namespace key in both modes. -Because the collection name becomes a file name under the configured directory, embedded mode validates it at construction with a contract owned here: ASCII letters, digits, underscore, and hyphen only, first character a letter or digit, at most 128 characters, no path separators, dots, or empty name; anything else is rejected with the configuration error before any file is touched, and a path-confinement test proves that separator and parent-directory inputs cannot escape the directory. +Because the collection name becomes a file name under the configured directory, embedded mode validates it at construction with a contract owned here: lowercase ASCII letters, digits, underscore, and hyphen only (lowercase so that names stay unique on the case-insensitive filesystems of desktop targets), first character a letter or digit, at most 128 characters, not a reserved device name on Windows (con, prn, aux, nul, com1 to com9, lpt1 to lpt9), no path separators, dots, or empty name; anything else is rejected with the configuration error before any file is touched, the name is recorded in the file's metadata table, and a path-confinement test proves that separator and parent-directory inputs cannot escape the directory. The composition root gains a vector-store mode switch mirroring the statistics-store switch; the vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds. ### Parity suite placement (ADR-I-0023) From fdf375cbdf83c36d66b7853887480de048dd1de3 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 19:44:30 +0900 Subject: [PATCH 13/44] =?UTF-8?q?=F0=9F=93=9D=20Technology=20ruling:=20the?= =?UTF-8?q?=20embedded=20vector=20store=20is=20Qdrant=20Edge;=20ADR-I-0023?= =?UTF-8?q?=20rewritten=20and=20renamed,=20phase=20document,=20plan,=20and?= =?UTF-8?q?=20roadmap=20aligned;=20Decision=20Log=20entry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Chosen by the decider as the more future-proof option for a portable memory expected to track years to decades across changing foundation models; two feasibility spikes (Qdrant Edge 0.8.0, sqlite-vec 0.1.9) are cited in the rejected alternatives; exactness becomes a threshold property, ties close through the shared loop, the type mapping is adapter-specific, and a contract canary pins the engine facts. Co-Authored-By: Claude Fable 5.1 --- .../v0-1-6-embedded-vector-recall-plan.md | 39 +++-- ...dded-qdrant-edge-vector-candidate-store.md | 136 ++++++++++++++++++ ...qlite-exact-scan-vector-candidate-store.md | 116 --------------- ...mpleteness-and-takes-a-scope-only-query.md | 13 +- ...I-0025-vector-record-is-a-read-contract.md | 6 +- ...v0_1_6_embedded_vector_candidate_recall.md | 91 +++++++----- docs/roadmap/development_roadmap.md | 8 +- 7 files changed, 231 insertions(+), 178 deletions(-) create mode 100644 docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md delete mode 100644 docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 043e539e..e3f5649a 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -6,7 +6,7 @@ - work_type: mixed ## Goal -- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0026: a redesigned vector port contract, a five-field vector record, an embedded SQLite exact-scan vector candidate store as the opt-in local mode, a shared contract suite over both adapters, and the evaluation repository's vector-only baseline moved onto the retrieval trace. +- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0026: a redesigned vector port contract, a five-field vector record, an embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) as the opt-in local mode, a shared contract suite over both adapters, and the evaluation repository's vector-only baseline moved onto the retrieval trace. ## Definition of Done - Every acceptance criterion in the phase document's "Acceptance criteria" section holds with recorded evidence. @@ -17,11 +17,12 @@ ## Scope / Non-goals - Scope: the phase document's deliverables and deletions, all in this repository. -- Non-goals: the phase document's non-goals (no default flip, no approximate index, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode). +- Non-goals: the phase document's non-goals (no default flip, no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode). ## Context (workspace) - Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. - As-built port: `src/ports/vector_candidate.rs`, `src/models/vector/candidate_record.rs`, `src/models/vector/record.rs`, `src/adapters/qdrant/{store,payload}.rs`, `src/policy/embedding_surface.rs`, `src/usecases/retrieve.rs`, `src/api/types/retrieval.rs`, `src/composition.rs`, `src/config/app_settings.rs`, `src/test_support.rs`. +- Prerequisite in this repository: the toolchain pin moves to the embedded engine's minimum (Rust 1.97.0 at decision time) in its own change; Task_4 cannot build before it merges. - Prerequisite tracked in the evaluation repository: its evidence-integrity fixes must be merged before this phase cites any harness measurement. - Repo reference docs consulted: the four ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. @@ -29,7 +30,7 @@ - none (the draft's five open questions were ruled by the decider on 2026-09-02 and are recorded in the phase document and the ADRs). ## Assumptions -- A1: `rusqlite` (already a dependency for the statistics store) is sufficient for the embedded adapter; no vector extension is added. +- A1: The embedded engine is `qdrant-edge` pinned exactly at 0.8.0 (beta); its API is guarded by a contract canary, and the pin is bumped only with a re-run of the canary and the parity suite. - A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison that gates the later default-flip decision. ## Tasks @@ -122,14 +123,15 @@ owner: reviewer detail: "Diff review vs ADR-I-0025; verify the payload design note's supersession note matches what landed" -### Task_4: Embedded SQLite vector candidate store, settings, and parity suite (ADR-I-0023) +### Task_4: Embedded Qdrant Edge vector candidate store, settings, and parity suite (ADR-I-0023) - type: impl - owns: - - src/adapters/sqlite_vector/** + - src/adapters/qdrant_edge/** - src/adapters.rs - src/composition.rs - src/config/app_settings.rs - src/errors.rs + - Cargo.toml - tests/vector_port_contract_tests.rs - tests/support/** - .env.example @@ -137,23 +139,25 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter per the phase document (schema keyed on object id and surface, normalised vector blobs, the query normalised once before scoring with zero-norm defined, object-type scope predicate, exact dot-product scan returning Exhaustive, restart safety), the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), composition mode switch with `collection_name` as the backend-neutral namespace key, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Measure and document corpus-size guidance from an in-phase benchmark. + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with a keyword index on object type, the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise; NotRequested at limit zero). The adapter constructs only object payloads and uses only the general shard type; every search and any index build runs on a blocking worker (or the engine's own worker pool where its API is asynchronous), never on an async executor thread. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). - acceptance: - - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets on the shared fixtures; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service). - - The collection name is validated to the phase document's allowlist before any file is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - The synchronous scan runs on a blocking worker with serialized connection access, never on an async executor thread; the benchmark records executor responsiveness under a concurrent scan. - - Restart test passes; repeated runs are byte-identical; reopening a file with a mismatched vector size or distance raises the collection-compatibility error, and reopening one with an unsupported stored schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. + - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. + - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). + - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. + - No scan or index build occupies an async executor thread; the benchmark records executor responsiveness under a concurrent scan. + - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - - Settings docs, single-process expectation, corpus-size guidance, and rebuild-from-graph-authority path are documented. + - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. + - The dependency-weight report records unstripped and stripped release deltas and the result of feature trimming; the latency guidance is measured and documented with the single-process expectation and the rebuild-from-graph-authority path. - validation: - kind: command required: true owner: worker - detail: "cargo test with no service (parity suite + embedded tests execute); service-up cargo test with the live switch set; benchmark numbers recorded in the report" + detail: "cargo test with no service (parity suite + embedded tests + canary execute); service-up cargo test with the live switch set; benchmark and weight numbers recorded in the report" - kind: review required: true owner: reviewer - detail: "Diff review vs ADR-I-0023; independent service-free and service-up runs" + detail: "Diff review vs ADR-I-0023; independent service-free and service-up runs; tie-closure reuse verified by reading, not by test names" ### Task_7: Fake retirement, closeout docs, and reconfirmation evidence - type: chore @@ -228,7 +232,12 @@ Append-only editing rule (applies to both logs below): when appending an entry, - Trigger / new insight: batch ingest produced phantom repair attempts and the evaluated rank was harness-invented; both would corrupt the parity and baseline evidence this phase cites. - Plan delta: none inside this plan; recorded as a prerequisite in Context. - User approval: yes, 2026-09-02. +- 2026-09-02 Decision: the embedded vector store is the in-process edition of the service engine (Qdrant Edge), not an in-house exact scan and not a SQLite vector extension. + - Trigger / new insight: the decider restated the objective as a portable memory that plugs into any foundation model and tracks years to decades of continuous character development; the draft's in-house exact scan violated the library-over-in-house principle; two feasibility spikes on the same probe set measured the two library-backed candidates (numbers in ADR-I-0023). + - Plan delta: Task_4 rewritten for the in-process engine (shard directory per collection, threshold-based exactness, shared tie-closure loop, contract canary, dependency-weight report); the toolchain pin moved to 1.97.0 as a prerequisite; approximate indexing leaves the non-goals; named-vector coexistence and shard-to-server sync are recorded as available but not exercised this phase. + - Tradeoffs considered: recorded in ADR-I-0023's rejected alternatives (in-house scan; sqlite-vec, lighter and deterministic but exhaustive-only in its stable release with a single-maintainer approximate-index future; LanceDB; in-memory only; default flip now); a scale probe was offered and declined as not worth the effort, so the interactive-latency assumption at decade scale is a recorded revisit trigger rather than a measurement. + - User approval: yes, 2026-09-02. ## Notes -- Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the exact-scan corpus-size guidance must be measured, not assumed. -- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` issues no search and reports the not-requested verdict in both adapters; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality (query normalised once, records normalised at write) is asserted, not assumed. +- Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the latency guidance and the stripped dependency weight must be measured, not assumed; the engine is beta, so its pin is exact and its bump is gated by the canary. +- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` issues no search and reports the not-requested verdict in both adapters; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality across adapters (both engines normalise cosine internally) is asserted, not assumed. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md new file mode 100644 index 00000000..2fa1fd8c --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -0,0 +1,136 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely replace the embedded engine with a lighter exhaustive-scan store the first time the dependency weight is questioned, or treat the embedded store as a test convenience whose semantics may drift from the service adapter, because both are the shortest path at small corpus sizes and both were measured as viable" + detected_signals: "cross-boundary contract shape with tempting alternatives; rejected alternatives likely to be re-proposed (the two spiked candidates); costly reversal (an engine switch rebuilds every embedded store); premises likely to expire (the engine is beta; the weight measurement is unstripped); deliberately bounded scope (single process, opt-in default, index tuning deferred)" + cost_of_violation: "an engine switch after embedded stores exist in the field rebuilds every character's recall index from graph authority and re-embeds it; two adapters with different admission semantics produce different continuity packs from the same memory, which evaluation evidence would attribute to retrieval regressions" + cost_of_wrong_preservation: "if the engine's beta API breaks or its footprint proves unacceptable on a target platform and this record is preserved as settled, local deployments carry a dependency that no longer earns its place" + cost_of_over_extension: "treating the embedded mode as validated for multi-process access or as the default before parity evidence exists misrepresents what the library has validated; treating the index knobs as tuned when this phase leaves them at their exact-scan setting would ship approximate recall nobody measured" +depends_on: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0023: Embedded Qdrant Edge vector candidate store as the opt-in local mode + +## Context and Problem Statement + +After the embedded persistent graph store became the validated default (ADR-I-0021) and retrieval statistics were already file-backed (ADR-I-0009), the vector candidate store was the only component that still required an external service. +That conflicts with the intended deployment shapes: desktop companions and game or simulation characters run on end-user machines where a container runtime cannot be assumed, and it keeps a service dependency in the default test path. +ADR-I-0003's own revisit clause, "operating two stores becomes too heavy for target users", was recorded as triggered at the close of the eval-driven family closeout. +The vector layer is candidate recall only: the vector store suggests, retrieval statistics guide fanout, and graph authority decides final inclusion, so an embedded adapter has a low correctness bar for any single query — it must prefilter and rank candidates well, never be authoritative for anything. +The bar that matters is longevity: a character's memory is expected to accumulate continuously for years to decades and to outlive several generations of embedding model, so the embedded recall index must be chosen for where a memory ends up, not for where it starts. +Two feasibility spikes were run on 2026-09-02 against the same probe set (build weight, lifecycle and reopen, exactness control, a thirty-way identical-vector tie cohort, five filtered queries against the live service, API shape): the in-process build of the service backend (Qdrant Edge 0.8.0, beta) and a SQLite vector extension (sqlite-vec 0.1.9, stable). + +## Decision Drivers + +- The decade standard: at continuous-accumulation scale the recall index plausibly reaches hundreds of thousands to a million vectors, where an exhaustive scan costs seconds per query, so approximate indexing, quantization, and memory-mapped read-only segments are the baseline for an interactive character, not an escalation. +- Embedding models will change several times over a memory's life; named vectors that let two embedding spaces coexist during lazy re-embedding are a required capability, not a nicety. +- The store is a rebuildable cache over graph authority, which bounds the cost of an engine switch but does not eliminate it: every embedded store in the field is rebuilt and re-embedded. +- Library over in-house: the library does not own a vector engine; it owns the port contract, the tie-closure loop, the canonical ordering, the verdict mapping, and the error classification, and it holds any engine to those. +- The embedded adapter must satisfy the same port contract as the service adapter, proven by a shared parity suite, or the evaluation suite stops being a regression instrument. +- Defaults must match validation evidence (ADR-I-0021's rule); flipping the default before parity evidence exists would repeat the mistake that record corrected. +- Portability across deployment shapes: an engine that can synchronise with the service backend keeps a path from a local character to a hosted one. + +## Decision + +Add an embedded vector candidate store mode behind the existing vector candidate port, implemented on the in-process build of the service backend (Qdrant Edge), selected by a dedicated store-mode setting. +The service adapter remains fully supported as the service and cloud mode; this decision adds a mode and deprecates nothing. +The default mode stays service until the parity suite and the evaluation suite have produced identical results across modes; flipping the default is a separate, evidence-gated decision. +The embedded store is single-process, matching the embedded graph store's expectation. + +Exactness is a threshold property, not a promise: below the configured indexing threshold a shard answers by exhaustive scan, above it the index answers, and in both cases the completeness verdict (ADR-I-0024) reports the boundary state of the returned top-K. +This phase ships the threshold at its exact-scan setting; index construction, quantization, and memory-mapped segments become available capabilities whose defaults are tuned on measured corpora in a later decision, never silently. + +The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. +The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. +The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). +Searches and index builds never run on an async executor thread: the adapter invokes the engine from a blocking worker, or through the engine's own worker pool where its API is asynchronous, and the in-phase benchmark records executor responsiveness while a scan or build is in progress. +A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. + +Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. +The path names a directory; each collection is one engine shard directory inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has shard directories). +Embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name and the record schema version in an adapter-owned marker inside the shard directory, and requires the path setting to be present, with a missing path a configuration error rather than an implicit default. +A reopened shard is validated against the configured embedding model (vector size and distance from the shard's own configuration) and the supported record schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires. + +## Implementation Impact + +- A new adapter module implementing the vector candidate port on the embedded engine; the composition root gains a mode switch mirroring the statistics-store switch. +- The library's toolchain pin moves to the minimum the engine compiles on (Rust 1.97 at decision time; the engine rejected the previous 1.95 pin). +- The settings type gains the mode and path keys; the service connection string becomes optional and is validated as present only in service mode. +- The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion is updated in the same wave. +- The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured; the deterministic test fake is retired in favour of the embedded adapter opened on a temporary directory, which removes the vector-service dependency from the default test path. +- Dependency weight is a recorded deliverable: the unstripped release delta is measured; the stripped delta and the effect of feature trimming are measured and recorded before closeout. +- Documentation states the single-process expectation, the threshold semantics, the measured latency guidance, and rebuild-from-graph-authority as the path between modes. + +## Considered Options + +1. The in-process build of the service backend (Qdrant Edge), opt-in, service mode retained. +2. An in-house SQLite exact cosine scan on the `rusqlite` dependency the statistics store already carries. +3. The SQLite vector extension (sqlite-vec). +4. A columnar embedded vector database (LanceDB). +5. An in-memory-only embedded store. +6. Flip the default to embedded in the same change. + +## Decision Outcome + +Chosen option: **Option 1**. +Measured on the shared probe set: lifecycle and reopen identical; the zero indexing threshold keeps a shard on a plain exhaustive scan while a threshold of one plus an optimise call builds the index; five filtered queries against the live service returned identical id sets and order with a maximum score delta of 0.0; cost of 489 additional dependency-tree lines and about 30.5 MB of unstripped release binary (stripped size unverified, feature trimming untested). +It is the only candidate that offers, in one engine, the capabilities the decade standard makes baseline — approximate indexing, quantization, named vectors, datetime payload indexes, memory-mapped read-only segments — plus synchronisation to the service backend, and it is the same engine family the service adapter already targets, so payload and filter conventions are shared rather than translated. + +### Rejected Alternatives + +Option 2 (in-house exact scan) violates the library-over-in-house driver: the library would own distance computation, blob encoding, and scan scheduling, and every capability the decade standard needs (index, quantization, named vectors) would have to be written or migrated to later; it is rejected outright, not deferred. +Option 3 (sqlite-vec 0.1.9) measured well — builds on both the previous and the new toolchain pin, 19 additional tree lines, about 6.2 MB unstripped, deterministic ties across fresh files and processes, parity five of five with score delta at or below 1.6e-7 — but its stable release is exhaustive-only with approximate indexing existing only in a pre-release, it carries limits on dimensions, result count, and metadata columns, and it is a pre-1.0 binding with a single maintainer; at decade scale its future is a second migration, so it is rejected for this role and reopened only if the chosen engine fails its Revisit When triggers and the extension has shipped a stable approximate index. +Option 4 (LanceDB) was rejected on dependency weight relative to the chosen engine without a spike, since the chosen engine already covers its capabilities; it is reopened only alongside Option 3's reopening. +Option 5 fails restart safety, which the persistent-graph-authority phase made a requirement for every store that survives a process; rejected outright. +Option 6 contradicts the defaults-match-evidence rule; it is reopened by the evidence named under Revisit When. + +## Consequences + +- Positive: a fully self-contained local deployment exists; the default test path needs no running service; both adapters are held to one contract by one suite; the embedded store can grow into indexed, quantized, and memory-mapped operation without an engine switch. +- Positive: shared engine family means payload and filter conventions are written once and the service parity result (score delta 0.0) is structural, not coincidental. +- Negative / tradeoffs: the engine is beta and its API may change; the canary test and the pinned version turn that into a build-time failure rather than a runtime one. +- Negative / tradeoffs: about 30.5 MB of unstripped binary and a higher toolchain floor; the weight deliverable exists to establish the real number. +- Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. + +## Decision Boundary + +Invariant: the embedded adapter implements the same port contract as the service adapter and is proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string; tie closure and canonical ordering come from the shared library loop and constructor, never from engine ordering; the indexing threshold ships at its exact-scan setting until a measured decision changes it. + +Not covered: the index, quantization, and memory-map tuning values (calibrated through a later measured decision), the latency guidance numbers (measured and revised through documentation), and the default mode (a separate evidence-gated decision). + +## Validation + +- Embedded mode constructs without any running service and survives process restart with identical search results. +- The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop; above the threshold a recall comparison against the exhaustive setting is recorded. +- A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. +- The contract canary passes on the pinned engine version and is re-run on every engine bump. +- The benchmark shows no scan or index build occupying an async executor thread. +- The default-mode construction test asserts service mode. + +## Revisit When + +- The engine leaves beta or changes its API — re-pin, re-run the canary, and re-measure parity before adopting the new version. +- A stripped-footprint measurement or feature trimming changes the weight picture materially in either direction — revisit the weight tradeoff recorded above and, if the footprint is unacceptable on a target platform, reopen Option 3. +- A corpus benchmark contradicts the interactive-latency assumption behind the decade standard (exhaustive scan acceptable far beyond the assumed scale, or the index insufficient at it) — revisit the threshold default and the tuning decision. +- The evaluation suite has run every dataset in embedded mode with results identical to service mode and one corpus at the guidance size — reopen the default mode. +- A multi-replica deployment shape is designed (the remote graph-authority phase ADR-I-0021 anticipates) — the single-process expectation is reconsidered together with the graph and statistics stores, never alone. + +## Consultation impact + +Question asked: which embedded engine, on the two spikes' evidence; the consult's earlier recommendation of an in-house exact scan was overruled by the decider on the decade-scale portability standard, and the settings shape and opt-in default were adopted as recommended. + +## More Information + +- ADR-I-0003 remains fully authoritative for the default backends; this record adds an opt-in mode in response to its revisit clause and changes no default, so it supersedes nothing. A later, evidence-gated record that flips the default would supersede ADR-I-0003's vector-backend default. +- ADR-I-0024 (port contract this adapter implements, including the tie-closure and verdict rules) and ADR-I-0025 (the record it stores). +- The two spike reports (2026-09-02) are transient working artifacts; the numbers above are their record. +- The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md deleted file mode 100644 index fe68c124..00000000 --- a/docs/decisions/implementation/ADR-I-0023-embedded-sqlite-exact-scan-vector-candidate-store.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -status: accepted -adr_type: implementation -date: 2026-09-02 -deciders: ["ebigunso"] -consulted: ["Claude Fable 5.1"] -informed: [] -warrant: - warranted_by: "without this record, future work would likely either treat the embedded vector store as a test convenience and let it drift from the service adapter's contract, or reach for an embedded approximate-nearest-neighbour library the moment a corpus feels large, discarding the exact-scan determinism the parity contract depends on" - detected_signals: "cross-boundary contract shape with a tempting alternative; rejected alternative likely to be re-proposed; premises likely to expire (corpus scale, in-process build of the service backend); deliberately bounded scope (single process, opt-in default)" - cost_of_violation: "two vector adapters with different admission semantics silently produce different continuity packs from the same memory, which the evaluation suite would attribute to retrieval regressions rather than backend divergence" - cost_of_wrong_preservation: "once corpora exceed the exact-scan guidance or the service backend ships a stable in-process build, keeping the exact scan as the only embedded option would make local deployments slow for no contractual reason" - cost_of_over_extension: "applying the single-process expectation to multi-replica deployments, or treating the embedded mode as the validated default before parity evidence exists, misrepresents what the library has validated" -depends_on: [implementation/ADR-I-0009-use-sqlite-as-default-retrieval-stats-store.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] -implements: [] -supersedes: [] -superseded_by: null -supersession_scope: null ---- - -# ADR-I-0023: Embedded SQLite exact-scan vector candidate store as the opt-in local mode - -## Context and Problem Statement - -After the embedded persistent graph store became the validated default (ADR-I-0021) and retrieval statistics were already file-backed (ADR-I-0009), the vector candidate store was the only component that still required an external service. -That conflicts with the intended deployment shapes: desktop companions and game or simulation characters run on end-user machines where a container runtime cannot be assumed, and it keeps a service dependency in the default test path. -ADR-I-0003's own revisit clause, "operating two stores becomes too heavy for target users", was recorded as triggered at the close of the eval-driven family closeout. -The vector layer is candidate recall only: the vector store suggests, retrieval statistics guide fanout, and graph authority decides final inclusion, so an embedded adapter has a low correctness bar — it must prefilter and rank candidates well, never be authoritative for anything. - -## Decision Drivers - -- Zero-infrastructure local deployment is a product requirement, not a convenience. -- The embedded adapter must satisfy the same port contract as the service adapter, proven by a shared parity suite; anything less makes the evaluation suite an unreliable regression instrument. -- No heavyweight dependency for a first implementation; `rusqlite` with the bundled engine is already a dependency through the statistics store. -- Exact scan at character-memory scale (tens of thousands of vectors) is honest, deterministic, and strictly better recall than approximate search. -- Defaults must match validation evidence (ADR-I-0021's rule); flipping the default before parity evidence exists would repeat the mistake that ADR-I-0021 corrected. - -## Decision - -Add an embedded vector candidate store mode behind the existing vector candidate port, implemented as a SQLite-backed exact cosine scan, selected by a dedicated store-mode setting. -The service adapter remains fully supported as the service and cloud mode; this decision adds a mode and deprecates nothing. -The default mode stays service until the parity suite and the evaluation suite have produced identical results across modes; flipping the default is a separate, evidence-gated decision. -The embedded store is single-process, matching the embedded graph store's expectation. - -Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. -The path names a directory; each collection is one SQLite file inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has files); embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name in the file's metadata table, and requires the path setting to be present, with a missing path a configuration error rather than an implicit default. - -Physical shape: one table keyed by object id and surface, one column per field of the vector record read contract (ADR-I-0025), the embedding stored as a fixed-width little-endian floating-point blob normalised at write, an index on object type for the scope predicate, and a metadata table recording vector size, distance, and record schema version so a reopened file is validated against the configured embedding model and the supported schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires; a restart test covers both. -Search normalises the query vector once before scanning (a zero-norm query scores every row zero and is reported, not rejected), scores the scoped rows by dot product in a fixed order so the score equals the cosine the service adapter reports, canonicalises through the shared constructor the port requires, and truncates to the requested limit; every issued search reports exhaustive completeness, and a zero limit issues no search and reports not requested (ADR-I-0024). -Score parity across adapters is asserted by a parity fixture whose query and record vectors are deliberately non-unit. - -## Implementation Impact - -- A new adapter module implementing the vector candidate port; the composition root gains a mode switch mirroring the statistics-store switch. -- The settings type gains the mode and path keys; the service connection string becomes optional and is validated as present only in service mode. -- The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion is updated in the same wave. -- The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured. -- The deterministic test fake that reimplements cosine scoring and scope filtering is retired in favour of the embedded adapter opened in memory, which removes the vector-service dependency from the default test path. -- Documentation states the single-process expectation, the measured corpus-size guidance, and rebuild-from-graph-authority as the path between modes. - -## Considered Options - -1. SQLite exact cosine scan behind the existing port, opt-in, service mode retained. -2. An embedded approximate-nearest-neighbour library as the first embedded implementation. -3. The in-process build of the service backend. -4. An in-memory-only embedded store. -5. Flip the default to embedded in the same change. - -## Decision Outcome - -Chosen option: **Option 1**. -It reuses an existing dependency, gives exact filter and ranking semantics that make the parity contract checkable by set equality, is deterministic by construction, and is restart-safe through an ordinary file. - -### Rejected Alternatives - -Option 2 adds a heavyweight dependency and approximate membership semantics before any corpus has demonstrated that exact scan is on the critical path; it is the recorded escalation path, reopened by a measured corpus exceeding the published exact-scan guidance or a benchmark showing the scan dominating retrieval latency. -Option 3 was not stable at decision time; it is a revisit candidate once it ships a stable release, because it would maximise reuse of the service adapter's conventions. -Option 4 fails restart safety, which the persistent-graph-authority phase made a requirement for every store that survives a process. -Option 5 contradicts the defaults-match-evidence rule; it is reopened by the evidence named under Revisit When. - -## Consequences - -- Positive: a fully self-contained local deployment exists; the default test path needs no running service; both adapters are held to one contract by one suite. -- Positive: the embedded adapter is the honest reference implementation for the port's semantics because it computes them exactly. -- Negative / tradeoffs: exact scan is linear in corpus size and reads every scoped embedding per query; the guidance number is measured, not engineered around. -- Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. - -## Decision Boundary - -Invariant: the embedded adapter implements the same port contract as the service adapter and is proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string. - -Not covered: the corpus-size guidance number (measured and revised through documentation), the choice to add an in-memory embedding cache in front of the scan (an implementation optimisation), and the default mode (a separate evidence-gated decision). - -## Validation - -- Embedded mode constructs without any running service and survives process restart with identical search results. -- The parity suite produces identical admitted candidate sets and orderings from both adapters across the full contract, including identical-vector tie cohorts. -- A reopened embedded store with a different vector size fails with the collection-compatibility error. -- The default-mode construction test asserts service mode. - -## Revisit When - -- A measured corpus exceeds the published exact-scan guidance, or a benchmark shows the scan on the retrieval critical path — reopen the approximate-index escalation. -- The service backend's in-process build reaches a stable release — reopen the choice of embedded engine. -- The evaluation suite has run every dataset in embedded mode with results identical to service mode and one corpus at the guidance size — reopen the default mode. -- A multi-replica deployment shape is designed (the remote graph-authority phase ADR-I-0021 anticipates) — the single-process expectation is reconsidered together with the graph and statistics stores, never alone. - -## Consultation impact - -Question asked: whether the embedded store should overload the service connection string or take its own settings, and whether to flip the default now; ruling adopted the separate settings and the opt-in default as recommended. - -## More Information - -- ADR-I-0003 remains fully authoritative for the default backends; this record adds an opt-in mode in response to its revisit clause and changes no default, so it supersedes nothing. A later, evidence-gated record that flips the default would supersede ADR-I-0003's vector-backend default. -- ADR-I-0024 (port contract this adapter implements) and ADR-I-0025 (the record it stores). -- The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index d1b909a7..4478b32c 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -26,7 +26,7 @@ The vector candidate port promised deterministic admission: at most `limit` uniq The service adapter closes the cohort by growing its fetch up to a bound, but when the bound is hit it returns the truncated set with no signal, and the port's result type — a bare candidate list — cannot carry the difference between "this top-K is determinate" and "membership may vary between runs". Separately, the port once carried a filter type whose currentness predicates were `Option` values implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake. Those filters were deleted as speculative in the structured-verdict phase because no caller used them; the query is now an embedding, a limit, and an object-type scope. -An embedded adapter (ADR-I-0023) makes the gap visible: an exact scan is exhaustive by construction and needs a way to say so, and a second adapter needs a query contract that cannot drift. +An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing threshold an embedded shard scans exhaustively and needs a way to say so, and a second adapter needs a query contract that cannot drift. ## Decision Drivers @@ -43,13 +43,14 @@ An embedded adapter (ADR-I-0023) makes the gap visible: an exact scan is exhaust ```rust pub enum VectorRecallCompleteness { NotRequested, // the limit was zero; no search was issued - Exhaustive { scanned: usize }, // every stored record in scope was scored + Exhaustive { scanned: usize }, // the scoped population was scored and returned in full (an unindexed shard returned fewer rows than the fetch limit) BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort was closed BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort open } ``` -Adapters own canonicalisation and must state the verdict truthfully: not requested only when the limit is zero and no search was issued (an admitted configuration, so the verdict must be total over it), exhaustive only when the whole scoped population was scored, closed only when the cutoff cohort was verified closed or the index returned fewer rows than asked, open at the bound. +Adapters own canonicalisation and must state the verdict truthfully: not requested only when the limit is zero and no search was issued (an admitted configuration, so the verdict must be total over it), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and returned, closed only when the cutoff cohort was verified closed or an index returned fewer rows than asked, open at the bound. +Every adapter answers through the shared tie-closure loop and the canonical constructor; engine ordering of equal-score cohorts is never relied on, because it is not stable across freshly built shards (ADR-I-0023). The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. @@ -82,7 +83,7 @@ Two re-entry paths are named now so the scope-only query is read as a current st ## Decision Outcome Chosen option: **Option 1**. -It makes the postcondition expressible by the type that owns it, distinguishes the exhaustive case the embedded adapter introduces from the closed-cohort case the service adapter can promise, and keeps every consumer a field access away from unchanged code. +It makes the postcondition expressible by the type that owns it, distinguishes the exhaustive case an unindexed embedded shard can report from the closed-cohort case an index can promise, and keeps every consumer a field access away from unchanged code. ### Rejected Alternatives @@ -107,7 +108,7 @@ Not covered: the service adapter's overfetch bound constants (calibrated values) - Unit tests on the service adapter's fetch decision assert the mapping to closed and open verdicts, including the all-tied cohort at the bound. - A retrieval test asserts the telemetry verdict for each variant using the fakes. -- The parity suite asserts exhaustive for the embedded adapter and closed for the service adapter on the identical-vector tie fixture. +- The parity suite asserts exhaustive for the embedded adapter below its indexing threshold and closed for the service adapter on the identical-vector tie fixture, both reached through the shared tie-closure loop. - A census of the vector adapters shows no match-or-unknown condition and no filter type beyond the object-type scope. ## Revisit When @@ -123,4 +124,4 @@ Question asked: whether the deleted hint filters should return for the embedded - ADR-I-0022 (tie-cohort closure and canonical ordering at the adapter boundary, the postcondition this record makes expressible). - ADR-I-0025 (the stored record whose columns the re-entry paths would extend). -- ADR-I-0023 (the embedded adapter that always reports exhaustive completeness). +- ADR-I-0023 (the embedded adapter, which reports exhaustive below its indexing threshold and the boundary verdicts above it). diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index 14600a28..64c8b4df 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -53,7 +53,7 @@ ADR-I-0024 names the two re-entry paths (a synchronised scope predicate; an immu ## Implementation Impact -- The vector record type and the surface builders lose the hint carriers; the payload map and the embedded schema serialise five fields. +- The vector record type and the surface builders lose the hint carriers; the payload map serialises five fields for both adapters, which share the engine family's payload conventions. - The service adapter stops creating per-field payload indexes for dropped fields. - The companion evaluation repository's vector-only baseline stops reading the readable text column and sources item text from its own ingest records (ADR-I-0026). - The payload design note's field categories and indexing policy are superseded by this record and carry a supersession note. @@ -81,7 +81,7 @@ Option 5 is the only subset with a forward-looking case that survives the synchr ## Consequences -- Positive: both adapters mirror one five-field manifest; the embedded schema stores those fields beside the vector blob with a single scope index (ADR-I-0023 owns the physical layout). +- Positive: both adapters mirror one five-field manifest; the embedded shard stores those fields as payload beside the vector with a single keyword index on object type (ADR-I-0023 owns the physical layout). - Positive: the embedded surface is preserved as vector provenance before surfaces become generated. - Negative / tradeoffs: a future scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the re-entry paths make that step predictable. @@ -111,4 +111,4 @@ Question asked: whether the unread hint families and the readable text column sh - ADR-I-0005 remains authoritative for graph authority over relationships; this record supersedes its payload field list and its "payload metadata as candidate filter" implementation guidance. - ADR-I-0002 remains authoritative for natural-language embedding surfaces; this record supersedes only its note to persist both text columns. -- ADR-I-0024 (query contract and re-entry paths), ADR-I-0023 (embedded schema), ADR-I-0026 (evaluation baseline reader). +- ADR-I-0024 (query contract and re-entry paths), ADR-I-0023 (embedded shard layout), ADR-I-0026 (evaluation baseline reader). diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index ece1b97d..8170a6b5 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -1,21 +1,23 @@ # v0.1.6 Design: Embedded Vector Candidate Recall -Status: decided 2026-09-02 (ADR-I-0023 through ADR-I-0026); supersedes the 2026-07 draft of this document. +Status: decided 2026-09-02 (ADR-I-0023 through ADR-I-0026); supersedes the 2026-07 draft of this document; the embedded engine ruling (Qdrant Edge over an in-house scan) was taken the same day on two feasibility spikes. ## Version intent Complete the zero-infrastructure local deployment story by adding an embedded vector candidate store mode behind the existing vector candidate port, and settle the port contract that both adapters must satisfy before a second adapter exists. With graph authority defaulting to embedded persistent storage (ADR-I-0021) and retrieval statistics already file-backed (ADR-I-0009), the vector candidate store is the only component that still requires an external service. That conflicts with the desktop-companion and game or simulation use cases, where end users cannot be expected to operate containers, and it keeps a service dependency in the default test path. +The embedded engine is chosen for where a character's memory ends up, not where it starts: a memory expected to accumulate for years to decades across several generations of embedding model needs approximate indexing, quantization, memory-mapped segments, and coexisting embedding spaces as baseline capabilities, so the in-process build of the service backend is adopted rather than an exhaustive scan the library would own (ADR-I-0023). Sequencing: this phase runs before scoped continuity, so the vector record is mirrored across two adapters while it is five fields, and so the scoped-continuity evaluation fixtures know which vector backend they validate against. ## Why this is safe to do before scoped continuity The vector layer is candidate recall only: the vector store suggests, statistics guide fanout, and graph authority decides final inclusion. -An embedded adapter therefore has a low correctness bar: it must prefilter and rank candidates well, never be authoritative for anything. +An embedded adapter therefore has a low correctness bar for any single query: it must prefilter and rank candidates well, never be authoritative for anything. The port is small (upsert, scoped search, delete), provider-neutral, and already exercised by deterministic fakes and a live smoke surface. The write path already removes the vectors of superseded and suppressed objects, so the live vector population is the active population by construction, and stale residue from failed maintenance is caught by graph verification; the embedded adapter inherits both guarantees without new code. +The embedded engine is the same family as the service backend, so payload and filter conventions are shared rather than translated, and the spike measured identical id sets, identical order, and a score delta of 0.0 on five filtered queries against the live service. ## Design direction @@ -26,8 +28,8 @@ This phase fixes the port contract deliberately, because two adapters cannot be Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. -Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero and no search was issued), exhaustive (every scoped record was scored), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). -The service adapter's existing tie-cohort loop maps onto the last two verdicts; before this phase it returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. +Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero and no search was issued), exhaustive (the scoped population was scored and returned in full), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). +Both adapters answer through the shared tie-closure loop and the canonical constructor; before this phase the service adapter returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. @@ -42,14 +44,20 @@ Two re-entry paths are named so the scope-only query and five-field record are r ### The embedded adapter (ADR-I-0023) -A SQLite-backed exact cosine scan, using the `rusqlite` dependency the statistics store already carries, with the same single-process, mutex-guarded connection model, except that the scan itself is a synchronous, potentially long operation behind an async port: it runs on a blocking worker (the runtime's blocking pool) with connection access still serialized, so a scan never occupies an async executor thread, and the in-phase benchmark records executor responsiveness while a scan is in progress. +The adapter runs the in-process build of the service backend (the `qdrant-edge` crate, pinned exactly; beta at adoption) as one engine shard per collection. -Schema: one table keyed by object id and surface with a column per contract field and the embedding as a fixed-width little-endian floating-point blob normalised at write; an index on object type for the scope predicate; a metadata table recording vector size, distance, and schema version. -Search: normalise the query once (a zero-norm query scores every row zero and is reported, not rejected), select the scoped rows, score by dot product in a fixed order so the score equals the service adapter's cosine, canonicalise through the shared constructor, truncate to the limit, and report exhaustive completeness with the scanned count; a parity fixture with non-unit query and record vectors pins score equality across adapters. +Exactness is a threshold property: a shard below its configured indexing threshold answers by exhaustive scan, a shard above it answers from its index, and the completeness verdict reports the boundary state either way. +This phase ships the threshold at its exact-scan setting (the spike confirmed that a zero threshold leaves a shard unindexed while a threshold of one plus an optimise call builds the index); index construction, quantization, and memory-mapped segments are available capabilities whose defaults are tuned in a later measured decision, never silently. + +Shard: cosine distance at the configured vector size; the five-field payload with a keyword index on object type; the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). +Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. +Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero. Delete: remove every surface of each object id, matching the service adapter's selector. -Restart safety: opening an existing file validates the recorded vector size and distance against the configured embedding model and raises the same collection-compatibility error the service adapter raises for a mismatched collection. -Determinism: same inputs, same scores, same total sort; equal-score cohorts are ordered by the shared comparator, so the embedded adapter satisfies deterministic admission by construction and never needs an overfetch loop. -Score parity across adapters is not bitwise (the service computes cosine on its own normalised copy), so parity compares membership and order with a small score tolerance, and tie fixtures use identical vectors. +Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. +Blocking discipline: a scan or an index build is a potentially long synchronous operation behind an async port; the adapter guarantees that neither ever occupies an async executor thread, by invoking the engine from a blocking worker or through the engine's own worker pool where its API is asynchronous, and the in-phase benchmark records executor responsiveness while a scan is in progress. +Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. +Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. +Score parity across adapters was measured at 0.0 delta on the spike; the parity suite still asserts it with non-unit query and record vectors rather than assuming it. ### Settings and composition (ADR-I-0023) @@ -61,13 +69,16 @@ VECTOR_STORE_PATH directory, required in embedded mode (missing is a configura QDRANT_CONNECTION_STRING required only in service mode ``` -`VECTOR_STORE_PATH` is a directory; each collection is one SQLite file inside it named by the collection name the public constructor already takes, so `collection_name` is the backend-neutral namespace key in both modes. -Because the collection name becomes a file name under the configured directory, embedded mode validates it at construction with a contract owned here: lowercase ASCII letters, digits, underscore, and hyphen only (lowercase so that names stay unique on the case-insensitive filesystems of desktop targets), first character a letter or digit, at most 128 characters, not a reserved device name on Windows (con, prn, aux, nul, com1 to com9, lpt1 to lpt9), no path separators, dots, or empty name; anything else is rejected with the configuration error before any file is touched, the name is recorded in the file's metadata table, and a path-confinement test proves that separator and parent-directory inputs cannot escape the directory. +`VECTOR_STORE_PATH` is a directory; each collection is one engine shard directory inside it named by the collection name the public constructor already takes, so `collection_name` is the backend-neutral namespace key in both modes. +Because the collection name becomes a directory name under the configured path, embedded mode validates it at construction with a contract owned here: lowercase ASCII letters, digits, underscore, and hyphen only (lowercase so that names stay unique on the case-insensitive filesystems of desktop targets), first character a letter or digit, at most 128 characters, not a reserved device name on Windows (con, prn, aux, nul, com1 to com9, lpt1 to lpt9), no path separators, dots, or empty name; anything else is rejected with the configuration error before any directory is touched, the name and the record schema version are recorded in an adapter-owned marker inside the shard directory, and a path-confinement test proves that separator and parent-directory inputs cannot escape the directory. +The engine requires the shard root directory to exist before opening; the adapter creates it, as the statistics store creates its parent directory. The composition root gains a vector-store mode switch mirroring the statistics-store switch; the vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds. +The toolchain pin moves to the engine's minimum (Rust 1.97 at adoption; the previous pin did not compile it), in its own change before the adapter lands. ### Parity suite placement (ADR-I-0023) -Library: a port-conformance suite in the integration tests — scope filtering, empty scope selects zero, canonical order, identical-vector tie cohort, best-score-per-object-and-surface deduplication, delete removes all surfaces, restart reopen, completeness verdict per adapter — run against the embedded adapter unconditionally and against the service adapter when a service connection is configured. +Library: a port-conformance suite in the integration tests — scope filtering, empty scope selects zero, canonical order, identical-vector tie cohort closed through the shared loop, best-score-per-object-and-surface deduplication, delete removes all surfaces, restart reopen, completeness verdict per adapter — run against the embedded adapter unconditionally and against the service adapter when a service connection is configured. +Parity acceptance is identical admitted sets and orderings while both adapters are below their indexing thresholds (the suite's fixtures are small enough that both are), and a recorded recall comparison of the embedded adapter above its threshold against its exhaustive setting, informational this phase. This follows the precedent that port conformance is enforced by contract tests run against every adapter, not by a runtime wrapper. Evaluation repository: no second contract suite; it adds an embedded-mode configuration to the continuity scenarios and requires identical scenario results between modes, the behaviour-level regression instrument. @@ -76,12 +87,14 @@ Evaluation repository: no second contract suite; it adds an embedded-mode config ```text port contract: completeness envelope, scope-only query with empty-scope-selects-zero, retrieval telemetry completeness field vector record read contract: five-field manifest shared by both adapters -SqliteVectorCandidateStore adapter: schema, upsert/delete, scoped exact-scan search, restart validation -VectorStoreMode and VectorStorePath settings; composition mode switch; service connection string required only in service mode -port-conformance parity suite in the library integration tests, run against both adapters +QdrantEdgeVectorCandidateStore adapter: shard per collection at the exact-scan threshold, upsert/delete, scoped search through the shared tie-closure loop, verdict mapping, restart validation, blocking discipline +engine contract canary test, re-run on every engine bump +VectorStoreMode and VectorStorePath settings; composition mode switch; service connection string required only in service mode; toolchain pin at the engine minimum +port-conformance parity suite in the library integration tests, run against both adapters; above-threshold recall comparison recorded restart-safety test for the embedded store; pipeline test over the embedded adapter with a deleted graph object -measured corpus-size guidance from an in-phase benchmark -documentation: settings, single-process expectation, corpus-size guidance, rebuild-from-graph-authority as the path between modes +dependency-weight report: unstripped and stripped release deltas, effect of feature trimming +latency benchmark: exhaustive scan across corpus sizes at the configured dimension; executor responsiveness under a concurrent scan +documentation: settings, single-process expectation, threshold semantics, measured latency guidance, rebuild-from-graph-authority as the path between modes four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0001, ADR-I-0002, and ADR-I-0005 ``` @@ -92,7 +105,7 @@ the hint carriers on the vector record type and the surface builders' hint popul the readable text column and the per-field payload index creation for dropped fields the test-only payload field constants and the prose-assertion note constant the service adapter's private enum token mappers, replaced by one Display/FromStr per enum in the domain (the embedded adapter must not add another copy) -the deterministic vector fake and its embedding-bearing record type, replaced by the embedded adapter opened in memory (failure-injecting and recording fakes stay) +the deterministic vector fake and its embedding-bearing record type, replaced by the embedded adapter opened on a temporary directory (failure-injecting and recording fakes stay) the port doc comment's "documented bounded-overfetch degradation policy" clause, now expressed by the type ``` @@ -101,10 +114,12 @@ the port doc comment's "documented bounded-overfetch degradation policy" clause, ```text changing the authority split or any retrieval semantics deprecating or altering the service adapter -approximate-nearest-neighbour indexing (the recorded escalation path if embedded ANN ever becomes necessary) +tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold, tuned by a later measured decision) +named-vector coexistence of two embedding spaces (an engine capability this decision was taken for; its use lands with the first embedding-model migration) migration tooling between modes or between record shapes (rebuild-from-graph-authority is the documented path) changing the default vector mode in this phase (embedded ships opt-in; flipping the default is a separate evidence-gated decision) multi-process access to the embedded store (same single-process expectation as embedded graph storage) +synchronisation between an embedded shard and a service collection (an engine capability; not exercised this phase) any vector-layer predicate beyond the object-type scope (the two named re-entry paths belong to later phases) any new public facade method (the evaluation baseline consumes the retrieval trace) reconciliation diagnostics (the reconciliation slice was deleted in the structured-verdict phase; graph verification is the guard) @@ -112,23 +127,28 @@ reconciliation diagnostics (the reconciliation slice was deleted in the structur ## Technology posture -- SQLite exact scan first: zero new dependencies, exact filter semantics, deterministic, restart-safe. -- The first escalation inside the embedded mode is an in-memory normalised matrix loaded at open with write-through, an implementation optimisation that changes no contract. -- An embedded approximate-nearest-neighbour library is the recorded escalation path if corpora outgrow exact scan; a measured corpus exceeding the guidance or a benchmark showing the scan on the critical path reopens it. -- The in-process build of the service backend is a revisit candidate once it ships a stable release; it would maximise reuse of the service adapter's conventions. -- Deployments that outgrow the embedded mode are exactly the deployments that should use the service mode; publish a measured corpus-size guidance number rather than engineering for it (at a 3072-dimension model each ten thousand vectors is about 120 MB of embedding data read per query, which bounds the honest number). +- The in-process build of the service backend (Qdrant Edge, pinned exactly at 0.8.0, beta) is the embedded engine: measured on the spike as lifecycle-and-reopen identical, exactness controllable by the indexing threshold, five-of-five service parity at score delta 0.0, at a cost of 489 additional dependency-tree lines and about 30.5 MB of unstripped release binary (stripped size and feature trimming are measured in-phase), requiring Rust 1.97 or later. +- The decade standard behind the choice: at continuous-accumulation scale the recall index plausibly reaches hundreds of thousands to a million vectors, where exhaustive scan is seconds per query, so approximate indexing, quantization, and memory-mapped segments are the baseline for an interactive character; embedding models will change several times, so named vectors that let two spaces coexist during lazy re-embedding matter; the store is a rebuildable cache over graph authority, which bounds but does not eliminate the cost of an engine switch. +- Library over in-house: the library owns the port contract, the tie-closure loop, the canonical ordering, the verdict mapping, and the error classification, and holds any engine to them; it does not own distance computation, storage layout, or index construction. +- Rejected on the same probe set: an in-house SQLite exact scan (violates library-over-in-house and would re-implement every capability above later); the SQLite vector extension sqlite-vec 0.1.9 (builds on both toolchains, 19 tree lines, about 6.2 MB unstripped, deterministic ties across fresh files, parity five of five at delta at or below 1.6e-7, but exhaustive-only in its stable release with approximate indexing only in a pre-release, a pre-1.0 binding, and a single maintainer); LanceDB (weight, without a spike, since the chosen engine covers its capabilities); an in-memory-only store (no restart safety). +- Beta risk is handled structurally: exact pin, contract canary, parity re-run on every bump; the engine leaving beta or changing its API, a stripped-footprint measurement that changes the weight picture, or a corpus benchmark contradicting the interactive-latency assumption each reopen ADR-I-0023. +- Deployments that outgrow the embedded mode are exactly the deployments that should use the service mode; the engine's synchronisation to a service collection is the recorded portability path from a local character to a hosted one. ## Acceptance criteria ```text Embedded mode is configurable and constructs without any running service. -The parity suite produces identical admitted candidate sets and orderings from both adapters across the full contract, including identical-vector tie cohorts. -Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical). -Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive, the service adapter reports closed on the tie fixture. -Embedded state survives process restart; a reopened store with a different vector size fails with the collection-compatibility error. +The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop. +A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded. +Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). +Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. +Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. +No scan or index build occupies an async executor thread; the benchmark records executor responsiveness under a concurrent scan. +The engine contract canary passes on the pinned version. +The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. The default test path requires no vector service; service-gated suites continue to pass unchanged. Both adapters persist exactly the five-field read contract; a census of both repositories shows no reader of a dropped field. -Documentation states the single-process expectation, the corpus-size guidance, and the rebuild-from-authority path. +Documentation states the single-process expectation, the threshold semantics, the latency guidance, and the rebuild-from-authority path. No public facade change; no retrieval behaviour change in service mode beyond the added telemetry field. ``` @@ -159,8 +179,8 @@ Each item was parked on this phase by the structured-verdict phase; each row sta Evidence: zero-hit census for the readable text column across both repositories; a vector-only run before and after produces identical item identities and text. 3. Search completeness. Parked claim: the port cannot express whether the top-K was determinate. - Re-verified: the only degradation site is the service adapter's fetch bound; no pipeline path inspects or retries on it. - Evidence: the fetch-decision unit test asserts the open verdict at the bound; a retrieval test asserts the telemetry field per variant; the live boundary test asserts the closed verdict. + Re-verified: the only degradation site is the service adapter's fetch bound; no pipeline path inspects or retries on it; the embedded engine's own tie order is not stable across fresh shards, which makes the shared loop a requirement for both adapters rather than a service-only workaround. + Evidence: the fetch-decision unit test asserts the open verdict at the bound; a retrieval test asserts the telemetry field per variant; the live boundary test asserts the closed verdict; the parity tie fixture asserts exhaustive versus closed through the same loop. 4. Hint filter semantics. Parked claim: query-side hint semantics belong to the port contract. Re-verified: the filter type and both match-or-unknown implementations were deleted in the structured-verdict phase, and no consumer asks for a vector-layer predicate (the evaluation surface policy carries object types and budgets only). @@ -172,10 +192,13 @@ Each item was parked on this phase by the structured-verdict phase; each row sta ## Decisions (the draft's open questions, resolved 2026-09-02) +- Embedded engine: the in-process build of the service backend (Qdrant Edge), on the decade-scale portability standard and the two spikes' measurements; the in-house exact scan and the SQLite vector extension are rejected alternatives (ADR-I-0023). +- Exactness: a threshold property shipped at the exact-scan setting; index, quantization, and memory-map tuning is a later measured decision (ADR-I-0023). - Default mode: stays opt-in this phase; the flip is reopened by the evaluation suite running every dataset in embedded mode with identical results and one corpus at the guidance size (ADR-I-0023, Revisit When). -- Corpus-size guidance: measured in-phase by a benchmark over a synthetic corpus at the configured dimension, published in documentation, revised through documentation. +- Latency guidance: measured in-phase by a benchmark over a synthetic corpus at the configured dimension, published in documentation, revised through documentation. +- Dependency weight: the unstripped delta is recorded; the stripped delta and feature trimming are measured in-phase, and a material change reopens ADR-I-0023. - Parity suite placement: contract parity in the library, behaviour parity in the evaluation repository (above). -- Settings shape: separate mode and path keys with `collection_name` as the backend-neutral namespace key, not a connection string interpreted by mode (ADR-I-0023). +- Settings shape: separate mode and path keys with `collection_name` as the backend-neutral namespace key naming one shard directory per collection, not a connection string interpreted by mode (ADR-I-0023). - Hint families: all dropped from the vector record, with the two named re-entry paths (ADR-I-0024, ADR-I-0025). - Text columns: readable text dropped, embedded text kept as provenance, governed by the three sentences in ADR-I-0025. - Evaluation baseline: trace-sourced, no facade change (ADR-I-0026). diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index 8f11285e..fd5004db 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -259,7 +259,7 @@ Assisted remember workflows may accept raw or semi-raw input as transient proces | v0.1.3 | Remember intake interfaces and deterministic write planning | Finished. Generation-ready write path with `RememberWritePlan`, memory candidates, validation, deterministic helpers, prepare/validate/commit flow, and shared manual/future-generated commit machinery. | | v0.1.4 | Continuity evaluation harness | Finished. Deterministic long-horizon evaluation harness implemented in the public companion `CharacterMemoryEvals` repository as a development aid, not core library functionality: synthetic interaction fixtures, a minimal example assistant loop, continuity-oriented retrieval-quality metrics, selectivity/fanout measurement, and hub-entity stress scenarios. | | v0.1.5 | Eval-driven v0.1 family closeout | Finished. Ran the evaluation harness across the v0.1 family, dispositioned eleven findings (none critical, none open), fixed deterministic vector admission and write-path warning diagnostics in the library, retained the measured defaults with a recorded basis (ADR-I-0022), adopted embedded persistent Oxigraph as the validated default (ADR-I-0021), and expanded the evaluation suite to 33 scenarios including benchmark-adapted and real-embedding fixtures. Closeout report: [`v0_1_5_closeout_report.md`](v0_1_5_closeout_report.md). | -| v0.1.6 | Embedded vector candidate recall | Planned. An embedded SQLite exact-scan vector candidate store behind the vector port as an opt-in local mode, so zero-infrastructure local deployments become possible and the default test path needs no external service while the service mode stays the default; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores only the fields a reader consumes; the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | +| v0.1.6 | Embedded vector candidate recall | Planned. An embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) behind the vector port as an opt-in local mode, shipped at its exact-scan indexing threshold, so zero-infrastructure local deployments become possible and the default test path needs no external service while the service mode stays the default; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores only the fields a reader consumes; the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | | v0.2 | Scoped continuity and reflection | `ContinuityScope`, scoped reflection, relationship state between arbitrary entities, character signals for continuing entities, open-loop/commitment lifecycle, and current continuity views. | | v0.3 | Factual rigor, temporal validity, and entity evolution | Assertions, claims, evidence links, belief assessments, source assessment, temporal validity, entity drift handling, and current-belief views. | | v0.4 | Retrieval observability and governance | Retrieval traces, context subgraphs, validation rules, graph health reports, policy diagnostics, rejected expansion traces, cluster/activation diagnostics, and retention assessment. | @@ -1200,7 +1200,7 @@ v0.2 entry is explicitly confirmed against the closed v0.1 family. Detailed draft: [`v0_1_6_embedded_vector_candidate_recall.md`](../design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md) -Decisions: ADR-I-0023 (embedded exact-scan vector store as the opt-in local mode), ADR-I-0024 (vector candidate recall reports completeness and takes a scope-only query), ADR-I-0025 (the vector record is a read contract), ADR-I-0026 (raw vector baselines read the retrieval trace). +Decisions: ADR-I-0023 (embedded Qdrant Edge vector store as the opt-in local mode), ADR-I-0024 (vector candidate recall reports completeness and takes a scope-only query), ADR-I-0025 (the vector record is a read contract), ADR-I-0026 (raw vector baselines read the retrieval trace). ## Intent @@ -1211,7 +1211,7 @@ Because a second adapter must implement the vector port, this phase also settles ## Goals ```text -add an embedded SQLite exact-scan vector candidate store behind the existing vector port, selected by a store-mode setting with its own path setting +add an embedded vector candidate store on the in-process build of the service backend behind the existing vector port, selected by a store-mode setting with its own path setting and shipped at its exact-scan indexing threshold make the port result carry a typed completeness verdict that the retrieval telemetry records and never repairs reduce the vector payload to its read contract: identity, surface, schema version, and the embedded text as provenance of what was ranked run one shared contract suite against both adapters, with the embedded adapter exercised unconditionally so the default test path needs no service @@ -1224,7 +1224,7 @@ record the re-entry paths for vector-layer predicates a later phase may need: a ```text changing the authority split or any retrieval semantics in the service mode deprecating or altering the service-mode adapter beyond the shared port contract -approximate-nearest-neighbor indexing in the embedded mode +tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold this phase) migration tooling between modes; rebuild from graph authority is the path flipping the default vector mode in this phase multi-process access to the embedded store From 8ecd1e6aeb698ef66c15b9848264eb82031a5d01 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 19:51:40 +0900 Subject: [PATCH 14/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?0:=20empty=20scope=20reports=20the=20not-requested=20verdict;?= =?UTF-8?q?=20service-adapter=20non-goal=20qualified=20by=20the=20shared?= =?UTF-8?q?=20contracts;=20toolchain=20prerequisite=20recorded=20as=20land?= =?UTF-8?q?ed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 ++-- ...all-reports-completeness-and-takes-a-scope-only-query.md | 6 +++--- .../v0_1_6_embedded_vector_candidate_recall.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index e3f5649a..bd040735 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -22,7 +22,7 @@ ## Context (workspace) - Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. - As-built port: `src/ports/vector_candidate.rs`, `src/models/vector/candidate_record.rs`, `src/models/vector/record.rs`, `src/adapters/qdrant/{store,payload}.rs`, `src/policy/embedding_surface.rs`, `src/usecases/retrieve.rs`, `src/api/types/retrieval.rs`, `src/composition.rs`, `src/config/app_settings.rs`, `src/test_support.rs`. -- Prerequisite in this repository: the toolchain pin moves to the embedded engine's minimum (Rust 1.97.0 at decision time) in its own change; Task_4 cannot build before it merges. +- Prerequisite in this repository, landed: the toolchain pin moved to the embedded engine's minimum (Rust 1.97.0) in its own change, merged 2026-09-02 as a88c117. - Prerequisite tracked in the evaluation repository: its evidence-integrity fixes must be merged before this phase cites any harness measurement. - Repo reference docs consulted: the four ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. @@ -240,4 +240,4 @@ Append-only editing rule (applies to both logs below): when appending an entry, ## Notes - Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the latency guidance and the stripped dependency weight must be measured, not assumed; the engine is beta, so its pin is exact and its bump is gated by the canary. -- Edge cases: empty object-type scope selects zero in both adapters; `limit == 0` issues no search and reports the not-requested verdict in both adapters; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality across adapters (both engines normalise cosine internally) is asserted, not assumed. +- Edge cases: an empty object-type scope or `limit == 0` issues no search and reports the not-requested verdict in both adapters; identical-vector tie fixtures must produce Exhaustive versus BoundaryTieClosed, never be encoded as expected parity of the bounded behavior; the parity suite includes non-unit query and record vectors so score equality across adapters (both engines normalise cosine internally) is asserted, not assumed. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index 4478b32c..01720fa9 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -42,20 +42,20 @@ An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing thres ```rust pub enum VectorRecallCompleteness { - NotRequested, // the limit was zero; no search was issued + NotRequested, // the limit was zero or the scope was empty; no search was issued Exhaustive { scanned: usize }, // the scoped population was scored and returned in full (an unindexed shard returned fewer rows than the fetch limit) BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort was closed BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort open } ``` -Adapters own canonicalisation and must state the verdict truthfully: not requested only when the limit is zero and no search was issued (an admitted configuration, so the verdict must be total over it), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and returned, closed only when the cutoff cohort was verified closed or an index returned fewer rows than asked, open at the bound. +Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and returned, closed only when the cutoff cohort was verified closed or an index returned fewer rows than asked, open at the bound. Every adapter answers through the shared tie-closure loop and the canonical constructor; engine ordering of equal-score cohorts is never relied on, because it is not stable across freshly built shards (ADR-I-0023). The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. The query is the embedding, the limit, and an object-type scope, and nothing else. -An empty scope selects zero candidates; wildcard-on-empty is prohibited, matching the graph query rule, and the retrieval context rejects an empty configured object-type set at the boundary. +An empty scope selects zero candidates and reports the not-requested verdict without issuing a search; wildcard-on-empty is prohibited, matching the graph query rule, and the retrieval context rejects an empty configured object-type set at the boundary. Zero-norm vectors are defined on both sides of the port: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure before any adapter sees it (so adapters may normalise at write without a division-by-zero path), and a zero-norm query scores every candidate zero and returns a truthful verdict; the parity suite carries both cases. Three-valued hint predicates are prohibited. Any future vector-layer predicate arrives as an explicit enum whose unknown arm is spelled out, an unknown or missing stored value never satisfies a positive predicate, and the predicate lands with its mapping in both adapters and a parity fixture in the same change. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 8170a6b5..e59b284c 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -28,7 +28,7 @@ This phase fixes the port contract deliberately, because two adapters cannot be Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. -Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero and no search was issued), exhaustive (the scoped population was scored and returned in full), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). +Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (the scoped population was scored and returned in full), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). Both adapters answer through the shared tie-closure loop and the canonical constructor; before this phase the service adapter returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. @@ -51,7 +51,7 @@ This phase ships the threshold at its exact-scan setting (the spike confirmed th Shard: cosine distance at the configured vector size; the five-field payload with a keyword index on object type; the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. -Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero. +Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. Blocking discipline: a scan or an index build is a potentially long synchronous operation behind an async port; the adapter guarantees that neither ever occupies an async executor thread, by invoking the engine from a blocking worker or through the engine's own worker pool where its API is asynchronous, and the in-phase benchmark records executor responsiveness while a scan is in progress. @@ -113,7 +113,7 @@ the port doc comment's "documented bounded-overfetch degradation policy" clause, ```text changing the authority split or any retrieval semantics -deprecating or altering the service adapter +deprecating the service adapter, or altering it beyond what the shared port and record contracts require tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold, tuned by a later measured decision) named-vector coexistence of two embedding spaces (an engine capability this decision was taken for; its use lands with the first embedding-model migration) migration tooling between modes or between record shapes (rebuild-from-graph-authority is the documented path) From 4aeb63a92744623d81e4ad7d55bf22d14a5a3e1a Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 19:57:48 +0900 Subject: [PATCH 15/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?1:=20cross-repository=20coordination=20gate=20for=20exhaustivel?= =?UTF-8?q?y=20converted=20vocabularies;=20fake=20retirement=20opens=20a?= =?UTF-8?q?=20temporary=20shard=20directory;=20ADR-I-0026=20keeps=20only?= =?UTF-8?q?=20the=20library-facing=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 5 +++-- ...R-I-0026-raw-vector-baselines-read-the-retrieval-trace.md | 1 - 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index bd040735..7aa8e9f6 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -169,7 +169,7 @@ - docs/roadmap/development_roadmap.md - depends_on: [Task_4] - description: | - Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened in memory (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; mark the roadmap row finished; move the plan to completed. + Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened on a temporary shard directory, as the phase document specifies, so tests exercise the persistence path (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; mark the roadmap row finished; move the plan to completed. - acceptance: - Zero-hit census for the retired fake and record type. - All five checklist rows cite evidence in the Progress Log. @@ -208,7 +208,8 @@ Task identifiers 5 and 6 were evaluation-repository work and moved to that repository's own plan; identifiers are not reused. -Each wave ends with reviewer approval and a PR per touched repository, merged by the decider before the next wave starts; the evaluation repository's sibling checkout is re-pinned to the merged library commit at every wave boundary. +Each wave ends with reviewer approval and a PR, merged by the decider before the next wave starts; the evaluation repository's sibling checkout is re-pinned to the merged library commit at every wave boundary. +Coordination gate: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository's plan before merge, and its sibling checkout is not re-pinned to the new library commit until its own conversion update has landed; the library wave itself does not wait. ## Rollback / Safety - Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict and the shrunken record, both covered by the parity suite. diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index 8203ef89..66b507b0 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -41,7 +41,6 @@ The evaluation repository's vector-only baseline issues one ordinary `retrieve` The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant declared by the embedding-surface policy; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore a closed or exhaustive surface-level verdict at that limit makes the object-level top-budget determinate, and the contract is pinned by a fixture whose objects carry every surface. Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). -The evaluation repository's vector-service client shrinks to collection lifecycle operations (existence and deletion), which the embedded mode replaces with file operations through the durable-store path list the adapter already maintains. What this record asks of the evaluation repository, recorded as the library-facing contract and nothing more: From 9fd1cf3f6b371445e5ff80e2ad1a0989b9957507 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:07:01 +0900 Subject: [PATCH 16/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?2:=20explicit=20shard=20close=20on=20a=20blocking=20worker;=20m?= =?UTF-8?q?ax=20surfaces=20per=20kind=20published=20as=20public=20policy;?= =?UTF-8?q?=20surface-enum=20consolidation=20ownership;=20schema-version?= =?UTF-8?q?=20exception=20in=20the=20record=20invariant;=20field-count=20c?= =?UTF-8?q?ensus=20reproducible?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 7 ++++++- ...R-I-0023-embedded-qdrant-edge-vector-candidate-store.md | 2 +- .../ADR-I-0025-vector-record-is-a-read-contract.md | 4 ++-- ...I-0026-raw-vector-baselines-read-the-retrieval-trace.md | 2 +- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 7aa8e9f6..4d080038 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -102,12 +102,17 @@ - src/usecases/retrieve.rs - src/usecases/vector_indexing.rs - src/errors.rs + - src/models/vector.rs + - src/models/vector/candidate_record.rs + - src/api/types/retrieval.rs + - src/api/** + - src/lib.rs - docs/design/database/vector_payload_design.md - docs/design/database/schema_cheat_sheet.md - docs/design/database/README.md - depends_on: [Task_2] - description: | - Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Update the database documentation that advertises the old record: the schema cheat sheet and the database README lose the hint fields, the graph URI, and the readable text column, and point at the five-field contract. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Consolidating the surface enum to one domain definition touches its internal definition and re-export and the public copy, all owned here. Publish the maximum number of embedding surfaces per object kind as public policy next to the surface policy that defines it (ADR-I-0026), with a test that the published value matches the builders. Update the database documentation that advertises the old record: the schema cheat sheet and the database README lose the hint fields, the graph URI, and the readable text column, and point at the five-field contract. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. - acceptance: - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository removes its reader under its own plan; its zero-hit census is consumed as closeout evidence, not ordered here). diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 2fa1fd8c..480ca3bc 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -52,7 +52,7 @@ This phase ships the threshold at its exact-scan setting; index construction, qu The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). -Searches and index builds never run on an async executor thread: the adapter invokes the engine from a blocking worker, or through the engine's own worker pool where its API is asynchronous, and the in-phase benchmark records executor responsiveness while a scan or build is in progress. +Searches, index builds, and shard shutdown never run on an async executor thread: the adapter invokes the engine from a blocking worker, or through the engine's own worker pool where its API is asynchronous; because dropping a shard flushes synchronously, the adapter owns an explicit close that performs the final drop on a blocking worker rather than letting the last reference fall on an executor thread; and the in-phase benchmark records executor responsiveness while a scan, a build, and a close are in progress. A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index 64c8b4df..dccc721c 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -22,7 +22,7 @@ supersession_scope: partial ## Context and Problem Statement -ADR-I-0005 decided that the vector payload stores filterable metadata and graph pointers, and the payload design note enumerated thirty-three fields with thirty of them indexed. +ADR-I-0005 decided that the vector payload stores filterable metadata and graph pointers, and the payload design note enumerated thirty-four fields with thirty-one of them indexed; the implemented manifest carried thirty-three with thirty indexed after the record-type field was dropped in the structured-verdict phase. By the time the embedded adapter (ADR-I-0023) was designed, the library read back exactly three of those fields — object id, object type, surface — and the only external reader was the companion evaluation repository's vector-only baseline reading the readable text column. The relationship hints were frozen at upsert and never updated by the link write path; the lifecycle hints described vectors the correction and forgetting paths delete; the readable text column duplicated graph text with a prefix removed; and every field was about to be mirrored into a second physical schema. ADR-I-0002's implementation note said to "persist both `embedding_text` and `content_text` where useful", which left the two text columns' meanings undefined. @@ -87,7 +87,7 @@ Option 5 is the only subset with a forward-looking case that survives the synchr ## Decision Boundary -Invariant: the vector record carries only fields a reader consumes, plus the embedded surface as provenance; readable content is hydrated from graph authority by object id; a returning hint arrives with its predicate and parity fixture through ADR-I-0024's re-entry paths. +Invariant: the vector record carries only fields a reader consumes, plus the embedded surface as provenance and the schema version ADR-I-0007 requires; readable content is hydrated from graph authority by object id; a returning hint arrives with its predicate and parity fixture through ADR-I-0024's re-entry paths. Not covered: the physical encoding of each column per adapter, and graph authority's own denormalised fields. diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index 66b507b0..229031d3 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -38,7 +38,7 @@ The question is what capability the library must expose so the baseline stops re The library exposes no raw candidate-search surface and no facade change. The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. -The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant declared by the embedding-surface policy; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. +The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant the library publishes as public policy per object kind, alongside the surface policy that defines it, so an ordinary caller never duplicates private policy and a new surface changes the published value in the same change; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore a closed or exhaustive surface-level verdict at that limit makes the object-level top-budget determinate, and the contract is pinned by a fixture whose objects carry every surface. Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). From 61dd723645c09a1e8fe6e8062e823a504d7b36dc Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:15:31 +0900 Subject: [PATCH 17/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?3:=20empty-scope=20change=20named=20as=20intended=20wherever=20?= =?UTF-8?q?service-mode=20behavior=20is=20claimed=20unchanged;=20zero-norm?= =?UTF-8?q?=20query=20rule=20and=20shard=20close=20carried=20into=20accept?= =?UTF-8?q?ance=20and=20validation;=20policy-value=20export=20distinguishe?= =?UTF-8?q?d=20from=20the=20prohibited=20search=20facade?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 7 ++++--- ...R-I-0023-embedded-qdrant-edge-vector-candidate-store.md | 2 +- ...I-0026-raw-vector-baselines-read-the-retrieval-trace.md | 4 ++-- .../v0_1_6_embedded_vector_candidate_recall.md | 7 ++++--- docs/roadmap/development_roadmap.md | 2 +- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 4d080038..cd66d853 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -17,7 +17,7 @@ ## Scope / Non-goals - Scope: the phase document's deliverables and deletions, all in this repository. -- Non-goals: the phase document's non-goals (no default flip, no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode). +- Non-goals: the phase document's non-goals (no default flip, no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode for non-empty scopes; ADR-I-0024's empty-scope change, zero candidates for an empty scope and boundary rejection of an empty configured scope, is an intended change and in scope). ## Context (workspace) - Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. @@ -78,6 +78,7 @@ Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; make the service adapter map its fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. - acceptance: - The envelope and enum match ADR-I-0024's Decision section; the canonical-candidates newtype is unchanged. + - Query-side zero-norm rule implemented in the service adapter: a zero-norm query scores every candidate zero and returns a truthful verdict, with a unit test and a parity fixture that Task_4 inherits. - Telemetry carries the verdict for every retrieval; a retrieval test asserts each variant. - Fetch-decision unit tests assert closed and open verdicts including the all-tied cohort at the bound. - Zero-hit census: no match-or-unknown condition, no filter type beyond object-type scope. @@ -149,7 +150,7 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - No scan or index build occupies an async executor thread; the benchmark records executor responsiveness under a concurrent scan. + - No scan, index build, or shard close occupies an async executor thread: the adapter exposes an explicit close that performs the synchronous final drop on a blocking worker; the benchmark records executor responsiveness during a concurrent scan, a build, and a close. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. @@ -217,7 +218,7 @@ Each wave ends with reviewer approval and a PR, merged by the decider before the Coordination gate: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository's plan before merge, and its sibling checkout is not re-pinned to the new library commit until its own conversion update has landed; the library wave itself does not wait. ## Rollback / Safety -- Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict and the shrunken record, both covered by the parity suite. +- Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict, the shrunken record, and the intended empty-scope change (an empty object-type scope selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary), all covered by the parity suite and the retrieval tests. - Stored service-mode payloads with dropped fields remain readable (extra fields tolerated unread); rebuild from graph authority is the recovery path. - Each wave is a separately revertible PR. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 480ca3bc..2201ff9b 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -113,7 +113,7 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t - The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop; above the threshold a recall comparison against the exhaustive setting is recorded. - A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. - The contract canary passes on the pinned engine version and is re-run on every engine bump. -- The benchmark shows no scan or index build occupying an async executor thread. +- The benchmark shows no scan, index build, or shard close occupying an async executor thread; the explicit close is exercised under load and its final drop is observed on a blocking worker. - The default-mode construction test asserts service mode. ## Revisit When diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index 229031d3..dbad4edf 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -44,7 +44,7 @@ Item text comes from the evaluation repository's own ingest records, keyed by th What this record asks of the evaluation repository, recorded as the library-facing contract and nothing more: -- The evaluation repository consumes the retrieval trace and the completeness telemetry as an ordinary caller; the library adds no surface for it. +- The evaluation repository consumes the retrieval trace and the completeness telemetry as an ordinary caller; the library adds no candidate-search surface for it, and the only public addition made for this reading is the published maximum-surfaces-per-object-kind policy value, which is a policy constant, not a query surface. - A raw-vector baseline that wants per-kind top-K uses one singleton-scoped traced retrieval per kind with the multiplied limit and object-level deduplication described above; any other reading of the trace is not covered by the parity claim. - How the evaluation repository migrates its baseline, mirrors telemetry, labels its rows, or guards its cleanup is planned and tracked in that repository. @@ -56,7 +56,7 @@ Keeping the baseline inside the traced retrieval path means the measurement of " ## Implementation Impact -- Library: none beyond ADR-I-0024's telemetry field; the acceptance criterion "no public facade change" holds. +- Library: ADR-I-0024's telemetry field plus one published policy value, the maximum number of embedding surfaces per object kind, exported beside the surface policy that defines it; no candidate-search facade, so the acceptance criterion "no public facade change beyond the telemetry field and this policy value" holds. - Evaluation repository: its baseline moves onto the trace under its own plan; nothing in this repository depends on how. ## Considered Options diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index e59b284c..c243371e 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -112,7 +112,7 @@ the port doc comment's "documented bounded-overfetch degradation policy" clause, ## Non-goals ```text -changing the authority split or any retrieval semantics +changing the authority split, or any retrieval semantics for non-empty scopes (the empty-scope change in ADR-I-0024 is intended and in scope) deprecating the service adapter, or altering it beyond what the shared port and record contracts require tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold, tuned by a later measured decision) named-vector coexistence of two embedding spaces (an engine capability this decision was taken for; its use lands with the first embedding-model migration) @@ -143,13 +143,14 @@ A recall comparison of the embedded adapter above its indexing threshold against Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. -No scan or index build occupies an async executor thread; the benchmark records executor responsiveness under a concurrent scan. +No scan, index build, or shard close occupies an async executor thread; the adapter's explicit close performs the synchronous final drop on a blocking worker, and the benchmark records executor responsiveness during a concurrent scan, a build, and a close. +A zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero with a truthful verdict, both proven in both adapters by parity fixtures. The engine contract canary passes on the pinned version. The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. The default test path requires no vector service; service-gated suites continue to pass unchanged. Both adapters persist exactly the five-field read contract; a census of both repositories shows no reader of a dropped field. Documentation states the single-process expectation, the threshold semantics, the latency guidance, and the rebuild-from-authority path. -No public facade change; no retrieval behaviour change in service mode beyond the added telemetry field. +No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behaviour change in service mode for non-empty scopes (an empty object-type scope now selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary, both intended). ``` ## What the evaluation repository provides and when it is used (ADR-I-0026) diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index fd5004db..4eee535f 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -1241,7 +1241,7 @@ Embedded state survives process restart. Retrieval telemetry reports the completeness verdict for every retrieval in both modes. The default test path requires no vector service; service-gated suites still execute under the service-backed CI job and cannot pass by skipping. The evaluation repository's vector-only baseline produces its rows from the retrieval trace in both modes. -No public facade change beyond the telemetry field; no retrieval behavior change in service mode. +No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behavior change in service mode for non-empty scopes (the intended empty-scope change: zero candidates, and boundary rejection of an empty configured scope). ``` --- From ff92fafa7115799854edc704bde6e1b8aaa5e21d Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:24:03 +0900 Subject: [PATCH 18/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?4:=20verdict=20enum=20wire=20shape=20recorded=20(internally=20t?= =?UTF-8?q?agged,=20snake=5Fcase);=20policy-value=20export=20named=20where?= =?UTF-8?q?=20public=20additions=20are=20summarised;=20roadmap=20row=20say?= =?UTF-8?q?s=20five-field=20record=20with=20provenance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- ...-recall-reports-completeness-and-takes-a-scope-only-query.md | 2 ++ .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 2 +- docs/roadmap/development_roadmap.md | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index 01720fa9..490dfc07 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -41,6 +41,8 @@ An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing thres `search_candidates` returns a result envelope: the canonical candidates (the constructor-owned canonical newtype survives as the field type) together with a typed completeness verdict. ```rust +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(tag = "kind", rename_all = "snake_case")] // internally tagged, snake_case: the wire shape the public telemetry enums already use pub enum VectorRecallCompleteness { NotRequested, // the limit was zero or the scope was empty; no search was issued Exhaustive { scanned: usize }, // the scoped population was scored and returned in full (an unindexed shard returned fewer rows than the fetch limit) diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index c243371e..eb1f3d0c 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -159,7 +159,7 @@ The companion evaluation repository is a development aid; its own work is planne - The library exposes, through an ordinary traced retrieval, everything a raw-vector baseline needs: the vector candidates with scores and the completeness verdict in telemetry; the honest way to use them is one singleton-scoped traced retrieval per measured object kind with a limit of the section budget multiplied by the maximum surfaces per object, deduplicated by object. - The cross-mode comparison (service mode against embedded mode on the continuity suite, identical baselines expected under the parity contract) is the evidence that gates the default flip recorded in ADR-I-0023; it is consumed at the closeout task and by that later decision, not produced by this plan. -- No public facade or configuration surface is added for the evaluation repository; if its measurements ever require one, that is a library decision taken on its own record. +- No candidate-search facade or configuration surface is added for the evaluation repository; the one public addition made for its trace reading is the published maximum-surfaces-per-object-kind policy value (ADR-I-0026), and if its measurements ever require more, that is a library decision taken on its own record. ## Evaluation tie-in diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index 4eee535f..f2885c87 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -259,7 +259,7 @@ Assisted remember workflows may accept raw or semi-raw input as transient proces | v0.1.3 | Remember intake interfaces and deterministic write planning | Finished. Generation-ready write path with `RememberWritePlan`, memory candidates, validation, deterministic helpers, prepare/validate/commit flow, and shared manual/future-generated commit machinery. | | v0.1.4 | Continuity evaluation harness | Finished. Deterministic long-horizon evaluation harness implemented in the public companion `CharacterMemoryEvals` repository as a development aid, not core library functionality: synthetic interaction fixtures, a minimal example assistant loop, continuity-oriented retrieval-quality metrics, selectivity/fanout measurement, and hub-entity stress scenarios. | | v0.1.5 | Eval-driven v0.1 family closeout | Finished. Ran the evaluation harness across the v0.1 family, dispositioned eleven findings (none critical, none open), fixed deterministic vector admission and write-path warning diagnostics in the library, retained the measured defaults with a recorded basis (ADR-I-0022), adopted embedded persistent Oxigraph as the validated default (ADR-I-0021), and expanded the evaluation suite to 33 scenarios including benchmark-adapted and real-embedding fixtures. Closeout report: [`v0_1_5_closeout_report.md`](v0_1_5_closeout_report.md). | -| v0.1.6 | Embedded vector candidate recall | Planned. An embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) behind the vector port as an opt-in local mode, shipped at its exact-scan indexing threshold, so zero-infrastructure local deployments become possible and the default test path needs no external service while the service mode stays the default; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores only the fields a reader consumes; the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | +| v0.1.6 | Embedded vector candidate recall | Planned. An embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) behind the vector port as an opt-in local mode, shipped at its exact-scan indexing threshold, so zero-infrastructure local deployments become possible and the default test path needs no external service while the service mode stays the default; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores a five-field record (identity, surface, schema version, and the embedded text kept as provenance); the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | | v0.2 | Scoped continuity and reflection | `ContinuityScope`, scoped reflection, relationship state between arbitrary entities, character signals for continuing entities, open-loop/commitment lifecycle, and current continuity views. | | v0.3 | Factual rigor, temporal validity, and entity evolution | Assertions, claims, evidence links, belief assessments, source assessment, temporal validity, entity drift handling, and current-belief views. | | v0.4 | Retrieval observability and governance | Retrieval traces, context subgraphs, validation rules, graph health reports, policy diagnostics, rejected expansion traces, cluster/activation diagnostics, and retention assessment. | From 8c847b73af8af9e296705b45e6ff889151d29134 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:32:16 +0900 Subject: [PATCH 19/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?5:=20keyword=20indexes=20on=20object=20id=20and=20object=20type?= =?UTF-8?q?=20(deletion=20stays=20index-backed=20at=20scale);=20lockfile?= =?UTF-8?q?=20and=20graph-schema=20note=20owned=20by=20their=20tasks;=20pa?= =?UTF-8?q?yload=20note's=20third=20design=20rule=20marked=20dated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 6 ++++-- .../ADR-I-0025-vector-record-is-a-read-contract.md | 2 +- docs/design/database/vector_payload_design.md | 2 +- .../v0_1_6_embedded_vector_candidate_recall.md | 2 +- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index cd66d853..7725d96c 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -111,9 +111,10 @@ - docs/design/database/vector_payload_design.md - docs/design/database/schema_cheat_sheet.md - docs/design/database/README.md + - docs/design/database/graph_schema_design.md - depends_on: [Task_2] - description: | - Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Consolidating the surface enum to one domain definition touches its internal definition and re-export and the public copy, all owned here. Publish the maximum number of embedding surfaces per object kind as public policy next to the surface policy that defines it (ADR-I-0026), with a test that the published value matches the builders. Update the database documentation that advertises the old record: the schema cheat sheet and the database README lose the hint fields, the graph URI, and the readable text column, and point at the five-field contract. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. + Shrink the record and the typed manifest to the five fields; drop the hint carriers, the readable text column, the per-field index creation for dropped fields, the test-only field constants and the prose-assertion note constant; replace the service adapter's private enum token mappers and the pipeline's copy with one Display/FromStr per enum in the domain. Also own ADR-I-0024's zero-norm rule on the write side: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure (adding the error-vocabulary variant it needs) before any adapter sees it, with a unit test on the service and a parity-suite fixture that Task_4 inherits, so no adapter ever normalises a zero vector. Consolidating the surface enum to one domain definition touches its internal definition and re-export and the public copy, all owned here. Publish the maximum number of embedding surfaces per object kind as public policy next to the surface policy that defines it (ADR-I-0026), with a test that the published value matches the builders. Update the database documentation that advertises the old record: the schema cheat sheet, the database README, and the graph schema design note's cross-store section lose the hint fields, the graph URI, the lifecycle-hint drift diagnostics, and the readable text column, and point at the five-field contract. Schema version ruling: the stored schema version is retained, because every field the new contract reads is present in records written under the current version and the removal only drops fields no reader consumes; existing stored payloads with extra fields are tolerated unread. A version bump is required only if a later change adds a read field that older records lack (the re-entry paths in ADR-I-0024), and that change owns the bump and its backfill. - acceptance: - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository removes its reader under its own plan; its zero-hit census is consumed as closeout evidence, not ordered here). @@ -138,6 +139,7 @@ - src/config/app_settings.rs - src/errors.rs - Cargo.toml + - Cargo.lock - tests/vector_port_contract_tests.rs - tests/support/** - .env.example @@ -145,7 +147,7 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with a keyword index on object type, the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise; NotRequested at limit zero). The adapter constructs only object payloads and uses only the general shard type; every search and any index build runs on a blocking worker (or the engine's own worker pool where its API is asynchronous), never on an async executor thread. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every search and any index build runs on a blocking worker (or the engine's own worker pool where its API is asynchronous), never on an async executor thread. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index dccc721c..45d829b2 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -81,7 +81,7 @@ Option 5 is the only subset with a forward-looking case that survives the synchr ## Consequences -- Positive: both adapters mirror one five-field manifest; the embedded shard stores those fields as payload beside the vector with a single keyword index on object type (ADR-I-0023 owns the physical layout). +- Positive: both adapters mirror one five-field manifest; the embedded shard stores those fields as payload beside the vector with keyword indexes on object id (the delete selector) and object type (the scope predicate); ADR-I-0023 owns the physical layout. - Positive: the embedded surface is preserved as vector provenance before surfaces become generated. - Negative / tradeoffs: a future scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the re-entry paths make that step predictable. diff --git a/docs/design/database/vector_payload_design.md b/docs/design/database/vector_payload_design.md index e7c25fcf..f964fa46 100644 --- a/docs/design/database/vector_payload_design.md +++ b/docs/design/database/vector_payload_design.md @@ -1,6 +1,6 @@ # Vector Database Payload Design -> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 records the re-entry paths for any returning prefilter column. The Record Shape, Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were (Record Shape still lists the graph URI, which the read contract dropped); the Design Goal, Why Natural-Language Surfaces, and Consistency Model sections remain current. +> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 records the re-entry paths for any returning prefilter column. The Record Shape, Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were (Record Shape still lists the graph URI, which the read contract dropped); the Design Goal (except its third rule, which named relationship and lifecycle hints the read contract dropped), Why Natural-Language Surfaces, and Consistency Model sections remain current. This document describes the Qdrant payload design for Character Memory. It is intentionally a design note, not a field-by-field copy of the Rust mapping code. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index eb1f3d0c..a8eae4a4 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -49,7 +49,7 @@ The adapter runs the in-process build of the service backend (the `qdrant-edge` Exactness is a threshold property: a shard below its configured indexing threshold answers by exhaustive scan, a shard above it answers from its index, and the completeness verdict reports the boundary state either way. This phase ships the threshold at its exact-scan setting (the spike confirmed that a zero threshold leaves a shard unindexed while a threshold of one plus an optimise call builds the index); index construction, quantization, and memory-mapped segments are available capabilities whose defaults are tuned in a later measured decision, never silently. -Shard: cosine distance at the configured vector size; the five-field payload with a keyword index on object type; the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). +Shard: cosine distance at the configured vector size; the five-field payload with keyword indexes on object id (the delete selector removes every surface of an object by it) and on object type (the scope predicate); the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. From 8b456591fa37f427443eb628541c8338227600b4 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:40:46 +0900 Subject: [PATCH 20/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?6:=20dedicated=20blocking=20owner=20holds=20the=20shard=20and?= =?UTF-8?q?=20runs=20every=20engine=20call=20including=20writes=20and=20sh?= =?UTF-8?q?utdown;=20facade=20drop=20reaches=20it=20without=20a=20new=20po?= =?UTF-8?q?rt=20or=20facade=20method;=20count-independent=20live-gate=20va?= =?UTF-8?q?lidation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 6 +++--- ...DR-I-0023-embedded-qdrant-edge-vector-candidate-store.md | 4 ++-- .../v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 7725d96c..d38f3c71 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -54,7 +54,7 @@ - kind: command required: true owner: worker - detail: "service-up cargo test with the switch set: all nine former skip sites execute; service-down with the switch set: the suites fail, not pass" + detail: "service-up cargo test with the switch set: every former skip site executes (census the skip sites before and after; the count is not fixed); service-down with the switch set: the suites fail, not pass" - kind: review required: true owner: reviewer @@ -147,12 +147,12 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every search and any index build runs on a blocking worker (or the engine's own worker pool where its API is asynchronous), never on an async executor thread. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - No scan, index build, or shard close occupies an async executor thread: the adapter exposes an explicit close that performs the synchronous final drop on a blocking worker; the benchmark records executor responsiveness during a concurrent scan, a build, and a close. + - No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; a test drops the facade inside an async runtime and asserts the executor was never blocked; the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 2201ff9b..92a60195 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -52,7 +52,7 @@ This phase ships the threshold at its exact-scan setting; index construction, qu The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). -Searches, index builds, and shard shutdown never run on an async executor thread: the adapter invokes the engine from a blocking worker, or through the engine's own worker pool where its API is asynchronous; because dropping a shard flushes synchronously, the adapter owns an explicit close that performs the final drop on a blocking worker rather than letting the last reference fall on an executor thread; and the in-phase benchmark records executor responsiveness while a scan, a build, and a close are in progress. +No engine call runs on an async executor thread: every synchronous engine operation (upsert and delete, which write the log and update the payload and field indexes; search; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that holds the shard and serialises access to it, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. @@ -113,7 +113,7 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t - The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop; above the threshold a recall comparison against the exhaustive setting is recorded. - A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. - The contract canary passes on the pinned engine version and is re-run on every engine bump. -- The benchmark shows no scan, index build, or shard close occupying an async executor thread; the explicit close is exercised under load and its final drop is observed on a blocking worker. +- The benchmark shows no scan, index build, or shard close occupying an async executor thread; the facade drop is exercised under load inside an async runtime and the shard's final drop is observed on the adapter's blocking owner; a write burst is included in the responsiveness measurement. - The default-mode construction test asserts service mode. ## Revisit When diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index a8eae4a4..7cb31f2b 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -54,7 +54,7 @@ Search: the query runs through the service adapter's tie-closure loop and the ca Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. -Blocking discipline: a scan or an index build is a potentially long synchronous operation behind an async port; the adapter guarantees that neither ever occupies an async executor thread, by invoking the engine from a blocking worker or through the engine's own worker pool where its API is asynchronous, and the in-phase benchmark records executor responsiveness while a scan is in progress. +Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. Score parity across adapters was measured at 0.0 delta on the spike; the parity suite still asserts it with non-unit query and record vectors rather than assuming it. @@ -143,7 +143,7 @@ A recall comparison of the embedded adapter above its indexing threshold against Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. -No scan, index build, or shard close occupies an async executor thread; the adapter's explicit close performs the synchronous final drop on a blocking worker, and the benchmark records executor responsiveness during a concurrent scan, a build, and a close. +No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread; all run on the adapter's dedicated blocking owner, dropping the facade signals that owner so the shard's final drop happens there, and the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. A zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero with a truthful verdict, both proven in both adapters by parity fixtures. The engine contract canary passes on the pinned version. The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. From 8e7f3e07a6bcaf77979859162066d7e68efb51a1 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:49:46 +0900 Subject: [PATCH 21/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?7:=20non-blocking=20drop=20made=20safe=20by=20write-ahead=20dur?= =?UTF-8?q?ability=20and=20a=20lock-aware=20constructor,=20proven=20by=20c?= =?UTF-8?q?lose-then-reopen;=20parity=20criterion=20bounded=20by=20indexin?= =?UTF-8?q?g=20thresholds;=20evaluation=20work=20described=20as=20consumed?= =?UTF-8?q?=20evidence;=20cross-repo=20conversion=20as=20a=20re-pin=20prer?= =?UTF-8?q?equisite?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 ++-- ...DR-I-0023-embedded-qdrant-edge-vector-candidate-store.md | 6 ++++-- .../v0_1_6_embedded_vector_candidate_recall.md | 4 +++- docs/roadmap/development_roadmap.md | 2 +- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index d38f3c71..5570a1fa 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -6,7 +6,7 @@ - work_type: mixed ## Goal -- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0026: a redesigned vector port contract, a five-field vector record, an embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) as the opt-in local mode, a shared contract suite over both adapters, and the evaluation repository's vector-only baseline moved onto the retrieval trace. +- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0026: a redesigned vector port contract, a five-field vector record, an embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) as the opt-in local mode, and a shared contract suite over both adapters; the evaluation repository's move of its vector-only baseline onto the retrieval trace is planned there and consumed here as closeout evidence. ## Definition of Done - Every acceptance criterion in the phase document's "Acceptance criteria" section holds with recorded evidence. @@ -152,7 +152,7 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; a test drops the facade inside an async runtime and asserts the executor was never blocked; the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. + - No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; durability does not depend on that final drop (writes are in the engine's write-ahead log before their call returns) and a constructor meeting a directory still locked by a closing owner waits with a bounded backoff; a close-then-reopen test drops the facade inside an async runtime, asserts the executor was never blocked, reopens the same directory immediately, and finds every write; the contract canary pins both engine facts; the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 92a60195..088b3755 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -52,7 +52,9 @@ This phase ships the threshold at its exact-scan setting; index construction, qu The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). -No engine call runs on an async executor thread: every synchronous engine operation (upsert and delete, which write the log and update the payload and field indexes; search; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that holds the shard and serialises access to it, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. +No engine call runs on an async executor thread: every synchronous engine operation (upsert and delete, which write the log and update the payload and field indexes; search; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that holds the shard and serialises access to it, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added. +Two facts make that non-blocking drop safe, and both are pinned by the contract canary: every write is durable in the engine's write-ahead log before the call returns, so the final flush on drop is compaction rather than persistence and a process exit that pre-empts it loses nothing; and a shard directory is locked while its owner holds it, so the constructor, on encountering a locked directory, waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. +A close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. @@ -65,7 +67,7 @@ A reopened shard is validated against the configured embedding model (vector siz - A new adapter module implementing the vector candidate port on the embedded engine; the composition root gains a mode switch mirroring the statistics-store switch. - The library's toolchain pin moves to the minimum the engine compiles on (Rust 1.97 at decision time; the engine rejected the previous 1.95 pin). - The settings type gains the mode and path keys; the service connection string becomes optional and is validated as present only in service mode. -- The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion is updated in the same wave. +- The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion, planned in that repository, is a prerequisite to re-pinning its checkout to this wave's merge — not work of this wave. - The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured; the deterministic test fake is retired in favour of the embedded adapter opened on a temporary directory, which removes the vector-service dependency from the default test path. - Dependency weight is a recorded deliverable: the unstripped release delta is measured; the stripped delta and the effect of feature trimming are measured and recorded before closeout. - Documentation states the single-process expectation, the threshold semantics, the measured latency guidance, and rebuild-from-graph-authority as the path between modes. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 7cb31f2b..06923d2b 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -54,7 +54,9 @@ Search: the query runs through the service adapter's tie-closure loop and the ca Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. -Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. +Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. +That non-blocking drop is safe because every write is durable in the engine's write-ahead log before the call returns (the final flush on drop is compaction, not persistence, so a process exit that pre-empts it loses nothing) and because a shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it; both facts are pinned by the contract canary and proven by a close-then-reopen test that drops the facade inside an async runtime, reopens the same directory immediately, and finds every write. +The in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. Score parity across adapters was measured at 0.0 delta on the spike; the parity suite still asserts it with non-unit query and record vectors rather than assuming it. diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index f2885c87..b72c43cf 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -1235,7 +1235,7 @@ a public candidate-search facade ```text Embedded mode constructs and serves retrieval without any running service. -The shared contract suite produces identical admitted candidate sets from both adapters across the port contract. +The shared contract suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds; above them a recall comparison is recorded. Deterministic admission holds in embedded mode; repeated runs are byte-identical. Embedded state survives process restart. Retrieval telemetry reports the completeness verdict for every retrieval in both modes. From e838bdd24b673c663b6c10da069b1f7aa443b0cc Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 20:57:04 +0900 Subject: [PATCH 22/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?8:=20exhaustive=20verdict=20means=20full=20scoring=20and=20a=20?= =?UTF-8?q?determinate=20top-K,=20never=20a=20full=20return;=20package=20v?= =?UTF-8?q?ersion=20bump=20owned=20by=20closeout;=20library=20plan=20keeps?= =?UTF-8?q?=20only=20a=20notification=20duty=20toward=20the=20evaluation?= =?UTF-8?q?=20repository?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 8 +++++--- ...l-reports-completeness-and-takes-a-scope-only-query.md | 4 ++-- .../v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 5570a1fa..a139bc8a 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -175,9 +175,11 @@ - docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md - docs/coding-agent/lessons.md - docs/roadmap/development_roadmap.md + - Cargo.toml + - Cargo.lock - depends_on: [Task_4] - description: | - Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened on a temporary shard directory, as the phase document specifies, so tests exercise the persistence path (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; mark the roadmap row finished; move the plan to completed. + Retire the deterministic vector fake and its embedding-bearing record type in favour of the embedded adapter opened on a temporary shard directory, as the phase document specifies, so tests exercise the persistence path (failure-injecting and recording fakes stay); collect the deferral-reconfirmation evidence for all five checklist rows; bump the package version to 0.1.6 in the manifest and lockfile as prior milestone closeouts did; mark the roadmap row finished; move the plan to completed. - acceptance: - Zero-hit census for the retired fake and record type. - All five checklist rows cite evidence in the Progress Log. @@ -216,8 +218,8 @@ Task identifiers 5 and 6 were evaluation-repository work and moved to that repository's own plan; identifiers are not reused. -Each wave ends with reviewer approval and a PR, merged by the decider before the next wave starts; the evaluation repository's sibling checkout is re-pinned to the merged library commit at every wave boundary. -Coordination gate: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository's plan before merge, and its sibling checkout is not re-pinned to the new library commit until its own conversion update has landed; the library wave itself does not wait. +Each wave ends with reviewer approval and a PR, merged by the decider before the next wave starts. +Notification duty: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository before merge; how and when that repository adopts the change is planned there, and this plan only consumes the resulting compatibility evidence at closeout. ## Rollback / Safety - Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict, the shrunken record, and the intended empty-scope change (an empty object-type scope selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary), all covered by the parity suite and the retrieval tests. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index 490dfc07..d04f8a3f 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -45,13 +45,13 @@ An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing thres #[serde(tag = "kind", rename_all = "snake_case")] // internally tagged, snake_case: the wire shape the public telemetry enums already use pub enum VectorRecallCompleteness { NotRequested, // the limit was zero or the scope was empty; no search was issued - Exhaustive { scanned: usize }, // the scoped population was scored and returned in full (an unindexed shard returned fewer rows than the fetch limit) + Exhaustive { scanned: usize }, // every stored record in scope was scored, so the requested top-K is determinate (an unindexed shard returned fewer rows than it asked for; the envelope is still capped at the limit) BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort was closed BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort open } ``` -Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and returned, closed only when the cutoff cohort was verified closed or an index returned fewer rows than asked, open at the bound. +Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and the requested top-K is determinate while the envelope stays capped at the limit, closed only when the cutoff cohort was verified closed or an index returned fewer rows than asked, open at the bound. Every adapter answers through the shared tie-closure loop and the canonical constructor; engine ordering of equal-score cohorts is never relied on, because it is not stable across freshly built shards (ADR-I-0023). The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 06923d2b..c9902bce 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -28,7 +28,7 @@ This phase fixes the port contract deliberately, because two adapters cannot be Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. -Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (the scoped population was scored and returned in full), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). +Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (every scoped record was scored, so the requested top-K is determinate), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). Both adapters answer through the shared tie-closure loop and the canonical constructor; before this phase the service adapter returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. @@ -51,7 +51,7 @@ This phase ships the threshold at its exact-scan setting (the spike confirmed th Shard: cosine distance at the configured vector size; the five-field payload with keyword indexes on object id (the delete selector removes every surface of an object by it) and on object type (the scope predicate); the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. -Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population came back), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. +Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population was scored, so the requested top-K is determinate; the envelope stays capped at the limit), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. From 5e7dee8f986966aa07079adc6a3d85f77d804eb7 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 21:04:14 +0900 Subject: [PATCH 23/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=201?= =?UTF-8?q?9:=20boundary=20verdicts=20describe=20the=20index's=20returned?= =?UTF-8?q?=20prefix,=20not=20global=20recall;=20only=20the=20exhaustive?= =?UTF-8?q?=20verdict=20asserts=20population-level=20determinacy;=20no=20e?= =?UTF-8?q?xact-search=20requirement=20on=20the=20service=20adapter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 2 +- ...l-reports-completeness-and-takes-a-scope-only-query.md | 8 +++++--- .../v0_1_6_embedded_vector_candidate_recall.md | 7 ++++--- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index a139bc8a..13d7be09 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -147,7 +147,7 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise, describing the closure of the cutoff cohort within the index's returned prefix and never global recall; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index d04f8a3f..456effb8 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -46,12 +46,14 @@ An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing thres pub enum VectorRecallCompleteness { NotRequested, // the limit was zero or the scope was empty; no search was issued Exhaustive { scanned: usize }, // every stored record in scope was scored, so the requested top-K is determinate (an unindexed shard returned fewer rows than it asked for; the envelope is still capped at the limit) - BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort was closed - BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort open + BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort WITHIN THAT PREFIX was closed: determinate relative to the index's answer, which an approximate index may have built without records it never surfaced + BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort within the prefix still open } ``` -Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and the requested top-K is determinate while the envelope stays capped at the limit, closed only when the cutoff cohort was verified closed or an index returned fewer rows than asked, open at the bound. +Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and the requested top-K is determinate while the envelope stays capped at the limit, closed only when the cutoff cohort within the index's returned prefix was verified closed or the index returned fewer rows than asked, open at the bound. +The two boundary verdicts describe the index's answer, not global recall: an approximate index may omit equal- or higher-scored records it never surfaced, and no amount of overfetch can prove their absence, so only the exhaustive verdict asserts determinate top-K membership over the scoped population; the boundary verdicts assert determinism of the returned set for a given index state (the same index answers the same query the same way) and whether its cutoff cohort was closed. +Callers that need population-level determinacy configure the adapter below its indexing threshold (ADR-I-0023) and read the exhaustive verdict; nothing in this record requires the service adapter to switch to exact search. Every adapter answers through the shared tie-closure loop and the canonical constructor; engine ordering of equal-score cohorts is never relied on, because it is not stable across freshly built shards (ADR-I-0023). The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index c9902bce..e7338e96 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -28,8 +28,9 @@ This phase fixes the port contract deliberately, because two adapters cannot be Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. -Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (every scoped record was scored, so the requested top-K is determinate), boundary tie closed (an index returned a prefix and the cutoff cohort was verified closed), or boundary tie open (the overfetch bound was reached with the cohort open). -Both adapters answer through the shared tie-closure loop and the canonical constructor; before this phase the service adapter returned the truncated set silently at its bound, and the port's bare list type could not say whether top-K membership was determinate. +Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (every scoped record was scored, so the requested top-K is determinate), boundary tie closed (an index returned a prefix and the cutoff cohort within that prefix was verified closed), or boundary tie open (the overfetch bound was reached with the cohort within the prefix still open). +The boundary verdicts describe the index's answer, not global recall: an approximate index may omit records it never surfaced, so only the exhaustive verdict asserts determinate membership over the scoped population, while the boundary verdicts assert that the returned set is deterministic for a given index state and whether its cutoff cohort was closed. +Both adapters answer through the shared tie-closure loop and the canonical constructor; before this phase the service adapter returned the truncated set silently at its bound, and the port's bare list type could not say whether even the returned prefix's cutoff was closed. The retrieval pipeline records the verdict in telemetry beside the returned candidate count and never repairs, retries, or fails on it, because candidate recall is non-authoritative. The canonical-candidates newtype introduced for deterministic admission survives as the envelope's candidates field; canonical ordering (score, object-type rank, object id, surface rank) is unchanged and applies identically to both adapters. @@ -51,7 +52,7 @@ This phase ships the threshold at its exact-scan setting (the spike confirmed th Shard: cosine distance at the configured vector size; the five-field payload with keyword indexes on object id (the delete selector removes every surface of an object by it) and on object type (the scope predicate); the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. -Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population was scored, so the requested top-K is determinate; the envelope stays capped at the limit), boundary tie closed or open from the loop otherwise, not requested at limit zero or empty scope. +Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population was scored, so the requested top-K is determinate; the envelope stays capped at the limit), boundary tie closed or open from the loop otherwise (a statement about the index's returned prefix, not about global recall), not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. From 6dc7b65630a8a2dfa680f254d9b4737eddf0f606 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 21:11:47 +0900 Subject: [PATCH 24/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?0:=20evaluation-facing=20wording=20distinguishes=20exhaustive?= =?UTF-8?q?=20determinacy=20from=20prefix=20closure;=20valid=20clippy=20in?= =?UTF-8?q?vocation=20in=20Task=5F3;=20roadmap=20non-goal=20scoped=20to=20?= =?UTF-8?q?non-empty=20scopes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 2 +- ...DR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md | 4 ++-- .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 2 +- docs/roadmap/development_roadmap.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 13d7be09..8aff6318 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -124,7 +124,7 @@ - kind: command required: true owner: worker - detail: "cargo fmt --check; clippy -D warnings; service-up cargo test; ignored qdrant_ lib tests; census commands recorded" + detail: "cargo fmt --check; cargo clippy --all-targets -- -D warnings; service-up cargo test; ignored qdrant_ lib tests; census commands recorded" - kind: review required: true owner: reviewer diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index dbad4edf..d8080f69 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -31,7 +31,7 @@ The question is what capability the library must expose so the baseline stops re - Evaluation tooling must not grow library surface that no product use case has demanded (ADR-I-0020's driver). - The library is not a vector-database abstraction, and vector-only candidates must never become behavior-influencing memory without graph verification (project philosophy; the persistent-graph-authority phase's acceptance criteria). -- The retrieval trace already is the raw vector recall: the canonical, pre-verification top-K with object reference, surface, score, and rank, scoped by the configured object types and sized by the candidate limit, and ADR-I-0024 adds the completeness verdict that says whether that top-K was determinate. +- The retrieval trace already is the raw vector recall: the canonical, pre-verification top-K with object reference, surface, score, and rank, scoped by the configured object types and sized by the candidate limit, and ADR-I-0024 adds the completeness verdict, which says either that the scoped population was scored exhaustively (top-K determinate over the population) or, for an indexed answer, only whether the cutoff cohort within the index's returned prefix was closed. - Every store's physical schema is adapter-private; two adapters must not create two baseline implementations. ## Decision @@ -39,7 +39,7 @@ The question is what capability the library must expose so the baseline stops re The library exposes no raw candidate-search surface and no facade change. The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant the library publishes as public policy per object kind, alongside the surface policy that defines it, so an ordinary caller never duplicates private policy and a new surface changes the published value in the same change; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. -That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore a closed or exhaustive surface-level verdict at that limit makes the object-level top-budget determinate, and the contract is pinned by a fixture whose objects carry every surface. +That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore an exhaustive surface-level verdict at that limit makes the object-level top-budget determinate over the scoped population, while a closed boundary verdict makes it determinate only relative to the index's returned prefix (an approximate index may have omitted records before tie closure), and the baseline records which of the two it obtained; the contract is pinned by a fixture whose objects carry every surface. Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). What this record asks of the evaluation repository, recorded as the library-facing contract and nothing more: diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index e7338e96..8c86fbe9 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -191,7 +191,7 @@ Each item was parked on this phase by the structured-verdict phase; each row sta Evidence: zero-hit census for the filter type and for empty-or-null match conditions in the service adapter; the prohibition and re-entry paths are recorded in ADR-I-0024. 5. Evaluation baseline capability. Parked claim: the baseline re-implements a hidden raw-vector capability against the payload schema. - Re-verified: one singleton-scoped traced retrieval per measured kind reproduces the direct per-kind search exactly, which a sliced mixed-kind top-K would not; each retrieval's completeness verdict reports whether that kind's top-K was determinate; the evaluation adapter can hold item text from ingest. + Re-verified: one singleton-scoped traced retrieval per measured kind reproduces the direct per-kind search exactly, which a sliced mixed-kind top-K would not; each retrieval's completeness verdict reports whether that kind's top-K was determinate over the scoped population (exhaustive) or only closed within the index's returned prefix (boundary verdicts), and the baseline records which; the evaluation adapter can hold item text from ingest. Evidence: the A/B run with row-level diff of item identities and ranks; after the switch, zero-hit census for vector-service search calls and payload constants in the evaluation adapter. ## Decisions (the draft's open questions, resolved 2026-09-02) diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index b72c43cf..e7d17d22 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -1222,7 +1222,7 @@ record the re-entry paths for vector-layer predicates a later phase may need: a ## Non-goals ```text -changing the authority split or any retrieval semantics in the service mode +changing the authority split, or any retrieval semantics in the service mode for non-empty scopes (the intended empty-scope change is in scope) deprecating or altering the service-mode adapter beyond the shared port contract tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold this phase) migration tooling between modes; rebuild from graph authority is the path From 9fac10ecd525b245e67b3e9666ab1c12e66d850a Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 21:19:21 +0900 Subject: [PATCH 25/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?1:=20exhaustive=20verdict=20requires=20an=20unindexed=20shard?= =?UTF-8?q?=20plus=20a=20closed=20cutoff=20cohort=20with=20scanned=20taken?= =?UTF-8?q?=20from=20a=20filtered=20scope=20count,=20so=20every=20exact=20?= =?UTF-8?q?scan=20is=20classified=20truthfully?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 2 +- ...ecall-reports-completeness-and-takes-a-scope-only-query.md | 4 ++-- .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 8aff6318..63b92b15 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -147,7 +147,7 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit; BoundaryTieClosed / BoundaryTieOpen otherwise, describing the closure of the cutoff cohort within the index's returned prefix and never global recall; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed per the adapter's own threshold configuration and the loop closed the cutoff cohort, with `scanned` from a filtered count of the scope, never from returned rows; BoundaryTieClosed when an indexed shard's cutoff cohort closed, describing the index's returned prefix and never global recall; BoundaryTieOpen whenever the bound is reached with the cohort open, on an exact scan too; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md index 456effb8..6de3c8d3 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md @@ -45,13 +45,13 @@ An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing thres #[serde(tag = "kind", rename_all = "snake_case")] // internally tagged, snake_case: the wire shape the public telemetry enums already use pub enum VectorRecallCompleteness { NotRequested, // the limit was zero or the scope was empty; no search was issued - Exhaustive { scanned: usize }, // every stored record in scope was scored, so the requested top-K is determinate (an unindexed shard returned fewer rows than it asked for; the envelope is still capped at the limit) + Exhaustive { scanned: usize }, // every stored record in scope was scored, so the requested top-K is determinate (the adapter knows the shard is unindexed and the cutoff cohort closed; scanned is the filtered scope count, not the rows returned; the envelope is still capped at the limit) BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort WITHIN THAT PREFIX was closed: determinate relative to the index's answer, which an approximate index may have built without records it never surfaced BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort within the prefix still open } ``` -Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed and the scan returned fewer rows than it asked for, so the whole scoped population was scored and the requested top-K is determinate while the envelope stays capped at the limit, closed only when the cutoff cohort within the index's returned prefix was verified closed or the index returned fewer rows than asked, open at the bound. +Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed (its own threshold configuration, pinned by the canary) so every scoped record was scored, AND the cutoff cohort was closed by the shared loop, with `scanned` taken from a filtered count of the scope rather than from the rows returned; the requested top-K is then determinate over the population while the envelope stays capped at the limit, and an exhaustive scan whose cutoff cohort stays open at the bound reports open, not exhaustive; closed only when the cutoff cohort within the index's returned prefix was verified closed or the index returned fewer rows than asked, open at the bound. The two boundary verdicts describe the index's answer, not global recall: an approximate index may omit equal- or higher-scored records it never surfaced, and no amount of overfetch can prove their absence, so only the exhaustive verdict asserts determinate top-K membership over the scoped population; the boundary verdicts assert determinism of the returned set for a given index state (the same index answers the same query the same way) and whether its cutoff cohort was closed. Callers that need population-level determinacy configure the adapter below its indexing threshold (ADR-I-0023) and read the exhaustive verdict; nothing in this record requires the service adapter to switch to exact search. Every adapter answers through the shared tie-closure loop and the canonical constructor; engine ordering of equal-score cohorts is never relied on, because it is not stable across freshly built shards (ADR-I-0023). diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 8c86fbe9..995d7292 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -52,7 +52,7 @@ This phase ships the threshold at its exact-scan setting (the spike confirmed th Shard: cosine distance at the configured vector size; the five-field payload with keyword indexes on object id (the delete selector removes every surface of an object by it) and on object type (the scope predicate); the object-type scope expressed as a filter; the general shard type only (the update-only shard type carries an unimplemented path); object payloads only (the engine's point constructor panics on any other JSON shape). Search: the query runs through the service adapter's tie-closure loop and the canonical constructor, because the spike found identical-vector cohorts stable within a shard and across reopen but not across freshly built shards — deterministic admission comes from closing the cutoff cohort and ordering it canonically, never from engine order. -Verdict mapping: exhaustive when the shard is unindexed and the scan returned fewer rows than the fetch limit (the whole scoped population was scored, so the requested top-K is determinate; the envelope stays capped at the limit), boundary tie closed or open from the loop otherwise (a statement about the index's returned prefix, not about global recall), not requested at limit zero or empty scope. +Verdict mapping: exhaustive when the shard is unindexed (known from the adapter's own threshold configuration) and the shared loop closed the cutoff cohort, with the scanned count taken from a filtered count of the scope rather than from the rows returned, so every exact scan with a closed cutoff reports exhaustive regardless of population size; boundary tie closed when an indexed shard's cutoff cohort closed (a statement about the index's returned prefix, not about global recall); boundary tie open whenever the bound is reached with the cohort still open, on an exact scan too; not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. From ee2d48f2d71f4a094af3006e359d6523906965bc Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 22:34:10 +0900 Subject: [PATCH 26/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?2:=20the=20embedded=20adapter=20flushes=20after=20every=20write?= =?UTF-8?q?=20and=20acknowledges=20only=20then,=20because=20the=20pinned?= =?UTF-8?q?=20engine=20persists=20only=20on=20flush=20and=20does=20not=20r?= =?UTF-8?q?eplay=20its=20log=20on=20load=20(measured:=20writers=20that=20s?= =?UTF-8?q?kipped=20the=20drop=20reopened=20empty);=20hard-exit=20test=20j?= =?UTF-8?q?oins=20Task=5F4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 6 +++++- ...I-0023-embedded-qdrant-edge-vector-candidate-store.md | 9 ++++++--- .../v0_1_6_embedded_vector_candidate_recall.md | 6 +++--- 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 63b92b15..8b5185a4 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -152,7 +152,7 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; durability does not depend on that final drop (writes are in the engine's write-ahead log before their call returns) and a constructor meeting a directory still locked by a closing owner waits with a bounded backoff; a close-then-reopen test drops the facade inside an async runtime, asserts the executor was never blocked, reopens the same directory immediately, and finds every write; the contract canary pins both engine facts; the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. + - No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; the owner flushes after every upsert or delete and acknowledges only then, so durability of acknowledged writes does not depend on that final drop (the engine persists only on flush and does not replay its log on load), and a constructor meeting a directory still locked by a closing owner waits with a bounded backoff; a close-then-reopen test drops the facade inside an async runtime, asserts the executor was never blocked, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens from a second process, and finds every acknowledged write; the contract canary pins both engine facts (no log replay on load; directory lock); the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. @@ -248,6 +248,10 @@ Append-only editing rule (applies to both logs below): when appending an entry, - Plan delta: Task_4 rewritten for the in-process engine (shard directory per collection, threshold-based exactness, shared tie-closure loop, contract canary, dependency-weight report); the toolchain pin moved to 1.97.0 as a prerequisite; approximate indexing leaves the non-goals; named-vector coexistence and shard-to-server sync are recorded as available but not exercised this phase. - Tradeoffs considered: recorded in ADR-I-0023's rejected alternatives (in-house scan; sqlite-vec, lighter and deterministic but exhaustive-only in its stable release with a single-maintainer approximate-index future; LanceDB; in-memory only; default flip now); a scale probe was offered and declined as not worth the effort, so the interactive-latency assumption at decade scale is a recorded revisit trigger rather than a measurement. - User approval: yes, 2026-09-02. +- 2026-09-02 — Durability rule for the embedded adapter (review round 22). + - Finding: the engine persists a write only on flush and does not replay its log on load; process-level probes that skipped the shard drop reopened with zero of two hundred points. + - Ruling: the blocking owner flushes after every write and acknowledges only then; the signal-only facade drop stays; no port or facade method is added; a hard-exit test joins Task_4's acceptance. + - Tradeoff: one synchronous disk sync per write, measured by the write burst in the benchmark; batching behind an acknowledgement is the named upgrade. ## Notes - Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the latency guidance and the stripped dependency weight must be measured, not assumed; the engine is beta, so its pin is exact and its bump is gated by the canary. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 088b3755..545a3b41 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -52,9 +52,9 @@ This phase ships the threshold at its exact-scan setting; index construction, qu The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). -No engine call runs on an async executor thread: every synchronous engine operation (upsert and delete, which write the log and update the payload and field indexes; search; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that holds the shard and serialises access to it, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added. -Two facts make that non-blocking drop safe, and both are pinned by the contract canary: every write is durable in the engine's write-ahead log before the call returns, so the final flush on drop is compaction rather than persistence and a process exit that pre-empts it loses nothing; and a shard directory is locked while its owner holds it, so the constructor, on encountering a locked directory, waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. -A close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. +No engine call runs on an async executor thread: every synchronous engine operation (upsert and delete, which write the log and update the payload and field indexes; search; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that holds the shard, serialises access to it, and acknowledges an upsert or delete only after the engine's flush has completed, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added. +Two facts make that non-blocking drop safe, and both are pinned by the contract canary: the engine persists a write only when its flush runs: the flush on shard drop is the persistence step, not compaction, and a load does not replay the write-ahead log, so a writer that skips the final drop reopens with none of its unflushed writes (measured on the pinned version: two process-level probes, one skipping the drop and one exiting the process, reopened with zero of two hundred points; the normal-drop control reopened with all of them), so the owner flushes after every write and acknowledges only then, which makes every acknowledged write durable independently of the final drop and means a process exit that pre-empts that drop loses nothing acknowledged; and a shard directory is locked while its owner holds it, so the constructor, on encountering a locked directory, waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. +A close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. @@ -102,6 +102,7 @@ Option 6 contradicts the defaults-match-evidence rule; it is reopened by the evi - Negative / tradeoffs: the engine is beta and its API may change; the canary test and the pinned version turn that into a build-time failure rather than a runtime one. - Negative / tradeoffs: about 30.5 MB of unstripped binary and a higher toolchain floor; the weight deliverable exists to establish the real number. - Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. +- Negative / tradeoffs: a flush after every write costs a synchronous disk sync per upsert or delete on the owner's thread; the write burst in the in-phase benchmark measures it, and batching acknowledged flushes is the named upgrade if it matters. ## Decision Boundary @@ -115,6 +116,7 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t - The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop; above the threshold a recall comparison against the exhaustive setting is recorded. - A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. - The contract canary passes on the pinned engine version and is re-run on every engine bump. +- The hard-exit test finds every acknowledged write after a process exit that skipped the shard drop, and the contract canary fails if the pinned engine ever starts replaying its log on load (the flush-per-write rule then becomes revisitable, not wrong). - The benchmark shows no scan, index build, or shard close occupying an async executor thread; the facade drop is exercised under load inside an async runtime and the shard's final drop is observed on the adapter's blocking owner; a write burst is included in the responsiveness measurement. - The default-mode construction test asserts service mode. @@ -125,6 +127,7 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t - A corpus benchmark contradicts the interactive-latency assumption behind the decade standard (exhaustive scan acceptable far beyond the assumed scale, or the index insufficient at it) — revisit the threshold default and the tuning decision. - The evaluation suite has run every dataset in embedded mode with results identical to service mode and one corpus at the guidance size — reopen the default mode. - A multi-replica deployment shape is designed (the remote graph-authority phase ADR-I-0021 anticipates) — the single-process expectation is reconsidered together with the graph and statistics stores, never alone. +- The write-burst measurement shows the per-write flush dominating ingestion cost — batch flushes behind an explicit acknowledgement rather than weakening the durability rule. ## Consultation impact diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 995d7292..bbee6d22 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -55,8 +55,8 @@ Search: the query runs through the service adapter's tie-closure loop and the ca Verdict mapping: exhaustive when the shard is unindexed (known from the adapter's own threshold configuration) and the shared loop closed the cutoff cohort, with the scanned count taken from a filtered count of the scope rather than from the rows returned, so every exact scan with a closed cutoff reports exhaustive regardless of population size; boundary tie closed when an indexed shard's cutoff cohort closed (a statement about the index's returned prefix, not about global recall); boundary tie open whenever the bound is reached with the cohort still open, on an exact scan too; not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. -Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard and serialises access, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. -That non-blocking drop is safe because every write is durable in the engine's write-ahead log before the call returns (the final flush on drop is compaction, not persistence, so a process exit that pre-empts it loses nothing) and because a shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it; both facts are pinned by the contract canary and proven by a close-then-reopen test that drops the facade inside an async runtime, reopens the same directory immediately, and finds every write. +Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard, serialises access, and acknowledges an upsert or delete only after the engine's flush has completed, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. +That non-blocking drop is safe because the owner flushes after every write and acknowledges only then (the engine persists a write only when its flush runs: the flush on shard drop is the persistence step, not compaction, and a load does not replay the write-ahead log, so a writer that skips the final drop reopens with none of its unflushed writes (measured on the pinned version: two process-level probes, one skipping the drop and one exiting the process, reopened with zero of two hundred points; the normal-drop control reopened with all of them)), so every acknowledged write is durable independently of the final drop and a process exit that pre-empts it loses nothing acknowledged, and because a shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it; both facts are pinned by the contract canary and proven by a close-then-reopen test that drops the facade inside an async runtime, reopens the same directory immediately, and finds every write, and by a hard-exit test that exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write. The in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. @@ -146,7 +146,7 @@ A recall comparison of the embedded adapter above its indexing threshold against Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. -No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread; all run on the adapter's dedicated blocking owner, dropping the facade signals that owner so the shard's final drop happens there, and the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. +No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread; all run on the adapter's dedicated blocking owner, dropping the facade signals that owner so the shard's final drop happens there; the owner acknowledges a write only after the engine's flush, the hard-exit test finds every acknowledged write after a process exit that skipped the drop, and the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. A zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero with a truthful verdict, both proven in both adapters by parity fixtures. The engine contract canary passes on the pinned version. The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. From ac81ce1f8e66b5d46c4d4cf6a8dcea1c0199f0f4 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Wed, 2 Sep 2026 22:41:57 +0900 Subject: [PATCH 27/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?3:=20shard=20open/load=20with=20lock=20backoff,=20payload=20ind?= =?UTF-8?q?ex=20creation,=20and=20the=20filtered=20scope=20count=20also=20?= =?UTF-8?q?run=20on=20the=20blocking=20owner,=20which=20opens=20the=20shar?= =?UTF-8?q?d=20itself;=20construction=20and=20reopen=20join=20the=20respon?= =?UTF-8?q?siveness=20measurement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 2 +- .../ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md | 4 ++-- .../roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 8b5185a4..4c6f2ae7 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -152,7 +152,7 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; the owner flushes after every upsert or delete and acknowledges only then, so durability of acknowledged writes does not depend on that final drop (the engine persists only on flush and does not replay its log on load), and a constructor meeting a directory still locked by a closing owner waits with a bounded backoff; a close-then-reopen test drops the facade inside an async runtime, asserts the executor was never blocked, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens from a second process, and finds every acknowledged write; the contract canary pins both engine facts (no log replay on load; directory lock); the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. + - No engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count behind the exhaustive verdict, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, which opens the shard itself so the async composition entry point never calls the engine directly, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; the owner flushes after every upsert or delete and acknowledges only then, so durability of acknowledged writes does not depend on that final drop (the engine persists only on flush and does not replay its log on load), and a constructor meeting a directory still locked by a closing owner waits with a bounded backoff; a close-then-reopen test drops the facade inside an async runtime, asserts the executor was never blocked, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens from a second process, and finds every acknowledged write; the contract canary pins both engine facts (no log replay on load; directory lock); the benchmark records executor responsiveness during construction and reopen (including a lock-backoff wait), a concurrent scan, a write burst, a build, and a close. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 545a3b41..fcade98e 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -52,7 +52,7 @@ This phase ships the threshold at its exact-scan setting; index construction, qu The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). -No engine call runs on an async executor thread: every synchronous engine operation (upsert and delete, which write the log and update the payload and field indexes; search; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that holds the shard, serialises access to it, and acknowledges an upsert or delete only after the engine's flush has completed, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added. +No engine call runs on an async executor thread: every synchronous engine operation (shard open and load, including the lock backoff; payload index creation; upsert and delete, which write the log and update the payload and field indexes; search; the filtered scope count behind the exhaustive verdict; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that opens the shard itself, holds it, serialises access to it, and acknowledges an upsert or delete only after the engine's flush has completed, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added. Two facts make that non-blocking drop safe, and both are pinned by the contract canary: the engine persists a write only when its flush runs: the flush on shard drop is the persistence step, not compaction, and a load does not replay the write-ahead log, so a writer that skips the final drop reopens with none of its unflushed writes (measured on the pinned version: two process-level probes, one skipping the drop and one exiting the process, reopened with zero of two hundred points; the normal-drop control reopened with all of them), so the owner flushes after every write and acknowledges only then, which makes every acknowledged write durable independently of the final drop and means a process exit that pre-empts that drop loses nothing acknowledged; and a shard directory is locked while its owner holds it, so the constructor, on encountering a locked directory, waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. A close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. @@ -117,7 +117,7 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t - A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. - The contract canary passes on the pinned engine version and is re-run on every engine bump. - The hard-exit test finds every acknowledged write after a process exit that skipped the shard drop, and the contract canary fails if the pinned engine ever starts replaying its log on load (the flush-per-write rule then becomes revisitable, not wrong). -- The benchmark shows no scan, index build, or shard close occupying an async executor thread; the facade drop is exercised under load inside an async runtime and the shard's final drop is observed on the adapter's blocking owner; a write burst is included in the responsiveness measurement. +- The benchmark shows no scan, index build, or shard close occupying an async executor thread; the facade drop is exercised under load inside an async runtime and the shard's final drop is observed on the adapter's blocking owner; construction and reopen, including a lock-backoff wait, and a write burst are included in the responsiveness measurement. - The default-mode construction test asserts service mode. ## Revisit When diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index bbee6d22..2670988d 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -55,7 +55,7 @@ Search: the query runs through the service adapter's tie-closure loop and the ca Verdict mapping: exhaustive when the shard is unindexed (known from the adapter's own threshold configuration) and the shared loop closed the cutoff cohort, with the scanned count taken from a filtered count of the scope rather than from the rows returned, so every exact scan with a closed cutoff reports exhaustive regardless of population size; boundary tie closed when an indexed shard's cutoff cohort closed (a statement about the index's returned prefix, not about global recall); boundary tie open whenever the bound is reached with the cohort still open, on an exact scan too; not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. -Blocking discipline: every engine call is synchronous (upsert and delete write the log and update payload and field indexes; search scans or traverses; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that holds the shard, serialises access, and acknowledges an upsert or delete only after the engine's flush has completed, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. +Blocking discipline: every engine call is synchronous (shard open and load with the lock backoff; payload index creation; upsert and delete write the log and update payload and field indexes; search scans or traverses; the filtered scope count behind the exhaustive verdict; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that opens the shard itself, holds it, serialises access, and acknowledges an upsert or delete only after the engine's flush has completed, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. That non-blocking drop is safe because the owner flushes after every write and acknowledges only then (the engine persists a write only when its flush runs: the flush on shard drop is the persistence step, not compaction, and a load does not replay the write-ahead log, so a writer that skips the final drop reopens with none of its unflushed writes (measured on the pinned version: two process-level probes, one skipping the drop and one exiting the process, reopened with zero of two hundred points; the normal-drop control reopened with all of them)), so every acknowledged write is durable independently of the final drop and a process exit that pre-empts it loses nothing acknowledged, and because a shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it; both facts are pinned by the contract canary and proven by a close-then-reopen test that drops the facade inside an async runtime, reopens the same directory immediately, and finds every write, and by a hard-exit test that exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write. The in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. @@ -146,7 +146,7 @@ A recall comparison of the embedded adapter above its indexing threshold against Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. -No engine call (upsert, delete, search, index build, or shutdown) occupies an async executor thread; all run on the adapter's dedicated blocking owner, dropping the facade signals that owner so the shard's final drop happens there; the owner acknowledges a write only after the engine's flush, the hard-exit test finds every acknowledged write after a process exit that skipped the drop, and the benchmark records executor responsiveness during a concurrent scan, a write burst, a build, and a close. +No engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count, index build, or shutdown) occupies an async executor thread; all run on the adapter's dedicated blocking owner, which opens the shard itself so the async composition entry point never touches the engine, dropping the facade signals that owner so the shard's final drop happens there; the owner acknowledges a write only after the engine's flush, the hard-exit test finds every acknowledged write after a process exit that skipped the drop, and the benchmark records executor responsiveness during construction and reopen (including a lock-backoff wait), a concurrent scan, a write burst, a build, and a close. A zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero with a truthful verdict, both proven in both adapters by parity fixtures. The engine contract canary passes on the pinned version. The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. From d756c5dd448ab55f5ed5a752f9b87ec44f06e488 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 18:39:04 +0900 Subject: [PATCH 28/44] =?UTF-8?q?=F0=9F=93=9D=20Decide=20the=20embedded=20?= =?UTF-8?q?default=20in=20ADR-I-0023,=20split=20engine=20discipline=20into?= =?UTF-8?q?=20ADR-I-0027,=20and=20narrow=20ADR-I-0024=20to=20the=20complet?= =?UTF-8?q?eness=20verdict=20and=20the=20prefilter=20rule?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-I-0023 now decides that embedded Qdrant Edge is the default vector mode from this phase, licensed by the phase's own parity suite and service-free integration path under the defaults-match-evidence rule; it partially supersedes ADR-I-0003's vector default (reciprocal frontmatter added). The blocking-owner and per-write-flush rules move to ADR-I-0027 as their own decision. ADR-I-0024 keeps the verdict and the unknown-never-matches prefilter rule, demotes the scope-only query to current state, moves the type shape to a non-binding appendix, and is renamed. Phase document, plan, roadmap, ADR-I-0025, and the payload design note follow. Co-Authored-By: Claude Fable 5.1 --- .../v0-1-6-embedded-vector-recall-plan.md | 18 ++- .../ADR-I-0003-qdrant-oxigraph-defaults.md | 4 +- ...dded-qdrant-edge-vector-candidate-store.md | 59 ++++---- ...ness-and-prefilters-never-match-unknown.md | 123 ++++++++++++++++ ...mpleteness-and-takes-a-scope-only-query.md | 131 ------------------ ...I-0025-vector-record-is-a-read-contract.md | 8 +- ...ctor-baselines-read-the-retrieval-trace.md | 2 +- ...blocking-owner-that-flushes-every-write.md | 101 ++++++++++++++ docs/design/database/vector_payload_design.md | 2 +- ...v0_1_6_embedded_vector_candidate_recall.md | 30 ++-- docs/roadmap/development_roadmap.md | 9 +- 11 files changed, 290 insertions(+), 197 deletions(-) create mode 100644 docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md delete mode 100644 docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md create mode 100644 docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 4c6f2ae7..bce0e155 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -6,7 +6,7 @@ - work_type: mixed ## Goal -- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0026: a redesigned vector port contract, a five-field vector record, an embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) as the opt-in local mode, and a shared contract suite over both adapters; the evaluation repository's move of its vector-only baseline onto the retrieval trace is planned there and consumed here as closeout evidence. +- Deliver the phase described in `docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md` under ADR-I-0023 through ADR-I-0027: a redesigned vector port contract, a five-field vector record, an embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) as the default vector mode with the service adapter retained as the explicit service mode, and a shared contract suite over both adapters; the evaluation repository's move of its vector-only baseline onto the retrieval trace is planned there and consumed here as closeout evidence. ## Definition of Done - Every acceptance criterion in the phase document's "Acceptance criteria" section holds with recorded evidence. @@ -17,14 +17,14 @@ ## Scope / Non-goals - Scope: the phase document's deliverables and deletions, all in this repository. -- Non-goals: the phase document's non-goals (no default flip, no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode for non-empty scopes; ADR-I-0024's empty-scope change, zero candidates for an empty scope and boundary rejection of an empty configured scope, is an intended change and in scope). +- Non-goals: the phase document's non-goals (no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode for non-empty scopes; ADR-I-0024's empty-scope change, zero candidates for an empty scope and boundary rejection of an empty configured scope, is an intended change and in scope). ## Context (workspace) - Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. - As-built port: `src/ports/vector_candidate.rs`, `src/models/vector/candidate_record.rs`, `src/models/vector/record.rs`, `src/adapters/qdrant/{store,payload}.rs`, `src/policy/embedding_surface.rs`, `src/usecases/retrieve.rs`, `src/api/types/retrieval.rs`, `src/composition.rs`, `src/config/app_settings.rs`, `src/test_support.rs`. - Prerequisite in this repository, landed: the toolchain pin moved to the embedded engine's minimum (Rust 1.97.0) in its own change, merged 2026-09-02 as a88c117. - Prerequisite tracked in the evaluation repository: its evidence-integrity fixes must be merged before this phase cites any harness measurement. -- Repo reference docs consulted: the four ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. +- Repo reference docs consulted: the five ADRs; ADR-I-0018 (dependency direction; ports may import the public retrieval vocabulary under its named exception); ADR-I-0007 (schema versioning); ADR-I-0021 (embedded default pattern); rules in `docs/coding-agent/rules/`. ## Open Questions (max 3) - none (the draft's five open questions were ruled by the decider on 2026-09-02 and are recorded in the phase document and the ADRs). @@ -77,7 +77,7 @@ - description: | Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; make the service adapter map its fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. - acceptance: - - The envelope and enum match ADR-I-0024's Decision section; the canonical-candidates newtype is unchanged. + - The envelope and verdict express the four situations in ADR-I-0024's Decision section (its appendix shape is a non-binding reference); the canonical-candidates newtype is unchanged. - Query-side zero-norm rule implemented in the service adapter: a zero-norm query scores every candidate zero and returns a truthful verdict, with a unit test and a parity fixture that Task_4 inherits. - Telemetry carries the verdict for every retrieval; a retrieval test asserts each variant. - Fetch-decision unit tests assert closed and open verdicts including the all-tied cohort at the bound. @@ -152,7 +152,7 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - No engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count behind the exhaustive verdict, index build, or shutdown) occupies an async executor thread: all run on the adapter's dedicated blocking owner, which opens the shard itself so the async composition entry point never calls the engine directly, and dropping the adapter through the existing facade drop signals that owner so the final shard drop happens there; the owner flushes after every upsert or delete and acknowledges only then, so durability of acknowledged writes does not depend on that final drop (the engine persists only on flush and does not replay its log on load), and a constructor meeting a directory still locked by a closing owner waits with a bounded backoff; a close-then-reopen test drops the facade inside an async runtime, asserts the executor was never blocked, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens from a second process, and finds every acknowledged write; the contract canary pins both engine facts (no log replay on load; directory lock); the benchmark records executor responsiveness during construction and reopen (including a lock-backoff wait), a concurrent scan, a write burst, a build, and a close. + - ADR-I-0027 holds: every engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count, index build, shutdown) runs on the adapter's dedicated blocking owner, which opens the shard itself; a write is acknowledged only after the engine's flush; the facade drop only signals the owner; a constructor meeting a locked directory waits with a bounded backoff; the close-then-reopen test, the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write), and the responsiveness benchmark (construction and reopen with a lock-backoff wait, a concurrent scan, a write burst, a build, a close) pass; the contract canary pins no-replay-on-load and the directory lock. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. @@ -222,7 +222,7 @@ Each wave ends with reviewer approval and a PR, merged by the decider before the Notification duty: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository before merge; how and when that repository adopts the change is planned there, and this plan only consumes the resulting compatibility evidence at closeout. ## Rollback / Safety -- Embedded mode is opt-in; the service mode's behavior is unchanged except for the reported verdict, the shrunken record, and the intended empty-scope change (an empty object-type scope selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary), all covered by the parity suite and the retrieval tests. +- Embedded mode is the default and the default-mode construction test asserts it; the service mode's behavior is unchanged except for the reported verdict, the shrunken record, and the intended empty-scope change (an empty object-type scope selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary), all covered by the parity suite and the retrieval tests. - Stored service-mode payloads with dropped fields remain readable (extra fields tolerated unread); rebuild from graph authority is the recovery path. - Each wave is a separately revertible PR. @@ -237,7 +237,7 @@ Append-only editing rule (applies to both logs below): when appending an entry, - 2026-09-02 Decision: the draft's port description was rewritten as an intentional new port contract. - Trigger / new insight: the draft (2026-07-20) described filter, diagnostics, and reconciliation capabilities that the structured-verdict phase deleted; thirty of thirty-three payload fields were write-only; the readable text column's only reader was the evaluation repository's direct store access. - Plan delta: contract-first waves (envelope and query, then record, then adapter); evaluation baseline moved to the trace; deletions promoted to deliverables. - - Tradeoffs considered: recorded in the four ADRs' rejected alternatives; the forward-looking keep case for hint fields (immutable time window) is recorded as a re-entry path rather than kept. + - Tradeoffs considered: recorded in the ADRs' rejected alternatives; the forward-looking keep case for hint fields (immutable time window) is recorded as a re-entry path rather than kept. - User approval: rulings on all five questions given 2026-09-02; plan approval pending. - 2026-09-02 Decision: evidence-integrity defects in the evaluation repository are fixed before this phase, outside this plan. - Trigger / new insight: batch ingest produced phantom repair attempts and the evaluated rank was harness-invented; both would corrupt the parity and baseline evidence this phase cites. @@ -252,6 +252,10 @@ Append-only editing rule (applies to both logs below): when appending an entry, - Finding: the engine persists a write only on flush and does not replay its log on load; process-level probes that skipped the shard drop reopened with zero of two hundred points. - Ruling: the blocking owner flushes after every write and acknowledges only then; the signal-only facade drop stays; no port or facade method is added; a hard-exit test joins Task_4's acceptance. - Tradeoff: one synchronous disk sync per write, measured by the write burst in the benchmark; batching behind an acknowledgement is the named upgrade. +- 2026-09-03 — Decision records restructured on the decider's review. + - ADR-I-0023 now decides the default: embedded is the default vector mode from this phase, licensed by the phase's own parity suite and service-free integration path (defaults-match-evidence, ADR-I-0021); ADR-I-0003's vector default is partially superseded; the evaluation repository's cross-mode run becomes a revisit trigger, not a gate. Task_4's default-mode test asserts embedded; Task_8's documentation leads with the local path. + - The blocking-owner and per-write-flush rules moved out of ADR-I-0023 into ADR-I-0027 as their own decision. + - ADR-I-0024 narrowed to the completeness verdict and the prefilter rule (unknown never matches); the scope-only query is a current state, the type shape is a non-binding appendix, and the two predicate paths are notes binding on no phase. ## Notes - Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the latency guidance and the stripped dependency weight must be measured, not assumed; the engine is beta, so its pin is exact and its bump is gated by the canary. diff --git a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md index 706b0aa8..a611f6ce 100644 --- a/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md +++ b/docs/decisions/implementation/ADR-I-0003-qdrant-oxigraph-defaults.md @@ -14,8 +14,8 @@ warrant: depends_on: [] implements: [] supersedes: [] -superseded_by: null -supersession_scope: null +superseded_by: implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +supersession_scope: partial # the vector-backend default only (embedded Qdrant Edge is the default); backend roles remain authoritative --- # ADR-I-0003: Use Qdrant and Oxigraph as default storage backends diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index fcade98e..0cecd575 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -7,18 +7,18 @@ consulted: ["Claude Fable 5.1"] informed: [] warrant: warranted_by: "without this record, future work would likely replace the embedded engine with a lighter exhaustive-scan store the first time the dependency weight is questioned, or treat the embedded store as a test convenience whose semantics may drift from the service adapter, because both are the shortest path at small corpus sizes and both were measured as viable" - detected_signals: "cross-boundary contract shape with tempting alternatives; rejected alternatives likely to be re-proposed (the two spiked candidates); costly reversal (an engine switch rebuilds every embedded store); premises likely to expire (the engine is beta; the weight measurement is unstripped); deliberately bounded scope (single process, opt-in default, index tuning deferred)" + detected_signals: "cross-boundary contract shape with tempting alternatives; rejected alternatives likely to be re-proposed (the two spiked candidates); costly reversal (an engine switch rebuilds every embedded store); premises likely to expire (the engine is beta; the weight measurement is unstripped); deliberately bounded scope (single process, index tuning deferred)" cost_of_violation: "an engine switch after embedded stores exist in the field rebuilds every character's recall index from graph authority and re-embeds it; two adapters with different admission semantics produce different continuity packs from the same memory, which evaluation evidence would attribute to retrieval regressions" cost_of_wrong_preservation: "if the engine's beta API breaks or its footprint proves unacceptable on a target platform and this record is preserved as settled, local deployments carry a dependency that no longer earns its place" - cost_of_over_extension: "treating the embedded mode as validated for multi-process access or as the default before parity evidence exists misrepresents what the library has validated; treating the index knobs as tuned when this phase leaves them at their exact-scan setting would ship approximate recall nobody measured" + cost_of_over_extension: "treating the embedded mode as validated for multi-process access misrepresents what the library has validated; treating the index knobs as tuned when this phase leaves them at their exact-scan setting would ship approximate recall nobody measured" depends_on: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] implements: [] -supersedes: [] +supersedes: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md] superseded_by: null -supersession_scope: null +supersession_scope: partial # ADR-I-0003's vector-backend default only; its backend roles stay authoritative and its graph default was already superseded by ADR-I-0021 --- -# ADR-I-0023: Embedded Qdrant Edge vector candidate store as the opt-in local mode +# ADR-I-0023: Embedded Qdrant Edge is the default vector candidate store; the service adapter remains the service mode ## Context and Problem Statement @@ -36,50 +36,50 @@ Two feasibility spikes were run on 2026-09-02 against the same probe set (build - The store is a rebuildable cache over graph authority, which bounds the cost of an engine switch but does not eliminate it: every embedded store in the field is rebuilt and re-embedded. - Library over in-house: the library does not own a vector engine; it owns the port contract, the tie-closure loop, the canonical ordering, the verdict mapping, and the error classification, and it holds any engine to those. - The embedded adapter must satisfy the same port contract as the service adapter, proven by a shared parity suite, or the evaluation suite stops being a regression instrument. -- Defaults must match validation evidence (ADR-I-0021's rule); flipping the default before parity evidence exists would repeat the mistake that record corrected. +- Defaults must match validation evidence (ADR-I-0021's rule): once the default test path runs on the embedded adapter, a service default would repeat the asymmetry that record corrected, so the default and its evidence land in the same change. - Portability across deployment shapes: an engine that can synchronise with the service backend keeps a path from a local character to a hosted one. ## Decision Add an embedded vector candidate store mode behind the existing vector candidate port, implemented on the in-process build of the service backend (Qdrant Edge), selected by a dedicated store-mode setting. -The service adapter remains fully supported as the service and cloud mode; this decision adds a mode and deprecates nothing. -The default mode stays service until the parity suite and the evaluation suite have produced identical results across modes; flipping the default is a separate, evidence-gated decision. +Embedded mode is the default vector mode from the phase that ships it. +The service adapter remains fully supported as the service and cloud mode, selected explicitly; this decision adds a mode and deprecates nothing. +The evidence that licenses the default is produced by the same phase: the shared parity suite proves both adapters identical below their indexing thresholds, including identical-vector tie cohorts, and the library's integration suite runs on the embedded adapter without a service, so the shipped default is the validated path (ADR-I-0021's rule). +The companion evaluation repository's cross-mode run is closeout evidence and a revisit trigger, not a gate: a difference between modes on its continuity suite reopens this record. The embedded store is single-process, matching the embedded graph store's expectation. Exactness is a threshold property, not a promise: below the configured indexing threshold a shard answers by exhaustive scan, above it the index answers, and in both cases the completeness verdict (ADR-I-0024) reports the boundary state of the returned top-K. This phase ships the threshold at its exact-scan setting; index construction, quantization, and memory-mapped segments become available capabilities whose defaults are tuned on measured corpora in a later decision, never silently. -The adapter reuses the service adapter's tie-closure loop and the canonical constructor, because the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards; deterministic admission comes from closing the cutoff cohort and ordering it canonically, exactly as in service mode. -The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific, while filter and payload conventions, the tie-closure loop, verdict mapping, and error classification are shared library logic that neither adapter may re-implement. +The adapter is held to the port contract by the shared library logic it may not re-implement: the tie-closure loop and the canonical constructor (the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards), the filter and payload conventions, the verdict mapping, and the error classification. +The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific. The adapter constructs only object payloads (the engine's point constructor panics on any other JSON shape) and uses only the general shard type (the update-only shard type carries an unimplemented path). -No engine call runs on an async executor thread: every synchronous engine operation (shard open and load, including the lock backoff; payload index creation; upsert and delete, which write the log and update the payload and field indexes; search; the filtered scope count behind the exhaustive verdict; index build; and shutdown, since dropping a shard flushes synchronously) is executed by a dedicated blocking owner the adapter creates at construction, a single blocking worker that opens the shard itself, holds it, serialises access to it, and acknowledges an upsert or delete only after the engine's flush has completed, so the adapter's own drop only signals that owner and the shard's final drop happens on the owner's thread, never on an executor thread, and no port or facade method is added. -Two facts make that non-blocking drop safe, and both are pinned by the contract canary: the engine persists a write only when its flush runs: the flush on shard drop is the persistence step, not compaction, and a load does not replay the write-ahead log, so a writer that skips the final drop reopens with none of its unflushed writes (measured on the pinned version: two process-level probes, one skipping the drop and one exiting the process, reopened with zero of two hundred points; the normal-drop control reopened with all of them), so the owner flushes after every write and acknowledges only then, which makes every acknowledged write durable independently of the final drop and means a process exit that pre-empts that drop loses nothing acknowledged; and a shard directory is locked while its owner holds it, so the constructor, on encountering a locked directory, waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. -A close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write; a hard-exit test writes, exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write; the in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. +How the adapter runs the engine inside an async host and what makes a write durable is a separate decision (ADR-I-0027). -Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`service` or `embedded`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. +Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`embedded` or `service`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. The path names a directory; each collection is one engine shard directory inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has shard directories). -Embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name and the record schema version in an adapter-owned marker inside the shard directory, and requires the path setting to be present, with a missing path a configuration error rather than an implicit default. +Embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name and the record schema version in an adapter-owned marker inside the shard directory, and requires the path setting to be present, with a missing path a configuration error rather than an implicit location, exactly as the embedded graph store's path. A reopened shard is validated against the configured embedding model (vector size and distance from the shard's own configuration) and the supported record schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires. ## Implementation Impact -- A new adapter module implementing the vector candidate port on the embedded engine; the composition root gains a mode switch mirroring the statistics-store switch. +- A new adapter module implementing the vector candidate port on the embedded engine; the composition root gains a mode switch mirroring the statistics-store switch, defaulting to embedded. - The library's toolchain pin moves to the minimum the engine compiles on (Rust 1.97 at decision time; the engine rejected the previous 1.95 pin). - The settings type gains the mode and path keys; the service connection string becomes optional and is validated as present only in service mode. - The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion, planned in that repository, is a prerequisite to re-pinning its checkout to this wave's merge — not work of this wave. - The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured; the deterministic test fake is retired in favour of the embedded adapter opened on a temporary directory, which removes the vector-service dependency from the default test path. - Dependency weight is a recorded deliverable: the unstripped release delta is measured; the stripped delta and the effect of feature trimming are measured and recorded before closeout. -- Documentation states the single-process expectation, the threshold semantics, the measured latency guidance, and rebuild-from-graph-authority as the path between modes. +- Documentation states the embedded default, the single-process expectation, the threshold semantics, the measured latency guidance, and rebuild-from-graph-authority as the path between modes; the README's backend setup leads with the local path and presents the service as the explicit alternative. ## Considered Options -1. The in-process build of the service backend (Qdrant Edge), opt-in, service mode retained. +1. The in-process build of the service backend (Qdrant Edge) as the default, service mode retained. 2. An in-house SQLite exact cosine scan on the `rusqlite` dependency the statistics store already carries. 3. The SQLite vector extension (sqlite-vec). 4. A columnar embedded vector database (LanceDB). 5. An in-memory-only embedded store. -6. Flip the default to embedded in the same change. +6. Ship embedded as an opt-in mode and decide the default separately. ## Decision Outcome @@ -93,7 +93,7 @@ Option 2 (in-house exact scan) violates the library-over-in-house driver: the li Option 3 (sqlite-vec 0.1.9) measured well — builds on both the previous and the new toolchain pin, 19 additional tree lines, about 6.2 MB unstripped, deterministic ties across fresh files and processes, parity five of five with score delta at or below 1.6e-7 — but its stable release is exhaustive-only with approximate indexing existing only in a pre-release, it carries limits on dimensions, result count, and metadata columns, and it is a pre-1.0 binding with a single maintainer; at decade scale its future is a second migration, so it is rejected for this role and reopened only if the chosen engine fails its Revisit When triggers and the extension has shipped a stable approximate index. Option 4 (LanceDB) was rejected on dependency weight relative to the chosen engine without a spike, since the chosen engine already covers its capabilities; it is reopened only alongside Option 3's reopening. Option 5 fails restart safety, which the persistent-graph-authority phase made a requirement for every store that survives a process; rejected outright. -Option 6 contradicts the defaults-match-evidence rule; it is reopened by the evidence named under Revisit When. +Option 6 defers a decision whose deciding evidence this very phase produces: the parity suite and the service-free integration path are phase acceptance, so after the phase the validated path is embedded and a service default would be the unvalidated one, the exact asymmetry ADR-I-0021 corrected; deferral would also leave every consumer-facing document describing a default the evidence no longer supports. Rejected outright; the default is reopened only by the triggers under Revisit When. ## Consequences @@ -102,13 +102,13 @@ Option 6 contradicts the defaults-match-evidence rule; it is reopened by the evi - Negative / tradeoffs: the engine is beta and its API may change; the canary test and the pinned version turn that into a build-time failure rather than a runtime one. - Negative / tradeoffs: about 30.5 MB of unstripped binary and a higher toolchain floor; the weight deliverable exists to establish the real number. - Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. -- Negative / tradeoffs: a flush after every write costs a synchronous disk sync per upsert or delete on the owner's thread; the write burst in the in-phase benchmark measures it, and batching acknowledged flushes is the named upgrade if it matters. +- Negative / tradeoffs: consumers who followed the service-first setup must now set the store path or select service mode explicitly; with no external consumers (Compatibility Policy) no migration hint is carried. ## Decision Boundary -Invariant: the embedded adapter implements the same port contract as the service adapter and is proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string; tie closure and canonical ordering come from the shared library loop and constructor, never from engine ordering; the indexing threshold ships at its exact-scan setting until a measured decision changes it. +Invariant: embedded is the default vector mode and the embedded adapter implements the same port contract as the service adapter, proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string; tie closure and canonical ordering come from the shared library loop and constructor, never from engine ordering; the indexing threshold ships at its exact-scan setting until a measured decision changes it. -Not covered: the index, quantization, and memory-map tuning values (calibrated through a later measured decision), the latency guidance numbers (measured and revised through documentation), and the default mode (a separate evidence-gated decision). +Not covered: the index, quantization, and memory-map tuning values (calibrated through a later measured decision) and the latency guidance numbers (measured and revised through documentation). ## Validation @@ -116,26 +116,23 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t - The parity suite produces identical admitted candidate sets and orderings from both adapters while both are below their indexing thresholds, including identical-vector tie cohorts closed through the shared loop; above the threshold a recall comparison against the exhaustive setting is recorded. - A reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. - The contract canary passes on the pinned engine version and is re-run on every engine bump. -- The hard-exit test finds every acknowledged write after a process exit that skipped the shard drop, and the contract canary fails if the pinned engine ever starts replaying its log on load (the flush-per-write rule then becomes revisitable, not wrong). -- The benchmark shows no scan, index build, or shard close occupying an async executor thread; the facade drop is exercised under load inside an async runtime and the shard's final drop is observed on the adapter's blocking owner; construction and reopen, including a lock-backoff wait, and a write burst are included in the responsiveness measurement. -- The default-mode construction test asserts service mode. +- The default-mode construction test asserts embedded mode; service mode requires the explicit mode value and a connection string. ## Revisit When - The engine leaves beta or changes its API — re-pin, re-run the canary, and re-measure parity before adopting the new version. - A stripped-footprint measurement or feature trimming changes the weight picture materially in either direction — revisit the weight tradeoff recorded above and, if the footprint is unacceptable on a target platform, reopen Option 3. - A corpus benchmark contradicts the interactive-latency assumption behind the decade standard (exhaustive scan acceptable far beyond the assumed scale, or the index insufficient at it) — revisit the threshold default and the tuning decision. -- The evaluation suite has run every dataset in embedded mode with results identical to service mode and one corpus at the guidance size — reopen the default mode. +- The companion evaluation repository's cross-mode run shows a difference between embedded and service mode on the continuity suite, or a corpus at the guidance size misses the interactive-latency assumption in embedded mode — reopen the default. - A multi-replica deployment shape is designed (the remote graph-authority phase ADR-I-0021 anticipates) — the single-process expectation is reconsidered together with the graph and statistics stores, never alone. -- The write-burst measurement shows the per-write flush dominating ingestion cost — batch flushes behind an explicit acknowledgement rather than weakening the durability rule. ## Consultation impact -Question asked: which embedded engine, on the two spikes' evidence; the consult's earlier recommendation of an in-house exact scan was overruled by the decider on the decade-scale portability standard, and the settings shape and opt-in default were adopted as recommended. +Question asked: which embedded engine, on the two spikes' evidence; the consult's earlier recommendation of an in-house exact scan was overruled by the decider on the decade-scale portability standard, the settings shape was adopted as recommended, and the consult's proposal to defer the default was overruled because the deciding evidence is produced by the same phase. ## More Information -- ADR-I-0003 remains fully authoritative for the default backends; this record adds an opt-in mode in response to its revisit clause and changes no default, so it supersedes nothing. A later, evidence-gated record that flips the default would supersede ADR-I-0003's vector-backend default. -- ADR-I-0024 (port contract this adapter implements, including the tie-closure and verdict rules) and ADR-I-0025 (the record it stores). +- ADR-I-0003 remains authoritative for the backend roles (vectors in the service backend family, graph authority in Oxigraph); this record supersedes only its vector-backend default, answering its own revisit clause; ADR-I-0021 already superseded its graph default. +- ADR-I-0024 (port contract this adapter implements, including the verdict rule), ADR-I-0025 (the record it stores), and ADR-I-0027 (how the adapter runs the engine and makes writes durable). - The two spike reports (2026-09-02) are transient working artifacts; the numbers above are their record. - The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md new file mode 100644 index 00000000..6379b93c --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md @@ -0,0 +1,123 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-02 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely add a vector-layer predicate as a three-valued hint filter that matches unknown values, or let an adapter truncate an unclosed equal-score cohort without saying so, because both are the natural first implementation and both have already happened in this repository" + detected_signals: "cross-boundary contract shape (port postcondition) with tempting alternatives; rejected alternative likely to be re-proposed; premises likely to expire (no vector-layer predicate is needed yet, and stored values are not yet guaranteed present)" + cost_of_violation: "a prefilter that matches unknown values admits stale candidates that graph verification then silently discards, and an unreported open cohort makes top-K membership vary between runs — both surface as unexplained retrieval nondeterminism in evaluation evidence long after the cause is forgotten" + cost_of_wrong_preservation: "if the unknown-never-matches rule is preserved after every stored value is guaranteed present and synchronised, adapters carry a defensive arm for a case that cannot occur" + cost_of_over_extension: "treating the completeness verdict as an error condition would fail retrieval on a determinism caveat about non-authoritative candidates" +depends_on: [implementation/ADR-I-0018-responsibility-boundary-modules-with-enforced-dependency-direction.md, implementation/ADR-I-0022-retain-measured-retrieval-defaults.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0024: Vector candidate recall reports its completeness, and a vector-layer prefilter never matches an unknown value + +## Context and Problem Statement + +The vector candidate port promised deterministic admission: at most `limit` unique object-and-surface matches in canonical order, with equal-score cohorts at the cutoff closed before truncation (ADR-I-0022 records the fix). +The service adapter closes the cohort by growing its fetch up to a bound, but when the bound is hit it returns the truncated set with no signal, so a caller cannot tell "this top-K is determinate" from "membership may vary between runs", and evaluation evidence later attributes the variation to retrieval. +Separately, the port once carried currentness predicates implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake, so a memory with a stale or missing value could be silently excluded from recall by a filter that was meant to include it. +Those filters were deleted as speculative when no caller used them. +A second adapter (ADR-I-0023) makes both gaps matter: below its indexing threshold an embedded shard scans exhaustively and needs a way to say so, and two adapters must agree on what a prefilter may do. + +## Decision Drivers + +- Retrieval rationale must be inspectable: the philosophy asks that a developer can see why a memory was or was not retrieved, and an unreported open cohort is unexplained recall. +- Candidate recall is non-authoritative; graph authority verifies every candidate, so a determinism caveat must never become a retrieval failure. +- A prefilter false negative is a memory that silently never returns — a continuity loss nobody can inspect — while a false positive costs one root slot and is discarded by verification; the asymmetry decides what a vector-layer predicate may do. +- Each port owns its stated postconditions; upper layers never repair lower-layer output. + +## Decision + +The search result carries, beside the canonical candidates, a completeness verdict stated by the adapter. +The verdict distinguishes four situations: no search was issued because the limit was zero or the scope was empty; every stored record in scope was scored, so the requested top-K is determinate over the population; an index answered with a prefix whose cutoff cohort was closed, so the returned set is determinate for that index state although an approximate index may have omitted records it never surfaced; and the overfetch bound was reached with the cutoff cohort still open, so membership may vary. +Adapters state the verdict truthfully: exhaustive only when the adapter knows the shard is unindexed and the cutoff cohort was closed, with the scanned count taken from the scope rather than the rows returned; an exhaustive scan whose cohort stays open at the bound reports open. +The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. + +A vector-layer predicate may be evaluated only over stored values that are immutable or synchronised on every mutation, and an unknown or missing stored value never satisfies a positive predicate. +A predicate that needs a stored value the write paths do not keep current is not a prefilter; it is a graph-authority question. + +## Character Memory Relevance + +Recall that silently drops memories, or silently varies between runs, is the unexplained recall the philosophy forbids: a character that forgets an episode because a filter matched a blank looks like a character that never lived it. +The verdict keeps determinism inspectable, and the prefilter rule keeps a candidate stage from ever being the reason a memory is unreachable. + +## Implementation Impact + +- The port's search method returns candidates plus verdict; the pipeline copies the verdict into telemetry; test fakes report exhaustive. +- The service adapter's existing fetch decision maps onto the closed and open situations. +- The evaluation repository mirrors the telemetry field (ADR-I-0026). + +## Considered Options + +1. A completeness verdict beside the candidates, plus the prefilter rule. +2. Silent degradation at the fetch bound (as built). +3. A boolean complete flag. +4. Fail with an error when the cohort is open at the bound. +5. Resurrect the deleted hint filters for the embedded adapter, which can evaluate them exactly. + +## Decision Outcome + +Chosen option: **Option 1**. +It makes the postcondition expressible by the layer that owns it, distinguishes the exhaustive case an unindexed shard can report from the closed-cohort case an index can promise, and keeps every consumer a field access away from unchanged code. + +### Rejected Alternatives + +Option 2 hides a determinism caveat that evaluation evidence later attributes to retrieval; rejected outright. +Option 3 loses the exhaustive-versus-closed distinction that tells a caller whether population-level determinacy was achieved; rejected outright. +Option 4 fails retrieval on a caveat about non-authoritative candidates that graph authority verifies anyway; rejected outright. +Option 5 recreates a prefilter over values that only the upsert path wrote, which the rule above forbids; a future predicate is admitted the moment its stored value is kept in sync or is immutable. + +## Consequences + +- Positive: top-K determinism is observable per retrieval and per adapter. +- Positive: any future prefilter has one admission test — is the value it reads always current — instead of a case-by-case argument. +- Negative / tradeoffs: callers wanting a scoped or time-bounded semantic search wait for a synchronised or immutable column rather than filtering on what happens to be stored. + +## Decision Boundary + +Invariant: the search result carries a completeness verdict stated truthfully by the adapter, and the pipeline never repairs, retries, or fails on it; a vector-layer predicate reads only immutable or synchronised values, and an unknown value never satisfies a positive predicate. + +Not covered: the current query shape (an embedding, a limit, and an object-type scope, with an empty scope selecting zero — a current state, not a rule), the verdict's type and wire shape (the appendix is a reference, not a contract), the telemetry field name, and the service adapter's overfetch bound. + +## Validation + +- Unit tests on the service adapter's fetch decision assert the closed and open verdicts, including the all-tied cohort at the bound. +- A retrieval test asserts the telemetry verdict for each situation using the fakes. +- The parity suite asserts exhaustive for the embedded adapter below its indexing threshold and closed for the service adapter on the identical-vector tie fixture. +- A census of the vector adapters shows no match-or-unknown condition. + +## Revisit When + +- An adapter appears that cannot classify its own cutoff (a remote index without a fetch count) — the verdict may need an "unknown" situation, which must still never be treated as an error. +- Every stored value a predicate could read is guaranteed present and synchronised — the unknown arm becomes unreachable and may be removed. + +## Consultation impact + +Question asked: whether the deleted hint filters should return for the embedded adapter; ruling adopted the prefilter rule instead. Revised 2026-09-03 on the decider's review: the type shape moved to an appendix and the scope-only query was demoted from rule to current state. + +## More Information + +- ADR-I-0022 (tie-cohort closure and canonical ordering, the postcondition this record makes expressible); ADR-I-0023 (the embedded adapter); ADR-I-0025 (the stored record a future predicate would extend); ADR-I-0026 (the evaluation reader of the verdict). +- Candidate predicates that satisfy the rule, noted for whichever phase needs them and binding on none: a scope id written at upsert and kept in sync by the link and reflection write paths (scoped continuity); an immutable time window over creation and observation time (a time-bounded retrieval route). + +## Appendix: reference shape at decision time (non-binding) + +```rust +pub struct VectorCandidateRecall { pub candidates: CanonicalCandidates, pub completeness: VectorRecallCompleteness } + +pub enum VectorRecallCompleteness { + NotRequested, + Exhaustive { scanned: usize }, + BoundaryTieClosed { fetched: usize }, + BoundaryTieOpen { fetched: usize, fetch_bound: usize }, +} +``` diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md deleted file mode 100644 index 6de3c8d3..00000000 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -status: accepted -adr_type: implementation -date: 2026-09-02 -deciders: ["ebigunso"] -consulted: ["Claude Fable 5.1"] -informed: [] -warrant: - warranted_by: "without this record, future work would likely add a vector-layer predicate as a three-valued hint filter that matches unknown values, or let an adapter truncate an unclosed equal-score cohort without saying so, because both are the natural first implementation and both have already happened in this repository" - detected_signals: "cross-boundary contract shape (port postcondition) with tempting alternatives; rejected alternative likely to be re-proposed; premises likely to expire (no vector-layer predicate is needed yet)" - cost_of_violation: "a prefilter that matches unknown values admits stale candidates that graph verification then silently discards, and an unreported open cohort makes top-K membership vary between runs — both surface as unexplained retrieval nondeterminism in evaluation evidence long after the cause is forgotten" - cost_of_wrong_preservation: "if a retrieval route needs a scoped or time-bounded semantic search and the scope-only query is preserved as a rule rather than a current state, retrieval will starve at scale (an unfiltered top-K contains only the in-scope fraction) and callers will overfetch instead of adding the predicate" - cost_of_over_extension: "treating the completeness verdict as an error condition would fail retrieval on a determinism caveat about non-authoritative candidates" -depends_on: [implementation/ADR-I-0018-responsibility-boundary-modules-with-enforced-dependency-direction.md, implementation/ADR-I-0022-retain-measured-retrieval-defaults.md] -implements: [] -supersedes: [] -superseded_by: null -supersession_scope: null ---- - -# ADR-I-0024: Vector candidate recall reports completeness and takes a scope-only query - -## Context and Problem Statement - -The vector candidate port promised deterministic admission: at most `limit` unique object-and-surface matches in canonical order, with equal-score cohorts at the cutoff closed before truncation (ADR-I-0022 records the fix). -The service adapter closes the cohort by growing its fetch up to a bound, but when the bound is hit it returns the truncated set with no signal, and the port's result type — a bare candidate list — cannot carry the difference between "this top-K is determinate" and "membership may vary between runs". -Separately, the port once carried a filter type whose currentness predicates were `Option` values implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake. -Those filters were deleted as speculative in the structured-verdict phase because no caller used them; the query is now an embedding, a limit, and an object-type scope. -An embedded adapter (ADR-I-0023) makes the gap visible: below its indexing threshold an embedded shard scans exhaustively and needs a way to say so, and a second adapter needs a query contract that cannot drift. - -## Decision Drivers - -- Each port owns its stated postconditions; upper layers never repair lower-layer output (the structured-verdict contract's ruling), so completeness must be stated by the adapter, not inferred by the pipeline. -- Candidate recall is non-authoritative; graph authority verifies every candidate, so a determinism caveat must never become a retrieval failure. -- Prefilter false negatives are unrecoverable while false positives cost one root slot, so a vector-layer predicate is only safe on data that is immutable or synchronised on every mutation. -- Two adapters must be held to one query contract with one parity suite. -- The retrieval telemetry and trace vocabulary is the one API surface ports may import (ADR-I-0018), and it is where callers already read the returned candidate count. - -## Decision - -`search_candidates` returns a result envelope: the canonical candidates (the constructor-owned canonical newtype survives as the field type) together with a typed completeness verdict. - -```rust -#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] -#[serde(tag = "kind", rename_all = "snake_case")] // internally tagged, snake_case: the wire shape the public telemetry enums already use -pub enum VectorRecallCompleteness { - NotRequested, // the limit was zero or the scope was empty; no search was issued - Exhaustive { scanned: usize }, // every stored record in scope was scored, so the requested top-K is determinate (the adapter knows the shard is unindexed and the cutoff cohort closed; scanned is the filtered scope count, not the rows returned; the envelope is still capped at the limit) - BoundaryTieClosed { fetched: usize }, // an index returned a prefix and the cutoff cohort WITHIN THAT PREFIX was closed: determinate relative to the index's answer, which an approximate index may have built without records it never surfaced - BoundaryTieOpen { fetched: usize, fetch_bound: usize }, // the overfetch bound was reached with the cohort within the prefix still open -} -``` - -Adapters own canonicalisation and must state the verdict truthfully: not requested only when no search was issued because the limit is zero or the scope is empty (both admitted inputs, so the verdict must be total over them), exhaustive only when the adapter knows the shard is unindexed (its own threshold configuration, pinned by the canary) so every scoped record was scored, AND the cutoff cohort was closed by the shared loop, with `scanned` taken from a filtered count of the scope rather than from the rows returned; the requested top-K is then determinate over the population while the envelope stays capped at the limit, and an exhaustive scan whose cutoff cohort stays open at the bound reports open, not exhaustive; closed only when the cutoff cohort within the index's returned prefix was verified closed or the index returned fewer rows than asked, open at the bound. -The two boundary verdicts describe the index's answer, not global recall: an approximate index may omit equal- or higher-scored records it never surfaced, and no amount of overfetch can prove their absence, so only the exhaustive verdict asserts determinate top-K membership over the scoped population; the boundary verdicts assert determinism of the returned set for a given index state (the same index answers the same query the same way) and whether its cutoff cohort was closed. -Callers that need population-level determinacy configure the adapter below its indexing threshold (ADR-I-0023) and read the exhaustive verdict; nothing in this record requires the service adapter to switch to exact search. -Every adapter answers through the shared tie-closure loop and the canonical constructor; engine ordering of equal-score cohorts is never relied on, because it is not stable across freshly built shards (ADR-I-0023). -The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. -The verdict type lives in the public retrieval telemetry vocabulary so the port can name it without a mirror type. - -The query is the embedding, the limit, and an object-type scope, and nothing else. -An empty scope selects zero candidates and reports the not-requested verdict without issuing a search; wildcard-on-empty is prohibited, matching the graph query rule, and the retrieval context rejects an empty configured object-type set at the boundary. -Zero-norm vectors are defined on both sides of the port: the vector indexing service rejects a zero-norm record embedding as a typed per-record indexing failure before any adapter sees it (so adapters may normalise at write without a division-by-zero path), and a zero-norm query scores every candidate zero and returns a truthful verdict; the parity suite carries both cases. -Three-valued hint predicates are prohibited. -Any future vector-layer predicate arrives as an explicit enum whose unknown arm is spelled out, an unknown or missing stored value never satisfies a positive predicate, and the predicate lands with its mapping in both adapters and a parity fixture in the same change. - -Two re-entry paths are named now so the scope-only query is read as a current state, not a rule: - -1. A synchronised scope predicate, owned by the scoped-continuity phase: a scope-id column written at upsert and kept in sync by the link and reflection write paths, because the existing relationship hints were frozen at upsert and never updated by linking, which made them unusable as a prefilter. -2. An immutable time-window predicate over `created_at` and `observed_at`, owned by whichever phase first ships a time-bounded retrieval route: immutability makes a write-time column correct without a sync path, and the columns are backfilled from graph authority if they are ever needed (ADR-I-0025). - -## Implementation Impact - -- The port trait's search method changes its return type; the pipeline reads the candidates field and copies the verdict into telemetry; the test fakes wrap their existing value in the exhaustive variant. -- The service adapter's fetch-decision enum maps one-to-one onto the closed and open variants. -- Retrieval telemetry gains a completeness field with a manual default; the companion evaluation repository mirrors the field in its telemetry record (ADR-I-0026 records the obligation). -- The port doc comment stops describing a "documented bounded-overfetch degradation policy" because the type now says it. - -## Considered Options - -1. A typed completeness verdict in a result envelope; scope-only query with the predicate rule and named re-entry paths. -2. Silent degradation at the fetch bound (as built). -3. A boolean `complete` flag on the result. -4. Fail closed with an error when the cohort is open at the bound. -5. Resurrect the deleted hint filters for the embedded adapter, which can evaluate them exactly. - -## Decision Outcome - -Chosen option: **Option 1**. -It makes the postcondition expressible by the type that owns it, distinguishes the exhaustive case an unindexed embedded shard can report from the closed-cohort case an index can promise, and keeps every consumer a field access away from unchanged code. - -### Rejected Alternatives - -Option 2 hides a determinism caveat that evaluation evidence later attributes to retrieval; rejected outright. -Option 3 loses the exhaustive-versus-closed distinction and the fetch counts that explain overfetch cost; rejected outright. -Option 4 fails retrieval on a caveat about non-authoritative candidates that graph authority verifies anyway; rejected outright. -Option 5 recreates a prefilter over hints that no write path other than upsert keeps in sync; it is reopened only through the named re-entry paths, each of which brings its own synchronisation obligation. - -## Consequences - -- Positive: top-K determinism is observable per retrieval; the parity suite can assert the verdict per adapter. -- Positive: the query contract is small enough to hold two adapters to by set equality. -- Negative / tradeoffs: callers that need scoped or time-bounded semantic recall must wait for the named predicate rather than overfetching; the re-entry paths exist to make that wait short and the shape predictable. - -## Decision Boundary - -Invariant: the search result carries a typed completeness verdict stated by the adapter; the pipeline never repairs or fails on it; the query carries no three-valued predicate; a new predicate lands in both adapters with a parity fixture. - -Not covered: the service adapter's overfetch bound constants (calibrated values), the exact telemetry field name, and the internal fetch-decision mechanics. - -## Validation - -- Unit tests on the service adapter's fetch decision assert the mapping to closed and open verdicts, including the all-tied cohort at the bound. -- A retrieval test asserts the telemetry verdict for each variant using the fakes. -- The parity suite asserts exhaustive for the embedded adapter below its indexing threshold and closed for the service adapter on the identical-vector tie fixture, both reached through the shared tie-closure loop. -- A census of the vector adapters shows no match-or-unknown condition and no filter type beyond the object-type scope. - -## Revisit When - -- A retrieval route needs a scoped or time-bounded semantic search — take the matching re-entry path above rather than reopening the predicate rule. -- An adapter appears that cannot classify its own cutoff (for example a remote index without a fetch count) — the verdict vocabulary may need a variant for "unknown", which must still never be treated as an error. - -## Consultation impact - -Question asked: whether the deleted hint filters should return for the embedded adapter; ruling adopted the scope-only query with the two named re-entry paths as recommended. - -## More Information - -- ADR-I-0022 (tie-cohort closure and canonical ordering at the adapter boundary, the postcondition this record makes expressible). -- ADR-I-0025 (the stored record whose columns the re-entry paths would extend). -- ADR-I-0023 (the embedded adapter, which reports exhaustive below its indexing threshold and the boundary verdicts above it). diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index 45d829b2..397ae90f 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -11,7 +11,7 @@ warrant: cost_of_violation: "every column that returns without a reader is mirrored across two adapters, indexed at every collection initialisation, and carried stale by write paths that never update it; a column deleted as unread would erase the only record of what a generated vector embedded" cost_of_wrong_preservation: "if a retrieval route needs a prefilter and the five-column rule is preserved as prohibition rather than current state, the predicate is blocked instead of landing through the named re-entry path" cost_of_over_extension: "extending the rule to the graph store would strip graph authority of denormalised fields it legitimately owns" -depends_on: [implementation/ADR-I-0007-schema-versioning.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md] +depends_on: [implementation/ADR-I-0007-schema-versioning.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md] implements: [] supersedes: [implementation/ADR-I-0005-qdrant-payload-vs-graph-authority.md, implementation/ADR-I-0002-natural-language-embedding-surfaces.md, implementation/ADR-I-0001-stable-cross-store-ids.md] superseded_by: null @@ -49,7 +49,7 @@ Consumers needing candidate content hydrate by object id. The relationship refs (episode, observation, thread, entity, participant, speaker, supersedes), the lifecycle and currentness flags, the time hints, the ranking and salience hints, the object-specific hints, the graph URI, and the raw source reference leave the vector write path. Dropping the graph URI partially supersedes ADR-I-0001's clause that every vector payload carries it: the stable object id remains the cross-store identity and the graph URI is derived from it by graph authority, so the pointer was a redundant copy of the id; ADR-I-0001's stable-id decision itself is unchanged. The typed field manifest introduced in the structured-verdict phase remains the single source of both adapters' column sets and shrinks to the five entries. -ADR-I-0024 names the two re-entry paths (a synchronised scope predicate; an immutable time-window predicate over `created_at` and `observed_at` backfilled from graph authority) so a returning column arrives with its predicate, its adapter mappings, and a parity fixture. +ADR-I-0024 rules that a predicate reads only synchronised or immutable values and notes the two candidate predicates (a synchronised scope id; an immutable time window over `created_at` and `observed_at` backfilled from graph authority), so a returning column arrives with its predicate and its reader. ## Implementation Impact @@ -99,7 +99,7 @@ Not covered: the physical encoding of each column per adapter, and graph authori ## Revisit When -- A retrieval route needs a scoped or time-bounded semantic search — take the ADR-I-0024 re-entry path; this record's invariant is satisfied by a column that arrives with its reader. +- A retrieval route needs a scoped or time-bounded semantic search — add the column under ADR-I-0024's prefilter rule; this record's invariant is satisfied by a column that arrives with its reader. - The assisted-remember phase makes the embedding surface a graph-authoritative provenance artifact — the vector copy becomes a cache and this record's provenance argument moves to the graph. - A re-indexing workflow appears that cannot rebuild from graph authority — the readable-text question reopens with that workflow as its reader. @@ -111,4 +111,4 @@ Question asked: whether the unread hint families and the readable text column sh - ADR-I-0005 remains authoritative for graph authority over relationships; this record supersedes its payload field list and its "payload metadata as candidate filter" implementation guidance. - ADR-I-0002 remains authoritative for natural-language embedding surfaces; this record supersedes only its note to persist both text columns. -- ADR-I-0024 (query contract and re-entry paths), ADR-I-0023 (embedded shard layout), ADR-I-0026 (evaluation baseline reader). +- ADR-I-0024 (completeness verdict and prefilter rule), ADR-I-0023 (embedded shard layout), ADR-I-0026 (evaluation baseline reader). diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index d8080f69..c63e0c3c 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -11,7 +11,7 @@ warrant: cost_of_violation: "a raw-recall facade makes the library a vector-database abstraction and exposes unverified candidates as if they were memory; a schema-reading baseline breaks silently the moment a second vector adapter ships a different physical schema, and it reimplements canonical ordering the library already owns" cost_of_wrong_preservation: "if a product use case for candidate-level recall arrives and this record is preserved as a blanket prohibition, the diagnostic surface the observability phase plans would be blocked instead of designed" cost_of_over_extension: "reading this record as forbidding evaluation tooling from using the trace at all would leave the baseline with no honest data source" -depends_on: [implementation/ADR-I-0020-restart-identity-via-caller-supplied-ids-not-a-lookup-surface.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-takes-a-scope-only-query.md] +depends_on: [implementation/ADR-I-0020-restart-identity-via-caller-supplied-ids-not-a-lookup-surface.md, implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md] implements: [] supersedes: [] superseded_by: null diff --git a/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md new file mode 100644 index 00000000..5d9c85b9 --- /dev/null +++ b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md @@ -0,0 +1,101 @@ +--- +status: accepted +adr_type: implementation +date: 2026-09-03 +deciders: ["ebigunso"] +consulted: ["Claude Fable 5.1"] +informed: [] +warrant: + warranted_by: "without this record, future work would likely call the embedded engine directly from the async retrieval path (it is a plain synchronous API and the first implementation compiles), and would rely on the shard's final drop for persistence because the engine's write call returns success before anything is durable; both were the natural first draft of this phase" + detected_signals: "cross-boundary contract shape (an engine with synchronous, non-durable writes inside an async host); premises likely to expire (the engine is beta and its persistence model may change); costly to detect (lost writes surface as a character forgetting after a crash, long after the cause)" + cost_of_violation: "an engine call on an executor thread stalls every other retrieval in the process for the duration of a scan, build, or flush; a write acknowledged before its flush is lost on any exit that skips the shard's drop, and the loss is silent — the store reopens cleanly and simply lacks the memories" + cost_of_wrong_preservation: "if the engine starts replaying its log on load or persisting on write and this record is preserved, every write keeps paying a synchronous disk sync it no longer needs" +depends_on: [implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md] +implements: [] +supersedes: [] +superseded_by: null +supersession_scope: null +--- + +# ADR-I-0027: The embedded vector engine runs on a dedicated blocking owner that flushes every write before acknowledging it + +## Context and Problem Statement + +The embedded vector engine (ADR-I-0023) exposes a synchronous API: opening a shard, creating payload indexes, upserting and deleting, searching, counting, building an index, and dropping a shard (which flushes) all block the calling thread. +The library's retrieval and write paths are async, and the composition entry point is itself async, so the natural first implementation calls the engine on an executor thread and stalls every other task in the process. +Separately, the engine's write call returns success before anything is durable. +Measured on the pinned version on 2026-09-02: a writer that skipped the shard's drop and a writer that exited the process after successful writes both reopened cleanly with zero of two hundred points, while the normal-drop control reopened with all of them; the engine persists only when its flush runs, and a load does not replay the write-ahead log. +An adapter that relies on the final drop for persistence therefore loses every unflushed memory on any exit that pre-empts that drop, and the loss is invisible at reopen. + +## Decision Drivers + +- No engine call may occupy an async executor thread; the library's other embedded stores already hold this line. +- A write the library has acknowledged must survive a process exit that skips orderly shutdown; a character that forgets after a crash violates continuity silently. +- No port or facade method is added for shutdown; the existing facade drop remains the only close path. +- The rule must be pinned to measured engine behaviour so a change in the engine reopens it rather than silently voiding it. + +## Decision + +The adapter creates a dedicated blocking owner at construction: one blocking worker that opens the shard itself, holds it, and serialises every engine call — shard open and load including the lock backoff, payload index creation, upsert and delete, search, the filtered scope count behind the exhaustive verdict, index build, and the final drop. +The async composition entry point never touches the engine; it only hands work to the owner and awaits the result. +The owner acknowledges an upsert or delete only after the engine's flush has completed, so every acknowledged write is durable independently of the shard's final drop. +Dropping the adapter through the existing facade drop only signals the owner; the shard's final drop happens on the owner's thread, and a process exit that pre-empts it loses nothing acknowledged. +A shard directory stays locked while an owner holds it; a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. +The contract canary (ADR-I-0023) additionally pins the two engine facts this record rests on: the engine does not persist a write until its flush runs, and a load does not replay the log. + +## Implementation Impact + +- The adapter owns a blocking thread and a request channel; every port method becomes a message to the owner. +- Each write costs one synchronous disk sync on the owner's thread; the in-phase benchmark's write burst measures it. +- The close-then-reopen test, the hard-exit test, and the responsiveness benchmark (construction and reopen including a lock-backoff wait, a scan, a write burst, a build, and a close) are the phase's evidence. + +## Considered Options + +1. A dedicated blocking owner that flushes after every write and acknowledges only then; signal-only facade drop. +2. Call the engine directly from the async paths and rely on the shard's drop for persistence. +3. A blocking owner with a signal-only drop and no per-write flush, documenting a weaker crash guarantee. +4. An explicit awaited close on the port or facade that flushes before returning. + +## Decision Outcome + +Chosen option: **Option 1**. +It is the only option that keeps executor threads free, makes every acknowledged write durable, and adds no API surface. + +### Rejected Alternatives + +Option 2 stalls the process on every scan, build, and flush, and was measured to lose every unflushed write on exit; rejected outright. +Option 3 makes the library's write acknowledgement a lie under crash or exit, and a character's lost memories are the cost; rejected outright. +Option 4 adds a close method every consumer must remember to call and still loses writes on any exit that skips it; the per-write flush makes it unnecessary; it is reopened only if the write-burst measurement shows the per-write flush dominating ingestion cost, in which case batched flushes behind an explicit acknowledgement are the shape, not a weaker guarantee. + +## Consequences + +- Positive: executor responsiveness is independent of corpus size and engine activity; acknowledged writes survive crashes and hard exits; no new API. +- Negative / tradeoffs: a synchronous disk sync per write; a serialised engine (one call at a time per adapter), which the candidate-recall role tolerates. + +## Decision Boundary + +Invariant: every engine call runs on the adapter's blocking owner; a write is acknowledged only after it is durable; the facade drop stays signal-only; the two engine facts are pinned by the canary. + +Not covered: the channel and thread mechanics, the backoff bound, and the batching of flushes behind an acknowledgement if measurement calls for it. + +## Validation + +- The hard-exit test writes, exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write. +- The close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write. +- The benchmark shows no engine call occupying an async executor thread and observes the shard's final drop on the owner's thread. +- The contract canary fails if the pinned engine starts replaying its log on load or persisting on write. + +## Revisit When + +- The engine persists on write or replays its log on load (the canary fails in that direction) — the per-write flush becomes optional and this record is revised. +- The write-burst measurement shows the per-write flush dominating ingestion cost — batch flushes behind an explicit acknowledgement rather than weakening the durability rule. +- A multi-process deployment shape is designed — the single-owner lock discipline is reconsidered with the graph and statistics stores, never alone. + +## Consultation impact + +Question asked (review round 22): whether the shard's drop-time flush is persistence or compaction; a process-level probe settled it as persistence, and the decider's rule "await the flush or document the weaker guarantee" was met by flushing per write. + +## More Information + +- ADR-I-0023 (the engine and the canary this record extends). +- The durability probe report of 2026-09-02 is a transient working artifact; the numbers above are its record. diff --git a/docs/design/database/vector_payload_design.md b/docs/design/database/vector_payload_design.md index f964fa46..9c00ba95 100644 --- a/docs/design/database/vector_payload_design.md +++ b/docs/design/database/vector_payload_design.md @@ -1,6 +1,6 @@ # Vector Database Payload Design -> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 records the re-entry paths for any returning prefilter column. The Record Shape, Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were (Record Shape still lists the graph URI, which the read contract dropped); the Design Goal (except its third rule, which named relationship and lifecycle hints the read contract dropped), Why Natural-Language Surfaces, and Consistency Model sections remain current. +> Supersession note (2026-09-02): ADR-I-0025 replaced the payload field inventory below with a five-field read contract (object id, object type, surface, schema version, embedded text) shared by the service and embedded adapters, and ADR-I-0024 rules what any returning prefilter column must satisfy (synchronised or immutable, unknown never matches). The Record Shape, Payload Categories, Indexing Policy, and relationship, lifecycle, time, and text-surface sections remain as the dated design rationale they were (Record Shape still lists the graph URI, which the read contract dropped); the Design Goal (except its third rule, which named relationship and lifecycle hints the read contract dropped), Why Natural-Language Surfaces, and Consistency Model sections remain current. This document describes the Qdrant payload design for Character Memory. It is intentionally a design note, not a field-by-field copy of the Rust mapping code. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 2670988d..5f10e6ac 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -1,6 +1,6 @@ # v0.1.6 Design: Embedded Vector Candidate Recall -Status: decided 2026-09-02 (ADR-I-0023 through ADR-I-0026); supersedes the 2026-07 draft of this document; the embedded engine ruling (Qdrant Edge over an in-house scan) was taken the same day on two feasibility spikes. +Status: decided 2026-09-02, revised 2026-09-03 (ADR-I-0023 through ADR-I-0027); supersedes the 2026-07 draft of this document; the embedded engine ruling (Qdrant Edge over an in-house scan) was taken the same day on two feasibility spikes. ## Version intent @@ -26,7 +26,7 @@ The embedded engine is the same family as the service backend, so payload and fi This phase fixes the port contract deliberately, because two adapters cannot be held to an implicit one. Query: the embedding, the limit, and an object-type scope, and nothing else; an empty scope selects zero candidates, and the retrieval context rejects an empty configured object-type set at the boundary. -Three-valued hint predicates are prohibited; a future predicate arrives as an explicit enum whose unknown arm never matches, in both adapters, with a parity fixture. +A vector-layer predicate reads only immutable or synchronised stored values, and an unknown or missing value never satisfies a positive predicate (ADR-I-0024); a future predicate lands in both adapters with a parity fixture, as every port change does. Result: a completeness envelope, the canonical candidates plus a typed verdict — not requested (the limit was zero or the scope was empty, so no search was issued), exhaustive (every scoped record was scored, so the requested top-K is determinate), boundary tie closed (an index returned a prefix and the cutoff cohort within that prefix was verified closed), or boundary tie open (the overfetch bound was reached with the cohort within the prefix still open). The boundary verdicts describe the index's answer, not global recall: an approximate index may omit records it never surfaced, so only the exhaustive verdict asserts determinate membership over the scoped population, while the boundary verdicts assert that the returned set is deterministic for a given index state and whether its cutoff cohort was closed. @@ -38,7 +38,7 @@ Record: both adapters persist exactly five fields — object id, object type, su Read-out text lives in graph authority; the vector record stores only the embedded surface, as provenance of what was ranked; consumers needing candidate content hydrate by object id. The relationship, lifecycle, time, ranking, object-specific, graph-URI, and raw-reference hints leave the write path: the library read none of them, the relationship hints were frozen at upsert and never updated by linking, the lifecycle hints described vectors the write path deletes, and the readable text column duplicated graph text. -Two re-entry paths are named so the scope-only query and five-field record are read as current state, not prohibition: +Two candidate predicates that satisfy that rule are noted so the scope-only query and five-field record are read as current state, not prohibition (binding on no phase): 1. A synchronised scope predicate, owned by the scoped-continuity phase: a scope-id column written at upsert and kept in sync by the link and reflection write paths. 2. An immutable time-window predicate over `created_at` and `observed_at`, owned by whichever phase first ships a time-bounded retrieval route; immutability makes a write-time column correct without a sync path, and the columns are backfilled from graph authority if ever needed. @@ -55,8 +55,8 @@ Search: the query runs through the service adapter's tie-closure loop and the ca Verdict mapping: exhaustive when the shard is unindexed (known from the adapter's own threshold configuration) and the shared loop closed the cutoff cohort, with the scanned count taken from a filtered count of the scope rather than from the rows returned, so every exact scan with a closed cutoff reports exhaustive regardless of population size; boundary tie closed when an indexed shard's cutoff cohort closed (a statement about the index's returned prefix, not about global recall); boundary tie open whenever the bound is reached with the cohort still open, on an exact scan too; not requested at limit zero or empty scope. Delete: remove every surface of each object id, matching the service adapter's selector. Restart safety: opening an existing shard validates its recorded vector size and distance against the configured embedding model and the adapter-owned marker's record schema version against the supported version before any query, raising the collection-compatibility error or the clear unsupported-schema failure ADR-I-0007 requires. -Blocking discipline: every engine call is synchronous (shard open and load with the lock backoff; payload index creation; upsert and delete write the log and update payload and field indexes; search scans or traverses; the filtered scope count behind the exhaustive verdict; index build; and shutdown flushes on drop), so the adapter creates a dedicated blocking owner at construction, one blocking worker that opens the shard itself, holds it, serialises access, and acknowledges an upsert or delete only after the engine's flush has completed, and routes every call through it; the adapter's own drop only signals that owner, so the shard's final drop happens on the owner's thread and no port or facade method is needed. -That non-blocking drop is safe because the owner flushes after every write and acknowledges only then (the engine persists a write only when its flush runs: the flush on shard drop is the persistence step, not compaction, and a load does not replay the write-ahead log, so a writer that skips the final drop reopens with none of its unflushed writes (measured on the pinned version: two process-level probes, one skipping the drop and one exiting the process, reopened with zero of two hundred points; the normal-drop control reopened with all of them)), so every acknowledged write is durable independently of the final drop and a process exit that pre-empts it loses nothing acknowledged, and because a shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it; both facts are pinned by the contract canary and proven by a close-then-reopen test that drops the facade inside an async runtime, reopens the same directory immediately, and finds every write, and by a hard-exit test that exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write. +Blocking discipline and durability (ADR-I-0027): every engine call — shard open and load with the lock backoff, payload index creation, upsert and delete, search, the filtered scope count behind the exhaustive verdict, index build, and the final drop — runs on a dedicated blocking owner the adapter creates at construction and that opens the shard itself, so the async composition entry point never touches the engine; the owner acknowledges a write only after the engine's flush, because the pinned engine persists only on flush and does not replay its log on load (writers that skipped the drop reopened with zero of two hundred points); the facade drop only signals the owner and no port or facade method is added. +A shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff; the two engine facts are pinned by the contract canary and proven by the close-then-reopen test and the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write). The in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. @@ -67,7 +67,7 @@ Score parity across adapters was measured at 0.0 delta on the spike; the parity Follow the one-key-per-backend pattern the graph and statistics stores already use rather than overloading the service connection string. ```text -VECTOR_STORE_MODE service | embedded (default: service) +VECTOR_STORE_MODE embedded | service (default: embedded) VECTOR_STORE_PATH directory, required in embedded mode (missing is a configuration error, never an implicit default), ignored in service mode QDRANT_CONNECTION_STRING required only in service mode ``` @@ -98,7 +98,7 @@ restart-safety test for the embedded store; pipeline test over the embedded adap dependency-weight report: unstripped and stripped release deltas, effect of feature trimming latency benchmark: exhaustive scan across corpus sizes at the configured dimension; executor responsiveness under a concurrent scan documentation: settings, single-process expectation, threshold semantics, measured latency guidance, rebuild-from-graph-authority as the path between modes -four implementation ADRs (ADR-I-0023 through ADR-I-0026) with reciprocal partial-supersession frontmatter on ADR-I-0001, ADR-I-0002, and ADR-I-0005 +five implementation ADRs (ADR-I-0023 through ADR-I-0027) with reciprocal partial-supersession frontmatter on ADR-I-0001, ADR-I-0002, ADR-I-0003, and ADR-I-0005 ``` Deletions that are deliverables, not side effects: @@ -120,10 +120,9 @@ deprecating the service adapter, or altering it beyond what the shared port and tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold, tuned by a later measured decision) named-vector coexistence of two embedding spaces (an engine capability this decision was taken for; its use lands with the first embedding-model migration) migration tooling between modes or between record shapes (rebuild-from-graph-authority is the documented path) -changing the default vector mode in this phase (embedded ships opt-in; flipping the default is a separate evidence-gated decision) multi-process access to the embedded store (same single-process expectation as embedded graph storage) synchronisation between an embedded shard and a service collection (an engine capability; not exercised this phase) -any vector-layer predicate beyond the object-type scope (the two named re-entry paths belong to later phases) +any vector-layer predicate beyond the object-type scope (the candidate predicates noted in ADR-I-0024 belong to later phases) any new public facade method (the evaluation baseline consumes the retrieval trace) reconciliation diagnostics (the reconciliation slice was deleted in the structured-verdict phase; graph verification is the guard) ``` @@ -146,7 +145,7 @@ A recall comparison of the embedded adapter above its indexing threshold against Deterministic admission holds in embedded mode (equal-score cohorts canonically ordered; repeated runs byte-identical; no engine ordering relied on). Retrieval telemetry reports the completeness verdict; the embedded adapter reports exhaustive below its threshold, the service adapter reports closed on the tie fixture. Embedded state survives process restart; a reopened shard with a different vector size or distance fails with the collection-compatibility error; an unsupported record schema version fails clearly. -No engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count, index build, or shutdown) occupies an async executor thread; all run on the adapter's dedicated blocking owner, which opens the shard itself so the async composition entry point never touches the engine, dropping the facade signals that owner so the shard's final drop happens there; the owner acknowledges a write only after the engine's flush, the hard-exit test finds every acknowledged write after a process exit that skipped the drop, and the benchmark records executor responsiveness during construction and reopen (including a lock-backoff wait), a concurrent scan, a write burst, a build, and a close. +ADR-I-0027 holds: no engine call occupies an async executor thread and every acknowledged write survives a hard exit, shown by the hard-exit test, the close-then-reopen test, and the responsiveness benchmark during construction and reopen (including a lock-backoff wait), a concurrent scan, a write burst, a build, and a close. A zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero with a truthful verdict, both proven in both adapters by parity fixtures. The engine contract canary passes on the pinned version. The dependency-weight report records unstripped and stripped release deltas and the effect of feature trimming. @@ -161,12 +160,12 @@ No public facade change beyond the telemetry field and the published maximum-sur The companion evaluation repository is a development aid; its own work is planned and tracked there, and this document records only what its measurements let this phase decide. - The library exposes, through an ordinary traced retrieval, everything a raw-vector baseline needs: the vector candidates with scores and the completeness verdict in telemetry; the honest way to use them is one singleton-scoped traced retrieval per measured object kind with a limit of the section budget multiplied by the maximum surfaces per object, deduplicated by object. -- The cross-mode comparison (service mode against embedded mode on the continuity suite, identical baselines expected under the parity contract) is the evidence that gates the default flip recorded in ADR-I-0023; it is consumed at the closeout task and by that later decision, not produced by this plan. +- The cross-mode comparison (service mode against embedded mode on the continuity suite, identical baselines expected under the parity contract) is closeout evidence and a revisit trigger for the embedded default recorded in ADR-I-0023 (a difference between modes reopens it); it is consumed at the closeout task, not produced by this plan. - No candidate-search facade or configuration surface is added for the evaluation repository; the one public addition made for its trace reading is the published maximum-surfaces-per-object-kind policy value (ADR-I-0026), and if its measurements ever require more, that is a library decision taken on its own record. ## Evaluation tie-in -The evaluation repository is expected to run its continuity suite in both vector modes; identical scenario results are what the parity contract predicts, and the comparison is the evidence that gates the later default-flip decision recorded in ADR-I-0023. +The evaluation repository is expected to run its continuity suite in both vector modes; identical scenario results are what the parity contract predicts, and a difference between modes is a revisit trigger for the embedded default (ADR-I-0023). How that configuration is built and run is planned in the evaluation repository; this phase consumes the comparison at closeout and cites nothing else from it. ## Deferral-reconfirmation checklist @@ -188,7 +187,7 @@ Each item was parked on this phase by the structured-verdict phase; each row sta 4. Hint filter semantics. Parked claim: query-side hint semantics belong to the port contract. Re-verified: the filter type and both match-or-unknown implementations were deleted in the structured-verdict phase, and no consumer asks for a vector-layer predicate (the evaluation surface policy carries object types and budgets only). - Evidence: zero-hit census for the filter type and for empty-or-null match conditions in the service adapter; the prohibition and re-entry paths are recorded in ADR-I-0024. + Evidence: zero-hit census for the filter type and for empty-or-null match conditions in the service adapter; the prefilter rule and candidate predicates are recorded in ADR-I-0024. 5. Evaluation baseline capability. Parked claim: the baseline re-implements a hidden raw-vector capability against the payload schema. Re-verified: one singleton-scoped traced retrieval per measured kind reproduces the direct per-kind search exactly, which a sliced mixed-kind top-K would not; each retrieval's completeness verdict reports whether that kind's top-K was determinate over the scoped population (exhaustive) or only closed within the index's returned prefix (boundary verdicts), and the baseline records which; the evaluation adapter can hold item text from ingest. @@ -198,11 +197,12 @@ Each item was parked on this phase by the structured-verdict phase; each row sta - Embedded engine: the in-process build of the service backend (Qdrant Edge), on the decade-scale portability standard and the two spikes' measurements; the in-house exact scan and the SQLite vector extension are rejected alternatives (ADR-I-0023). - Exactness: a threshold property shipped at the exact-scan setting; index, quantization, and memory-map tuning is a later measured decision (ADR-I-0023). -- Default mode: stays opt-in this phase; the flip is reopened by the evaluation suite running every dataset in embedded mode with identical results and one corpus at the guidance size (ADR-I-0023, Revisit When). +- Default mode: embedded from this phase, licensed by the phase's own parity suite and service-free integration path under the defaults-match-evidence rule (ADR-I-0021); the evaluation repository's cross-mode run is a revisit trigger, not a gate (ADR-I-0023). +- Engine discipline: every engine call on a dedicated blocking owner; a write acknowledged only after the engine's flush; signal-only facade drop (ADR-I-0027). - Latency guidance: measured in-phase by a benchmark over a synthetic corpus at the configured dimension, published in documentation, revised through documentation. - Dependency weight: the unstripped delta is recorded; the stripped delta and feature trimming are measured in-phase, and a material change reopens ADR-I-0023. - Parity suite placement: contract parity in the library, behaviour parity in the evaluation repository (above). - Settings shape: separate mode and path keys with `collection_name` as the backend-neutral namespace key naming one shard directory per collection, not a connection string interpreted by mode (ADR-I-0023). -- Hint families: all dropped from the vector record, with the two named re-entry paths (ADR-I-0024, ADR-I-0025). +- Hint families: all dropped from the vector record, with the two candidate predicates noted (ADR-I-0024, ADR-I-0025). - Text columns: readable text dropped, embedded text kept as provenance, governed by the three sentences in ADR-I-0025. - Evaluation baseline: trace-sourced, no facade change (ADR-I-0026). diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index e7d17d22..b4c082c3 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -259,7 +259,7 @@ Assisted remember workflows may accept raw or semi-raw input as transient proces | v0.1.3 | Remember intake interfaces and deterministic write planning | Finished. Generation-ready write path with `RememberWritePlan`, memory candidates, validation, deterministic helpers, prepare/validate/commit flow, and shared manual/future-generated commit machinery. | | v0.1.4 | Continuity evaluation harness | Finished. Deterministic long-horizon evaluation harness implemented in the public companion `CharacterMemoryEvals` repository as a development aid, not core library functionality: synthetic interaction fixtures, a minimal example assistant loop, continuity-oriented retrieval-quality metrics, selectivity/fanout measurement, and hub-entity stress scenarios. | | v0.1.5 | Eval-driven v0.1 family closeout | Finished. Ran the evaluation harness across the v0.1 family, dispositioned eleven findings (none critical, none open), fixed deterministic vector admission and write-path warning diagnostics in the library, retained the measured defaults with a recorded basis (ADR-I-0022), adopted embedded persistent Oxigraph as the validated default (ADR-I-0021), and expanded the evaluation suite to 33 scenarios including benchmark-adapted and real-embedding fixtures. Closeout report: [`v0_1_5_closeout_report.md`](v0_1_5_closeout_report.md). | -| v0.1.6 | Embedded vector candidate recall | Planned. An embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) behind the vector port as an opt-in local mode, shipped at its exact-scan indexing threshold, so zero-infrastructure local deployments become possible and the default test path needs no external service while the service mode stays the default; a redesigned port contract that reports recall completeness, takes a scope-only query, and stores a five-field record (identity, surface, schema version, and the embedded text kept as provenance); the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0026. | +| v0.1.6 | Embedded vector candidate recall | Planned. An embedded vector candidate store on the in-process build of the service backend (Qdrant Edge) behind the vector port as the default vector mode, shipped at its exact-scan indexing threshold, so zero-infrastructure local deployments are the default and the default test path needs no external service, with the service adapter retained as the explicit service mode; a redesigned port contract that reports recall completeness, forbids prefilters that match unknown values, and stores a five-field record (identity, surface, schema version, and the embedded text kept as provenance); the evaluation repository's vector-only baseline moves onto the retrieval trace. Decisions: ADR-I-0023 through ADR-I-0027. | | v0.2 | Scoped continuity and reflection | `ContinuityScope`, scoped reflection, relationship state between arbitrary entities, character signals for continuing entities, open-loop/commitment lifecycle, and current continuity views. | | v0.3 | Factual rigor, temporal validity, and entity evolution | Assertions, claims, evidence links, belief assessments, source assessment, temporal validity, entity drift handling, and current-belief views. | | v0.4 | Retrieval observability and governance | Retrieval traces, context subgraphs, validation rules, graph health reports, policy diagnostics, rejected expansion traces, cluster/activation diagnostics, and retention assessment. | @@ -1200,7 +1200,7 @@ v0.2 entry is explicitly confirmed against the closed v0.1 family. Detailed draft: [`v0_1_6_embedded_vector_candidate_recall.md`](../design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md) -Decisions: ADR-I-0023 (embedded Qdrant Edge vector store as the opt-in local mode), ADR-I-0024 (vector candidate recall reports completeness and takes a scope-only query), ADR-I-0025 (the vector record is a read contract), ADR-I-0026 (raw vector baselines read the retrieval trace). +Decisions: ADR-I-0023 (embedded Qdrant Edge is the default vector candidate store), ADR-I-0024 (vector candidate recall reports completeness and prefilters never match unknown values), ADR-I-0025 (the vector record is a read contract), ADR-I-0026 (raw vector baselines read the retrieval trace), ADR-I-0027 (the embedded engine runs on a blocking owner that flushes every write). ## Intent @@ -1211,12 +1211,12 @@ Because a second adapter must implement the vector port, this phase also settles ## Goals ```text -add an embedded vector candidate store on the in-process build of the service backend behind the existing vector port, selected by a store-mode setting with its own path setting and shipped at its exact-scan indexing threshold +add an embedded vector candidate store on the in-process build of the service backend behind the existing vector port, selected by a store-mode setting with its own path setting, shipped at its exact-scan indexing threshold, and made the default vector mode on the phase's own parity evidence make the port result carry a typed completeness verdict that the retrieval telemetry records and never repairs reduce the vector payload to its read contract: identity, surface, schema version, and the embedded text as provenance of what was ranked run one shared contract suite against both adapters, with the embedded adapter exercised unconditionally so the default test path needs no service move the evaluation repository's vector-only baseline onto the retrieval trace so no consumer depends on a store's private schema -record the re-entry paths for vector-layer predicates a later phase may need: a synchronized scope predicate, and an immutable time-window predicate +rule that a vector-layer predicate reads only synchronized or immutable values and never matches an unknown one, noting the candidate predicates a later phase may need ``` ## Non-goals @@ -1226,7 +1226,6 @@ changing the authority split, or any retrieval semantics in the service mode for deprecating or altering the service-mode adapter beyond the shared port contract tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold this phase) migration tooling between modes; rebuild from graph authority is the path -flipping the default vector mode in this phase multi-process access to the embedded store a public candidate-search facade ``` From 12497f7b7ad223a0cde8001a9b355039009dc17c Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 18:44:15 +0900 Subject: [PATCH 29/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?4:=20Task=5F2=20extracts=20the=20tie-closure=20loop=20into=20a?= =?UTF-8?q?=20shared=20crate-visible=20module=20it=20owns,=20so=20Task=5F4?= =?UTF-8?q?=20reuses=20it=20without=20crossing=20ownership?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index bce0e155..cbb52191 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -66,7 +66,9 @@ - src/ports/vector_candidate.rs - src/models/vector/candidate_record.rs - src/api/types/retrieval.rs + - src/adapters/qdrant.rs - src/adapters/qdrant/store.rs + - src/adapters/qdrant/tie_closure.rs - src/usecases/retrieve.rs - src/usecases/remember.rs - src/usecases/correct_forget.rs @@ -75,11 +77,12 @@ - src/adapters/oxigraph/tests.rs - depends_on: [] - description: | - Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; make the service adapter map its fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. + Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; extract the service adapter's private tie-closure loop (fetch decision, fetch bound, cohort closure, canonical construction) into `src/adapters/qdrant/tie_closure.rs` as crate-visible shared logic that takes an engine-neutral fetch callback, so the embedded adapter (Task_4) calls it rather than re-implementing it; make the service adapter map the shared fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. - acceptance: - The envelope and verdict express the four situations in ADR-I-0024's Decision section (its appendix shape is a non-binding reference); the canonical-candidates newtype is unchanged. - Query-side zero-norm rule implemented in the service adapter: a zero-norm query scores every candidate zero and returns a truthful verdict, with a unit test and a parity fixture that Task_4 inherits. - Telemetry carries the verdict for every retrieval; a retrieval test asserts each variant. + - The tie-closure loop lives in `src/adapters/qdrant/tie_closure.rs`, the service adapter calls it, and its existing unit tests (fetch decision, all-tied cohort at the bound) move with it; nothing in `store.rs` closes a cohort on its own. - Fetch-decision unit tests assert closed and open verdicts including the all-tied cohort at the bound. - Zero-hit census: no match-or-unknown condition, no filter type beyond object-type scope. - validation: @@ -147,7 +150,7 @@ - docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md - depends_on: [Task_3] - description: | - Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the service adapter's tie-closure loop and the canonical constructor with the verdict mapping (Exhaustive when the shard is unindexed per the adapter's own threshold configuration and the loop closed the cutoff cohort, with `scanned` from a filtered count of the scope, never from returned rows; BoundaryTieClosed when an indexed shard's cutoff cohort closed, describing the index's returned prefix and never global recall; BoundaryTieOpen whenever the bound is reached with the cohort open, on an exact scan too; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). + Implement the embedded adapter on `qdrant-edge` pinned at 0.8.0 per the phase document: one engine shard directory per collection under `VECTOR_STORE_PATH`, cosine distance at the configured vector size, the indexing threshold shipped at its exact-scan setting (zero) with no optimise call, the five-field payload with keyword indexes on object id (the delete selector) and object type (the scope predicate), the object-type scope as a filter, and search through the shared tie-closure loop Task_2 extracts into `src/adapters/qdrant/tie_closure.rs` and the canonical constructor, with the verdict mapping (Exhaustive when the shard is unindexed per the adapter's own threshold configuration and the loop closed the cutoff cohort, with `scanned` from a filtered count of the scope, never from returned rows; BoundaryTieClosed when an indexed shard's cutoff cohort closed, describing the index's returned prefix and never global recall; BoundaryTieOpen whenever the bound is reached with the cohort open, on an exact scan too; NotRequested at limit zero or empty scope). The adapter constructs only object payloads and uses only the general shard type; every engine call (upsert, delete, search, index build, and shutdown) runs on a dedicated blocking owner the adapter creates at construction and that holds the shard, never on an async executor thread; the adapter's drop signals the owner so the shard's final drop happens on the owner's thread, and no port or facade method is added. Add a contract canary test in the pattern of the service client's erased-connect canary pinning the engine facts the adapter relies on (zero threshold means unindexed; object-payload precondition; shard-directory precondition; crate-local type provenance). Implement the `VECTOR_STORE_MODE` and `VECTOR_STORE_PATH` settings with mode-specific validation (service connection string required only in service mode), the composition mode switch with `collection_name` as the backend-neutral namespace key, the adapter-owned marker recording collection name and record schema version, and the port-conformance parity suite run against both adapters (embedded unconditionally, service under the live gate). Extend the vector error vocabulary only where the embedded adapter needs a kind the service adapter lacks. Produce the dependency-weight report (unstripped and stripped release deltas, effect of feature trimming) and the latency benchmark (exhaustive scan at the configured dimension across corpus sizes; executor responsiveness under a concurrent scan). - acceptance: - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). From 83bd885abccf74e29d57fa95ff66ca26e3d0d8c1 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 18:47:26 +0900 Subject: [PATCH 30/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?5:=20plan=20metadata=20date,=20ADR-I-0003=20relationship=20stat?= =?UTF-8?q?ed=20exactly,=20and=20the=20match-or-unknown=20failure=20descri?= =?UTF-8?q?bed=20truthfully=20in=20ADR-I-0024?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 ++-- ...DR-I-0023-embedded-qdrant-edge-vector-candidate-store.md | 4 ++-- ...ports-completeness-and-prefilters-never-match-unknown.md | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index cbb52191..4143a842 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -2,7 +2,7 @@ - status: draft - generated: 2026-09-02 -- last_updated: 2026-09-02 +- last_updated: 2026-09-03 - work_type: mixed ## Goal @@ -31,7 +31,7 @@ ## Assumptions - A1: The embedded engine is `qdrant-edge` pinned exactly at 0.8.0 (beta); its API is guarded by a contract canary, and the pin is bumped only with a re-run of the canary and the parity suite. -- A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison that gates the later default-flip decision. +- A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison that gates the later embedded-default revisit decision. ## Tasks diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 0cecd575..074ad4dd 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -15,7 +15,7 @@ depends_on: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md, implementati implements: [] supersedes: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md] superseded_by: null -supersession_scope: partial # ADR-I-0003's vector-backend default only; its backend roles stay authoritative and its graph default was already superseded by ADR-I-0021 +supersession_scope: partial # ADR-I-0003's vector-backend default only; its backend roles and its embedded graph description stay authoritative --- # ADR-I-0023: Embedded Qdrant Edge is the default vector candidate store; the service adapter remains the service mode @@ -132,7 +132,7 @@ Question asked: which embedded engine, on the two spikes' evidence; the consult' ## More Information -- ADR-I-0003 remains authoritative for the backend roles (vectors in the service backend family, graph authority in Oxigraph); this record supersedes only its vector-backend default, answering its own revisit clause; ADR-I-0021 already superseded its graph default. +- ADR-I-0003 remains authoritative for the backend roles (vectors in the service backend family, graph authority in Oxigraph); this record supersedes only its vector-backend default, answering its own revisit clause; its embedded graph description is untouched (ADR-I-0021 made persistent embedded storage the validated graph default without superseding it). - ADR-I-0024 (port contract this adapter implements, including the verdict rule), ADR-I-0025 (the record it stores), and ADR-I-0027 (how the adapter runs the engine and makes writes durable). - The two spike reports (2026-09-02) are transient working artifacts; the numbers above are their record. - The embedded vector candidate recall phase document in the roadmap-phases design directory. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md index 6379b93c..969e15ec 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md @@ -24,7 +24,7 @@ supersession_scope: null The vector candidate port promised deterministic admission: at most `limit` unique object-and-surface matches in canonical order, with equal-score cohorts at the cutoff closed before truncation (ADR-I-0022 records the fix). The service adapter closes the cohort by growing its fetch up to a bound, but when the bound is hit it returns the truncated set with no signal, so a caller cannot tell "this top-K is determinate" from "membership may vary between runs", and evaluation evidence later attributes the variation to retrieval. -Separately, the port once carried currentness predicates implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake, so a memory with a stale or missing value could be silently excluded from recall by a filter that was meant to include it. +Separately, the port once carried currentness predicates implemented as match-or-unknown: a record whose payload lacked the field satisfied a positive predicate in both the service adapter and the test fake, so the filter admitted records under a rationale ("current") that was not true of them; and because the field was written only at upsert, a record whose value had since changed in graph authority was filtered on a stale value, admitted when it should not have been or excluded when it should have been returned. Those filters were deleted as speculative when no caller used them. A second adapter (ADR-I-0023) makes both gaps matter: below its indexing threshold an embedded shard scans exhaustively and needs a way to say so, and two adapters must agree on what a prefilter may do. @@ -47,8 +47,8 @@ A predicate that needs a stored value the write paths do not keep current is not ## Character Memory Relevance -Recall that silently drops memories, or silently varies between runs, is the unexplained recall the philosophy forbids: a character that forgets an episode because a filter matched a blank looks like a character that never lived it. -The verdict keeps determinism inspectable, and the prefilter rule keeps a candidate stage from ever being the reason a memory is unreachable. +Recall that silently varies between runs, or that admits and excludes memories on values nobody keeps true, is the unexplained recall the philosophy forbids: a character that forgets an episode because a stale column excluded it looks like a character that never lived it, and a filter that admits on a blank gives a rationale that is false. +The verdict keeps determinism inspectable; the prefilter rule keeps a candidate stage from being the reason a memory is unreachable, and keeps every stated filter rationale true. ## Implementation Impact From df11e00daa920e8174e585ca7c7b29d8e1bfc480 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 18:50:12 +0900 Subject: [PATCH 31/44] =?UTF-8?q?=F0=9F=93=9D=20Make=20the=20five=20v0.1.6?= =?UTF-8?q?=20decision=20records=20read=20the=20same=20at=20any=20time:=20?= =?UTF-8?q?change-relative=20and=20phase-relative=20wording=20replaced=20b?= =?UTF-8?q?y=20standing=20statements?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../v0-1-6-embedded-vector-recall-plan.md | 2 +- ...dded-qdrant-edge-vector-candidate-store.md | 46 +++++++++---------- ...ness-and-prefilters-never-match-unknown.md | 10 ++-- ...I-0025-vector-record-is-a-read-contract.md | 40 ++++++++-------- ...ctor-baselines-read-the-retrieval-trace.md | 10 ++-- ...blocking-owner-that-flushes-every-write.md | 4 +- 6 files changed, 56 insertions(+), 56 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 4143a842..7f8c58e9 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -31,7 +31,7 @@ ## Assumptions - A1: The embedded engine is `qdrant-edge` pinned exactly at 0.8.0 (beta); its API is guarded by a contract canary, and the pin is bumped only with a re-run of the canary and the parity suite. -- A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison that gates the later embedded-default revisit decision. +- A2: The evaluation repository plans and tracks its own work; this plan consumes two of its outputs only: the trace-sourced baseline's A/B evidence (deferral-reconfirmation row 5) and the cross-mode comparison consumed as closeout evidence and as a revisit trigger for the embedded default. ## Tasks diff --git a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md index 074ad4dd..9e8ae070 100644 --- a/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md +++ b/docs/decisions/implementation/ADR-I-0023-embedded-qdrant-edge-vector-candidate-store.md @@ -10,7 +10,7 @@ warrant: detected_signals: "cross-boundary contract shape with tempting alternatives; rejected alternatives likely to be re-proposed (the two spiked candidates); costly reversal (an engine switch rebuilds every embedded store); premises likely to expire (the engine is beta; the weight measurement is unstripped); deliberately bounded scope (single process, index tuning deferred)" cost_of_violation: "an engine switch after embedded stores exist in the field rebuilds every character's recall index from graph authority and re-embeds it; two adapters with different admission semantics produce different continuity packs from the same memory, which evaluation evidence would attribute to retrieval regressions" cost_of_wrong_preservation: "if the engine's beta API breaks or its footprint proves unacceptable on a target platform and this record is preserved as settled, local deployments carry a dependency that no longer earns its place" - cost_of_over_extension: "treating the embedded mode as validated for multi-process access misrepresents what the library has validated; treating the index knobs as tuned when this phase leaves them at their exact-scan setting would ship approximate recall nobody measured" + cost_of_over_extension: "treating the embedded mode as validated for multi-process access misrepresents what the library has validated; treating the index knobs as tuned while they sit at their untuned exact-scan setting would ship approximate recall nobody measured" depends_on: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md, implementation/ADR-I-0021-embedded-persistent-oxigraph-default.md] implements: [] supersedes: [implementation/ADR-I-0003-qdrant-oxigraph-defaults.md] @@ -22,9 +22,9 @@ supersession_scope: partial # ADR-I-0003's vector-backend default only; its ba ## Context and Problem Statement -After the embedded persistent graph store became the validated default (ADR-I-0021) and retrieval statistics were already file-backed (ADR-I-0009), the vector candidate store was the only component that still required an external service. +With the graph store embedded by default (ADR-I-0021) and retrieval statistics file-backed (ADR-I-0009), the vector candidate store was the one component that required an external service. That conflicts with the intended deployment shapes: desktop companions and game or simulation characters run on end-user machines where a container runtime cannot be assumed, and it keeps a service dependency in the default test path. -ADR-I-0003's own revisit clause, "operating two stores becomes too heavy for target users", was recorded as triggered at the close of the eval-driven family closeout. +ADR-I-0003's own revisit clause, "operating two stores becomes too heavy for target users", was recorded as triggered in the v0.1.5 closeout report. The vector layer is candidate recall only: the vector store suggests, retrieval statistics guide fanout, and graph authority decides final inclusion, so an embedded adapter has a low correctness bar for any single query — it must prefilter and rank candidates well, never be authoritative for anything. The bar that matters is longevity: a character's memory is expected to accumulate continuously for years to decades and to outlive several generations of embedding model, so the embedded recall index must be chosen for where a memory ends up, not for where it starts. Two feasibility spikes were run on 2026-09-02 against the same probe set (build weight, lifecycle and reopen, exactness control, a thirty-way identical-vector tie cohort, five filtered queries against the live service, API shape): the in-process build of the service backend (Qdrant Edge 0.8.0, beta) and a SQLite vector extension (sqlite-vec 0.1.9, stable). @@ -36,20 +36,20 @@ Two feasibility spikes were run on 2026-09-02 against the same probe set (build - The store is a rebuildable cache over graph authority, which bounds the cost of an engine switch but does not eliminate it: every embedded store in the field is rebuilt and re-embedded. - Library over in-house: the library does not own a vector engine; it owns the port contract, the tie-closure loop, the canonical ordering, the verdict mapping, and the error classification, and it holds any engine to those. - The embedded adapter must satisfy the same port contract as the service adapter, proven by a shared parity suite, or the evaluation suite stops being a regression instrument. -- Defaults must match validation evidence (ADR-I-0021's rule): once the default test path runs on the embedded adapter, a service default would repeat the asymmetry that record corrected, so the default and its evidence land in the same change. +- Defaults must match validation evidence (ADR-I-0021's rule): when the default test path runs on the embedded adapter, a service default would repeat the asymmetry that record corrected, so the default and its evidence land in the same change. - Portability across deployment shapes: an engine that can synchronise with the service backend keeps a path from a local character to a hosted one. ## Decision Add an embedded vector candidate store mode behind the existing vector candidate port, implemented on the in-process build of the service backend (Qdrant Edge), selected by a dedicated store-mode setting. -Embedded mode is the default vector mode from the phase that ships it. +Embedded mode is the default vector mode. The service adapter remains fully supported as the service and cloud mode, selected explicitly; this decision adds a mode and deprecates nothing. -The evidence that licenses the default is produced by the same phase: the shared parity suite proves both adapters identical below their indexing thresholds, including identical-vector tie cohorts, and the library's integration suite runs on the embedded adapter without a service, so the shipped default is the validated path (ADR-I-0021's rule). +The evidence that licenses the default is produced by the same change that introduces the mode: the shared parity suite proves both adapters identical below their indexing thresholds, including identical-vector tie cohorts, and the library's integration suite runs on the embedded adapter without a service, so the shipped default is the validated path (ADR-I-0021's rule). The companion evaluation repository's cross-mode run is closeout evidence and a revisit trigger, not a gate: a difference between modes on its continuity suite reopens this record. The embedded store is single-process, matching the embedded graph store's expectation. Exactness is a threshold property, not a promise: below the configured indexing threshold a shard answers by exhaustive scan, above it the index answers, and in both cases the completeness verdict (ADR-I-0024) reports the boundary state of the returned top-K. -This phase ships the threshold at its exact-scan setting; index construction, quantization, and memory-mapped segments become available capabilities whose defaults are tuned on measured corpora in a later decision, never silently. +The threshold ships at its exact-scan setting; index construction, quantization, and memory-mapped segments are available capabilities whose defaults are set by a separate measured decision on real corpora, never silently. The adapter is held to the port contract by the shared library logic it may not re-implement: the tie-closure loop and the canonical constructor (the spike showed identical-vector cohorts stable within a shard and across reopen but not across fresh shards), the filter and payload conventions, the verdict mapping, and the error classification. The engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the type mapping is adapter-specific. @@ -57,25 +57,25 @@ The adapter constructs only object payloads (the engine's point constructor pani A contract canary test, in the pattern of the service client's erased-connect canary, pins the engine facts the adapter relies on — the meaning of the zero indexing threshold, the object-payload precondition, the shard-directory precondition, and the crate-local type provenance — so an upstream change fails a test rather than a character. How the adapter runs the engine inside an async host and what makes a write durable is a separate decision (ADR-I-0027). -Configuration follows the one-key-per-backend pattern the graph and statistics stores already use: a mode setting (`embedded` or `service`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. -The path names a directory; each collection is one engine shard directory inside it, named by the collection name the public constructor already takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has shard directories). +Configuration follows the one-key-per-backend pattern the graph and statistics stores use: a mode setting (`embedded` or `service`) plus a path setting read only in embedded mode, with the service connection string required only in service mode. +The path names a directory; each collection is one engine shard directory inside it, named by the collection name the public constructor takes, so the constructor's collection name is the backend-neutral namespace key in both modes (a server has collections, a directory has shard directories). Embedded mode admits only names that are portable and unique on case-insensitive filesystems (lowercase allowlist and reserved-name rejection, specified in the phase document), records the name and the record schema version in an adapter-owned marker inside the shard directory, and requires the path setting to be present, with a missing path a configuration error rather than an implicit location, exactly as the embedded graph store's path. A reopened shard is validated against the configured embedding model (vector size and distance from the shard's own configuration) and the supported record schema version before any query, failing with the same collection-compatibility error the service adapter raises and, for an unsupported schema version, the clear failure ADR-I-0007 requires. ## Implementation Impact -- A new adapter module implementing the vector candidate port on the embedded engine; the composition root gains a mode switch mirroring the statistics-store switch, defaulting to embedded. -- The library's toolchain pin moves to the minimum the engine compiles on (Rust 1.97 at decision time; the engine rejected the previous 1.95 pin). -- The settings type gains the mode and path keys; the service connection string becomes optional and is validated as present only in service mode. -- The vector database error vocabulary gains an engine-error kind for the embedded backend and reuses the existing filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion, planned in that repository, is a prerequisite to re-pinning its checkout to this wave's merge — not work of this wave. -- The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured; the deterministic test fake is retired in favour of the embedded adapter opened on a temporary directory, which removes the vector-service dependency from the default test path. -- Dependency weight is a recorded deliverable: the unstripped release delta is measured; the stripped delta and the effect of feature trimming are measured and recorded before closeout. +- An adapter module implements the vector candidate port on the embedded engine; the composition root has a mode switch mirroring the statistics-store switch, defaulting to embedded. +- The library's toolchain pin is the minimum the pinned engine compiles on (Rust 1.97 for engine 0.8.0, which does not build on 1.95). +- The settings type carries the mode and path keys; the service connection string is optional and validated as present only in service mode. +- The vector database error vocabulary includes an engine-error kind for the embedded backend beside the filesystem and payload-shape kinds; the vocabulary is closed, so the companion evaluation repository's exhaustive conversion, planned in that repository, is its prerequisite for re-pinning to a library version that carries the embedded backend, not part of the library change. +- The port-conformance parity suite lives in the library's integration tests and runs against the embedded adapter unconditionally and against the service adapter when a service is configured; the embedded adapter opened on a temporary directory replaces the deterministic test fake, so the default test path has no vector-service dependency. +- Dependency weight is a recorded deliverable: the unstripped release delta, the stripped delta, and the effect of feature trimming are measured and recorded with the adapter. - Documentation states the embedded default, the single-process expectation, the threshold semantics, the measured latency guidance, and rebuild-from-graph-authority as the path between modes; the README's backend setup leads with the local path and presents the service as the explicit alternative. ## Considered Options 1. The in-process build of the service backend (Qdrant Edge) as the default, service mode retained. -2. An in-house SQLite exact cosine scan on the `rusqlite` dependency the statistics store already carries. +2. An in-house SQLite exact cosine scan on the `rusqlite` dependency the statistics store carries. 3. The SQLite vector extension (sqlite-vec). 4. A columnar embedded vector database (LanceDB). 5. An in-memory-only embedded store. @@ -85,15 +85,15 @@ A reopened shard is validated against the configured embedding model (vector siz Chosen option: **Option 1**. Measured on the shared probe set: lifecycle and reopen identical; the zero indexing threshold keeps a shard on a plain exhaustive scan while a threshold of one plus an optimise call builds the index; five filtered queries against the live service returned identical id sets and order with a maximum score delta of 0.0; cost of 489 additional dependency-tree lines and about 30.5 MB of unstripped release binary (stripped size unverified, feature trimming untested). -It is the only candidate that offers, in one engine, the capabilities the decade standard makes baseline — approximate indexing, quantization, named vectors, datetime payload indexes, memory-mapped read-only segments — plus synchronisation to the service backend, and it is the same engine family the service adapter already targets, so payload and filter conventions are shared rather than translated. +It is the only candidate that offers, in one engine, the capabilities the decade standard makes baseline — approximate indexing, quantization, named vectors, datetime payload indexes, memory-mapped read-only segments — plus synchronisation to the service backend, and it is the same engine family the service adapter targets, so payload and filter conventions are shared rather than translated. ### Rejected Alternatives Option 2 (in-house exact scan) violates the library-over-in-house driver: the library would own distance computation, blob encoding, and scan scheduling, and every capability the decade standard needs (index, quantization, named vectors) would have to be written or migrated to later; it is rejected outright, not deferred. -Option 3 (sqlite-vec 0.1.9) measured well — builds on both the previous and the new toolchain pin, 19 additional tree lines, about 6.2 MB unstripped, deterministic ties across fresh files and processes, parity five of five with score delta at or below 1.6e-7 — but its stable release is exhaustive-only with approximate indexing existing only in a pre-release, it carries limits on dimensions, result count, and metadata columns, and it is a pre-1.0 binding with a single maintainer; at decade scale its future is a second migration, so it is rejected for this role and reopened only if the chosen engine fails its Revisit When triggers and the extension has shipped a stable approximate index. -Option 4 (LanceDB) was rejected on dependency weight relative to the chosen engine without a spike, since the chosen engine already covers its capabilities; it is reopened only alongside Option 3's reopening. +Option 3 (sqlite-vec 0.1.9) measured well — builds on Rust 1.95 and 1.97, 19 additional tree lines, about 6.2 MB unstripped, deterministic ties across fresh files and processes, parity five of five with score delta at or below 1.6e-7 — but its stable release is exhaustive-only with approximate indexing existing only in a pre-release, it carries limits on dimensions, result count, and metadata columns, and it is a pre-1.0 binding with a single maintainer; at decade scale its future is a second migration, so it is rejected for this role and reopened only if the chosen engine fails its Revisit When triggers and the extension has shipped a stable approximate index. +Option 4 (LanceDB) was rejected on dependency weight relative to the chosen engine without a spike, since the chosen engine covers its capabilities; it is reopened only alongside Option 3's reopening. Option 5 fails restart safety, which the persistent-graph-authority phase made a requirement for every store that survives a process; rejected outright. -Option 6 defers a decision whose deciding evidence this very phase produces: the parity suite and the service-free integration path are phase acceptance, so after the phase the validated path is embedded and a service default would be the unvalidated one, the exact asymmetry ADR-I-0021 corrected; deferral would also leave every consumer-facing document describing a default the evidence no longer supports. Rejected outright; the default is reopened only by the triggers under Revisit When. +Option 6 defers a decision whose deciding evidence the same change produces: the parity suite and the service-free integration path are its acceptance, so with that change merged the validated path is embedded and a service default would be the unvalidated one, the exact asymmetry ADR-I-0021 corrected; deferral would also leave every consumer-facing document describing a default the evidence does not support. Rejected outright; the default is reopened only by the triggers under Revisit When. ## Consequences @@ -102,13 +102,13 @@ Option 6 defers a decision whose deciding evidence this very phase produces: the - Negative / tradeoffs: the engine is beta and its API may change; the canary test and the pinned version turn that into a build-time failure rather than a runtime one. - Negative / tradeoffs: about 30.5 MB of unstripped binary and a higher toolchain floor; the weight deliverable exists to establish the real number. - Negative / tradeoffs: two adapters must be kept in parity for every port change; the parity suite is the cost of that guarantee. -- Negative / tradeoffs: consumers who followed the service-first setup must now set the store path or select service mode explicitly; with no external consumers (Compatibility Policy) no migration hint is carried. +- Negative / tradeoffs: consumers who followed the service-first setup must set the store path or select service mode explicitly; with no external consumers (Compatibility Policy) no migration hint is carried. ## Decision Boundary Invariant: embedded is the default vector mode and the embedded adapter implements the same port contract as the service adapter, proven by the shared parity suite; the store is single-process; the mode is selected by configuration, never inferred from the connection string; tie closure and canonical ordering come from the shared library loop and constructor, never from engine ordering; the indexing threshold ships at its exact-scan setting until a measured decision changes it. -Not covered: the index, quantization, and memory-map tuning values (calibrated through a later measured decision) and the latency guidance numbers (measured and revised through documentation). +Not covered: the index, quantization, and memory-map tuning values (calibrated through a separate measured decision) and the latency guidance numbers (measured and revised through documentation). ## Validation @@ -128,7 +128,7 @@ Not covered: the index, quantization, and memory-map tuning values (calibrated t ## Consultation impact -Question asked: which embedded engine, on the two spikes' evidence; the consult's earlier recommendation of an in-house exact scan was overruled by the decider on the decade-scale portability standard, the settings shape was adopted as recommended, and the consult's proposal to defer the default was overruled because the deciding evidence is produced by the same phase. +Question asked: which embedded engine, on the two spikes' evidence; the consult's initial recommendation of an in-house exact scan was overruled by the decider on the decade-scale portability standard, the settings shape was adopted as recommended, and the consult's proposal to defer the default was overruled because the deciding evidence is produced by the same phase. ## More Information diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md index 969e15ec..3fa8ca12 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md @@ -7,7 +7,7 @@ consulted: ["Claude Fable 5.1"] informed: [] warrant: warranted_by: "without this record, future work would likely add a vector-layer predicate as a three-valued hint filter that matches unknown values, or let an adapter truncate an unclosed equal-score cohort without saying so, because both are the natural first implementation and both have already happened in this repository" - detected_signals: "cross-boundary contract shape (port postcondition) with tempting alternatives; rejected alternative likely to be re-proposed; premises likely to expire (no vector-layer predicate is needed yet, and stored values are not yet guaranteed present)" + detected_signals: "cross-boundary contract shape (port postcondition) with tempting alternatives; rejected alternative likely to be re-proposed; premises likely to expire (no vector-layer predicate has a caller, and stored values are not guaranteed present)" cost_of_violation: "a prefilter that matches unknown values admits stale candidates that graph verification then silently discards, and an unreported open cohort makes top-K membership vary between runs — both surface as unexplained retrieval nondeterminism in evaluation evidence long after the cause is forgotten" cost_of_wrong_preservation: "if the unknown-never-matches rule is preserved after every stored value is guaranteed present and synchronised, adapters carry a defensive arm for a case that cannot occur" cost_of_over_extension: "treating the completeness verdict as an error condition would fail retrieval on a determinism caveat about non-authoritative candidates" @@ -74,7 +74,7 @@ It makes the postcondition expressible by the layer that owns it, distinguishes Option 2 hides a determinism caveat that evaluation evidence later attributes to retrieval; rejected outright. Option 3 loses the exhaustive-versus-closed distinction that tells a caller whether population-level determinacy was achieved; rejected outright. Option 4 fails retrieval on a caveat about non-authoritative candidates that graph authority verifies anyway; rejected outright. -Option 5 recreates a prefilter over values that only the upsert path wrote, which the rule above forbids; a future predicate is admitted the moment its stored value is kept in sync or is immutable. +Option 5 recreates a prefilter over values that only the upsert path wrote, which the rule above forbids; a predicate is admitted when its stored value is kept in sync or is immutable. ## Consequences @@ -102,14 +102,14 @@ Not covered: the current query shape (an embedding, a limit, and an object-type ## Consultation impact -Question asked: whether the deleted hint filters should return for the embedded adapter; ruling adopted the prefilter rule instead. Revised 2026-09-03 on the decider's review: the type shape moved to an appendix and the scope-only query was demoted from rule to current state. +Question asked: whether the deleted hint filters should return for the embedded adapter; ruling adopted the prefilter rule instead. Revised 2026-09-03 on the decider's review: the type shape is an appendix and the scope-only query is recorded as current state, not as a rule. ## More Information -- ADR-I-0022 (tie-cohort closure and canonical ordering, the postcondition this record makes expressible); ADR-I-0023 (the embedded adapter); ADR-I-0025 (the stored record a future predicate would extend); ADR-I-0026 (the evaluation reader of the verdict). +- ADR-I-0022 (tie-cohort closure and canonical ordering, the postcondition this record makes expressible); ADR-I-0023 (the embedded adapter); ADR-I-0025 (the stored record a predicate would extend); ADR-I-0026 (the evaluation reader of the verdict). - Candidate predicates that satisfy the rule, noted for whichever phase needs them and binding on none: a scope id written at upsert and kept in sync by the link and reflection write paths (scoped continuity); an immutable time window over creation and observation time (a time-bounded retrieval route). -## Appendix: reference shape at decision time (non-binding) +## Appendix: reference shape (non-binding) ```rust pub struct VectorCandidateRecall { pub candidates: CanonicalCandidates, pub completeness: VectorRecallCompleteness } diff --git a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md index 397ae90f..0ade46f9 100644 --- a/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md +++ b/docs/decisions/implementation/ADR-I-0025-vector-record-is-a-read-contract.md @@ -23,8 +23,8 @@ supersession_scope: partial ## Context and Problem Statement ADR-I-0005 decided that the vector payload stores filterable metadata and graph pointers, and the payload design note enumerated thirty-four fields with thirty-one of them indexed; the implemented manifest carried thirty-three with thirty indexed after the record-type field was dropped in the structured-verdict phase. -By the time the embedded adapter (ADR-I-0023) was designed, the library read back exactly three of those fields — object id, object type, surface — and the only external reader was the companion evaluation repository's vector-only baseline reading the readable text column. -The relationship hints were frozen at upsert and never updated by the link write path; the lifecycle hints described vectors the correction and forgetting paths delete; the readable text column duplicated graph text with a prefix removed; and every field was about to be mirrored into a second physical schema. +When the embedded adapter (ADR-I-0023) was designed, the library read back exactly three of those fields — object id, object type, surface — and the only external reader was the companion evaluation repository's vector-only baseline reading the readable text column. +The relationship hints were frozen at upsert and never updated by the link write path; the lifecycle hints described vectors the correction and forgetting paths delete; the readable text column duplicated graph text with a prefix removed; and the embedded adapter would have mirrored every field into a second physical schema. ADR-I-0002's implementation note said to "persist both `embedding_text` and `content_text` where useful", which left the two text columns' meanings undefined. The forward-looking case for each family was analysed against the planned phases (scoped continuity, factual rigor and temporal validity, retrieval observability, associative recall, assisted remember, multimodal) before deciding. @@ -32,8 +32,8 @@ The forward-looking case for each family was analysed against the planned phases - A column earns its place when a reader exists; carrying it unread costs two adapter mappings, index creation per collection, a parity fixture, and the sync discipline ADR-I-0005 named in its own tradeoffs. - Prefilter hints are only safe on immutable or synchronised data; the relationship, lifecycle, ranking, and mutable time hints were none of those. -- Re-adding an immutable column later is a backfill from graph authority, not a re-index. -- Once embedding surfaces are generated or caller-supplied (the assisted-remember phase; the write plan already carries a caller-supplied surface), the text a vector embeds is no longer re-derivable from graph authority, so it is provenance in the philosophy's sense. +- Re-adding an immutable column is a backfill from graph authority, not a re-index. +- When embedding surfaces are generated or caller-supplied (the assisted-remember phase; the write plan carries a caller-supplied surface), the text a vector embeds is not re-derivable from graph authority, so it is provenance in the philosophy's sense. - Read-out text is graph authority's job; the vector layer suggests, it does not describe. ## Decision @@ -45,23 +45,23 @@ Read-out text lives in graph authority. The vector record stores only the embedded surface, as provenance of what was ranked. Consumers needing candidate content hydrate by object id. -`content_text` is dropped. -The relationship refs (episode, observation, thread, entity, participant, speaker, supersedes), the lifecycle and currentness flags, the time hints, the ranking and salience hints, the object-specific hints, the graph URI, and the raw source reference leave the vector write path. +There is no readable-text column (`content_text`). +The relationship refs (episode, observation, thread, entity, participant, speaker, supersedes), the lifecycle and currentness flags, the time hints, the ranking and salience hints, the object-specific hints, the graph URI, and the raw source reference are not part of the vector record. Dropping the graph URI partially supersedes ADR-I-0001's clause that every vector payload carries it: the stable object id remains the cross-store identity and the graph URI is derived from it by graph authority, so the pointer was a redundant copy of the id; ADR-I-0001's stable-id decision itself is unchanged. -The typed field manifest introduced in the structured-verdict phase remains the single source of both adapters' column sets and shrinks to the five entries. +The typed field manifest is the single source of both adapters' column sets and holds exactly the five entries. ADR-I-0024 rules that a predicate reads only synchronised or immutable values and notes the two candidate predicates (a synchronised scope id; an immutable time window over `created_at` and `observed_at` backfilled from graph authority), so a returning column arrives with its predicate and its reader. ## Implementation Impact -- The vector record type and the surface builders lose the hint carriers; the payload map serialises five fields for both adapters, which share the engine family's payload conventions. -- The service adapter stops creating per-field payload indexes for dropped fields. -- The companion evaluation repository's vector-only baseline stops reading the readable text column and sources item text from its own ingest records (ADR-I-0026). -- The payload design note's field categories and indexing policy are superseded by this record and carry a supersession note. +- The vector record type and the surface builders carry no hint fields; the payload map serialises five fields for both adapters, which share the engine family's payload conventions. +- The service adapter creates payload indexes only for object id and object type. +- The companion evaluation repository's vector-only baseline sources item text from its own ingest records, not from a payload column (ADR-I-0026). +- The payload design note's field categories and indexing policy are superseded by this record and carry a supersession note saying so. - No migration: under the Compatibility Policy, existing stores are rebuilt from graph authority. ## Considered Options -1. Five-column read contract; keep `embedding_text` only; drop the hint families with named re-entry paths. +1. Five-column read contract; keep `embedding_text` only; no hint families, with the candidate predicates noted in ADR-I-0024. 2. Keep both text columns. 3. Drop both text columns. 4. Keep the unread hints for the planned phases. @@ -70,24 +70,24 @@ ADR-I-0024 rules that a predicate reads only synchronised or immutable values an ## Decision Outcome Chosen option: **Option 1**. -It stores what is read, keeps the one column that becomes non-re-derivable, and prices re-entry honestly. +It stores what is read, keeps the one column that ceases to be re-derivable when surfaces are generated, and prices a returning column honestly. ### Rejected Alternatives -Option 2: `content_text` is a deterministic function of graph object fields at every surface builder, so it never carries information graph authority lacks, and its one reader moves to its own ingest records; rejected outright. -Option 3: `embedding_text` is cheap and becomes the only record of what a generated or caller-supplied vector embeds; rejected outright. -Option 4: no planned phase names a vector-layer predicate the existing fields could serve without new synchronisation work — scoped retrieval needs a scope id kept in sync by linking, temporal validity is a ranking property of new claim objects, salience evolves by reinforcement, and lifecycle hints describe vectors the write path deletes; reopened only through the re-entry paths. +Option 2: `content_text` is a deterministic function of graph object fields at every surface builder, so it never carries information graph authority lacks, and its one reader has its own ingest records; rejected outright. +Option 3: `embedding_text` is cheap and is the only record of what a generated or caller-supplied vector embeds; rejected outright. +Option 4: no planned phase names a vector-layer predicate the existing fields could serve without new synchronisation work — scoped retrieval needs a scope id kept in sync by linking, temporal validity is a ranking property of new claim objects, salience evolves by reinforcement, and lifecycle hints describe vectors the write path deletes; reopened only under ADR-I-0024's prefilter rule. Option 5 is the only subset with a forward-looking case that survives the synchronisation test; it was declined because no phase document asks for the predicate and the columns backfill cheaply when one does. ## Consequences - Positive: both adapters mirror one five-field manifest; the embedded shard stores those fields as payload beside the vector with keyword indexes on object id (the delete selector) and object type (the scope predicate); ADR-I-0023 owns the physical layout. -- Positive: the embedded surface is preserved as vector provenance before surfaces become generated. -- Negative / tradeoffs: a future scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the re-entry paths make that step predictable. +- Positive: the embedded surface is preserved as vector provenance, which matters once surfaces are generated rather than derived. +- Negative / tradeoffs: a scoped or time-bounded prefilter requires a backfill and a schema-version step rather than a query-only change; the candidate predicates noted in ADR-I-0024 make that step predictable. ## Decision Boundary -Invariant: the vector record carries only fields a reader consumes, plus the embedded surface as provenance and the schema version ADR-I-0007 requires; readable content is hydrated from graph authority by object id; a returning hint arrives with its predicate and parity fixture through ADR-I-0024's re-entry paths. +Invariant: the vector record carries only fields a reader consumes, plus the embedded surface as provenance and the schema version ADR-I-0007 requires; readable content is hydrated from graph authority by object id; a returning hint arrives with its predicate and parity fixture under ADR-I-0024's prefilter rule. Not covered: the physical encoding of each column per adapter, and graph authority's own denormalised fields. @@ -105,7 +105,7 @@ Not covered: the physical encoding of each column per adapter, and graph authori ## Consultation impact -Question asked: whether the unread hint families and the readable text column should be kept for planned phases; ruling adopted the five-column contract with the three governing sentences and the named re-entry paths. +Question asked: whether the unread hint families and the readable text column should be kept for planned phases; ruling adopted the five-column contract with the three governing sentences and the noted candidate predicates. ## More Information diff --git a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md index c63e0c3c..cedd0578 100644 --- a/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md +++ b/docs/decisions/implementation/ADR-I-0026-raw-vector-baselines-read-the-retrieval-trace.md @@ -23,7 +23,7 @@ supersession_scope: null ## Context and Problem Statement The companion evaluation repository (a development aid, not core library functionality) runs a vector-only baseline: ingest through the library, then rank by plain vector similarity to measure what hybrid retrieval adds. -As built, that baseline held its own client to the vector service, ran one filtered search per object kind against the library's collection, read three payload fields by hard-coded name, re-implemented best-score-per-object deduplication and score ordering, and took item text from a payload column. +Before this record, that baseline held its own client to the vector service, ran one filtered search per object kind against the library's collection, read three payload fields by hard-coded name, re-implemented best-score-per-object deduplication and score ordering, and took item text from a payload column. That is a hidden capability: the baseline depended on an adapter-private schema, duplicated ordering the library owns, and could not run at all against an embedded store (ADR-I-0023). The question is what capability the library must expose so the baseline stops reaching into a store. @@ -31,7 +31,7 @@ The question is what capability the library must expose so the baseline stops re - Evaluation tooling must not grow library surface that no product use case has demanded (ADR-I-0020's driver). - The library is not a vector-database abstraction, and vector-only candidates must never become behavior-influencing memory without graph verification (project philosophy; the persistent-graph-authority phase's acceptance criteria). -- The retrieval trace already is the raw vector recall: the canonical, pre-verification top-K with object reference, surface, score, and rank, scoped by the configured object types and sized by the candidate limit, and ADR-I-0024 adds the completeness verdict, which says either that the scoped population was scored exhaustively (top-K determinate over the population) or, for an indexed answer, only whether the cutoff cohort within the index's returned prefix was closed. +- The retrieval trace is the raw vector recall: the canonical, pre-verification top-K with object reference, surface, score, and rank, scoped by the configured object types and sized by the candidate limit, and ADR-I-0024 adds the completeness verdict, which says either that the scoped population was scored exhaustively (top-K determinate over the population) or, for an indexed answer, only whether the cutoff cohort within the index's returned prefix was closed. - Every store's physical schema is adapter-private; two adapters must not create two baseline implementations. ## Decision @@ -40,7 +40,7 @@ The library exposes no raw candidate-search surface and no facade change. The evaluation repository's vector-only baseline issues one ordinary `retrieve` with tracing enabled per measured object kind, each with a singleton object-type scope, and reads each retrieval's completeness verdict from telemetry; a single mixed-kind top-K is not used, because a global cutoff can exclude an underrepresented kind's valid candidates without any open verdict. The trace's vector candidates are object-and-surface pairs recorded before the pipeline's object-level deduplication, so the baseline's candidate limit per kind is that kind's section budget multiplied by the maximum number of embedding surfaces one object of that kind can have (a constant the library publishes as public policy per object kind, alongside the surface policy that defines it, so an ordinary caller never duplicates private policy and a new surface changes the published value in the same change; one for every kind at the time of this record), and the baseline deduplicates by object keeping the best-scoring surface and truncates to the budget. That limit is sufficient by construction: an object ranked within the budget by its best surface has that surface inside the surface top-K of budget times surfaces, because the surfaces above it belong to fewer than the budget's worth of objects; therefore an exhaustive surface-level verdict at that limit makes the object-level top-budget determinate over the scoped population, while a closed boundary verdict makes it determinate only relative to the index's returned prefix (an approximate index may have omitted records before tie closure), and the baseline records which of the two it obtained; the contract is pinned by a fixture whose objects carry every surface. -Item text comes from the evaluation repository's own ingest records, keyed by the external identity it already reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). +Item text comes from the evaluation repository's own ingest records, keyed by the external identity it reverse-maps, never from a store payload (ADR-I-0025's third sentence: consumers needing candidate content hydrate by object id). What this record asks of the evaluation repository, recorded as the library-facing contract and nothing more: @@ -57,7 +57,7 @@ Keeping the baseline inside the traced retrieval path means the measurement of " ## Implementation Impact - Library: ADR-I-0024's telemetry field plus one published policy value, the maximum number of embedding surfaces per object kind, exported beside the surface policy that defines it; no candidate-search facade, so the acceptance criterion "no public facade change beyond the telemetry field and this policy value" holds. -- Evaluation repository: its baseline moves onto the trace under its own plan; nothing in this repository depends on how. +- Evaluation repository: its baseline reads the trace under its own plan; nothing in this repository depends on how. ## Considered Options @@ -92,7 +92,7 @@ Not covered: any headroom the baseline adds to a kind's limit, the shape of the ## Validation - An A/B run of the vector-only configuration (direct search versus trace-derived) shows identical item identities and ranks per question before the direct path is deleted. -- After the switch, the evaluation adapter contains no search call against the vector service and no payload field constant. +- The evaluation adapter contains no search call against the vector service and no payload field constant. - The baseline runs unchanged in embedded mode. ## Revisit When diff --git a/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md index 5d9c85b9..0b590e8f 100644 --- a/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md +++ b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md @@ -29,7 +29,7 @@ An adapter that relies on the final drop for persistence therefore loses every u ## Decision Drivers -- No engine call may occupy an async executor thread; the library's other embedded stores already hold this line. +- No engine call may occupy an async executor thread; the library's other embedded stores hold this line. - A write the library has acknowledged must survive a process exit that skips orderly shutdown; a character that forgets after a crash violates continuity silently. - No port or facade method is added for shutdown; the existing facade drop remains the only close path. - The rule must be pinned to measured engine behaviour so a change in the engine reopens it rather than silently voiding it. @@ -93,7 +93,7 @@ Not covered: the channel and thread mechanics, the backoff bound, and the batchi ## Consultation impact -Question asked (review round 22): whether the shard's drop-time flush is persistence or compaction; a process-level probe settled it as persistence, and the decider's rule "await the flush or document the weaker guarantee" was met by flushing per write. +Question asked in review: whether the shard's drop-time flush is persistence or compaction; a process-level probe settled it as persistence, and the decider's rule "await the flush or document the weaker guarantee" was met by flushing per write. ## More Information From 22d5a5a9c7218046bfcabf09987fd2f75b06980e Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 18:51:30 +0900 Subject: [PATCH 32/44] =?UTF-8?q?=F0=9F=93=9D=20Decision=20log:=20the=20ph?= =?UTF-8?q?ase=20merges=20as=20one=20change;=20waves=20land=20against=20th?= =?UTF-8?q?e=20planning=20branch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 7f8c58e9..df3ccc68 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -259,6 +259,7 @@ Append-only editing rule (applies to both logs below): when appending an entry, - ADR-I-0023 now decides the default: embedded is the default vector mode from this phase, licensed by the phase's own parity suite and service-free integration path (defaults-match-evidence, ADR-I-0021); ADR-I-0003's vector default is partially superseded; the evaluation repository's cross-mode run becomes a revisit trigger, not a gate. Task_4's default-mode test asserts embedded; Task_8's documentation leads with the local path. - The blocking-owner and per-write-flush rules moved out of ADR-I-0023 into ADR-I-0027 as their own decision. - ADR-I-0024 narrowed to the completeness verdict and the prefilter rule (unknown never matches); the scope-only query is a current state, the type shape is a non-binding appendix, and the two predicate paths are notes binding on no phase. +- 2026-09-03 — Merge shape: the records and plan are not merged ahead of the implementation, because ADR-I-0023's embedded default rests on evidence the implementation produces. Implementation waves land as PRs against the planning branch (`plan/v0-1-6-embedded-vector-recall`), which stays the readable home of the records; the phase merges to main as one change once the decision is solidified by its evidence. ## Notes - Risks: the row/summary schema move in the evaluation repository (typed backend identity) is a clean break under its compatibility policy and must not touch sealed evidence; the latency guidance and the stripped dependency weight must be measured, not assumed; the engine is beta, so its pin is exact and its bump is gated by the canary. From 514bea563016dafd293e3dbb7ad1ff79d51d3c23 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 19:01:02 +0900 Subject: [PATCH 33/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?6:=20canary=20pins=20pre-flush=20non-persistence=20in=20the=20p?= =?UTF-8?q?lan=20and=20phase=20document,=20documentation=20ownership=20cor?= =?UTF-8?q?rected=20to=20Task=5F4,=20and=20ADR-I-0024=20requires=20a=20ful?= =?UTF-8?q?ly=20populated=20column=20before=20a=20predicate=20is=20enabled?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 ++-- ...rts-completeness-and-prefilters-never-match-unknown.md | 8 ++++---- .../v0_1_6_embedded_vector_candidate_recall.md | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index df3ccc68..aa527ad2 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -155,7 +155,7 @@ - Embedded mode constructs and retrieves with no service running; the parity suite yields identical admitted sets and orderings on the shared fixtures while both adapters are below their indexing thresholds; the tie fixture yields Exhaustive (embedded) and BoundaryTieClosed (service), both through the shared tie-closure loop, with no engine ordering relied on. - A recall comparison of the embedded adapter above its indexing threshold against its exhaustive setting is recorded on the benchmark corpus (informational this phase; index tuning is a later decision). - The collection name is validated to the phase document's allowlist before any directory is touched, and a path-confinement test proves separator and parent-directory inputs cannot escape the configured directory. - - ADR-I-0027 holds: every engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count, index build, shutdown) runs on the adapter's dedicated blocking owner, which opens the shard itself; a write is acknowledged only after the engine's flush; the facade drop only signals the owner; a constructor meeting a locked directory waits with a bounded backoff; the close-then-reopen test, the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write), and the responsiveness benchmark (construction and reopen with a lock-backoff wait, a concurrent scan, a write burst, a build, a close) pass; the contract canary pins no-replay-on-load and the directory lock. + - ADR-I-0027 holds: every engine call (shard open and load with its lock backoff, payload index creation, upsert, delete, search, the filtered scope count, index build, shutdown) runs on the adapter's dedicated blocking owner, which opens the shard itself; a write is acknowledged only after the engine's flush; the facade drop only signals the owner; a constructor meeting a locked directory waits with a bounded backoff; the close-then-reopen test, the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write), and the responsiveness benchmark (construction and reopen with a lock-backoff wait, a concurrent scan, a write burst, a build, a close) pass; the contract canary pins the three engine facts ADR-I-0027 rests on: no persistence before flush (a probe that writes, skips flush and drop, and reopens empty), no log replay on load, and the directory lock. - Restart test passes; repeated runs are byte-identical; reopening a shard with a mismatched vector size or distance raises the collection-compatibility error, and reopening one whose marker carries an unsupported record schema version raises the clear failure ADR-I-0007 requires, each covered by its own test. - Embedded mode with no `VECTOR_STORE_PATH` is a configuration error at construction, never an implicit default; covered by a settings test. - The contract canary passes on the pinned engine version and is documented as the gate for every engine bump. @@ -256,7 +256,7 @@ Append-only editing rule (applies to both logs below): when appending an entry, - Ruling: the blocking owner flushes after every write and acknowledges only then; the signal-only facade drop stays; no port or facade method is added; a hard-exit test joins Task_4's acceptance. - Tradeoff: one synchronous disk sync per write, measured by the write burst in the benchmark; batching behind an acknowledgement is the named upgrade. - 2026-09-03 — Decision records restructured on the decider's review. - - ADR-I-0023 now decides the default: embedded is the default vector mode from this phase, licensed by the phase's own parity suite and service-free integration path (defaults-match-evidence, ADR-I-0021); ADR-I-0003's vector default is partially superseded; the evaluation repository's cross-mode run becomes a revisit trigger, not a gate. Task_4's default-mode test asserts embedded; Task_8's documentation leads with the local path. + - ADR-I-0023 now decides the default: embedded is the default vector mode from this phase, licensed by the phase's own parity suite and service-free integration path (defaults-match-evidence, ADR-I-0021); ADR-I-0003's vector default is partially superseded; the evaluation repository's cross-mode run becomes a revisit trigger, not a gate. Task_4's default-mode test asserts embedded and its README and phase-document deliverables lead with the local path. - The blocking-owner and per-write-flush rules moved out of ADR-I-0023 into ADR-I-0027 as their own decision. - ADR-I-0024 narrowed to the completeness verdict and the prefilter rule (unknown never matches); the scope-only query is a current state, the type shape is a non-binding appendix, and the two predicate paths are notes binding on no phase. - 2026-09-03 — Merge shape: the records and plan are not merged ahead of the implementation, because ADR-I-0023's embedded default rests on evidence the implementation produces. Implementation waves land as PRs against the planning branch (`plan/v0-1-6-embedded-vector-recall`), which stays the readable home of the records; the phase merges to main as one change once the decision is solidified by its evidence. diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md index 3fa8ca12..53018e7c 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md @@ -42,8 +42,8 @@ The verdict distinguishes four situations: no search was issued because the limi Adapters state the verdict truthfully: exhaustive only when the adapter knows the shard is unindexed and the cutoff cohort was closed, with the scanned count taken from the scope rather than the rows returned; an exhaustive scan whose cohort stays open at the bound reports open. The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. -A vector-layer predicate may be evaluated only over stored values that are immutable or synchronised on every mutation, and an unknown or missing stored value never satisfies a positive predicate. -A predicate that needs a stored value the write paths do not keep current is not a prefilter; it is a graph-authority question. +A vector-layer predicate may be evaluated only over a column that is fully populated for every searchable record (backfilled from graph authority before the predicate is enabled, per ADR-I-0025) and that is immutable or synchronised on every mutation; under those two conditions a missing or unknown value is a defect, not a state, and it never satisfies a positive predicate, so the rule produces no false negative on a correctly populated column and turns an incorrectly populated one into a visible failure rather than a silent widening. +A predicate that needs a stored value the write paths do not keep current, or that is not populated for every record, is not a prefilter; it is a graph-authority question. ## Character Memory Relevance @@ -79,12 +79,12 @@ Option 5 recreates a prefilter over values that only the upsert path wrote, whic ## Consequences - Positive: top-K determinism is observable per retrieval and per adapter. -- Positive: any future prefilter has one admission test — is the value it reads always current — instead of a case-by-case argument. +- Positive: any future prefilter has one admission test — is the column populated for every record and always current — instead of a case-by-case argument. - Negative / tradeoffs: callers wanting a scoped or time-bounded semantic search wait for a synchronised or immutable column rather than filtering on what happens to be stored. ## Decision Boundary -Invariant: the search result carries a completeness verdict stated truthfully by the adapter, and the pipeline never repairs, retries, or fails on it; a vector-layer predicate reads only immutable or synchronised values, and an unknown value never satisfies a positive predicate. +Invariant: the search result carries a completeness verdict stated truthfully by the adapter, and the pipeline never repairs, retries, or fails on it; a vector-layer predicate reads only a fully populated column that is immutable or synchronised, and an unknown value never satisfies a positive predicate. Not covered: the current query shape (an embedding, a limit, and an object-type scope, with an empty scope selecting zero — a current state, not a rule), the verdict's type and wire shape (the appendix is a reference, not a contract), the telemetry field name, and the service adapter's overfetch bound. diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 5f10e6ac..19ec8532 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -59,7 +59,7 @@ Blocking discipline and durability (ADR-I-0027): every engine call — shard ope A shard directory stays locked while an owner holds it, so a constructor that meets a locked directory waits with a bounded backoff; the two engine facts are pinned by the contract canary and proven by the close-then-reopen test and the hard-exit test (exit without dropping the shard, reopen from a second process, find every acknowledged write). The in-phase benchmark records executor responsiveness while a scan, a write burst, a build, and a close are in progress. Type mapping: the engine's point, filter, condition, and scored-point types are crate-local engine types, not the service client's protocol types, so the conversion is adapter-specific; payload and filter conventions, the tie-closure loop, the verdict mapping, and the error classification are shared library logic that neither adapter re-implements. -Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. +Contract canary: a test in the pattern of the service client's erased-connect canary pins the engine facts the adapter relies on — zero threshold means unindexed, object-payload precondition, shard-directory precondition, crate-local type provenance, and the three facts ADR-I-0027 rests on: no persistence before flush, no log replay on load, and the directory lock — so an upstream change fails a test rather than a character; the pin is bumped only with a canary and parity re-run. Score parity across adapters was measured at 0.0 delta on the spike; the parity suite still asserts it with non-unit query and record vectors rather than assuming it. ### Settings and composition (ADR-I-0023) From 013a6d9cd6917d5b66c2b676b845d41b8f9846ef Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 19:06:57 +0900 Subject: [PATCH 34/44] =?UTF-8?q?=F0=9F=93=9D=20Review=20fixes=20round=202?= =?UTF-8?q?7:=20zero-norm=20rules=20restored=20to=20ADR-I-0024=20and=20car?= =?UTF-8?q?ved=20out=20of=20the=20service-mode=20non-goal;=20directory=20l?= =?UTF-8?q?ock=20pinned=20throughout=20ADR-I-0027?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 6 +++--- ...orts-completeness-and-prefilters-never-match-unknown.md | 4 +++- ...ne-runs-on-a-blocking-owner-that-flushes-every-write.md | 7 ++++--- .../v0_1_6_embedded_vector_candidate_recall.md | 4 ++-- docs/roadmap/development_roadmap.md | 4 ++-- 5 files changed, 14 insertions(+), 11 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index aa527ad2..117a1640 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -17,7 +17,7 @@ ## Scope / Non-goals - Scope: the phase document's deliverables and deletions, all in this repository. -- Non-goals: the phase document's non-goals (no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode for non-empty scopes; ADR-I-0024's empty-scope change, zero candidates for an empty scope and boundary rejection of an empty configured scope, is an intended change and in scope). +- Non-goals: the phase document's non-goals (no index tuning beyond the exact-scan threshold, no migration tooling, no multi-process embedded access, no public candidate-search facade, no retrieval semantics change in service mode for non-empty scopes and non-degenerate queries; ADR-I-0024's empty-scope change, zero candidates for an empty scope and boundary rejection of an empty configured scope, is an intended change and in scope). ## Context (workspace) - Design memo and audits: `.agent-work/orchestrator/` (v016-port-design-consult.md sections A-G; cm-design-audit.md; cme-design-audit.md; v016-consolidated-triage.md) and the researcher censuses under `.agent-work/researcher/` and the evaluation repository's `.agent-work/evals-researcher/`; all transient, consumed into this plan and the ADRs. @@ -80,7 +80,7 @@ Introduce the result envelope (canonical candidates plus the typed completeness verdict) and the verdict enum in the public retrieval telemetry vocabulary; extract the service adapter's private tie-closure loop (fetch decision, fetch bound, cohort closure, canonical construction) into `src/adapters/qdrant/tie_closure.rs` as crate-visible shared logic that takes an engine-neutral fetch callback, so the embedded adapter (Task_4) calls it rather than re-implementing it; make the service adapter map the shared fetch decision onto the verdict; make the query scope-only with empty-scope-selects-zero and boundary rejection of an empty configured object-type set; record the verdict in retrieval telemetry beside the returned count; update every fake store. No repair, retry, or failure on the verdict. - acceptance: - The envelope and verdict express the four situations in ADR-I-0024's Decision section (its appendix shape is a non-binding reference); the canonical-candidates newtype is unchanged. - - Query-side zero-norm rule implemented in the service adapter: a zero-norm query scores every candidate zero and returns a truthful verdict, with a unit test and a parity fixture that Task_4 inherits. + - Query-side zero-norm rule (ADR-I-0024) implemented in the service adapter: a zero-norm query scores every candidate zero and returns a truthful verdict, with a unit test and a parity fixture that Task_4 inherits. - Telemetry carries the verdict for every retrieval; a retrieval test asserts each variant. - The tie-closure loop lives in `src/adapters/qdrant/tie_closure.rs`, the service adapter calls it, and its existing unit tests (fetch decision, all-tied cohort at the bound) move with it; nothing in `store.rs` closes a cohort on its own. - Fetch-decision unit tests assert closed and open verdicts including the all-tied cohort at the bound. @@ -122,7 +122,7 @@ - The manifest test asserts exactly five entries; both text-column producers except `embedding_text` are gone. - Zero-hit census across both repositories for the dropped fields and for `content_text` readers (the evaluation repository removes its reader under its own plan; its zero-hit census is consumed as closeout evidence, not ordered here). - One token mapping per enum; census shows no copy in adapters or use cases. - - A zero-norm record embedding yields a typed per-record indexing failure and never reaches an adapter; unit test present. + - A zero-norm record embedding yields a typed per-record indexing failure and never reaches an adapter (ADR-I-0024); unit test present. - validation: - kind: command required: true diff --git a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md index 53018e7c..e7e4df52 100644 --- a/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md +++ b/docs/decisions/implementation/ADR-I-0024-vector-candidate-recall-reports-completeness-and-prefilters-never-match-unknown.md @@ -41,6 +41,7 @@ The search result carries, beside the canonical candidates, a completeness verdi The verdict distinguishes four situations: no search was issued because the limit was zero or the scope was empty; every stored record in scope was scored, so the requested top-K is determinate over the population; an index answered with a prefix whose cutoff cohort was closed, so the returned set is determinate for that index state although an approximate index may have omitted records it never surfaced; and the overfetch bound was reached with the cutoff cohort still open, so membership may vary. Adapters state the verdict truthfully: exhaustive only when the adapter knows the shard is unindexed and the cutoff cohort was closed, with the scanned count taken from the scope rather than the rows returned; an exhaustive scan whose cohort stays open at the bound reports open. The retrieval pipeline records the verdict in retrieval telemetry beside the returned candidate count and never repairs, retries, or fails on it. +Degenerate vectors are defined on both sides of the port so that neither adapter has an undefined path: a zero-norm record embedding is rejected at indexing as a typed per-record failure before any adapter sees it, and a zero-norm query scores every candidate zero and reports a truthful verdict. A vector-layer predicate may be evaluated only over a column that is fully populated for every searchable record (backfilled from graph authority before the predicate is enabled, per ADR-I-0025) and that is immutable or synchronised on every mutation; under those two conditions a missing or unknown value is a defect, not a state, and it never satisfies a positive predicate, so the rule produces no false negative on a correctly populated column and turns an incorrectly populated one into a visible failure rather than a silent widening. A predicate that needs a stored value the write paths do not keep current, or that is not populated for every record, is not a prefilter; it is a graph-authority question. @@ -86,7 +87,7 @@ Option 5 recreates a prefilter over values that only the upsert path wrote, whic Invariant: the search result carries a completeness verdict stated truthfully by the adapter, and the pipeline never repairs, retries, or fails on it; a vector-layer predicate reads only a fully populated column that is immutable or synchronised, and an unknown value never satisfies a positive predicate. -Not covered: the current query shape (an embedding, a limit, and an object-type scope, with an empty scope selecting zero — a current state, not a rule), the verdict's type and wire shape (the appendix is a reference, not a contract), the telemetry field name, and the service adapter's overfetch bound. +Not covered: the current query shape (an embedding, a limit, and an object-type scope, with an empty scope selecting zero — a current state, not a rule; the zero-norm rules above are the one query-shape behaviour this record fixes), the verdict's type and wire shape (the appendix is a reference, not a contract), the telemetry field name, and the service adapter's overfetch bound. ## Validation @@ -94,6 +95,7 @@ Not covered: the current query shape (an embedding, a limit, and an object-type - A retrieval test asserts the telemetry verdict for each situation using the fakes. - The parity suite asserts exhaustive for the embedded adapter below its indexing threshold and closed for the service adapter on the identical-vector tie fixture. - A census of the vector adapters shows no match-or-unknown condition. +- Parity fixtures cover the zero-norm query and the rejected zero-norm record in both adapters. ## Revisit When diff --git a/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md index 0b590e8f..ea7c8dbd 100644 --- a/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md +++ b/docs/decisions/implementation/ADR-I-0027-embedded-vector-engine-runs-on-a-blocking-owner-that-flushes-every-write.md @@ -41,7 +41,7 @@ The async composition entry point never touches the engine; it only hands work t The owner acknowledges an upsert or delete only after the engine's flush has completed, so every acknowledged write is durable independently of the shard's final drop. Dropping the adapter through the existing facade drop only signals the owner; the shard's final drop happens on the owner's thread, and a process exit that pre-empts it loses nothing acknowledged. A shard directory stays locked while an owner holds it; a constructor that meets a locked directory waits with a bounded backoff for the previous owner to release it rather than failing or opening a second handle. -The contract canary (ADR-I-0023) additionally pins the two engine facts this record rests on: the engine does not persist a write until its flush runs, and a load does not replay the log. +The contract canary (ADR-I-0023) additionally pins the three engine facts this record rests on: the engine does not persist a write until its flush runs, a load does not replay the log, and a shard directory held by one owner refuses a second open until it is released. ## Implementation Impact @@ -74,7 +74,7 @@ Option 4 adds a close method every consumer must remember to call and still lose ## Decision Boundary -Invariant: every engine call runs on the adapter's blocking owner; a write is acknowledged only after it is durable; the facade drop stays signal-only; the two engine facts are pinned by the canary. +Invariant: every engine call runs on the adapter's blocking owner; a write is acknowledged only after it is durable; the facade drop stays signal-only; a constructor waits for a locked directory rather than opening a second handle; the three engine facts are pinned by the canary. Not covered: the channel and thread mechanics, the backoff bound, and the batching of flushes behind an acknowledgement if measurement calls for it. @@ -83,12 +83,13 @@ Not covered: the channel and thread mechanics, the backoff bound, and the batchi - The hard-exit test writes, exits the process without dropping the shard, reopens the directory from a second process, and finds every acknowledged write. - The close-then-reopen test drops the facade inside an async runtime, reopens the same directory immediately, and finds every write. - The benchmark shows no engine call occupying an async executor thread and observes the shard's final drop on the owner's thread. -- The contract canary fails if the pinned engine starts replaying its log on load or persisting on write. +- The contract canary fails if the pinned engine starts replaying its log on load, persisting on write, or admitting a second open of a held shard directory. ## Revisit When - The engine persists on write or replays its log on load (the canary fails in that direction) — the per-write flush becomes optional and this record is revised. - The write-burst measurement shows the per-write flush dominating ingestion cost — batch flushes behind an explicit acknowledgement rather than weakening the durability rule. +- The engine's directory lock changes semantics (the canary fails in that direction) — the constructor's wait-for-release rule is re-derived before the pin moves. - A multi-process deployment shape is designed — the single-owner lock discipline is reconsidered with the graph and statistics stores, never alone. ## Consultation impact diff --git a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md index 19ec8532..cac763fa 100644 --- a/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md +++ b/docs/design/roadmap-phases/v0_1_6_embedded_vector_candidate_recall.md @@ -115,7 +115,7 @@ the port doc comment's "documented bounded-overfetch degradation policy" clause, ## Non-goals ```text -changing the authority split, or any retrieval semantics for non-empty scopes (the empty-scope change in ADR-I-0024 is intended and in scope) +changing the authority split, or any retrieval semantics for non-empty scopes and non-degenerate queries (the empty-scope and zero-norm rules in ADR-I-0024 are intended and in scope) deprecating the service adapter, or altering it beyond what the shared port and record contracts require tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold, tuned by a later measured decision) named-vector coexistence of two embedding spaces (an engine capability this decision was taken for; its use lands with the first embedding-model migration) @@ -152,7 +152,7 @@ The dependency-weight report records unstripped and stripped release deltas and The default test path requires no vector service; service-gated suites continue to pass unchanged. Both adapters persist exactly the five-field read contract; a census of both repositories shows no reader of a dropped field. Documentation states the single-process expectation, the threshold semantics, the latency guidance, and the rebuild-from-authority path. -No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behaviour change in service mode for non-empty scopes (an empty object-type scope now selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary, both intended). +No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behaviour change in service mode for non-empty scopes and non-degenerate queries (an empty object-type scope now selects zero instead of searching unfiltered, and an empty configured scope is rejected at the boundary, both intended). ``` ## What the evaluation repository provides and when it is used (ADR-I-0026) diff --git a/docs/roadmap/development_roadmap.md b/docs/roadmap/development_roadmap.md index b4c082c3..00525f49 100644 --- a/docs/roadmap/development_roadmap.md +++ b/docs/roadmap/development_roadmap.md @@ -1222,7 +1222,7 @@ rule that a vector-layer predicate reads only synchronized or immutable values a ## Non-goals ```text -changing the authority split, or any retrieval semantics in the service mode for non-empty scopes (the intended empty-scope change is in scope) +changing the authority split, or any retrieval semantics in the service mode for non-empty scopes and non-degenerate queries (the intended empty-scope and zero-norm rules are in scope) deprecating or altering the service-mode adapter beyond the shared port contract tuning the embedded index, quantization, or memory-mapping defaults (available in the engine; shipped at the exact-scan threshold this phase) migration tooling between modes; rebuild from graph authority is the path @@ -1240,7 +1240,7 @@ Embedded state survives process restart. Retrieval telemetry reports the completeness verdict for every retrieval in both modes. The default test path requires no vector service; service-gated suites still execute under the service-backed CI job and cannot pass by skipping. The evaluation repository's vector-only baseline produces its rows from the retrieval trace in both modes. -No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behavior change in service mode for non-empty scopes (the intended empty-scope change: zero candidates, and boundary rejection of an empty configured scope). +No public facade change beyond the telemetry field and the published maximum-surfaces-per-object-kind policy value; no retrieval behavior change in service mode for non-empty scopes and non-degenerate queries (the intended empty-scope change: zero candidates, and boundary rejection of an empty configured scope). ``` --- From 7b5d9ed987b0decfbc3698f2bad3147ea842b270 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 19:20:59 +0900 Subject: [PATCH 35/44] =?UTF-8?q?=F0=9F=93=9D=20Plan=20approved=20for=20im?= =?UTF-8?q?plementation;=20wave=20PRs=20target=20the=20planning=20branch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 117a1640..231e3fcf 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -1,6 +1,6 @@ # Plan: v0.1.6 Embedded Vector Candidate Recall -- status: draft +- status: approved - generated: 2026-09-02 - last_updated: 2026-09-03 - work_type: mixed @@ -221,7 +221,7 @@ Task identifiers 5 and 6 were evaluation-repository work and moved to that repository's own plan; identifiers are not reused. -Each wave ends with reviewer approval and a PR, merged by the decider before the next wave starts. +Each wave ends with reviewer approval and a PR against the planning branch `plan/v0-1-6-embedded-vector-recall`, merged by the decider before the next wave starts; the planning branch merges to main as one change at phase end (Decision Log, 2026-09-03). Notification duty: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository before merge; how and when that repository adopts the change is planned there, and this plan only consumes the resulting compatibility evidence at closeout. ## Rollback / Safety From 586374b7e0fcca86f756b78fec95e35485f47386 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 20:21:04 +0900 Subject: [PATCH 36/44] =?UTF-8?q?=F0=9F=93=9D=20Decision=20records:=20auth?= =?UTF-8?q?oring=20rules=20ruled=20during=20the=20v0.1.6=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/decisions/README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 23c7e8ee..58480e2a 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -28,5 +28,12 @@ Separate numbering per track; IDs are never reused. - Partial supersession stays in place: the record remains authoritative for its surviving clauses, with `supersession_scope` and reciprocal frontmatter links recording the split. - In records predating the current template, a blank warrant means it was not recorded at decision time, not an authoring omission. Fill blank newer frontmatter fields only when the record is substantively revisited. +## Authoring rules +- One decision per record. A choice of component and a choice of how that component is run or made durable are separate decisions with separate records. +- A record encodes a locked-in decision. A deciding factor is deferred only when it is out of reach because of external factors, never when the same change produces the evidence. +- The decision body pins the system's design and, at most, the behaviour of a narrow part of the implementation. Code shapes go in a non-binding appendix, never in the decision. +- Every protected clause is checked against the project philosophy: it stays only if a philosophy goal (continuity, provenance, inspectable recall, correction) is what it protects. Current state is recorded under "Not covered", not as an invariant. +- Records read the same at any time. No wording that hinges on when the record was written ("this phase", "later decision", "at decision time", "once", "already", change verbs such as "gains" or "moves"); history is anchored to named records, versions, and absolute dates, and Implementation Impact describes the resulting state. + ## Status values `accepted`, `rejected`, `superseded`, `deprecated`. Records capture decisions, not undecided proposals. From 677dd74caacaee5b1d7c0c1c8489f2d3cdae9b67 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Thu, 3 Sep 2026 20:22:59 +0900 Subject: [PATCH 37/44] =?UTF-8?q?=F0=9F=93=9D=20Rule:=20PR=20titles=20stat?= =?UTF-8?q?e=20the=20outcome,=20contents=20go=20in=20the=20body?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/rules/orchestrator.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/coding-agent/rules/orchestrator.md b/docs/coding-agent/rules/orchestrator.md index fc2dd857..910690f8 100644 --- a/docs/coding-agent/rules/orchestrator.md +++ b/docs/coding-agent/rules/orchestrator.md @@ -9,6 +9,7 @@ last_updated: "2026-07-23" ## Repo-Specific Orchestrator Policies - When creating or updating a PR, follow the format specified in `.github/pull_request_template.md`. +- PR titles state what the change achieves, not a list of its contents; the contents go in the body. Never use bare version numbers or milestone labels as titles. - Layer-boundary reorganizations must include a `use crate::` dependency-direction audit as required Reviewer evidence per ADR-I-0018 (ports/policy/models never import usecases, and import api only for the ADR's one named exception: the `api::types::retrieval` trace/telemetry vocabulary; errors/domain import no upper layer); file-placement conformance alone does not catch inverted edges hidden behind re-export shims. - Scope the ADR-I-0018 dependency-direction audit to the diff under review (e.g. `git diff | grep '^+.*use crate::'`) when reviewing incremental changes: pre-existing ports/policy/models imports of domain types via `crate::api::types` are grandfathered debt awaiting a one-time sweep to `crate::domain`, and a blanket grep forces per-line disambiguation between old and newly introduced edges. - PR feedback monitors must include terminal merged/closed state (harness pr-review-monitoring owns arming for reviews/comments but does not cover terminal-state watch) (user-directed 2026-07-19). From af5c89fd87d1facc76d5c2b2af29ef4b3bd9d03e Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 00:40:26 +0900 Subject: [PATCH 38/44] =?UTF-8?q?=F0=9F=93=9D=20Rule:=20wave=20PRs=20are?= =?UTF-8?q?=20registered=20as=20a=20GitHub=20stack=20on=20the=20planning?= =?UTF-8?q?=20PR?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/rules/orchestrator.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/coding-agent/rules/orchestrator.md b/docs/coding-agent/rules/orchestrator.md index 910690f8..a9681518 100644 --- a/docs/coding-agent/rules/orchestrator.md +++ b/docs/coding-agent/rules/orchestrator.md @@ -10,6 +10,7 @@ last_updated: "2026-07-23" - When creating or updating a PR, follow the format specified in `.github/pull_request_template.md`. - PR titles state what the change achieves, not a list of its contents; the contents go in the body. Never use bare version numbers or milestone labels as titles. +- When a phase merges as one change, wave PRs target the planning branch and are registered as a GitHub stack on the planning PR with `gh stack link ...` (bottom to top); later waves are appended with `gh stack link ...`. The stack decides merge order. - Layer-boundary reorganizations must include a `use crate::` dependency-direction audit as required Reviewer evidence per ADR-I-0018 (ports/policy/models never import usecases, and import api only for the ADR's one named exception: the `api::types::retrieval` trace/telemetry vocabulary; errors/domain import no upper layer); file-placement conformance alone does not catch inverted edges hidden behind re-export shims. - Scope the ADR-I-0018 dependency-direction audit to the diff under review (e.g. `git diff | grep '^+.*use crate::'`) when reviewing incremental changes: pre-existing ports/policy/models imports of domain types via `crate::api::types` are grandfathered debt awaiting a one-time sweep to `crate::domain`, and a blanket grep forces per-line disambiguation between old and newly introduced edges. - PR feedback monitors must include terminal merged/closed state (harness pr-review-monitoring owns arming for reviews/comments but does not cover terminal-state watch) (user-directed 2026-07-19). From e1a8a7dde2afc213c1b23022fb6f0add78d6a607 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 01:02:42 +0900 Subject: [PATCH 39/44] =?UTF-8?q?=F0=9F=93=9D=20Progress=20log:=20wave=201?= =?UTF-8?q?=20approved=20and=20stacked;=20wave=202=20dispatched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 231e3fcf..fc029a5b 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -235,6 +235,8 @@ Append-only editing rule (applies to both logs below): when appending an entry, - 2026-09-02 Planning wave completed: five parallel inputs (design consult, two altitude audits, two forensic censuses) consolidated; decider ruled the five design questions; ADR-I-0023 through ADR-I-0026, the rewritten phase document, and the roadmap section authored on branch `plan/v0-1-6-embedded-vector-recall`. Plan awaits approval. +- 2026-09-04 Wave 1 done: Task_1 (a633b2e, PR #74) and Task_2 (5b30856, PR #75) approved by the Tier D reviewer; Task_2 needed one revision (checked backend-limit conversion with a BoundaryTieOpen regression at the u32 cap; public re-exports) and carries one post-review fix (dimension check before the zero-norm scroll). PRs stacked on the planning PR as stack #76; merges pending. Wave 2 (Task_3) dispatched on a branch stacked on Task_2. + ## Decision Log (append-only; re-plans and major discoveries) - 2026-09-02 Decision: the draft's port description was rewritten as an intentional new port contract. From 6ac2bef637ae9a858f10a29b92185f34fe94593a Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 02:48:43 +0900 Subject: [PATCH 40/44] =?UTF-8?q?=F0=9F=93=9D=20Lesson:=20reconcile=20the?= =?UTF-8?q?=20companion=20pin=20before=20filing=20cross-repository=20break?= =?UTF-8?q?age?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/lessons.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/coding-agent/lessons.md b/docs/coding-agent/lessons.md index 4075d068..e323a3dc 100644 --- a/docs/coding-agent/lessons.md +++ b/docs/coding-agent/lessons.md @@ -454,3 +454,25 @@ Prevention: ## Purge note (2026-07-23) Eleven entries purged per the user-directed low-value/invalid sweep (Codex purge map, agmsg 2026-07-23T12:28Z): ten PURGE-LOW-VALUE (restatements of now-mandatory harness/rule content — plan-format task records, PR monitoring, canonical-byte verification, compatibility policy, module layout, evidenced-scope rulebook default, parallel dispatch — plus two cheaply rediscovered one-off quirks and one unstructured batch-notes bundle) and one PURGE-INVALID (the phase-bounded v0.1 compatibility ruling, superseded by the repo-wide Compatibility Policy). Full entries recoverable from git history at 4997bdc. + +## 2026-09-03 — Reconcile The Companion Pin Before Filing Cross-Repository Breakage [tags: review, scope, cross-repo, assumptions] + +Context: +- Plan: `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` +- Task/Wave: Task_3 / Wave 2 +- Roles involved: Reviewer | Orchestrator + +Symptom: +- The reviewer filed a preliminary HIGH because a new variant in the closed error vocabulary would not compile in the evaluation repository's exhaustive match once that repository re-pins. + +Root cause: +- The finding treated the companion repository's current main as if it were already pinned to the change under review, although the companion stays pinned at an older library commit until its own migration plan runs; ADR-I-0023's impact section names that conversion as the companion's re-pin prerequisite. + +Fix applied: +- The orchestrator ruled the item out of scope; the reviewer recorded it as the already-tracked re-pin obligation and completed the review on the in-PR acceptance bullets. + +Prevention: +- Before filing cross-repository breakage, reconcile the companion's pin and the plan that owns its migration; when the companion intentionally remains pinned until its own migration, record an obligation, not an in-PR defect. + +Evidence: +- Reviewer messages of 2026-09-03 (preliminary HIGH, ruling acknowledgement, final REVIEW3_DONE) on Task_3 at 773d65e. From 55dca32aaba62ca3ddf47cfc6264717d5678f9e0 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 03:35:42 +0900 Subject: [PATCH 41/44] =?UTF-8?q?=F0=9F=93=9D=20Progress=20log:=20wave=202?= =?UTF-8?q?=20approved=20and=20stacked?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index fc029a5b..97845be5 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -236,6 +236,7 @@ Append-only editing rule (applies to both logs below): when appending an entry, - 2026-09-02 Planning wave completed: five parallel inputs (design consult, two altitude audits, two forensic censuses) consolidated; decider ruled the five design questions; ADR-I-0023 through ADR-I-0026, the rewritten phase document, and the roadmap section authored on branch `plan/v0-1-6-embedded-vector-recall`. Plan awaits approval. - 2026-09-04 Wave 1 done: Task_1 (a633b2e, PR #74) and Task_2 (5b30856, PR #75) approved by the Tier D reviewer; Task_2 needed one revision (checked backend-limit conversion with a BoundaryTieOpen regression at the u32 cap; public re-exports) and carries one post-review fix (dimension check before the zero-norm scroll). PRs stacked on the planning PR as stack #76; merges pending. Wave 2 (Task_3) dispatched on a branch stacked on Task_2. +- 2026-09-04 Wave 2 done: Task_3 (36ef8c8, PR #77) approved by the Tier D reviewer after two revisions on the zero-norm rejected-record fixture (final shape: one definition behind the non-default `test-fixtures` feature, enabled for integration tests by a self dev-dependency, doc-hidden). The evaluation repository's exhaustive error-vocabulary conversion is recorded as its re-pin obligation (ADR-I-0023), not a finding. PR appended to stack #76. ## Decision Log (append-only; re-plans and major discoveries) From c3987390a9f5fea22e2c5269e9f8d9e91dbc4240 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 04:09:29 +0900 Subject: [PATCH 42/44] =?UTF-8?q?=F0=9F=93=9D=20Lesson:=20PR=20watchers=20?= =?UTF-8?q?must=20not=20depend=20on=20tools=20absent=20from=20the=20monito?= =?UTF-8?q?r=20shell?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/lessons.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/coding-agent/lessons.md b/docs/coding-agent/lessons.md index e323a3dc..47684a28 100644 --- a/docs/coding-agent/lessons.md +++ b/docs/coding-agent/lessons.md @@ -476,3 +476,25 @@ Prevention: Evidence: - Reviewer messages of 2026-09-03 (preliminary HIGH, ruling acknowledgement, final REVIEW3_DONE) on Task_3 at 773d65e. + +## 2026-09-04 — PR Watchers Must Not Depend On Tools Absent From The Monitor Shell [tags: workflow, monitoring, tooling, orchestrator] + +Context: +- Plan: `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` +- Task/Wave: Wave 1–2 PR monitoring +- Roles involved: Orchestrator + +Symptom: +- Copilot reviews on the wave PRs and on the evaluation repository's PR arrived without any watcher notification; the decider noticed the review before the orchestrator did. + +Root cause: +- The watcher scripts piped GitHub API output through `jq`, which is not on the PATH of the background monitor shell; every poll failed on stderr, which the monitor does not surface, so the watchers stayed silent while appearing armed. + +Fix applied: +- Watchers re-armed using only `gh api --jq` (no external `jq`), one loop covering the whole stack, printing on any change in review count, review-comment count, or merge state. + +Prevention: +- A watcher script uses only tools proven available in the monitor shell (`gh --jq`, POSIX sh); before trusting a new watcher, read its output file once to confirm it produced a first sample rather than errors. + +Evidence: +- Monitor output file for the PR #77 watcher on 2026-09-04: eighteen consecutive `jq: command not found` lines and no events, while Copilot's "approval recommended" review was already posted. From 67aef4e53677bfe1f641ab8362007e608035118c Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 04:15:18 +0900 Subject: [PATCH 43/44] =?UTF-8?q?=F0=9F=93=9D=20Merge=20shape:=20the=20who?= =?UTF-8?q?le=20stack=20merges=20in=20one=20go=20at=20phase=20end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- .../plans/active/v0-1-6-embedded-vector-recall-plan.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md index 97845be5..a6e5df1c 100644 --- a/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md +++ b/docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md @@ -221,7 +221,7 @@ Task identifiers 5 and 6 were evaluation-repository work and moved to that repository's own plan; identifiers are not reused. -Each wave ends with reviewer approval and a PR against the planning branch `plan/v0-1-6-embedded-vector-recall`, merged by the decider before the next wave starts; the planning branch merges to main as one change at phase end (Decision Log, 2026-09-03). +Each wave ends with reviewer approval and a PR stacked on the previous wave's PR (GitHub stack on the planning PR); the next wave branches from the approved tip without waiting for a merge, and the decider merges the entire stack in one go at phase end (Decision Log, 2026-09-03 and 2026-09-04). Notification duty: any wave that changes a public vocabulary the evaluation repository converts exhaustively (the vector database error kinds in Wave 3, the telemetry field in Wave 1) is announced to that repository before merge; how and when that repository adopts the change is planned there, and this plan only consumes the resulting compatibility evidence at closeout. ## Rollback / Safety @@ -237,6 +237,7 @@ Append-only editing rule (applies to both logs below): when appending an entry, - 2026-09-04 Wave 1 done: Task_1 (a633b2e, PR #74) and Task_2 (5b30856, PR #75) approved by the Tier D reviewer; Task_2 needed one revision (checked backend-limit conversion with a BoundaryTieOpen regression at the u32 cap; public re-exports) and carries one post-review fix (dimension check before the zero-norm scroll). PRs stacked on the planning PR as stack #76; merges pending. Wave 2 (Task_3) dispatched on a branch stacked on Task_2. - 2026-09-04 Wave 2 done: Task_3 (36ef8c8, PR #77) approved by the Tier D reviewer after two revisions on the zero-norm rejected-record fixture (final shape: one definition behind the non-default `test-fixtures` feature, enabled for integration tests by a self dev-dependency, doc-hidden). The evaluation repository's exhaustive error-vocabulary conversion is recorded as its re-pin obligation (ADR-I-0023), not a finding. PR appended to stack #76. +- 2026-09-04 — Merge shape refined: the stack (planning PR at the bottom, one PR per wave above it) is merged in one go at phase end; GitHub's bottom-up stack order blocks interim merges of wave PRs while the planning PR is a draft, which is the intended shape. Waves proceed by branching from the previous wave's approved tip. ## Decision Log (append-only; re-plans and major discoveries) From 106b3c052ddeef8707f24883c012f2aa782cfe88 Mon Sep 17 00:00:00 2001 From: ebigunso Date: Fri, 4 Sep 2026 05:15:30 +0900 Subject: [PATCH 44/44] =?UTF-8?q?=F0=9F=93=9D=20Lessons:=20absolute=20work?= =?UTF-8?q?tree=20paths;=20broad=20suites=20only=20after=20the=20live=20wi?= =?UTF-8?q?ndow=20is=20granted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 --- docs/coding-agent/lessons.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/coding-agent/lessons.md b/docs/coding-agent/lessons.md index 47684a28..51662500 100644 --- a/docs/coding-agent/lessons.md +++ b/docs/coding-agent/lessons.md @@ -498,3 +498,25 @@ Prevention: Evidence: - Monitor output file for the PR #77 watcher on 2026-09-04: eighteen consecutive `jq: command not found` lines and no events, while Copilot's "approval recommended" review was already posted. + +## 2026-09-04 — Edit Only Through Absolute Worktree Paths, And Run Broad Suites Only After The Live Window Is Granted [tags: workflow, worktrees, live-mutex, worker] + +Context: +- Plan: `docs/coding-agent/plans/active/v0-1-6-embedded-vector-recall-plan.md` +- Task/Wave: Task_4 / Wave 3 +- Roles involved: Worker | Orchestrator + +Symptom: +- A relative-path patch briefly modified files in the shared main checkout instead of the task worktree before being reverted; and a full test suite started before the exclusive service window was confirmed, so service-backed tests could have collided with another agent's run. + +Root cause: +- Relative paths resolve against whatever the current directory happens to be, and several checkouts of the same repository share identical relative paths; the live-window protocol grants exclusivity only on the explicit `WINDOW_YOURS` reply, not on sending `LIVE_START`. + +Fix applied: +- The stray edits were reverted and verified clean; the suite was re-run inside the granted window. + +Prevention: +- Every edit and every git command names an absolute path inside the task worktree, and the current directory is checked before each edit; a suite that may reach a shared service starts only after `WINDOW_YOURS` is observed, otherwise run the service-free command. + +Evidence: +- Worker Task_4 report of 2026-09-03 (commit d232830) and the orchestrator's clean `git status` check on the main checkout.