Skip to content

✨ The vector record becomes a five-field read contract, and zero-norm embeddings are rejected before any adapter #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ebigunso wants to merge 3 commits into from
Open

✨ The vector record becomes a five-field read contract, and zero-norm embeddings are rejected before any adapter #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
035d044
feat: enforce five-field vector record contract
ebigunso Sep 3, 2026
22ee7c6
test: expose rejected vector record fixture
ebigunso Sep 3, 2026
7751d90
test: expose vector fixture to integration crates
ebigunso Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ name = "character_memory"
version = "0.1.5"
edition = "2021"

[features]
default = []
test-fixtures = []

[dependencies]
# Core
config = "0.15.22"
Expand All @@ -29,6 +33,7 @@ uuid = { version = "1.23.1", features = ["v4", "v5", "serde"] }
chrono = { version = "0.4.44", features = ["serde"] }

[dev-dependencies]
character_memory = { path = ".", features = ["test-fixtures"] }
mockall = "0.14.0"
tonic = "0.14.6"
tempfile = "3.20.0"
2 changes: 1 addition & 1 deletion docs/design/database/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
These documents explain the storage-schema rationale behind the graph-authoritative architecture.

- [Database Schema Cheat Sheet](schema_cheat_sheet.md): compact reference for Qdrant payload fields, Oxigraph classes, graph predicates, and cross-store authority.
- [Vector Database Payload Design](vector_payload_design.md): why Qdrant stores candidate-recall payload hints rather than authoritative memory state.
- [Vector Database Payload Design](vector_payload_design.md): why Qdrant uses an exact five-field candidate record while Oxigraph retains memory authority.
- [Graph Database Schema Design](graph_schema_design.md): why Oxigraph/RDF stores canonical objects, typed links, provenance, lifecycle state, bounded expansion context, and how derived retrieval stats support selectivity and fanout policy without becoming graph truth.
9 changes: 4 additions & 5 deletions docs/design/database/graph_schema_design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ urn:cmem:link:<uuid>

The graph also stores the UUID, object type, graph URI, and schema version as literal properties.

This is redundant by design. The URI is efficient for graph edges, while the literal fields make debugging, migration checks, and cross-store joins easier. Qdrant carries the same `object_id` and `graph_uri` so vector candidates can be joined back to graph truth without guessing.
This is redundant inside the graph representation by design. The URI is efficient for graph edges, while the literal fields make debugging and migration checks easier. Qdrant carries `object_id`; retrieval uses that stable identity to hydrate graph truth without duplicating the graph URI in vector payloads.

## Object Classes

Expand Down Expand Up @@ -370,20 +370,19 @@ This means a reopened graph store can answer object queries, link queries, prove
Qdrant, Oxigraph, and the retrieval stats store share stable object IDs, but they do not share authority.

```text
Qdrant recalls candidates and applies coarse payload filters
Qdrant recalls candidates and filters by canonical object type
Stats supplies derived selectivity/fanout inputs
Oxigraph verifies existence, relationships, provenance, lifecycle, and context
```

This is why the vector payload intentionally duplicates some graph-derived hints. Duplication is acceptable for speed as long as retrieval treats those hints as non-authoritative.
The vector payload does not duplicate graph-derived hints. Retrieval joins candidates to Oxigraph by stable object ID, then applies relationships, provenance, lifecycle, currentness, and context from graph authority.

Internal reconciliation diagnostics can report cross-store drift:

- vector point exists but graph object is missing
- graph object exists but vector point is missing
- vector payload `graph_uri` does not match the canonical graph URI
- vector lifecycle/currentness hints disagree with graph authority
- vector payload schema version is unsupported
- vector payload identity or surface token is malformed
- graph object is missing required provenance
- stats counter refers to graph edge/object state that no longer exists
- stats health indicates conservative fallback should be used
Expand Down
64 changes: 8 additions & 56 deletions docs/design/database/schema_cheat_sheet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This is the compact schema reference. The companion design notes explain why the

| Store | Role | Authoritative For | Not Authoritative For |
|---|---|---|---|
| Qdrant | Vector candidate recall and coarse payload filtering | Vector points, embedding surfaces, payload hints | Memory existence, relationships, provenance, lifecycle, currentness, entity selectivity |
| Qdrant | Vector candidate recall and object-type prefiltering | Vector points and embedding-surface provenance | Memory content, existence, relationships, provenance, lifecycle, currentness, entity selectivity |
| Oxigraph | Graph authority | Memory objects, typed links, provenance, lifecycle, currentness, expansion context | Semantic nearest-neighbor ranking, derived selectivity counters |
| RetrievalStatsStore | Derived retrieval-policy statistics | Entity/relation counters, global counters, selectivity inputs, fanout diagnostics | Memory existence, relationships, provenance, lifecycle, currentness, semantic ranking |
| Raw store / caller storage | Source material | Raw transcript or source content behind `raw_ref` | Canonical memory state |
Expand All @@ -27,7 +27,7 @@ Oxigraph decides.
| Field | Stored In | Purpose |
|---|---|---|
| `object_id` / `objectId` | Qdrant payload and graph literal | Stable object UUID |
| `graph_uri` / `graphUri` | Qdrant payload and graph literal | Stable graph resource pointer |
| `graphUri` | Graph literal | Stable graph resource pointer |
| `schema_version` / `schemaVersion` | Qdrant payload and graph literal | Persistence and migration marker |

Graph URI pattern:
Expand All @@ -45,63 +45,15 @@ Retrieval stats store keys refer to the same `object_id` / entity ID values, but

## Qdrant Payload Fields

### Identity And Surface

| Field | Type / Shape | Notes |
|---|---|---|
| `object_id` | keyword UUID string | Stable vector-to-graph join id |
| `graph_uri` | keyword URI string | Stable graph resource pointer |
| `object_type` | keyword enum | Canonical memory object type |
| `record_type` | keyword enum | Indexed vector record kind |
| `schema_version` | keyword string | Payload migration marker |
| `object_id` | indexed keyword UUID string | Stable vector-to-graph join id |
| `object_type` | indexed keyword enum | Canonical memory object type |
| `surface` | keyword enum | Embedded semantic surface |
| `embedding_text` | string | Text used to generate the vector |
| `content_text` | string | Compact readable/debug text |

### Object-Specific Hints

| Field | Type / Shape | Notes |
|---|---|---|
| `derived_type` | keyword enum | Derived memory subtype |
| `entity_type` | keyword enum | Entity subtype |
| `thread_status` | keyword enum | Thread lifecycle/status hint |
| `modality` | keyword enum | Source modality |
| `source_conversation_id` | keyword string | Source conversation filter |
| `canonical_key` | keyword string | Stable caller/domain key |

### Relationship Hints

Relationship fields in Qdrant are filter hints only. Oxigraph remains authoritative.
| `schema_version` | keyword string | Record compatibility marker |
| `embedding_text` | text | Exact text used to generate the vector; not read-out content |

| Field | Type / Shape | Notes |
|---|---|---|
| `episode_ids` | keyword array | Related episode ids |
| `observation_ids` | keyword array | Related observation ids |
| `thread_ids` | keyword array | Related thread ids |
| `entity_ids` | keyword array | Related entity ids |
| `participant_entity_ids` | keyword array | Episode participant ids |
| `speaker_entity_id` | keyword UUID string | Observation speaker id |
| `supersedes` | keyword array | Supersession hint |

Do not compute entity selectivity from Qdrant relationship hints. Qdrant may identify candidate entities from vector hits, but selectivity counts must come from graph-authoritative writes or graph-derived stats.

### Lifecycle, Ranking, And Time Hints

| Field | Type / Shape | Notes |
|---|---|---|
| `retention_state` | keyword enum | Lifecycle filter hint |
| `is_current` | bool | Currentness hint |
| `is_superseded` | bool | Supersession/currentness hint |
| `salience_score` | float | Ranking/filter hint |
| `confidence` | float | Ranking/filter hint |
| `stability` | keyword enum | Derived memory stability |
| `created_at` | datetime | Creation time |
| `updated_at` | datetime | Update time |
| `started_at` | datetime | Episode start time |
| `ended_at` | datetime | Episode end time |
| `observed_at` | datetime | Observation time |
| `last_touched_at` | datetime | Thread recency |
| `raw_ref` | keyword string | Source pointer, not raw transcript content |
These are the only Qdrant payload fields. Readable content, graph URI, object-specific state, relationships, lifecycle/currentness, ranking, timestamps, provenance, and raw references are hydrated from Oxigraph by `object_id`. Existing obsolete extra fields may remain on old points but readers ignore them.

## Qdrant Indexed Object Types

Expand Down Expand Up @@ -264,4 +216,4 @@ The final context pack follows Oxigraph state.

## Reconciliation Diagnostics

Internal diagnostics can report vector-only records, graph-only records, graph URI mismatch, stale lifecycle/currentness hints, unsupported vector schema versions, graph records with missing required provenance, stats records missing graph authority, stats health failures, and low-selectivity expansions rejected by policy. The initial boundary is report-only; diagnostics do not repair stores or expose a public facade API by default.
Internal diagnostics can report vector-only records, graph-only records, unsupported vector schema versions, malformed vector identity or surface tokens, graph records with missing required provenance, stats records missing graph authority, stats health failures, and low-selectivity expansions rejected by policy. The initial boundary is report-only; diagnostics do not repair stores or expose a public facade API by default.
Loading
Loading