Skip to content

Embed by format, not type: link facets stay out of the recall vector #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
tieguy wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Embed by format, not type: link facets stay out of the recall vector #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ _files:_
| served-content inertness | Routing | `inertHeaders` | `served-content inertness totality` |
| served-read gating | Routing | `denyUnlessBearerScope` | `served bearer-gating totality` |

<sub>Generated at 2026-08-03 15:26Z.</sub>
<sub>Generated at 2026-08-11 16:20Z.</sub>
<!-- coherence:end -->

## Current state
Expand Down
6 changes: 3 additions & 3 deletions mnemion-js/docs/coherence/_graph.html
View file
Open in desktop

Large diffs are not rendered by default.

60 changes: 33 additions & 27 deletions mnemion-js/docs/coherence/graph.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"generatedAt": "2026-08-03 15:26Z",
"generatedAt": "2026-08-11 16:20Z",
"root": "mnemion-js",
"absRoot": "/Users/daniloc/Documents/Development/mnemion/mnemion-js",
"absRoot": "/tmp/claude-1000/-var-home-louie-Projects-Self-mnemion/67f571c5-a9bd-427e-b564-8c9bd44e2072/scratchpad/pr-embed/mnemion-js",
"nodes": [
{
"id": "c:.",
Expand Down Expand Up @@ -2291,7 +2291,7 @@
"kind": "symbol",
"sub": "interface",
"path": "entities/Hive/prime.ts",
"line": 21
"line": 22
},
{
"id": "s:entities/Hive/prime.ts#patternExists()",
Expand All @@ -2300,7 +2300,7 @@
"kind": "symbol",
"sub": "method",
"path": "entities/Hive/prime.ts",
"line": 24
"line": 25
},
{
"id": "s:entities/Hive/prime.ts#PrimeResult",
Expand All @@ -2309,7 +2309,7 @@
"kind": "symbol",
"sub": "interface",
"path": "entities/Hive/prime.ts",
"line": 27
"line": 28
},
{
"id": "s:entities/Hive/prime.ts#MemoryPolicy",
Expand All @@ -2318,7 +2318,7 @@
"kind": "symbol",
"sub": "interface",
"path": "entities/Hive/prime.ts",
"line": 38
"line": 39
},
{
"id": "s:entities/Hive/prime.ts#CONFLICT_SIMILARITY",
Expand All @@ -2327,7 +2327,7 @@
"kind": "symbol",
"sub": "const",
"path": "entities/Hive/prime.ts",
"line": 50
"line": 51
},
{
"id": "s:entities/Hive/prime.ts#getPatternClass",
Expand All @@ -2336,7 +2336,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 56,
"line": 57,
"prose": "A pattern's class: \"knowledge\" (default — recalled by meaning) or \"dataset\"\n (structured records aggregated by computation, exempt from the memory machinery)."
},
{
Expand All @@ -2346,7 +2346,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 78,
"line": 79,
"prose": "Read a pattern's memory policy from _objects, applying opinionated defaults."
},
{
Expand All @@ -2356,7 +2356,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 97,
"line": 98,
"prose": "Recall-weight multiplier: halves per half-life elapsed, floored so old-but-relevant survives."
},
{
Expand All @@ -2366,8 +2366,8 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 106,
"prose": "Build embeddable text from an entry's text/select facets."
"line": 116,
"prose": "Build embeddable text from the facets whose FORMAT says they carry prose.\n\n What an entry MEANS is decided by each facet's format, not its storage type:\n a URL and a paragraph are both TEXT, but only one is language. So this reads\n every facet and derives inclusion from FORMAT_PALETTE[…].embed — the same\n declaration the renderers key off — rather than re-listing types here.\n The INTRINSIC format only (`_fields.format` ?? the type default); a view's\n per-facet override is presentation, and must not change what an entry means.\n Facets are read in declaration order, so put prose first: the MAX_EMBED_CHARS\n budget is spent in that order."
},
{
"id": "s:entities/Hive/prime.ts#embedEntry",
Expand All @@ -2376,7 +2376,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 144,
"line": 154,
"prose": "Embed an entry and upsert its vector. Fire-and-forget safe.\n A precomputed vector (e.g. from the write-time conflict check) skips the AI call."
},
{
Expand All @@ -2386,7 +2386,7 @@
"kind": "symbol",
"sub": "interface",
"path": "entities/Hive/prime.ts",
"line": 179
"line": 189
},
{
"id": "s:entities/Hive/prime.ts#findNeighbors",
Expand All @@ -2395,7 +2395,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 190,
"line": 200,
"prose": "Find same-pattern semantic neighbors of not-yet-written entry data.\n Returns matches ≥ CONFLICT_SIMILARITY plus the computed vector (reusable for\n the post-write upsert, avoiding a second AI call). Best-effort: any failure\n returns empty with no vector."
},
{
Expand All @@ -2405,7 +2405,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 228,
"line": 238,
"prose": "Remove a vector when an entry is archived."
},
{
Expand All @@ -2415,7 +2415,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 242,
"line": 252,
"prose": "Prime: partial cue activates a full constellation."
},
{
Expand All @@ -2425,7 +2425,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 380,
"line": 390,
"prose": "SQLite datetime('now') emits \"YYYY-MM-DD HH:MM:SS\" in UTC with no zone marker — parse as UTC."
},
{
Expand All @@ -2435,7 +2435,7 @@
"kind": "symbol",
"sub": "function",
"path": "entities/Hive/prime.ts",
"line": 388,
"line": 398,
"prose": "Follow foreign key links one hop from an entry."
},
{
Expand Down Expand Up @@ -5055,7 +5055,7 @@
"kind": "symbol",
"sub": "const",
"path": "shared/core/format-palette.ts",
"line": 22,
"line": 29,
"prose": "Add a format here and the enum, the agent contract, and validation pick it up;\nthe SPA won't compile until it has a matching renderer (Record<FormatId,…>)."
},
{
Expand All @@ -5065,7 +5065,7 @@
"kind": "symbol",
"sub": "type",
"path": "shared/core/format-palette.ts",
"line": 33
"line": 40
},
{
"id": "s:shared/core/format-palette.ts#FORMAT_IDS",
Expand All @@ -5074,7 +5074,7 @@
"kind": "symbol",
"sub": "const",
"path": "shared/core/format-palette.ts",
"line": 35
"line": 42
},
{
"id": "s:shared/core/format-palette.ts#isFormat",
Expand All @@ -5083,7 +5083,7 @@
"kind": "symbol",
"sub": "function",
"path": "shared/core/format-palette.ts",
"line": 37
"line": 44
},
{
"id": "s:shared/core/format-palette.ts#defaultFormatForType",
Expand All @@ -5092,7 +5092,7 @@
"kind": "symbol",
"sub": "function",
"path": "shared/core/format-palette.ts",
"line": 44,
"line": 51,
"prose": "The default rendering for a facet that carries no explicit format — derived\nfrom its declared type, so a datetime is friendly and a boolean is a check\nwithout anyone having to say so. Truth (the type) drives presentation."
},
{
Expand All @@ -5102,7 +5102,7 @@
"kind": "symbol",
"sub": "function",
"path": "shared/core/format-palette.ts",
"line": 59,
"line": 66,
"prose": "The resolve chain: view override ?? facet intrinsic ?? foreign-key reference ??\ntype default. A declared foreign key (hasLink) wins over the type default — an\nFK facet is a reference by nature — but an explicit format still overrides it.\nUnknown ids fall through (a stale format never crashes a render — it degrades\nto the next source), so the resolver always returns a real FormatId."
},
{
Expand All @@ -5112,7 +5112,7 @@
"kind": "symbol",
"sub": "function",
"path": "shared/core/format-palette.ts",
"line": 74,
"line": 81,
"prose": "Validate a view's per-facet `formats` override map: an object of\nfacet-name → format-id. `hasFacet` null = pattern unknown → skip the\nfacet-existence check (format-id checks still run). Returns [] when valid."
},
{
Expand All @@ -5122,7 +5122,7 @@
"kind": "symbol",
"sub": "function",
"path": "shared/core/format-palette.ts",
"line": 93,
"line": 100,
"prose": "Agent-facing prose, generated from the palette so the contract can't drift\nfrom what's enforced. Embedded in the _views + set_facet_format docs."
},
{
Expand Down Expand Up @@ -5826,6 +5826,12 @@
"target": "f:shared/core/constants.ts",
"kind": "imports"
},
{
"id": "f:entities/Hive/prime.ts->f:shared/core/format-palette.ts:imports",
"source": "f:entities/Hive/prime.ts",
"target": "f:shared/core/format-palette.ts",
"kind": "imports"
},
{
"id": "f:entities/Hive/prime.ts->f:shared/core/log.ts:imports",
"source": "f:entities/Hive/prime.ts",
Expand Down
18 changes: 14 additions & 4 deletions mnemion-js/entities/Hive/prime.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
// the decay clock), never from a stored counter, per data-is-destiny.

import { uri } from "../../shared/core/constants";
import { FORMAT_PALETTE, resolveFormat } from "../../shared/core/format-palette";
import { logWarn } from "../../shared/core/log";
import { isKernelPattern, primeIncluded, seal } from "./policy";

Expand Down Expand Up @@ -102,16 +103,25 @@ export function decayMultiplier(ageDays: number, halfLifeDays: number | null): n

// === Embedding ===

/** Build embeddable text from an entry's text/select facets. */
/** Build embeddable text from the facets whose FORMAT says they carry prose.
*
* What an entry MEANS is decided by each facet's format, not its storage type:
* a URL and a paragraph are both TEXT, but only one is language. So this reads
* every facet and derives inclusion from FORMAT_PALETTE[…].embed — the same
* declaration the renderers key off — rather than re-listing types here.
* The INTRINSIC format only (`_fields.format` ?? the type default); a view's
* per-facet override is presentation, and must not change what an entry means.
* Facets are read in declaration order, so put prose first: the MAX_EMBED_CHARS
* budget is spent in that order. */
export function buildEmbedText(db: any, pattern: string, entry: Record<string, unknown>): string {
// Get text and select facets for this pattern
const facets = db.exec(
"SELECT name FROM _fields WHERE object_name = ? AND type IN ('text', 'select') ORDER BY id",
"SELECT name, type, format FROM _fields WHERE object_name = ? ORDER BY id",
pattern
).toArray() as { name: string }[];
).toArray() as { name: string; type: string; format: string | null }[];

const parts: string[] = [pattern];
for (const f of facets) {
if (!FORMAT_PALETTE[resolveFormat(undefined, f.format, f.type)].embed) continue;
const val = entry[f.name];
if (val != null && typeof val === "string" && val.length > 0) {
parts.push(val);
Expand Down
23 changes: 15 additions & 8 deletions mnemion-js/shared/core/format-palette.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,26 @@ export interface FormatType {
label: string;
/** Agent-facing: what this format does / when to use it. */
help: string;
/** Does this value contribute PROSE to the recall vector? A format is the
* data's nature, so it — not the storage type — decides what an entry MEANS
* for `prime`. Required, so a new format cannot be added without deciding:
* the `satisfies` below stops compiling until it is classified. False for
* values that are addresses or scalars rather than language (a URL dilutes
* the vector and crowds the embed budget without adding meaning). */
embed: boolean;
}

// Add a format here and the enum, the agent contract, and validation pick it up;
// the SPA won't compile until it has a matching renderer (Record<FormatId,…>).
export const FORMAT_PALETTE = {
text: { label: "Text", help: "Plain text (the default for text facets)." },
link: { label: "Link", help: "Render the value as a clickable link (href = the value)." },
tags: { label: "Tags", help: "Split a comma-separated value into chips." },
date: { label: "Date", help: "Render a timestamp as a friendly relative date." },
boolean: { label: "Boolean", help: "Render a truthy/falsy value as ✓ / ✗." },
select: { label: "Select", help: "An interactive dropdown that changes the value inline (options: the facet's declared options, else the values already in use)." },
number: { label: "Number", help: "Format with thousands separators; right-aligned and sorted numerically (not lexically) in tables. The default for integer/number facets." },
reference: { label: "Reference", help: "A navigable link to another entry — shows that entry's label, click to open it. Auto-applied to facets that declare a foreign key (links)." },
text: { label: "Text", help: "Plain text (the default for text facets).", embed: true },
link: { label: "Link", help: "Render the value as a clickable link (href = the value). Also excludes the value from recall: a URL is an address, not prose, so it never feeds prime's embedding.", embed: false },
tags: { label: "Tags", help: "Split a comma-separated value into chips.", embed: true },
date: { label: "Date", help: "Render a timestamp as a friendly relative date.", embed: false },
boolean: { label: "Boolean", help: "Render a truthy/falsy value as ✓ / ✗.", embed: false },
select: { label: "Select", help: "An interactive dropdown that changes the value inline (options: the facet's declared options, else the values already in use).", embed: true },
number: { label: "Number", help: "Format with thousands separators; right-aligned and sorted numerically (not lexically) in tables. The default for integer/number facets.", embed: false },
reference: { label: "Reference", help: "A navigable link to another entry — shows that entry's label, click to open it. Auto-applied to facets that declare a foreign key (links).", embed: true },
} satisfies Record<string, FormatType>;

export type FormatId = keyof typeof FORMAT_PALETTE;
Expand Down
46 changes: 46 additions & 0 deletions mnemion-js/src/__tests__/formats.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
FORMAT_PALETTE,
isFormat,
} from "../../shared/core/format-palette";
import { buildEmbedText } from "../../entities/Hive/prime";

function getStore(): DurableObjectStub<HiveDO> {
const id = env.MNEMION_HIVE.idFromName(`user:test:${crypto.randomUUID()}`);
Expand Down Expand Up @@ -87,6 +88,51 @@ describe("format palette totality", () => {
}
expect(isFormat("definitely-not-a-format")).toBe(false);
});

it("every format declares whether its value feeds the recall vector", () => {
for (const id of FORMAT_IDS) expect(typeof FORMAT_PALETTE[id].embed).toBe("boolean");
});
});

// === What feeds the recall vector (format-derived, not type-derived) ===
//
// buildEmbedText resolves each facet's INTRINSIC format (_fields.format ?? the
// type default) and consults the palette. A view override is deliberately not
// consulted — what a desk displays is presentation; what an entry means is data.

describe("buildEmbedText", () => {
// The rows _fields would yield; buildEmbedText's only dependency is this shape.
const fieldsDb = (rows: { name: string; type: string; format?: string | null }[]) =>
({ exec: () => ({ toArray: () => rows }) }) as any;

it("omits a link-formatted facet — a URL is an address, not prose", () => {
const db = fieldsDb([
{ name: "summary", type: "text", format: null },
{ name: "source_link", type: "text", format: "link" },
]);
const text = buildEmbedText(db, "topics", {
summary: "the licensing question that keeps resurfacing",
source_link: "https://mail.google.com/mail/u/0/#inbox/FMfcgzabc123",
});
expect(text).toContain("the licensing question that keeps resurfacing");
expect(text).not.toContain("mail.google.com");
});

it("still embeds text and select facets, and still skips dates", () => {
const db = fieldsDb([
{ name: "title", type: "text", format: null },
{ name: "status", type: "select", format: null },
{ name: "reviewed_at", type: "datetime", format: null },
]);
const text = buildEmbedText(db, "topics", {
title: "patent pledges",
status: "dormant",
reviewed_at: "2026-08-11T00:00:00Z",
});
expect(text).toContain("patent pledges");
expect(text).toContain("dormant");
expect(text).not.toContain("2026-08-11");
});
});

// === set_facet_format change type (intrinsic format authoring) ===
Expand Down