feat: add conflict-preserving, diversity-aware candidate packing (#192)

[ezutfen]

Closes #192 (ENG-RECALL-003E). Parent #160, epic #153. Baseline c1d5f24 (post #190/#191).

What

Replaces the governed/exploratory shadow profiles' rank-then-truncate selection with one versioned, deterministic packing contract — recall-packing-v1 (engram/recall_packing.py) — operating strictly after admission (#186), admission-first relationship expansion (#190), and separated relevance/utility ranking, and only inside the read-only shadow comparison surface:

eligible corpus -> exact V2 admission -> retrieval + expansion
  -> separated ranking -> conflict/diversity-aware packing  <- this PR
  -> byte/token/item budgets -> shadow packet

Algorithm (pure, in-memory, phased — no global packing score)

• Known-root groups via union-find over exactly one durable fact: explicit derived_from edges between admitted candidates. Canonical content equality (identical content_hash) proves only equal canonicalized text — the write-path dedup index (idx_memitems_dedup, scoped over tenant/workspace/principal/hash) is scoped duplicate prevention, not a global root assertion — so it is not a root signal and never groups. Similarity is never root identity either; unknown root stays unknown and is never coerced to independent or redundant (ENG-EVIDENCE-001 — Evidence-root-aware corroboration and usefulness lane #161 keeps sole ownership of corroboration).
• Phase 1 (representation + conflict co-pack, rank order): one representative per group (ranked); selecting one side of an explicit conflict immediately attempts to co-pack its admitted counterpart (conflict_pair_preserved) ahead of lower-ranked redundancy, under the same hard budgets.
• Phase 2 (fill, rank order): leftover capacity fills from the existing ranked order (diversity_fill) — siblings are crowd-out protected, not forbidden.
• Budgets: the exact rendered accounting the legacy path uses (len(content.encode()) bytes, max(1, bytes//4) tokens), same skip-not-break discipline — never exceeded, deterministic boundaries.
• Accounting reconciles mechanically: selected + Σ omitted == admitted count; omission reasons conflict_counterpart_budget > redundant_known_root > budget; packet-level packing summary is counts only (version, selected count, preserved conflict-pair count, omission counts) — no rejected content, no counterpart ids.
• Selection only, never mutation: per-item output is one additive packing_reason; relevance/utility/admission/evidence/epistemic/risk/warning codes/relationship blocks are not packer inputs it can rewrite.

Merge-blocker correction (this push): content_hash is not a root signal

Review correctly flagged that bare content_hash equality was treated as mechanically-known shared-root identity. That is too broad: Engram's durable duplicate identity is scoped over (tenant_id, workspace_id, principal_id, content_hash), so two admitted rows may legally share a hash across workspace/principal scope with no derivation relationship — equal canonical content proves the same canonicalized proposition text, not same evidence root, provenance, or a copied/derived relationship. #192 requires unknown root to remain unknown and leaves evidence-root-aware corroboration/independence to #161.

Correction (selection behavior, phases, conflicts, budgets, reasons, accounting, ordering, and the recall-packing-v1 contract identity are unchanged):

• content_hash removed from PackCandidate entirely — content identity cannot even reach the packer (pinned by test_packer_input_surface_excludes_content_identity).
• _known_root_groups() now unions only over explicit derived_from edges (transitively through chains). No substitute heuristic.
• New negative regression test_same_hash_different_scope_is_not_a_known_root: two admitted candidates, identical durable hash, different workspaces, no derivation — both selected by rank under budget 2, the distinct item omitted by budget alone, redundant_known_root absent.
• The former cross-workspace same-hash tunnel test now links its direct + tunnel-expanded family with an explicit derived_from edge (same direct+expanded scenario, genuine durable relation).
• ADR-160 doctrine and module docstrings updated: known-root diversity uses explicit durable derivation relationships only; idx_memitems_dedup never described as proving cross-scope duplicate/root identity.

Before/after evidence (live packets, item budget 2)

Scenario	Legacy (before)	Governed (after)
3-item known-root family (explicit derived_from) + distinct	[alpha one, alpha two] — family crowd-out	[alpha one, distinct fact], omitted: {redundant_known_root: 2}
Explicit conflict + redundant sibling ranked between sides	sibling kept, contra side erased	[alpha, contra], conflict_pairs_preserved: 1, sibling deferred
Same content_hash, different workspace, no derivation	—	[A, B] both by rank, omitted: {budget: 1} — root stays unknown, never redundant_known_root
Hard byte budget (40) fits one side only	—	30-byte packet, omitted: {conflict_counterpart_budget: 1, redundant_known_root: 1} — bounded and honest, never a silent resolution

Boundedness / security

• Relation loading is one bulk edge query with both endpoints inside the already-admitted set — query count constant as candidate volume grows; no provider call, no N+1, no graph walk, no second V2 evaluation or ENG-CLASSIFY-003 — Separate retention value from epistemic confidence and support versioned reassessment #157 selection, no writes (shadow stays read-only).
• Withheld counterparts are not resurrected; inaccessible/cross-tenant counterparts affect neither selection nor diagnostics and never leak ids; exposure counters cannot move packing; repeated evaluation is fully deterministic.

Rollout boundary (unchanged)

Legacy packet is byte-for-byte unchanged (packing: None, no packing_reason keys, /v1/recall output identical); CERTIFIED_SERVING_PROFILES == {"legacy"}; requesting a candidate profile on ordinary recall still 422s; no MCP selection; no #161 corroboration, no Context Ledger receipts, no #162 certification/cutover.

Tests

tests/test_recall_packing.py — 32 tests covering the issue's full matrix: known-root diversity via explicit derivation (1–5) plus transitive derivation chains and the packer input-surface pin, conflict preservation (6–11), direct+expanded integration including the explicit-derivation cross-workspace family and the same-hash/different-scope negative regression (12–14), budget boundaries + reconciliation (15–18), semantic separation (19–22), security/perf/compat (23–30). DB-backed tests skip without PostgreSQL, mirroring tests/test_recall_profile_semantic.py.

Validation

• Focused: tests/test_recall_packing.py 32 passed against live PostgreSQL 16 + pgvector; tests/test_recall_profile_semantic.py + tests/test_scope_completeness.py 64 passed.
• make check — lint + strict mypy green; root suite green except two portal failures (test_rotation_http_replay_status_and_credential_cutover, test_rotation_and_termination_functions_are_owner_only) reproduced identically on the unmodified PR head (environment-only; they need the full CI compose env).
• make compose-ci — green against real PostgreSQL 16 + pgvector under the non-owner app role.
• Fresh exact-head GitHub CI green.

Docs

ADR-160 supplement records the #192 contract (packing after admission+ranking; mechanically-known roots = explicit derived_from only, canonical content equality is not provenance; conflicts as representation obligations; hard budgets; no mutation; #161 ownership preserved) and the landed packing follow-ups are removed from Known limitations.

Post-merge follow-up (needs the merge SHA): update #160 progress with landed PR/head/merge SHA and remaining slices.