Skip to content

dedupe content merges leave stale content_hash and poison enrichment #623

Description

@EtanHey

Problem

merge_duplicate_chunk and merge_existing_chunk_content replace chunks.content with [Merged duplicate originals]... and recompute dedupe/simhash fields, but do not update chunks.content_hash.

The enrichment controller hashes the merged content. Drain prefers the non-null stored hash and rejects every resulting update as stale:

Skipping stale enrichment for chunk_id=... content_hash mismatch

Live evidence

Three M1 rows showed the same invariant break:

  • content starts with the merged-duplicate marker;
  • dedupe_audit records a simhash merge;
  • stored content_hash differs from SHA-256 of current stripped content;
  • enrich_status and enriched_at remain null;
  • drain cycles those same IDs continuously.

Expected

Any transaction that changes chunks.content must atomically update every content-derived field, including content_hash.

Acceptance

  • Red/green regression tests cover both merge functions.
  • content_hash == sha256(content.strip()) after a content-changing merge.
  • Existing stale rows have a bounded repair path.
  • Fresh enrichment updates apply once and retire instead of looping.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions