fix(witan): report the merge divergences newest-record-wins used to swallow

[blarghmatey]

What are the relevant tickets?

No GitHub issue. Tracked in the witan work graph as
tk-witan-migrate-merge-silently-drops-divergent-edi-1f1453, under
wp-witan-multi-user-service-deployment-dcf6ee (delivery phase).

Filed from a real incident on 2026-08-19, during the re-merge that followed the
witan serve split-brain (#261).

Description (What does it do?)

• witan migrate merge discarded divergent edits with no signal.
  Reconciliation is newest-record-wins per node, which is right when one side is
  stale. It is a different thing when BOTH stores have written the same node
  since they last agreed: there the loser's edit is deleted, and several witan
  fields are append-only logs rather than values —
  WorkflowProject.description, which accretes status blocks, above all.
• The loss landed in the kept bucket, indistinguishable from the 3000-odd
  nodes that genuinely needed no action, and --dry-run printed kept-target
  for it exactly as it did for them. On 2026-08-19 the local store held the
  fourth update to a project description and production held the fifth; merging
  either way would have dropped one permanently. It was caught by hand, and
  nothing in the tool would have surfaced it.
• Divergence is most likely exactly when a merge is most needed. A merge is
  run because writes went somewhere they should not have, and that same window
  is when the other side keeps being edited.

Record-level timestamps cannot separate "the target is ahead" from "both sides
advanced" on their own — that is the task's own point 2, and it is why this
stores something at merge time.

• A merge now records a per-side watermark for the pair of stores: the
  newest comparison timestamp in the source, and the newest that will be in the
  target once this merge's winners land. The next merge marks any collision past
  both marks diverged, and the CLI names it with both timestamps and which
  side was kept.
• The merge rule does not change and nothing is auto-merged. A divergence is
  still resolved newest-record-wins; diverged cuts across
  updated/kept_target rather than partitioning with them. What changes is
  that the slugs to reconcile by hand are nameable — the manual fix on 08-19
  took minutes once it was known where to look.
• Each side is compared only against its own mark. A source is a laptop's
  clock and a deployed target is a cluster's; comparing across them would turn
  skew into invented divergences.
• The target mark includes the winners this merge is about to load. They
  land carrying their own source timestamps, so a mark taken from the pre-merge
  target alone sits below them, and every row the merge added would come back as
  a divergence next run. This is the false-positive that would have made the
  feature useless in practice.
• No watermark means "cannot tell", not "nothing diverged", and the first
  merge of a pair says so — but only when something actually collided, so a
  clean cutover into an empty graph stays quiet. --dry-run reports divergence
  and records nothing, having written none of the winners its mark would
  describe.
• Over the deployment the mark crosses as a store_merge parameter and is
  threaded batch to batch, so what the client records covers the whole merge
  rather than its last batch. A deployment too old to return one yields no
  mark rather than a partial one, and the CLI says the next merge will be blind.
• Marks live in ~/.config/witan/merge-watermarks.json
  ($WITAN_MERGE_WATERMARKS), beside the token cache. Client-side because the
  pairing is "this store, that deployment" and only the machine holding the
  source knows it. Fails soft throughout: a missing or corrupt file costs one
  merge's reporting, never the merge.
• The report renders source_at/target_at (ISO) rather than the raw
  source_ts/target_ts. Both real stores in a cutover are on omnigraph >= 0.9,
  which exports timestamps as epoch millis — the first smoke run printed two
  13-digit integers for a human to compare by eye. They are rendered server-side,
  where _parse_ts already lives, rather than adding a second parser to the CLI.

Also in this PR: the omnigraph edge digest refresh

Unrelated to the merge fix, but it was blocking this PR's witan-code job (and
every other PR's). edge is force-updated on every push to upstream main, so
the pinned digest goes stale on upstream's schedule rather than ours. It had.

All three tiers move together — the installer and both Dockerfiles — from the
2026-08-21T00:11Z build (62a9c3fe6b) to 2026-08-24T12:50Z (972f1666c5).

Three upstream commits are in the gap and only #522 carries any Rust
(perf(changes): bound candidate scans to transaction footprints); #543 and
#549 are docs and CI hygiene. That diff is confined to
crates/omnigraph/src/changes/* — the change feed, which witan does not use.
One part of it is not change-feed-internal and is on witan's write path:
table_store.rs now stamps omnigraph.no_by_source_delete=v1 on every keyed
merge_insert, so on every write witan makes. Upstream documents it as
read-advisory and only its own pruning path reads it, so no behaviour change —
but it is why this is not simply "docs and a perf fix".

Verified rather than assumed:

• each tarball downloaded and hashed locally, cross-checked against the
  published .sha256 in the same sitting — all three matched;
• upstream head and all three asset timestamps read before and after the
  downloads, identical both times, so the triple describes one build rather
  than a window;
• the linux/x86_64 value is exactly the got digest CI reported when it
  refused the stale pin — corroboration independent of my machine;
• version 0.10.0 and internal-schema 6 read off the new binary via
  bin/check_omnigraph_format.py, so not a rebuild-every-graph event;
• the error prose witan's classifier keys on checked against the artifact with
  strings, not only the diff — no rename;
• _download_omnigraph driven against the refreshed pin into a throwaway path
  (the exact step CI failed on) and it now succeeds.

One trap worth recording: manifest table version and ahead of manifest do
not appear in the binary — and did not in the previous build either. They are
assembled at runtime, so their absence from strings is not a regression. I
checked them against the old binary before reporting anything.

Still no upstream v0.10.0 release (latest tag is v0.9.0, 2026-08-07), which
remains the only reason this is on a moving tag at all. Refs
tk-omnigraph-0-10-0-edge-halved-the-write-ceiling-r-7ba7c2.

witan-core goes to 0.32.1: the pin ships to PyPI, where a stale digest fails
witan setup the same way it failed CI.

How can this be tested?

uv run --isolated --package witan-council --group test pytest mcp/servers/witan

966 pass, 23 of them new here. The pre-existing merge tests that assert
(added, updated, kept_target) pass unchanged — the decision strings did not
move, diverged is an added field and an added count.

Smoked end to end against two real local stores, driving the actual CLI render
path, not the tool return value. Seed one node, merge, edit both sides, then
merge again:

===== merge 2, dry run =====
Dry run against /tmp/.../wm/target.omni:
  kept-target  Memory           mem-shared-note-aa11bb
0 to add, 0 to update, 1 kept (target already newer-or-equal).

1 node(s) changed on BOTH sides since the last merge (watermark 2026-08-24T13:46:00.624423+00:00).
Newest-record-wins will keep one side and drop the other's edit. Reconcile these before you merge
for real — for an append-only field (a WorkflowProject description) this is lost content, not a
stale value:
  Memory           mem-shared-note-aa11bb
    source 2026-06-02T00:00:00Z  target 2026-06-03T00:00:00Z  -> kept target

(verbatim at COLUMNS=100, with only the scratch store path elided)

Note the line above the report: 1 kept is exactly what this merge printed
before the change, and is the whole complaint.

Coverage worth naming, because the interesting cases here are the ones that must
NOT be reported:

• a target-only edit since the last merge (the common shape of a kept-target
  on a shared graph) is not divergence;
• a row this merge itself loaded is not divergence on the next run;
• no watermark marks nothing, rather than guessing;
• clock skew between the two stores does not manufacture one;
• divergence resolved in the source's favour is reported too — that is the
  direction that overwrites the shared graph, and it counts as a perfectly
  ordinary-looking updated.

The MCP path is covered end to end through the in-memory server
(test_remote_proxy.py), which also pins that a dict parameter round-trips
through the binding — a merge that silently dropped since would report nothing
and look exactly like a clean merge.

Additional Context

On a real merge the report is post-hoc. By the time it prints, the losing
edit is already out of the target; the message says so in those words, and
points at the store that lost, which still has the text. The place this protects
you is --dry-run, which the runbook already makes mandatory before a cutover.
Refusing to merge on detected divergence was considered and not done — it would
block a cutover mid-flight on a judgment the operator is better placed to make.

This is fix 1 of the three the task lists, and deliberately not fix 3.
Field-aware reconciliation for known append-only text is a bigger change with
its own failure modes; naming the slugs was assessed in the task itself as most
of the value, and it is what a person can act on today.

The watermark is per-machine and not backed up. Losing it degrades to the
current behaviour (one merge with no report) rather than to anything wrong. A
watermark stored in the graph would travel with the store and be restored by
the very export/load cycle it exists to describe, which is why it is not.

Checklist:

• Roll the deployment to witan-council 0.29.0 before expecting a remote
  merge to report anything. Until then store_merge returns no watermark,
  the proxy passes None up, and the CLI prints the "next merge will be
  blind" warning — correct, but it means the first post-roll merge is still
  the one that establishes the mark, and the one after it is the first that
  can report.
• The operational rule in the task stands until then: before any
  witan migrate merge where both stores may have been written since they
  last agreed, diff WorkflowProject.description on both sides for the
  projects you care about. The row counts will not tell you.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BZBrdCenMsbribvmY6zVpe

[Copilot]

Pull request overview

Adds watermark-based divergence reporting to witan migrate merge while preserving newest-record-wins behavior.

Changes:

• Detects and reports records edited on both sides.
• Persists merge watermarks and threads them through remote batches.
• Adds tests, documentation, and the 0.29.0 release bump.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
uv.lock	Updates the workspace package version.
mcp/servers/witan/witan/server.py	Implements divergence detection and watermark generation.
mcp/servers/witan/witan/remote/proxy.py	Carries watermarks across remote batches.
mcp/servers/witan/witan/merge_watermark.py	Adds client-side watermark persistence.
mcp/servers/witan/witan/cli/migrate.py	Reads, reports, and records merge watermarks.
mcp/servers/witan/tests/test_remote_proxy.py	Tests remote divergence and batching behavior.
mcp/servers/witan/tests/test_migrate.py	Tests reconciliation, persistence, and CLI reporting.
mcp/servers/witan/pyproject.toml	Bumps witan-council to 0.29.0.
mcp/servers/witan/docs/migration-runbook.md	Documents divergence handling.
mcp/servers/witan/CHANGELOG.md	Records the 0.29.0 fix.
docs/reference/mcp-tools/memory.md	Documents new MCP parameters.
docs/reference/environment.md	Documents the watermark environment variable.
docs/reference/cli.md	Documents CLI divergence behavior.
docs/guides/migration-runbook.md	Mirrors the updated migration guide.
docs/_data/environment.toml	Adds generated environment-variable metadata.

Suppressed comments (1)

mcp/servers/witan/witan/cli/migrate.py:340

• _next_watermark always returns a dict, so an empty/local merge produces {"source_ts": None, "target_ts": None}, which is truthy and gets persisted here. On a later merge, that entry is also truthy to _report_divergence, suppressing the “cannot tell” warning, while _reconcile_nodes parses both marks as None and detects nothing. Treat a watermark as usable only when both timestamps are present before storing it.

    watermark = result.get("watermark")
    if not watermark:

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[blarghmatey]

Addressed all 7 review threads in b222ac9 — 6 fixed, 1 accepted deliberately with the reasoning recorded on the thread. Every finding was reproduced before it was fixed rather than taken on the description.

Fixed

• Stale mark after a partial merge. Batches commit independently, so a half-finished merge leaves rows the standing mark predates; the next run read exactly those as an independent target edit. The mark is now retired before the first batch commits and reinstalled only on success. A dry run commits nothing and keeps it.
• Non-UTF-8 watermark file crashed the merge. UnicodeDecodeError is a ValueError but not an OSError, so it escaped the catch entirely — from a module whose docstring promises to fail soft.
• Pair keyed by CLI spelling, not store identity. graph.omni from two directories was two stores under one key; /tmp/g.omni, ../tmp/g.omni and file:///tmp/g.omni were one store under three. Local paths now key on their resolved absolute path.
• First-run notice promised too much. "The next one will report divergence" is false under --dry-run, which records nothing. Corrected in the CLI, the runbook, and the regenerated mirror.
• The suppressed comment was right too (no thread to reply on): {"source_ts": null, "target_ts": null} was the worst possible stored value — truthy, so it suppressed the "cannot tell" notice, but unparseable, so it detected nothing. Silence reading as "nothing diverged" is the one outcome this feature exists to prevent. is_usable now gates both the write and the read.

Accepted, not fixed — the target watermark folds in winner timestamps, mixing the source's clock into the target's threshold. Reproduced: a source 1h ahead hides a genuine target edit made 30 min after the merge, where synchronised clocks report it. Kept because the alternative is worse in the common case — a pure target mark reports a divergence on every row the merge itself loaded as soon as the source touches it again. The bend and its cost are now documented in _next_watermark instead of claiming the per-side rule holds strictly, and pinned by a test asserting current behaviour so closing the window is deliberate. The real fix (per-record baselines) is tracked as tk-merge-divergence-the-target-watermark-mixes-in-t-794bd1.

One test fake was also wrong and had been hiding a bug: kept_target: 1 with an empty decision list is a shape a real result cannot have, since the counts are derived from the decisions.

974 tests pass (8 new here).