fix(replication): stop the resume-cursor write freezing the apply worker

[kriszyp]

What

The replication apply loop in resources/Table.ts persists the per-peer resume cursor with dbisDb.put(). On RocksDB that call is not asynchronous — openRocksDatabase (resources/databases.ts:202) deliberately aliases put onto putSync so thrown errors aren't masked by a promise:

db.put = db.putSync as any;
db.remove = db.removeSync as any;

So the cursor write absorbs RocksDB write-stall back-pressure on the worker's event loop. Measured locally during bulk catch-up at Fabric's RocksDB sizing: up to 101s inside a single call, and ~700s of a 725s catch-up window spent inside these writes in aggregate.

A frozen event loop stops everything else on that thread, including the replication keep-alive — so the sender's receive watchdog concludes the receiver is dead and tears the subscription down (4 reconnects per local run, 9 in the failing nightly), discarding in-flight work.

This change stages the cursor into a transaction and commits that. Staging is an in-memory WriteBatch append; Transaction.commit() is the one natively-async write path in rocksdb-js. The same stall is now awaited off-thread instead of blocking. Either half failing (staging or commit) aborts the transaction rather than leaking a native handle that would pin a snapshot and hold off compaction — the shape evict() and commitItems() already use in this file.

What this does not do

It does not make catch-up faster. The wait is genuine storage back-pressure and is unchanged in duration — this only stops it rendering the worker unresponsive. The nightly Large-data large-catchup job will not be turned green by this alone; catch-up throughput is bounded by RocksDB flush capacity at the configured WriteBufferManager size, which is a provisioning question tracked separately.

Why the write stays where it is

The obvious alternative — folding the cursor write into the data transaction it describes — is unsafe. harper-pro's replicationConnection.ts assigns endTxnEvent.localTime = lastDurableSequenceId inside onCommit, and may await flushCopyRowsDurable() first. Core reads event.localTime only after awaiting onCommit. That clamp is what stops the persisted cursor advancing past a blob that is not yet durable, so writing the cursor inside the data transaction (which commits before onCommit runs) would persist the unclamped value and break the no-data-loss guarantee.

So the write keeps its exact position in the sequence; only the mechanism changes.

Review follow-ups (second round)

• A failed seqTransaction.commit() now aborts. It previously returned its rejection with the handle still open. Raised by both review bots; real, and the same leak the staging-failure branch immediately above it was already guarding against.

• The RocksDB branch now has coverage — unitTests/resources/replicationSeqCursor.test.js drives an end_txn event carrying remoteNodeIds through the apply loop (an intermediate/replication source, the only path that reaches this closure) and asserts:

  1. the cursor is staged into a transaction and never written with a blocking store write (put/unstaged putSync on __dbis__ are both spied);
  2. a forced cursor-commit failure aborts that transaction and the apply loop keeps applying the next transaction's writes and cursor.

  Both assertions were checked against the pre-fix code: stripping the isRocksDB branch fails (1); removing just the commit .catch fails (2). Skipped under HARPER_STORAGE_ENGINE=lmdb, where put is genuinely async.

Test notes

4 local 4GB catch-up runs per arm against a real 2-node cluster, at Fabric's RocksDB sizing (8GB instance → blockCache 15% / WBM 5%), alternating builds to control for machine drift:

variant	throughput	converged	event-loop-lag warnings	watchdog teardowns
this change	5.7ᵀ, 6.5, 7.5, 8.3 MB/s	3/4	0, 0, 0, 0	0, 0, 0, 0
baseline	5.3ᵀ, 5.6ᵀ, 11.3, 16.4 MB/s	2/4	11, 11, 4, 5	4, 4, 3, 1

(ᵀ = exceeded the test's budget)

The freeze/teardown columns are the claim. The throughput column is explicitly not — the spread is wide enough on both arms that n=4 supports no throughput conclusion, and none is claimed. Instrumenting the commit directly showed why the two arms should be equivalent there: the cursor commit costs ~1.2ms mean when RocksDB is healthy and blocks only when the shared WriteBufferManager is stalled — the identical condition the data commits face. Baseline performs the same operations in the same order (putSync after await committingTxn.committed), so it already paid a data-commit stall followed by a cursor-write stall, sequentially. Only the thread-blocking property differs.

One further 4 GB run on the current head (post-merge with main, at the same Fabric RocksDB sizing — blockCache=1228 MB wbm=409 MB): converged in 528.8s of a 720s budget at 7.7 MB/s, with 0 event-loop-lag warnings and 0 watchdog teardowns. Longest no-progress gap 44s, against the harness's 300s wedge guard. Full core unitTests/resources suite: 1240 passing.

Also run: harper-pro test:unit (394 passing), and fullyConnectedReplication / directionalFlowReplication / receiveBacklogMemory (11/11, including the LMDB replication suite which exercises the non-RocksDB branch).

integrationTests/cloneNode/cloneResume.test.mjs fails on this branch — but it fails identically on baseline (same error, same ~184s), so it is pre-existing and unrelated.

Reviewer notes / open concerns

• The LMDB branch now awaits dbisDb.put() where it previously left the promise floating. That's a small deliberate behaviour change (it also stops a potential unhandled rejection); lmdb-js batches writes so the latency is negligible, and the LMDB replication suite passes. Happy to revert to fire-and-forget if you'd rather I not touch that path.
• main has since been merged in, so this is current; the paired harper-pro PR (fix(replication): unblock the apply worker's resume-cursor write; make large-catchup failures legible harper-pro#604) points its core submodule at this branch's head.

Refs harper-pro#603

🤖 Generated with Claude Code (Claude Opus 4.8)

[gemini-code-assist]

Code Review

This pull request modifies resources/Table.ts to prevent RocksDB write-stall back-pressure on the event loop during bulk catch-up. It stages the sequence ID update into a RocksTransaction and commits it asynchronously when isRocksDB is true, and awaits the update. The review feedback points out that if seqTransaction.commit() fails, the transaction handle remains open, which can leak native resources, pin a snapshot, and prevent compaction. It suggests catching the error on commit and aborting the transaction.

[claude]

Reviewed; no blockers found. The previously-flagged unaborted seqTransaction.commit() failure is now fixed in 9b1a254 (aborts + rethrows, symmetric with the staging-failure branch), with new test coverage for the abort path.

[cb1kenobi]

Crazy find.