Skip to content

feat: recover by owner and heartbeat, not on open - #28

Open
fahimfaisaal wants to merge 10 commits into
mainfrom
feat/owner-heartbeat-recovery
Open

fahimfaisaal wants to merge 10 commits into
mainfrom
feat/owner-heartbeat-recovery

Conversation

@fahimfaisaal

@fahimfaisaal fahimfaisaal commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Closes #27

Cause

file.open ran requeueNoAck on a table's first open in each process: every processing row with ack = 0 went back to pending. Rows had no owner, the file had no liveness signal, and fin matched by row id only. So any second process on the file (consumer, producer, inspector, pauser) requeued jobs a live process was running, and a stale ack could end another process's run. A crashed process's rows came back only on the next open.

What changed

  • Owner. Random id per file (process x database file), made in newFile. New nullable column owner on queue tables (migration guarded by hasColumn, next to addJobID). The claim sets it; a requeue clears it.
  • Liveness. File-level tables __sqliteq_owners (id, expires_at, shifted) and __sqliteq_clock (id, swept_at, moved). Only a process that claims has an owner row.
  • Sweep. One write transaction under file.mu: gap guard, own heartbeat, delete of expired owner rows, clock write. Runs on a table's first open (replaces requeueNoAck) and on a ticker goroutine per file, every timeout / 6. It requeues processing rows whose owner has no live row, one scan per open table, only when it has to: on a first open, when it deleted an expired owner row, when the moved count shows another process deleted one, and once per timeout as a safety net.
  • Legacy rows. processing with owner IS NULL requeued on a table's first open only, never by the ticker.
  • Finish. fin has AND owner = ? for every caller (finish, finishDirect, finishTx, the cancel drop in claimGated); all go through end. An ack for a lost row returns false.
  • Close. Last release of a file: stop the goroutine, requeue rows WHERE owner = <me>, delete the owner row.
  • Option. WithHeartbeatTimeout(d) on the store: default 30s, New refuses under 5s. Mismatch on a file already open in the process is an error, as for checkpoint pages. All three acquire call sites pass it.
  • No new index. EXPLAIN QUERY PLAN shows the requeue is SCAN q without a partial index on processing rows. An index fixed that but cost every claim and ack (+12% ack, +15% batch pop), so it is gone and the sweep scans rarely instead (above). fin stays SEARCH q USING INTEGER PRIMARY KEY (rowid=?) in both forms.
  • The plain claim is still one statement with no transaction.
  • README and CHANGELOG: multi-process rules, mixed-version warning, the accepted same-owner stale-ack limit. The ponytail: comment in file.open is gone.

Deviations from the issue's design

The design came from source reading; these are the places the code disagreed.

  1. Every process with a queue open sweeps, also one that never claims. The issue said the first claim starts the goroutine and a producer needs none. VarMQ's directManager.Next returns early when Len() == 0, so a consumer that waits on an empty queue never claims, would never sweep, and a dead owner's rows would stay. A producer still writes no owner row; it does write the clock row each interval.
  2. Wake goes through OnPauseChange. sqliteq has no enqueue wake to reuse (VarMQ wakes its own worker on Add). The only adapter-side wake is the pause hook, so a sweep that sees a requeue calls the hooks with false on unpaused queues. Marked ponytail:; upgrade path is a wake interface in VarMQ's contract.
  3. Counter in the clock row (moved). Not in the issue. It moves when a sweep deletes an expired owner row or requeues. Every process that sees it move scans its open tables once and wakes its workers. Without it only the process whose sweep did the requeue woke its worker (review finding), and the sweep would need an index or a scan on every tick.
  4. A heartbeat is shifted once between two beats of its owner (shifted). The issue's guard shifted on every long gap, which blocked recovery for good when every later process was short-lived (review finding).
  5. First claim, and a claim after a stall of two intervals, runs the sweep transaction, not a bare insert of the owner row, so the gap guard runs before the row is written (review finding). The claim statement itself is unchanged.
  6. Negative gap (clock set back) shifts too, past the same limit.
  7. Crash then restart waits about one timeout, also for a single process. Follows from the issue's design (heartbeat is the only liveness), but it is the most visible behaviour change: before, the restart's open returned the rows at once. In README, CHANGELOG, and TestOwnerRestart.
  8. No index on processing rows; owner id is 8 random bytes (16 hex chars), to keep claim and ack cost down. The issue said to add the index if the requeue scans. It scans, but the index cost more than scanning rarely.
  9. No benchmarks existed in the repo. Added bench_test.go (claim, ack, batch pop, each with a 100k pending backlog) and ran the same file on origin/main.
  10. The 5s minimum: the issue asked for a shorter interval or a note. It is a note (option doc, README): a beat can wait the 5s busy timeout behind another process's write.

Decided before this PR, per the issue's open questions: same-owner stale ack is accepted and documented (no attempt counter); owner id in inspection is out of scope; VarMQ docs (plan step 9) are not touched.

Review findings

Two independent reviews of the whole diff against origin/main (correctness, security), a second correctness pass on the fix commits, and a third on the index removal (464bae4). The last fix commit (8bed901) applies the reviewer's proposed fixes and was not re-reviewed.

Finding Severity Result
Gap guard shifted a dead owner on every open after a gap; short-lived processes never got its rows (both reviews, reproduced) high Fixed: single shift per beat. TestOwnerDeadShortLived
Consumer not woken when another process's sweep or hand-back did the requeue (reproduced, 8 of 20 runs) medium Fixed: moved counter. TestWakeAfterOthersRequeue
Clock set back, then a first claim before the next sweep: the fresh owner row was shifted and looked dead low Fixed: beat is a sweep. TestClockGap/claim_after_clock_set_back
Sweep errors dropped; one failing table stopped the heartbeat low Fixed: errors logged, a table's failure does not stop the rest
sweepLoop took registry.mu; a slow open of another file delayed this file's heartbeat low Fixed: lock-free snapshot of the open queues
Non-integer swept_at failed every sweep for good low Fixed: CAST. TestClockGap/unreadable_clock
NULL id in __sqliteq_owners disabled reaping low Fixed: PRIMARY KEY NOT NULL
A first-open sweep used up the requeue count before the loop saw it (single file) low Fixed. TestOpenKeepsWake
After the index removal: a tick with a table list from before an open used up the count and never scanned the new table (single file) medium Fixed: the list holds a table before its first sweep, and the tick reads it under file.mu. TestOpenListsTableBeforeSweep
After the index removal: a row claimed just as its owner row was deleted, by a process that died before its next beat, was never scanned low Fixed: every process scans its open tables once per timeout. TestSweepReadsTablesOnDeath
NULL NOT IN (empty set) is true: the ticker sweep took legacy rows found by own test Fixed: owner IS NOT NULL AND
Docs: "Close of the last Queues" wrong with a WorkflowStore on the same file low Fixed in README and CHANGELOG
Two stalls with no beat of a live owner between them requeue its row (cost of the single shift) low Not fixed. Double run, not loss. Documented
Per-tick work under the write lock grows with processing rows and owner rows (25 ms at 300k processing rows, 161 ms with 300k owner rows; measured with the index) low Changed by the index removal: a tick now touches only the two small tables; a table scan runs once per timeout and per death, and grows with the whole table
Two New on one folder with different timeouts share __sqliteq_workflows.db: the second NewWorkflowStore fails, Purge from the other keeps its jobs low Not fixed. Same as checkpoint pages today
A permanently broken table logs each tick note Not fixed

Clean in both reviews: SQL construction (identifiers through quoteIdent, values bound), the reserved __sqliteq_ prefix against the new names, lock order (queues.mu -> registry.mu -> file.mu, the loop now takes only file.mu), no wait on the goroutine under a lock it needs, no double close, no goroutine leak on release including the failed-open path, no path that loses a job.

Benchmarks

go test -run '^$' -bench . -benchtime=20000x, 16 runs each, the three versions interleaved in one session, i7-13700, Linux. 100k pending rows behind the ones taken. Medians.

Benchmark origin/main with the index (d96785a) this branch (no index)
Ack (Acknowledge) 15407 ns 17304 ns (+12%) 16136 ns (+5%)
Batch pop, 64 per call, per item 1744 ns 1999 ns (+15%) 1820 ns (+4%)
Claim (DequeueWithAckId) 35105 ns 42424 ns (+21%) 42133 ns (+20%)

The single-claim benchmark is too noisy to read: one version varies between 22 and 46 µs from run to run, and an earlier session showed this branch 10% under origin/main. Batch pop does the same row write and costs 4% more per item.

What remains is the fix itself: a claim writes the owner id into the row, an ack matches it.

Cost moved off the hot path: each process scans each open queue table once per timeout and once per process death, under the write lock. Before this PR every first open did that scan.

Tests

go vet ./... clean. go test -race -count=1 ./...: ok github.com/goptics/sqliteq 22.1s. Owner tests also pass with -count=3.

From the issue's list:

  • Live owner untouched: TestOwnerLive, and TestKilledProcess (two real processes, a VarMQ worker in each).
  • Dead owner without reopen: TestOwnerDead (survivor opened first and never claimed), TestKilledProcess (helper process killed mid-job, job runs in the survivor's worker).
  • Stale ack from another owner: TestStaleAck (Acknowledge, Reject, finishDirect).
  • Legacy NULL rows: TestLegacyRows, and the v0.2.3 file in TestMigrateV02File.
  • Clean Close: TestCleanClose.
  • Clock gap: TestClockGap (long gap, clock set back, no clock row, unreadable clock, claim after a step back).
  • Producer-only: TestProducerOnly (no owner row; see deviation 1 for the goroutine).
  • Priority queue, WithRemoveOnComplete(false), single file: the live, dead, stale-ack and close cases run on all four variants.
  • Also: TestOwnerRestart, TestOwnerDeadShortLived, TestBeatAfterStall, TestHeartbeatOption, TestWakeAfterOthersRequeue, TestOpenKeepsWake, TestOpenListsTableBeforeSweep, TestSweepReadsTablesOnDeath.

Two owners on one file: otherProcess opens a second file on the same path through newFile (the constructor acquire uses); TestKilledProcess re-executes the test binary.

Not verified

  • A real host suspend; the guard is tested by moving swept_at and expires_at.
  • Operating systems other than Linux.
  • Two new-version processes that open an old-schema file at the same instant: both can pass hasColumn and one ALTER TABLE ADD COLUMN owner then fails. addJobID has the same pattern.
  • Sweep scan time on a very large table (millions of rows); measured only at 100k.
  • Mixed timeouts: a process with a short timeout that opens a file swept at a longer interval shifts heartbeats once and delays a reap by that gap (ponytail: in owner.go).

Each file in a process gets an owner id, stored on the rows it claims.
A claiming process keeps a heartbeat row; every process with a queue
open sweeps and requeues only rows of owners without a live heartbeat.
A finish matches the owner. The last release hands own rows back.

Refs #27
Review findings:
- a dead owner's heartbeat moved on at every open after a gap, so
  short-lived processes never got its rows; it now moves once until
  the owner beats again
- only the process whose sweep requeued woke its queues; the clock
  row now counts requeues and each process wakes at its next sweep
- a first claim after a clock step back was moved with the old gap;
  the claim's beat is now a sweep, so the guard runs first
- sweep errors are logged, one failing table no longer stops the beat
- the sweep loop takes no registry lock
- unreadable clock value and NULL owner id no longer stop reaping

Refs #27
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.92308% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
owner.go 96.92% 4 Missing ⚠️
queue.go 90.00% 1 Missing ⚠️
registry.go 97.05% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

A claim and an ack maintained the index on every call (+8% ack, +16%
batch pop). A sweep now reads a queue table on its first open, when it
drops an expired owner row, or when the file's count shows another
process dropped one. The count moves on each drop, so every process
reads its open tables once per death.

Refs #27
Review findings on the index removal:
- a tick with a table list from before an open could use up the count
  and never read the new table; the list now holds the table before
  its first sweep and the tick reads it under the write mutex
- a row claimed just as its owner row was dropped had no later read;
  the loop now reads every open table once per timeout

Refs #27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Second process on one database file requeues jobs another process is running

1 participant