Repository navigation
feat: recover by owner and heartbeat, not on open - #28
Open
fahimfaisaal wants to merge 10 commits into
Open
fahimfaisaal wants to merge 10 commits into
fahimfaisaal wants to merge 10 commits into
Conversation
Each file in a process gets an owner id, stored on the rows it claims. A claiming process keeps a heartbeat row; every process with a queue open sweeps and requeues only rows of owners without a live heartbeat. A finish matches the owner. The last release hands own rows back. Refs #27
Review findings: - a dead owner's heartbeat moved on at every open after a gap, so short-lived processes never got its rows; it now moves once until the owner beats again - only the process whose sweep requeued woke its queues; the clock row now counts requeues and each process wakes at its next sweep - a first claim after a clock step back was moved with the old gap; the claim's beat is now a sweep, so the guard runs first - sweep errors are logged, one failing table no longer stops the beat - the sweep loop takes no registry lock - unreadable clock value and NULL owner id no longer stop reaping Refs #27
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
A claim and an ack maintained the index on every call (+8% ack, +16% batch pop). A sweep now reads a queue table on its first open, when it drops an expired owner row, or when the file's count shows another process dropped one. The count moves on each drop, so every process reads its open tables once per death. Refs #27
Review findings on the index removal: - a tick with a table list from before an open could use up the count and never read the new table; the list now holds the table before its first sweep and the tick reads it under the write mutex - a row claimed just as its owner row was dropped had no later read; the loop now reads every open table once per timeout Refs #27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #27
Cause
file.openranrequeueNoAckon a table's first open in each process: everyprocessingrow withack = 0went back topending. Rows had no owner, the file had no liveness signal, andfinmatched by row id only. So any second process on the file (consumer, producer, inspector, pauser) requeued jobs a live process was running, and a stale ack could end another process's run. A crashed process's rows came back only on the next open.What changed
file(process x database file), made innewFile. New nullable columnowneron queue tables (migration guarded byhasColumn, next toaddJobID). The claim sets it; a requeue clears it.__sqliteq_owners (id, expires_at, shifted)and__sqliteq_clock (id, swept_at, moved). Only a process that claims has an owner row.file.mu: gap guard, own heartbeat, delete of expired owner rows, clock write. Runs on a table's first open (replacesrequeueNoAck) and on a ticker goroutine per file, every timeout / 6. It requeuesprocessingrows whose owner has no live row, one scan per open table, only when it has to: on a first open, when it deleted an expired owner row, when themovedcount shows another process deleted one, and once per timeout as a safety net.processingwithowner IS NULLrequeued on a table's first open only, never by the ticker.finhasAND owner = ?for every caller (finish,finishDirect,finishTx, the cancel drop inclaimGated); all go throughend. An ack for a lost row returnsfalse.releaseof a file: stop the goroutine, requeue rowsWHERE owner = <me>, delete the owner row.WithHeartbeatTimeout(d)on the store: default 30s,Newrefuses under 5s. Mismatch on a file already open in the process is an error, as for checkpoint pages. All threeacquirecall sites pass it.EXPLAIN QUERY PLANshows the requeue isSCAN qwithout a partial index onprocessingrows. An index fixed that but cost every claim and ack (+12% ack, +15% batch pop), so it is gone and the sweep scans rarely instead (above).finstaysSEARCH q USING INTEGER PRIMARY KEY (rowid=?)in both forms.ponytail:comment infile.openis gone.Deviations from the issue's design
The design came from source reading; these are the places the code disagreed.
directManager.Nextreturns early whenLen() == 0, so a consumer that waits on an empty queue never claims, would never sweep, and a dead owner's rows would stay. A producer still writes no owner row; it does write the clock row each interval.OnPauseChange. sqliteq has no enqueue wake to reuse (VarMQ wakes its own worker onAdd). The only adapter-side wake is the pause hook, so a sweep that sees a requeue calls the hooks withfalseon unpaused queues. Markedponytail:; upgrade path is a wake interface in VarMQ's contract.moved). Not in the issue. It moves when a sweep deletes an expired owner row or requeues. Every process that sees it move scans its open tables once and wakes its workers. Without it only the process whose sweep did the requeue woke its worker (review finding), and the sweep would need an index or a scan on every tick.shifted). The issue's guard shifted on every long gap, which blocked recovery for good when every later process was short-lived (review finding).TestOwnerRestart.processingrows; owner id is 8 random bytes (16 hex chars), to keep claim and ack cost down. The issue said to add the index if the requeue scans. It scans, but the index cost more than scanning rarely.bench_test.go(claim, ack, batch pop, each with a 100k pending backlog) and ran the same file onorigin/main.Decided before this PR, per the issue's open questions: same-owner stale ack is accepted and documented (no attempt counter); owner id in inspection is out of scope; VarMQ docs (plan step 9) are not touched.
Review findings
Two independent reviews of the whole diff against
origin/main(correctness, security), a second correctness pass on the fix commits, and a third on the index removal (464bae4). The last fix commit (8bed901) applies the reviewer's proposed fixes and was not re-reviewed.TestOwnerDeadShortLivedmovedcounter.TestWakeAfterOthersRequeueTestClockGap/claim_after_clock_set_backsweepLooptookregistry.mu; a slow open of another file delayed this file's heartbeatswept_atfailed every sweep for goodCAST.TestClockGap/unreadable_clockidin__sqliteq_ownersdisabled reapingPRIMARY KEY NOT NULLTestOpenKeepsWakefile.mu.TestOpenListsTableBeforeSweepTestSweepReadsTablesOnDeathNULL NOT IN (empty set)is true: the ticker sweep took legacy rowsowner IS NOT NULL ANDCloseof the lastQueues" wrong with aWorkflowStoreon the same fileNewon one folder with different timeouts share__sqliteq_workflows.db: the secondNewWorkflowStorefails,Purgefrom the other keeps its jobsClean in both reviews: SQL construction (identifiers through
quoteIdent, values bound), the reserved__sqliteq_prefix against the new names, lock order (queues.mu->registry.mu->file.mu, the loop now takes onlyfile.mu), no wait on the goroutine under a lock it needs, no double close, no goroutine leak onreleaseincluding the failed-open path, no path that loses a job.Benchmarks
go test -run '^$' -bench . -benchtime=20000x, 16 runs each, the three versions interleaved in one session, i7-13700, Linux. 100k pending rows behind the ones taken. Medians.origin/maind96785a)Acknowledge)DequeueWithAckId)The single-claim benchmark is too noisy to read: one version varies between 22 and 46 µs from run to run, and an earlier session showed this branch 10% under
origin/main. Batch pop does the same row write and costs 4% more per item.What remains is the fix itself: a claim writes the owner id into the row, an ack matches it.
Cost moved off the hot path: each process scans each open queue table once per timeout and once per process death, under the write lock. Before this PR every first open did that scan.
Tests
go vet ./...clean.go test -race -count=1 ./...:ok github.com/goptics/sqliteq 22.1s. Owner tests also pass with-count=3.From the issue's list:
TestOwnerLive, andTestKilledProcess(two real processes, a VarMQ worker in each).TestOwnerDead(survivor opened first and never claimed),TestKilledProcess(helper process killed mid-job, job runs in the survivor's worker).TestStaleAck(Acknowledge,Reject,finishDirect).TestLegacyRows, and the v0.2.3 file inTestMigrateV02File.TestCleanClose.TestClockGap(long gap, clock set back, no clock row, unreadable clock, claim after a step back).TestProducerOnly(no owner row; see deviation 1 for the goroutine).WithRemoveOnComplete(false), single file: the live, dead, stale-ack and close cases run on all four variants.TestOwnerRestart,TestOwnerDeadShortLived,TestBeatAfterStall,TestHeartbeatOption,TestWakeAfterOthersRequeue,TestOpenKeepsWake,TestOpenListsTableBeforeSweep,TestSweepReadsTablesOnDeath.Two owners on one file:
otherProcessopens a secondfileon the same path throughnewFile(the constructoracquireuses);TestKilledProcessre-executes the test binary.Not verified
swept_atandexpires_at.hasColumnand oneALTER TABLE ADD COLUMN ownerthen fails.addJobIDhas the same pattern.ponytail:inowner.go).