Repository navigation
feat: queue inspection and durable pause - #23
Merged
Merged
Conversation
Go 1.27 (varmq root needs it). contract pinned too: varmq's replace is not inherited.
OutcomeWatcher, ResultAcknowledgeable, JobCanceller, CancelNotifier, CancelAcknowledgeable on Queue and PriorityQueue (spec 6.13, 6.14). Sidecar tables <name>__sqliteq_jobs (outcome per Job ID, retention) and <name>__sqliteq_cancelled (marks, AUTOINCREMENT seq). Job ID = job_id, else item JSON id. Finish writes outcome + clears mark in its tx. Pops gate on live marks, drop with outcome cancelled, cap 1000. EnqueueOnce re-add clears both via AFTER INSERT trigger. Purge clears both. No pub/sub: one poll goroutine per queue object, 50 ms, only while watches or hooks exist. Txs begin IMMEDIATE.
A poll from before an OnCancel reset no longer advances lastSeq past the new hook's marks. A batch pop stopped at maxDrops reports one item left so the caller pops again. Go 1.27 under Breaking.
go.mod needs 1.27 now. Note that a multi-statement op opens its own tx in a round of one.
- Outcome poll reads records by seq cursor (AUTOINCREMENT), not a 500-id sample: a burst of finishes no longer loses watches. - Count trim spares records under 1 min old; first record of a queue object trims; poll sweeps retention + expired marks each minute. Index on marks expires_at. - OnCancel: per-hook from seq, hooks picked after the read: a hook added mid-poll gets its marks, stopped or closed ones get none. A failed first read gives a no-op hook, not a 24 h replay. - deliver reaches only watches started before the read (re-add). - Drop cap per tx: claims never pass it; single pops go on past it. - Queue names must not contain __sqliteq_.
Batched finishes (shared and fallback round), Close with live watch and hook, hooks during a poll and stale marks, repeat cancel, gated Dequeue to empty, kept-row ends, deliver generation, 3000-watch burst, reserved names, sidecars on a v0.2 file, retention floor and first record trim. Closed-queue watch check no longer vacuous.
Codecov: sweep was uncovered; DequeueWithAckId and Dequeue loop again past maxDrops, now tested.
varmq#179 merged; contract tagged.
Bad JSON, non-string or empty id: handed out by a gated pop, finishes as usual, no record.
QueueID sqliteq:<h>:fifo|prio:<name>, <h> = first 8 hex of sha256 of the file's resolved path. QueueStats: counts per row status in one statement (cancelled under failed), totals from new <name>__sqliteq_meta (insert trigger counts adds, finish counts in its tx, Purge resets). Jobs: pending hand-out order, processing by ack id, kept ended rows newest first, envelope data verbatim, errors from outcome record.
Pause stores row paused_at in <name>__sqliteq_meta; every pop statement selects nothing while it exists, in every process on the file, across restarts and Purge. An empty pop reads the row: refused, Paused set, PauseRefusals moves, batch reports 0 left. While Paused, the poll reads the row every pollEvery to see a Resume from another process. OnPauseChange hooks run outside package locks. A queue opened on a paused file starts paused.
observe checked gen, then set the cache: a Pause or Resume of the same object between the two was overwritten by a stale poll or refusal read (cache false while the row exists, poll stopped). setPaused now checks gen and swaps under pz.mu; poll start and hooks run after unlock.
Outside a tx the empty claim or pop and the pause row read are two statements; a Resume from another process between them made a refused pop look unrefused, so varmq raised ErrFailedToDequeue. An empty pop found unpaused now pops once more.
…tats, Purge keeps paused_at
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #22 (feat/outcome-cancel).
sqliteq queues become inspectable and pausable from varmq's API and vartop, same as redisq.
QueueandPriorityQueue:QueueID:sqliteq:<h>:fifo|prio:<name>,<h>= first 8 hex of sha256 of file's resolved path. Same file + name -> same ID in every process.QueueStats: counts per row status in one statement (cancelled under failed). Totals persist in new<name>__sqliteq_metatable: insert trigger counts adds, finish counts in its tx, Purge resets. Older table: totals seeded from its rows (8707769).Jobs: pending in hand-out order, processing by ack id, kept ended rows newest first, cancelled page (openapi 1.4.0), envelope data verbatim, errors from outcome record.QueuePauser,PauseNotifier):paused_atin meta table. Every pop selects nothing while it exists: every process on the file, across restarts and Purge.Pausedset,PauseRefusalsmoves, batch reports 0 left.OnPauseChangehooks run outside package locks. Queue opened on paused file starts paused.pz.mu(f0f82b5); empty pop found unpaused pops once more, so a Resume between pop and row read is not ErrFailedToDequeue (7165e94).Evidence
go vet ./...clean;go test -race -count=1 ./...green.paused_at), Jobs pages incl. cancelled, pause across handles on one file, stale refusal racing Resume.Merge Danger
<name>__sqliteq_metatable + insert trigger on first open. Older sqliteq ignores both.NOT EXISTSon the meta pause row. Paused file stops consumers in every process until Resume.