Skip to content

feat: queue inspection and durable pause - #23

Merged
fahimfaisaal merged 23 commits into
mainfrom
feat/queue-inspector
Nov 9, 2026
Merged

fahimfaisaal merged 23 commits into
mainfrom
feat/queue-inspector

Conversation

@fahimfaisaal

Copy link
Copy Markdown
Member

Summary

Stacked on #22 (feat/outcome-cancel).

sqliteq queues become inspectable and pausable from varmq's API and vartop, same as redisq.

  • QueueInspector on Queue and PriorityQueue:
    • QueueID: sqliteq:<h>:fifo|prio:<name>, <h> = first 8 hex of sha256 of file's resolved path. Same file + name -> same ID in every process.
    • QueueStats: counts per row status in one statement (cancelled under failed). Totals persist in new <name>__sqliteq_meta table: insert trigger counts adds, finish counts in its tx, Purge resets. Older table: totals seeded from its rows (8707769).
    • Jobs: pending in hand-out order, processing by ack id, kept ended rows newest first, cancelled page (openapi 1.4.0), envelope data verbatim, errors from outcome record.
  • Durable pause (QueuePauser, PauseNotifier):
    • Pause stores row paused_at in meta table. Every pop selects nothing while it exists: every process on the file, across restarts and Purge.
    • Empty pop reads the row: refused, Paused set, PauseRefusals moves, batch reports 0 left.
    • While paused, poll reads row every 50 ms to see Resume from another process. OnPauseChange hooks run outside package locks. Queue opened on paused file starts paused.
    • Fixes from review: gen check + cache swap in one step under pz.mu (f0f82b5); empty pop found unpaused pops once more, so a Resume between pop and row read is not ErrFailedToDequeue (7165e94).
  • README "Processes and files" (recovery runs on every process's first open), CHANGELOG.

Evidence

  • go vet ./... clean; go test -race -count=1 ./... green.
  • Tests appended to queue_test.go and priority_queue_test.go: stats/totals (no-op finishes, seeded ended rows, Purge keeps paused_at), Jobs pages incl. cancelled, pause across handles on one file, stale refusal racing Resume.
  • 6-lens review, 2 fix rounds.

Merge Danger

  • Door: two-way for code; one-way-ish for data: adds <name>__sqliteq_meta table + insert trigger on first open. Older sqliteq ignores both.
  • Blast Radius: every pop statement gains a NOT EXISTS on the meta pause row. Paused file stops consumers in every process until Resume.
  • Known gaps: recovery still runs on every process's first open (owner/heartbeat later). Totals of rows added before upgrade are seeded, not exact.

Go 1.27 (varmq root needs it). contract pinned too: varmq's replace
is not inherited.
OutcomeWatcher, ResultAcknowledgeable, JobCanceller, CancelNotifier,
CancelAcknowledgeable on Queue and PriorityQueue (spec 6.13, 6.14).

Sidecar tables <name>__sqliteq_jobs (outcome per Job ID, retention)
and <name>__sqliteq_cancelled (marks, AUTOINCREMENT seq). Job ID =
job_id, else item JSON id. Finish writes outcome + clears mark in its
tx. Pops gate on live marks, drop with outcome cancelled, cap 1000.
EnqueueOnce re-add clears both via AFTER INSERT trigger. Purge clears
both. No pub/sub: one poll goroutine per queue object, 50 ms, only
while watches or hooks exist. Txs begin IMMEDIATE.
A poll from before an OnCancel reset no longer advances lastSeq past
the new hook's marks. A batch pop stopped at maxDrops reports one item
left so the caller pops again. Go 1.27 under Breaking.
go.mod needs 1.27 now. Note that a multi-statement op opens its own
tx in a round of one.
- Outcome poll reads records by seq cursor (AUTOINCREMENT), not a
  500-id sample: a burst of finishes no longer loses watches.
- Count trim spares records under 1 min old; first record of a queue
  object trims; poll sweeps retention + expired marks each minute.
  Index on marks expires_at.
- OnCancel: per-hook from seq, hooks picked after the read: a hook
  added mid-poll gets its marks, stopped or closed ones get none. A
  failed first read gives a no-op hook, not a 24 h replay.
- deliver reaches only watches started before the read (re-add).
- Drop cap per tx: claims never pass it; single pops go on past it.
- Queue names must not contain __sqliteq_.
Batched finishes (shared and fallback round), Close with live watch
and hook, hooks during a poll and stale marks, repeat cancel, gated
Dequeue to empty, kept-row ends, deliver generation, 3000-watch burst,
reserved names, sidecars on a v0.2 file, retention floor and first
record trim. Closed-queue watch check no longer vacuous.
Codecov: sweep was uncovered; DequeueWithAckId and Dequeue loop again
past maxDrops, now tested.
varmq#179 merged; contract tagged.
Bad JSON, non-string or empty id: handed out by a gated pop, finishes
as usual, no record.
QueueID sqliteq:<h>:fifo|prio:<name>, <h> = first 8 hex of sha256 of
the file's resolved path. QueueStats: counts per row status in one
statement (cancelled under failed), totals from new <name>__sqliteq_meta
(insert trigger counts adds, finish counts in its tx, Purge resets).
Jobs: pending hand-out order, processing by ack id, kept ended rows
newest first, envelope data verbatim, errors from outcome record.
Pause stores row paused_at in <name>__sqliteq_meta; every pop statement
selects nothing while it exists, in every process on the file, across
restarts and Purge. An empty pop reads the row: refused, Paused set,
PauseRefusals moves, batch reports 0 left. While Paused, the poll reads
the row every pollEvery to see a Resume from another process.
OnPauseChange hooks run outside package locks. A queue opened on a
paused file starts paused.
observe checked gen, then set the cache: a Pause or Resume of the same
object between the two was overwritten by a stale poll or refusal read
(cache false while the row exists, poll stopped). setPaused now checks
gen and swaps under pz.mu; poll start and hooks run after unlock.
Outside a tx the empty claim or pop and the pause row read are two
statements; a Resume from another process between them made a refused
pop look unrefused, so varmq raised ErrFailedToDequeue. An empty pop
found unpaused now pops once more.
@fahimfaisaal
fahimfaisaal changed the base branch from feat/outcome-cancel to main November 9, 2026 08:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant