Repository navigation
feat: job outcome and cancel for varmq - #22
Merged
Merged
Conversation
Go 1.27 (varmq root needs it). contract pinned too: varmq's replace is not inherited.
OutcomeWatcher, ResultAcknowledgeable, JobCanceller, CancelNotifier, CancelAcknowledgeable on Queue and PriorityQueue (spec 6.13, 6.14). Sidecar tables <name>__sqliteq_jobs (outcome per Job ID, retention) and <name>__sqliteq_cancelled (marks, AUTOINCREMENT seq). Job ID = job_id, else item JSON id. Finish writes outcome + clears mark in its tx. Pops gate on live marks, drop with outcome cancelled, cap 1000. EnqueueOnce re-add clears both via AFTER INSERT trigger. Purge clears both. No pub/sub: one poll goroutine per queue object, 50 ms, only while watches or hooks exist. Txs begin IMMEDIATE.
A poll from before an OnCancel reset no longer advances lastSeq past the new hook's marks. A batch pop stopped at maxDrops reports one item left so the caller pops again. Go 1.27 under Breaking.
go.mod needs 1.27 now. Note that a multi-statement op opens its own tx in a round of one.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
- Outcome poll reads records by seq cursor (AUTOINCREMENT), not a 500-id sample: a burst of finishes no longer loses watches. - Count trim spares records under 1 min old; first record of a queue object trims; poll sweeps retention + expired marks each minute. Index on marks expires_at. - OnCancel: per-hook from seq, hooks picked after the read: a hook added mid-poll gets its marks, stopped or closed ones get none. A failed first read gives a no-op hook, not a 24 h replay. - deliver reaches only watches started before the read (re-add). - Drop cap per tx: claims never pass it; single pops go on past it. - Queue names must not contain __sqliteq_.
Batched finishes (shared and fallback round), Close with live watch and hook, hooks during a poll and stale marks, repeat cancel, gated Dequeue to empty, kept-row ends, deliver generation, 3000-watch burst, reserved names, sidecars on a v0.2 file, retention floor and first record trim. Closed-queue watch check no longer vacuous.
Codecov: sweep was uncovered; DequeueWithAckId and Dequeue loop again past maxDrops, now tested.
varmq#179 merged; contract tagged.
Bad JSON, non-string or empty id: handed out by a gated pop, finishes as usual, no record.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Job outcome and job cancel for varmq durable handles (varmq spec/redis-storage.md 1.6.0 §6.13, 1.7.0 §6.14). Without them, varmq handles on sqliteq report
ErrDetached/ErrCancelUnsupported. Pairs with goptics/varmq#179 (merged; pinscontract v0.6.0, varmq main 6bc553d); redisq equivalent goptics/redisq#33.What
QueueandPriorityQueueimplementcontract.OutcomeWatcher,ResultAcknowledgeable,JobCanceller,CancelNotifier,CancelAcknowledgeable(compile-time asserts).<name>__sqliteq_jobs(outcome per Job ID) and<name>__sqliteq_cancelled(marks, 24 h each), bothAUTOINCREMENTseq.cancelled, cap 1000 drops per tx. Batch pop at cap reports 1 left; single pops go on.EnqueueOncethat stores an id clears old record + mark (AFTER INSERTtrigger).Purgeclears both.OnCancelpoll by seq cursor (new rows only, 50 ms), one goroutine per queue object, only while watches or hooks exist. Same goroutine sweeps retention + expired marks each minute.OnCancelhooks carry own start seq, picked after the read: hook added mid-poll gets its marks, stopped hook none. Outcome reaches only watches started before the read (re-add safe).SQLite vs Redis spec
OnCancelper worker, so consumer queue objects poll marks for life (light SELECT ~20/s).job_idcolumn, else item JSONidparsed in Go.CancelJoband the sweep.Dequeuegated too. Kept rows (WithRemoveOnComplete(false)) get statuscancelled.BEGIN IMMEDIATE. Finish always in a tx, also rounds of one. One EXISTS read per pop.Blast Radius: sqliteq API, go.mod
idget outcome records.Queues.Close(last ref) now closes its queue handles.__sqliteq_(both layouts): sidecar names use it.Review
Six parallel reviews (Matt Pocock code-review, code-reviewer, feature-dev reviewer, security audit, Go concurrency, test engineer). Fixed in 7cc1fd6 / e009366: lost watches under a finish burst (reproduced: 2787 of 4000), hook races, re-add delivery, drop cap overrun, sweep, reserved names, test gaps. Not fixed: varmq
batchAckSource.fetchdropsremainingwhen n == 0 (varmq side); case-insensitiveidkey (matches varmq decode); varmq root is a pseudo-version of main 6bc553d (contract is tagged v0.6.0), tag before release.Tests
go test -race -count=3 ./...green,go vetclean. New tests inqueue_test.go/priority_queue_test.go: outcome via ack/reject/result, invalid JSON, ended-before-watch, two queue objects on one file, cancel waiting/running/ended, re-add, Purge, OnCancel, retention, drop cap, batched group-commit finishes, Close with live watch/hook, 3000-watch burst, sidecars on v0.2 file, undecodable envelopes as plain jobs, varmq end to end (NewResultQueue.Add->Result()across queue objects, remoteCancelwaiting + running).