Repository navigation
feat: EnqueueOnce dedupes jobs by id while their row exists - #16
Merged
Merged
Conversation
Queue.EnqueueOnce(item, id) and PriorityQueue.EnqueueOnce(item, priority, id) implement varmq's IDedupQueue and IDedupPriorityQueue. Opening a table adds a job_id TEXT column when it is missing and a unique partial index on it over the rows that have one. The insert ends in ON CONFLICT (job_id) WHERE job_id IS NOT NULL DO NOTHING, so a repeat stores nothing and returns true, while a NOT NULL failure such as Enqueue(nil) still fails. Enqueue and an empty id bind NULL. The index is <name>__sqliteq_job_id rather than <name>_job_id, for the reason the pending index has its suffix: tables and indexes share one namespace, and a queue named orders_job_id could not open beside orders.
A short EnqueueOnce section: a repeat stores nothing and returns true, and the id is remembered while its row exists (pending, processing, or completed with WithRemoveOnComplete(false)). The Compatibility line now says batch pops and EnqueueOnce need varmq v1.10.0 or later, and the schema list names job_id.
A nil option panicked in New, in the registry's option loop and in newQueue. All three now return one error, so New, NewQueue and NewPriorityQueue never panic on it; a rejected call creates no folder, file or table. Two tests pin the EnqueueOnce id: a job claimed and never acked keeps its id through the requeue on the next open, so a repeat stores nothing and the job runs once; Purge frees the ids of pending and processing rows. README: any name SQLite accepts as a table name under WithSingleFile, the same handle only while it is open, the single-file layout does not create its folder, batch claims only for a worker whose sole source is the sqliteq queue, a Durability section for WAL with synchronous=NORMAL, and the Windows device names as queue names in the folder layout.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #15 (
perf/group-commit-barge). Last of five for sqliteq v0.3.0.What changes
EnqueueOnce.Queue.EnqueueOnce(item, id)andPriorityQueue.EnqueueOnce(item, priority, id)implement varmq'sIDedupQueueandIDedupPriorityQueue.job_idcolumn if it is missing, plus a unique partial index<name>__sqliteq_job_idover the rows that have an id.ON CONFLICT (job_id) WHERE job_id IS NOT NULL DO NOTHING. A repeat stores nothing and returns true, while a NOT NULL failure such asEnqueue(nil)still fails.WithRemoveOnComplete(false)). It survives crash recovery, andPurgefrees it.Add.Enqueueand an empty id store NULL, which the partial index skips. varmq's default job-id generator returns an empty id, so only callers that set ids pay for the index.synchronous=NORMAL, and the Windows reserved names.Batch pops and
EnqueueOnceneed varmq v1.10.0. That version isn't tagged yet; it is current varmqmainplus the remaining-cap doc PR. With an older varmq the queue still works, just without batch pops orEnqueueOnce.Before release (not in this PR)
Newreturns an error; folder layout by default; ack ids are row ids;RequeueNoAckRowsis no longer exported; queue order is by id, notcreated_at;go 1.24;lucsky/cuiddropped; folder name rules; path and checkpoint errors surface atNewQueue/NewPriorityQueue.Tests
go test -race ./...passes. New tests cover:Purge.Mutation checks confirm the tests catch the likely mistakes: a wrong index name,
INSERT OR IGNORE, binding""instead of NULL, clearingjob_idon requeue, and skipping the delete inPurge.