Skip to content

feat: EnqueueOnce dedupes jobs by id while their row exists - #16

Merged
fahimfaisaal merged 3 commits into
perf/group-commit-bargefrom
feat/enqueue-once
Oct 23, 2026
Merged

fahimfaisaal merged 3 commits into
perf/group-commit-bargefrom
feat/enqueue-once

Conversation

@fahimfaisaal

Copy link
Copy Markdown
Member

Stacked on #15 (perf/group-commit-barge). Last of five for sqliteq v0.3.0.

What changes

  • EnqueueOnce. Queue.EnqueueOnce(item, id) and PriorityQueue.EnqueueOnce(item, priority, id) implement varmq's IDedupQueue and IDedupPriorityQueue.
  • Schema. Opening a table adds a job_id column if it is missing, plus a unique partial index <name>__sqliteq_job_id over the rows that have an id.
  • Insert. It ends in ON CONFLICT (job_id) WHERE job_id IS NOT NULL DO NOTHING. A repeat stores nothing and returns true, while a NOT NULL failure such as Enqueue(nil) still fails.
  • How long an id is remembered: while its row exists (pending, processing, or completed with WithRemoveOnComplete(false)). It survives crash recovery, and Purge frees it.
  • Cost for plain Add. Enqueue and an empty id store NULL, which the partial index skips. varmq's default job-id generator returns an empty id, so only callers that set ids pay for the index.
  • Review fixes:
    • a nil option now returns an error instead of panicking;
    • README wording: name rules, the single-file folder must already exist, batch claims need sqliteq to be the worker's only source, a Durability section on synchronous=NORMAL, and the Windows reserved names.

Batch pops and EnqueueOnce need varmq v1.10.0. That version isn't tagged yet; it is current varmq main plus the remaining-cap doc PR. With an older varmq the queue still works, just without batch pops or EnqueueOnce.

Before release (not in this PR)

  • Pin varmq v1.10.0.
  • Regenerate the v0.2.3 and v0.3.0 benchmark results.
  • README performance numbers.
  • CHANGELOG, with the full breaking-change list: New returns an error; folder layout by default; ack ids are row ids; RequeueNoAckRows is no longer exported; queue order is by id, not created_at; go 1.24; lucsky/cuid dropped; folder name rules; path and checkpoint errors surface at NewQueue/NewPriorityQueue.
  • Tag v0.3.0.

Tests

go test -race ./... passes. New tests cover:

  • a repeat in a later round, in the same grouped round, and under 32 concurrent callers;
  • while processing, after ack in both modes, and the priority path;
  • a v0.2.3 file;
  • an empty id;
  • recovery after a claim, and Purge.

Mutation checks confirm the tests catch the likely mistakes: a wrong index name, INSERT OR IGNORE, binding "" instead of NULL, clearing job_id on requeue, and skipping the delete in Purge.

Queue.EnqueueOnce(item, id) and PriorityQueue.EnqueueOnce(item,
priority, id) implement varmq's IDedupQueue and IDedupPriorityQueue.
Opening a table adds a job_id TEXT column when it is missing and a
unique partial index on it over the rows that have one. The insert
ends in ON CONFLICT (job_id) WHERE job_id IS NOT NULL DO NOTHING, so a
repeat stores nothing and returns true, while a NOT NULL failure such
as Enqueue(nil) still fails. Enqueue and an empty id bind NULL.

The index is <name>__sqliteq_job_id rather than <name>_job_id, for the
reason the pending index has its suffix: tables and indexes share one
namespace, and a queue named orders_job_id could not open beside orders.
A short EnqueueOnce section: a repeat stores nothing and returns true,
and the id is remembered while its row exists (pending, processing, or
completed with WithRemoveOnComplete(false)). The Compatibility line now
says batch pops and EnqueueOnce need varmq v1.10.0 or later, and the
schema list names job_id.
A nil option panicked in New, in the registry's option loop and in
newQueue. All three now return one error, so New, NewQueue and
NewPriorityQueue never panic on it; a rejected call creates no folder,
file or table.

Two tests pin the EnqueueOnce id: a job claimed and never acked keeps
its id through the requeue on the next open, so a repeat stores nothing
and the job runs once; Purge frees the ids of pending and processing
rows.

README: any name SQLite accepts as a table name under WithSingleFile,
the same handle only while it is open, the single-file layout does not
create its folder, batch claims only for a worker whose sole source is
the sqliteq queue, a Durability section for WAL with synchronous=NORMAL,
and the Windows device names as queue names in the folder layout.
@fahimfaisaal
fahimfaisaal added this pull request to stack #17 October 23, 2026 12:08
@codecov

codecov Bot commented Oct 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.31034% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
queue.go 73.91% 3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@fahimfaisaal
fahimfaisaal merged commit 98ce80c into main Oct 23, 2026
1 of 2 checks passed
@fahimfaisaal
fahimfaisaal deleted the feat/enqueue-once branch October 23, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant