Skip to content

perf: batch pops for varmq and one write mutex per file - #14

Merged
fahimfaisaal merged 3 commits into
perf/lean-statementsfrom
perf/batch-pop
Oct 23, 2026
Merged

fahimfaisaal merged 3 commits into
perf/lean-statementsfrom
perf/batch-pop

Conversation

@fahimfaisaal

@fahimfaisaal fahimfaisaal commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Stacked on #13 (perf/lean-statements). Third of five for sqliteq v0.3.0.

What changes

  • Batch pops for varmq. DequeueBatchWithAckIds implements varmq's IBatchAcknowledgeable. It claims up to min(len(items), len(ackIDs)) rows in one statement and reports the remaining pending count, capped at 256. varmq only checks whether that count is zero (docs: let batch storage cap the remaining count varmq#127).
  • One write mutex per file. Every write on a file takes it: enqueue, claim, pop, ack, purge, recovery and schema setup. Writers on one file now wait in Go instead of sleeping in SQLite's busy handler. Len and Values do not take it.
  • README: batch pops, the write lock, and WAL peaks quoted from the published benchmark's walMB metric.

Effect

The WAL stays bounded under load: about 4.2 MB with WithCheckpointPages(1000) and about 41 MB with 10000. Before this PR, the soak test reached about 400 MB.

In the end-to-end varmq benchmark, single-queue rows gain +46% to +189% over PR 2.

Tests

go test -race ./... passes. New tests cover batch pops, concurrent writers on one file, and a crash after a batch claim.

DequeueBatchWithAckIds claims up to min(len(items), len(ackIDs)) rows in
one statement and reports the pending count after it, capped at 256.

Every write on a file (enqueue, claim, pop, ack, purge, recovery, schema
setup) holds the file's mutex, so writers on one file queue in Go instead
of sleeping in SQLite's busy handler. Len and Values do not take it.
@codecov

codecov Bot commented Oct 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.95652% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
queue.go 86.95% 3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@fahimfaisaal
fahimfaisaal merged commit ab54d6d into main Oct 23, 2026
2 checks passed
@fahimfaisaal
fahimfaisaal deleted the perf/batch-pop branch October 23, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant