Skip to content

perf: group-commit acks and enqueues per file - #15

Merged
fahimfaisaal merged 4 commits into
perf/batch-popfrom
perf/group-commit-barge
Oct 23, 2026
Merged

fahimfaisaal merged 4 commits into
perf/batch-popfrom
perf/group-commit-barge

Conversation

@fahimfaisaal

Copy link
Copy Markdown
Member

Stacked on #14 (perf/batch-pop). Fourth of five for sqliteq v0.3.0.

What changes

Enqueue and Acknowledge calls that run at the same time on one file now commit together in one transaction (group commit). See commit.go.

  • Leading a round. The first caller with no round running leads. It commits what has queued up under the file's write mutex, at most its own round plus one more.
  • Handing off. At the end of its turn the leader gives up the lead and wakes the oldest waiting caller. A caller that arrives first may take the lead, and then commits the woken caller's op in its own round.
  • Results. Every caller gets its own result. If the grouped transaction fails, it rolls back and each op runs again in its own transaction, so one bad call fails alone.
  • One-op rounds run as a single autocommit statement, which avoids the cost of a database/sql Tx.
  • No user code inside a round. Items are converted in the calling goroutine, so a panicking driver.Valuer stays with its own caller and cannot wedge the batcher.
  • Claims, pops and purge still take the write mutex directly.

The branch keeps the two earlier hand-off designs as separate commits. The first passed the lead to a fixed successor, which left the file idle between rounds and lost about 20% at 24 CPUs. The second added the one-op autocommit. Both are kept for review history; squash on merge if you prefer.

Effect

From the end-to-end varmq benchmark, A/B/A against PR 3 with 8 samples per row:

  • no row regresses;
  • 10 queues in one file (WithSingleFile) gain +47% to +108%;
  • single-file, 10 queues, checkpoint 1000 is +25% to +33% above PR 1, with the WAL still bounded at about 4.2 MB.

Tests

go test -race ./... passes, including the batcher tests at GOMAXPROCS 1, 2 and 24. Tests cover:

  • the leader cap and hand-off;
  • the per-op fallback after a failed round;
  • per-caller results from one committed round;
  • a panicking item;
  • Close while rounds are in flight.

Concurrent Enqueue and Acknowledge calls on one file now share a
transaction. The first caller with no round running leads: it commits
what has queued up, at most its own round plus one more, under the
file's write mutex, then hands the lead to the oldest waiting caller.

Each caller still gets its own result. If the grouped transaction
fails, the round rolls back and each call runs in a transaction of its
own, so one bad call fails alone. Claims, pops and purge keep taking
the write mutex directly.
A database/sql Tx costs a cancelable context and a goroutine per
transaction. A round of one op has nothing to share, and in the folder
layout about half the rounds hold one op, so at GOMAXPROCS=24 the Tx
setup cost more than grouping saved (Mixed10Q/Folder-ckpt1000 fell
about 19% below the per-call commits). A lone op now runs as one
autocommit statement, still under the write mutex.
A leader ending its turn used to keep the lead set and pass it to the
oldest waiting caller, so the next round waited for that goroutine to
be woken and scheduled. Now the lead is given up and the oldest waiter
is woken for it; a caller that arrives first takes it instead and
commits the waiter's op in its own round. A leader still commits at
most its own round plus one more.

An op's done channel holds two: at most one wake-up and the outcome,
so a commit never blocks on a caller that has not woken yet.
database/sql converts each argument inside Exec, and Exec now runs in
whichever goroutine leads the round. An item whose driver.Valuer
panicked there unwound past the batcher, so the lead stayed taken, the
round's other callers never got a result, and a grouped round left its
write transaction open. Enqueue now converts the item with the default
parameter converter, as database/sql would for this driver, before the
op joins a round, so such a panic stays with its own caller.

Tests: a panicking item panics in its caller and later writes still
work; a committing grouped round gives each op its own result from one
transaction; the batching test queues writes behind a held round
instead of counting rounds under a racing scheduler.
@codecov

codecov Bot commented Oct 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
queue.go 87.87% 2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@fahimfaisaal
fahimfaisaal merged commit d306604 into main Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant