Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,22 +156,21 @@ On case-insensitive filesystems (macOS, Windows), spell a path the same way ever

## Checkpoints

SQLite appends writes to a write-ahead log (WAL) and folds them back into the database file in a checkpoint. `WithCheckpointPages(n)` sets how many pages the WAL may reach before SQLite checkpoints on its own. The default is 1000 pages, SQLite's own setting, which keeps the WAL near 4 MB per file. With it, an `Enqueue` occasionally waits for a checkpoint's disk sync.
SQLite appends writes to a write-ahead log (WAL) and folds them back into the database file in a checkpoint. `WithCheckpointPages(n)` sets how many pages the WAL may reach before SQLite checkpoints on its own. The default is 1000 pages, SQLite's own setting. A page is 4 KB, so a checkpoint starts once the WAL reaches about 4 MB. That size is where a checkpoint is triggered, not a hard limit on the file. A checkpoint runs inside the commit of the write that crosses that size, while that write holds the file's write lock. So with the default, any write on that file (an `Enqueue`, a claim, an ack or a purge) occasionally waits for a checkpoint's disk sync.

```go
queuesManager, err := sqliteq.New("data", sqliteq.WithCheckpointPages(10000))
```

Raising it makes those waits rarer, which cuts `Enqueue` tail latency and raises throughput, at the price of a larger WAL on disk. For example:
Raising it makes those waits rarer, which cuts `Enqueue` tail latency and raises throughput, at the price of a larger WAL on disk: 10000 pages triggers a checkpoint at about 41 MB.

- 4000 pages: the WAL peaks near 16 MB per file.
- 10000 pages: the WAL peaks near 41 MB per file.
In the `Mixed1Q` and `Drain1Q` rows of the [varmq-benchmarks `sqlite` benchmark](https://github.com/goptics/varmq-benchmarks/tree/main/sqlite) (one queue in one file, 20,000 jobs), its `walMB` metric, the largest size the WAL file reached, was 4.1 MB with 1000 pages and 41 MB with 10000 pages, measured on one machine.

The WAL is a log, not extra stored data: the database holds the same items either way. `n` must be at least 1, because `New` returns an error otherwise. Everything that reaches one file must use the same value: a `NewQueue` or `NewPriorityQueue` that reaches a file already open with a different value returns an error naming the value the file is open with. `New` itself does not open files, so it does not detect this.

## Compatibility

sqliteq does not import varmq. Batch pops and `EnqueueOnce` are being added for v0.3 and need varmq ≥ v1.10.0.
sqliteq does not import varmq. Queues and priority queues implement `DequeueBatchWithAckIds`, so with varmq ≥ v1.10.0 a worker claims several jobs in one statement. An older varmq never calls it and claims one job per call, which still works. `EnqueueOnce` comes in a later release.

## How It Works

Expand All @@ -192,6 +191,8 @@ SQLiteQ uses a SQLite database to store queue items with the following schema:
- Operations leverage SQLite's indexing for logarithmic time complexity rather than true constant-time
- SQLite's WAL (Write-Ahead Logging) mode is enabled for better concurrent access
- One partial index covers the pending items in queue order; opening a v0.2 table replaces its old indexes with it
- Writes to one file (adds, claims, acks, purges) take one lock in the process, so they wait for each other in Go rather than in SQLite's busy handler. `Len` and `Values` do not take it
- A batch pop reports the items left after it, counting at most 256; `Len` gives the exact count

## 👤 Author

Expand Down
30 changes: 30 additions & 0 deletions priority_queue_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -623,3 +623,33 @@ func TestPriorityOrderByID(t *testing.T) {
}
}
}

func TestBatchPriorityOrder(t *testing.T) {
qs := mustNew(t, t.TempDir())
defer qs.Close()
pq, err := qs.NewPriorityQueue("q")
if err != nil {
t.Fatalf("Failed to create priority queue: %v", err)
}

// Job names are "<priority>/<insert index>"
for i, p := range []int{3, 1, 1, 0, 2} {
if !pq.Enqueue([]byte(fmt.Sprintf("%d/%d", p, i)), p) {
t.Fatalf("Enqueue %d failed", i)
}
}

items, ackIDs := make([]any, 5), make([]string, 5)
n, remaining, err := pq.DequeueBatchWithAckIds(items, ackIDs)
if err != nil || n != 5 || remaining != 0 {
t.Fatalf("n=%d remaining=%d err=%v, want 5, 0, nil", n, remaining, err)
}
for i, want := range []string{"0/3", "1/1", "1/2", "2/4", "3/0"} {
if got := string(items[i].([]byte)); got != want {
t.Errorf("items[%d] = %q, want %q", i, got, want)
}
if !pq.Acknowledge(ackIDs[i]) {
t.Errorf("Acknowledge(%s) failed", ackIDs[i])
}
}
}
100 changes: 83 additions & 17 deletions queue.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,8 @@ func newQueue(f *file, name string, priority bool, opts ...Option) (*Queue, erro
}

// The db is shared by every queue of the file, so it is not closed on errors
if err := q.initTable(); err != nil {
return nil, fmt.Errorf("failed to initialize table: %w", err)
}
if err := q.migrate(); err != nil {
return nil, fmt.Errorf("failed to migrate indexes: %w", err)
if err := q.initSchema(); err != nil {
return nil, err
}
if err := q.prepare(); err != nil {
return nil, fmt.Errorf("failed to prepare statements: %w", err)
Expand All @@ -71,6 +68,20 @@ func newQueue(f *file, name string, priority bool, opts ...Option) (*Queue, erro
return q, nil
}

// initSchema creates or migrates the table and its index under the write mutex
func (q *Queue) initSchema() error {
q.f.mu.Lock()
defer q.f.mu.Unlock()

if err := q.initTable(); err != nil {
return fmt.Errorf("failed to initialize table: %w", err)
}
if err := q.migrate(); err != nil {
return fmt.Errorf("failed to migrate indexes: %w", err)
}
return nil
}

// initTable creates the table as v0.2 defines it, so old tables need no
// rebuild, and adds the priority column to a priority queue's table when
// it is missing
Expand Down Expand Up @@ -200,6 +211,9 @@ func (q *Queue) closeStmts() {

// requeueNoAck puts rows left in processing without an ack back to pending
func (q *Queue) requeueNoAck() error {
q.f.mu.Lock()
defer q.f.mu.Unlock()

_, err := q.f.db.Exec(
fmt.Sprintf("UPDATE %s SET status = 'pending', updated_at = ? WHERE status = 'processing' AND ack = 0", q.table),
time.Now().UTC(),
Expand Down Expand Up @@ -258,6 +272,9 @@ func (q *Queue) enqueue(item any, priority int) bool {
}

now := time.Now().UTC()
q.f.mu.Lock()
defer q.f.mu.Unlock()

var err error
if q.priority {
_, err = q.insert.Exec(item, now, now, priority)
Expand All @@ -270,6 +287,26 @@ func (q *Queue) enqueue(item any, priority int) bool {
// claimN moves up to n pending rows to processing in one statement and
// returns them in queue order
func (q *Queue) claimN(n int) ([]claimed, error) {
q.f.mu.Lock()
out, err := q.claimRows(n)
q.f.mu.Unlock()
if err != nil {
return nil, err
}

// RETURNING order is unspecified
slices.SortFunc(out, func(a, b claimed) int {
if c := cmp.Compare(a.priority, b.priority); c != 0 {
return c
}
return cmp.Compare(a.id, b.id)
})
return out, nil
}

// claimRows runs the claim statement and reads every row it returns; the
// caller holds the write mutex
func (q *Queue) claimRows(n int) ([]claimed, error) {
rows, err := q.claim.Query(time.Now().UTC(), n)
if err != nil {
return nil, err
Expand All @@ -290,18 +327,7 @@ func (q *Queue) claimN(n int) ([]claimed, error) {
out = append(out, c)
}
// Reading to the end runs the statement to completion, which commits it
if err := rows.Err(); err != nil {
return nil, err
}

// RETURNING order is unspecified
slices.SortFunc(out, func(a, b claimed) int {
if c := cmp.Compare(a.priority, b.priority); c != 0 {
return c
}
return cmp.Compare(a.id, b.id)
})
return out, nil
return out, rows.Err()
}

// Dequeue removes and returns the next item from the queue
Expand All @@ -311,6 +337,10 @@ func (q *Queue) Dequeue() (any, bool) {
return nil, false
}

q.f.mu.Lock()
defer q.f.mu.Unlock()

// Deferred last, so the rows close (and the statement ends) before unlocking
rows, err := q.pop.Query()
if err != nil {
return nil, false
Expand Down Expand Up @@ -345,6 +375,36 @@ func (q *Queue) DequeueWithAckId() (any, bool, string) {
return c[0].data, true, strconv.FormatInt(c[0].id, 10)
}

// DequeueBatchWithAckIds implements varmq.IBatchAcknowledgeable: it moves up
// to min(len(items), len(ackIDs)) items to processing in one statement, in
// queue order, each under its row id as ack id. remaining is the pending
// count after the move, capped at remainingCap. n is 0 with a nil error when
// the queue is empty or closed.
func (q *Queue) DequeueBatchWithAckIds(items []any, ackIDs []string) (n, remaining int, err error) {
want := min(len(items), len(ackIDs))
if q.closed.Load() || want == 0 {
return 0, 0, nil
}

c, err := q.claimN(want)
if err != nil {
return 0, 0, err
}
if len(c) == 0 {
return 0, 0, nil
}
for i, row := range c {
items[i], ackIDs[i] = row.data, strconv.FormatInt(row.id, 10)
}

// The rows are claimed already: an error here would make varmq drop them,
// so a failed count reports items left and the next pop finds out
if err := q.countCapped.QueryRow().Scan(&remaining); err != nil {
return len(c), 1, nil
}
return len(c), remaining, nil
}

// Acknowledge marks a processing item as completed, deleting it unless
// WithRemoveOnComplete(false) is set
// Returns true only if exactly one processing row had this ack ID
Expand All @@ -354,6 +414,9 @@ func (q *Queue) Acknowledge(ackID string) bool {
return false
}

q.f.mu.Lock()
defer q.f.mu.Unlock()

var (
result sql.Result
err error
Expand Down Expand Up @@ -405,6 +468,9 @@ func (q *Queue) Values() []any {

// Purge removes all items from the queue
func (q *Queue) Purge() {
q.f.mu.Lock()
defer q.f.mu.Unlock()

q.f.db.Exec(fmt.Sprintf("DELETE FROM %s", q.table))
}

Expand Down
Loading
Loading