Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
35cf8ad
build: pin varmq and contract at 0a9288e
fahimfaisaal Nov 8, 2026
010dd7c
feat: job outcome and cancel for varmq
fahimfaisaal Nov 8, 2026
c51876e
docs: changelog for job outcome and cancel
fahimfaisaal Nov 8, 2026
5ea8a08
fix: cancel poll era, report items left at drop cap
fahimfaisaal Nov 8, 2026
9374f80
chore: Go 1.27 in CI and README, commitRound doc
fahimfaisaal Nov 8, 2026
7cc1fd6
fix: review findings on outcome and cancel
fahimfaisaal Nov 8, 2026
e009366
test: outcome and cancel review gaps
fahimfaisaal Nov 8, 2026
665759f
test: sweep, single pops past the drop cap
fahimfaisaal Nov 8, 2026
f42847f
build: pin contract v0.6.0, varmq main 6bc553d
fahimfaisaal Nov 8, 2026
e159469
test: undecodable envelope is a plain job
fahimfaisaal Nov 8, 2026
b4594f8
feat: QueueInspector on Queue and PriorityQueue
fahimfaisaal Nov 8, 2026
46c3caa
feat: durable queue pause (varmq QueuePauser, PauseNotifier)
fahimfaisaal Nov 8, 2026
5f0b799
docs: queue inspection and pause in README, CHANGELOG
fahimfaisaal Nov 8, 2026
8707769
fix: seed an older table's totals from its rows
fahimfaisaal Nov 8, 2026
f0f82b5
fix: pause cache gen check and swap in one step
fahimfaisaal Nov 8, 2026
7165e94
fix: pop again when a Resume lands after an empty pop
fahimfaisaal Nov 8, 2026
75726d9
feat: Jobs pages cancelled rows (openapi 1.4.0)
fahimfaisaal Nov 8, 2026
31b2c22
test: totals on no-op finishes and seeded ended rows, file pause in s…
fahimfaisaal Nov 8, 2026
42fb1ee
docs: cancelled page, seeded totals gap, processes sharing a file
fahimfaisaal Nov 8, 2026
734eccb
test: stale refusal racing Resume; no-op finish totals with kept rows
fahimfaisaal Nov 9, 2026
584a58f
refactor: setPaused takes only the gen; own calls pass gen.Add(1)
fahimfaisaal Nov 9, 2026
2bf6272
docs: recovery runs on every process's first open of a queue
fahimfaisaal Nov 9, 2026
52b28fc
merge main (#22 squash) into feat/queue-inspector
fahimfaisaal Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `Reject(ackID, reason)` on `Queue` and `PriorityQueue`, required by varmq's `IAcknowledgeable` since varmq#173: finishes an item that failed for good, never requeued. Deleted like an ack, or with `WithRemoveOnComplete(false)` kept with `status` `failed`. Same group commit as `Acknowledge`. The reason is the job's outcome error.
- Job outcome (varmq spec 6.13): `WatchOutcome` (`contract.OutcomeWatcher`) and `AcknowledgeResult` (`contract.ResultAcknowledgeable`) on both queues. A job with a Job ID (`EnqueueOnce` id, else the item's JSON `id`) gets a record in `<name>__sqliteq_jobs` when it ends, same transaction as the finish. Watches poll it by seq every 50 ms (new records only), from any queue object on the file, any process. Retention: completed 1000 / 24 h, failed and cancelled 10000 / 7 d; the count limit spares records under 1 min old. A polling queue object also sweeps retention and expired marks once a minute.
- Job cancel (varmq spec 6.14): `CancelJob`, `OnCancel`, `AcknowledgeCancelled` (`contract.JobCanceller`, `CancelNotifier`, `CancelAcknowledgeable`). Marks in `<name>__sqliteq_cancelled`, 24 h each. Every pop drops a marked waiting job (outcome `cancelled`, at most 1000 per transaction; a single-item pop goes on past that, a batch pop reports an item left); `OnCancel` polls new marks. `ErrJobEnded` when the record shows it ended.
- Queue inspection (`contract.QueueInspector`) on both queues, so varmq's REST API and vartop list sqliteq queues. `QueueID` is `sqliteq:<h>:fifo|prio:<name>`, `<h>` the first 8 hex of the SHA-256 of the file's resolved path, `<name>` in lower case. `QueueStats`: counts per row status in one statement (cancelled under failed; completed and failed only with `WithRemoveOnComplete(false)`), totals submitted / completed / failed from new table `<name>__sqliteq_meta` (an insert trigger counts adds; a finish counts in its transaction; an older table starts from its rows, so jobs deleted before the upgrade are missing; `Purge` resets them), and the pause. `Jobs`: pending in hand-out order, processing by ack id, kept completed, failed (with cancelled) and cancelled rows newest first, payload the envelope's `data` verbatim, errors from the outcome record. Attempts are not counted (0 pending, 1 after).
- Durable queue pause (varmq spec 6.7, 6.8): `Pause`, `Resume`, `Paused`, `PauseRefusals` (`varmq.QueuePauser`) and `OnPauseChange` (`varmq.PauseNotifier`). The flag is a row of `<name>__sqliteq_meta`; every pop statement selects nothing while it exists, in every process on the file; it outlives restarts and `Purge`. A refused pop sets `Paused`; while set, the queue object polls the row every 50 ms to see a `Resume` from another process.
- `EnqueueOnce` that stores an id drops its old record and mark (trigger). `Purge` clears records and marks too. `Queue.Close` drops watches and hooks.

### Breaking
Expand Down
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,16 @@ queue := worker.WithPersistentQueue(q)
queue.Add("hello")
```

### Inspection and pause

Both queue types implement varmq's `QueueInspector`, `QueuePauser` and `PauseNotifier`, so a bound queue shows up in varmq's REST API (`GET /queues`, `GET /queues/{id}/jobs`, `PATCH /queues/{id}/actions/pause|resume`) and in vartop, as a redisq queue does.

- **Queue ID**: `sqliteq:<h>:fifo:<name>` or `sqliteq:<h>:prio:<name>`. `<h>` is the first 8 hex digits of the SHA-256 of the database file's resolved path, so every process on the file agrees, two files with one queue name never collide, and no path shows. `<name>` is in lower case. sqliteq has no in-memory mode, so every queue has a file. A file reached through two different paths (another mount, say) gets two IDs.
- **Counts**: rows per `status`, read in one statement. Cancelled rows count as failed. Completed and failed rows exist only with `WithRemoveOnComplete(false)`; by default they are deleted, so those counts are 0. Kept rows have no retention.
- **Totals**: `submitted` counts stored adds (a repeat `EnqueueOnce` is not one), `completed` and `failed` count finishes (`AcknowledgeCancelled` counts as failed; a waiting job dropped for a cancel counts as neither). They live in `<name>__sqliteq_meta`, start from the rows an older table holds when this version first opens it, and `Purge` resets them. Seeded totals count the rows the table holds then, so jobs deleted before the upgrade (the default `WithRemoveOnComplete(true)`) are missing from them.
- **Jobs**: pending in hand-out order, processing by ack id, completed, failed (with cancelled, `state` `cancelled`) and cancelled newest finish first. `payload` is the varmq envelope's `data`, or the stored item itself (as a JSON string when it is not JSON). A job without an id shows its ack id once handed out. sqliteq does not count attempts: they read 0 while pending and 1 after. A kept row loses its start time when it ends. A failed job's `errors` holds its reason when the job has an id and its outcome record is still kept.
- **Pause**: `Pause()` stores a `paused_at` row in `<name>__sqliteq_meta`. Every pop selects nothing while it exists, so the pause reaches every process on the file at once, outlives restarts, and survives `Purge`; only `Resume()` ends it. Adds, acks and rejects go on. `Paused()` is this object's cache: a pop refused for the pause sets it, and while it is set the queue polls the row every 50 ms, so a `Resume` from another process wakes this one's workers within that time.

sqliteq does not import varmq. Batch pops and `EnqueueOnce` need varmq ≥ v1.10.0. From that release, a worker whose only source is the sqliteq queue claims several jobs in one statement through `DequeueBatchWithAckIds` (a worker bound to several queues claims one job per call), and a job added with an id (`WithJobId`, or one from `WithJobIdGenerator`) goes through `EnqueueOnce`. An older varmq calls neither: it claims one job per call and adds every job with `Enqueue`, which still works.

## Storage
Expand Down Expand Up @@ -228,11 +238,11 @@ In the folder layout the queue name becomes a file name, so it may only contain

Names are case-insensitive in both layouts: `Jobs` and `jobs` are the same queue, stored in `jobs.db` in the folder layout.

### One process per file
### Processes and files

Run one process per database file. sqliteq keeps a registry in the process, keyed by the file's real path (made absolute, with symlinks resolved). Every `New` in the process that reaches the same file shares one connection pool and one write lock, so two `New` calls on one path never write through two competing pools. Two `Queues` values can therefore hold the same queue, and you get the same handle from both. The file closes when the last `Queues` using it is closed.
sqliteq keeps a registry in the process, keyed by the file's real path (made absolute, with symlinks resolved). Every `New` in the process that reaches the same file shares one connection pool and one write lock, so two `New` calls on one path never write through two competing pools. Two `Queues` values can therefore hold the same queue, and you get the same handle from both. The file closes when the last `Queues` using it is closed.

Another process is not coordinated with this one. The first time a process opens a queue, it puts every item still marked as processing back to pending, so a second process opening a file that is in use would rerun the first process's running jobs. sqliteq does not take an OS lock to stop that.
Other processes can open the same file. SQLite's lock orders their writes, and what lives in the file reaches every process: a pause stops every process's pops at once, and a `Resume`, a cancel or a job outcome reaches the others within 50 ms (they poll). Recovery is the limit. Every process that opens a queue runs recovery on its first open of that queue, whatever it does with it: consumers, producers, observers (a process that only serves the REST API) and `Pause`/`Resume` callers alike. Recovery puts every item still marked as processing back to pending, so a process that opens the queue while another one runs its jobs makes those jobs run again. sqliteq does not take an OS lock to stop that. Sharing a file is safe only when every process opens the queue before any of them takes jobs, and no process opens it, or opens it again after a restart, later.

On macOS, spell a path the same way every time: the registry key keeps the letter case you pass, so `Data` and `data` would be two entries for one folder. On Windows, Go resolves existing folders and files to their on-disk letter case, so different spellings of them share one entry.

Expand Down Expand Up @@ -290,7 +300,7 @@ Each queue is a table with these columns:
- `id`: autoincrement primary key. Items come out in `id` order, and a priority queue's in `priority`, then `id` order. The ack id `DequeueWithAckId` returns is this id in decimal.
- `priority`: priority queues only; a lower number comes out first. Opening an older table as a priority queue adds it.
- `data`: the stored item.
- `status`: `pending`, `processing` or `completed`.
- `status`: `pending`, `processing`, or on a kept row `completed`, `failed` or `cancelled`.
- `ack`: 0 when an item is added, 1 once it is acknowledged or rejected and kept with `WithRemoveOnComplete(false)` (`status` `completed` or `failed`). On open, only `processing` rows with `ack` 0 go back to pending.
- `ack_id`: kept from v0.2 so old tables need no rebuild; no longer used.
- `job_id`: the id an item was added with through `EnqueueOnce`, unique among the rows that have one; NULL for `Enqueue`. Opening an older table adds it.
Expand All @@ -299,7 +309,8 @@ Each queue is a table with these columns:
On top of that table:

- Two partial indexes: one covers the pending items in queue order, and a unique one covers `job_id` on the rows that have one, which `EnqueueOnce` relies on.
- A claim or a pop is one `UPDATE ... RETURNING` or `DELETE ... RETURNING` statement.
- A claim or a pop is one `UPDATE ... RETURNING` or `DELETE ... RETURNING` statement. It selects nothing while the queue is paused.
- `<name>__sqliteq_meta` holds the totals and the pause; a trigger counts each stored add.
- Writes to one file (adds, claims, acks, purges) take one lock in the process, so they wait for each other in Go rather than in SQLite's busy handler. `Len` and `Values` do not take it.
- Concurrent `Enqueue` and `Acknowledge` calls on one file are committed together in one transaction. Each call still gets its own result, and if the shared transaction fails, each call is retried on its own so one bad call fails alone.

Expand Down
253 changes: 253 additions & 0 deletions inspect.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,253 @@
package sqliteq

import (
"bytes"
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"fmt"
"strconv"
"strings"
"time"

"github.com/goptics/varmq/contract"
)

// Queue inspection (varmq contract.QueueInspector; spec redis-storage.md 7)
// on SQLite: counts from the queue's table, totals from <name>__sqliteq_meta.
// Completed and failed jobs are rows kept by WithRemoveOnComplete(false);
// by default ended rows are deleted, so those counts and pages are empty.

var (
_ contract.QueueInspector = (*Queue)(nil)
_ contract.QueueInspector = (*PriorityQueue)(nil)
)

const metaSuffix = "__sqliteq_meta"

// initMeta creates the meta table (key -> integer: the totals submitted,
// completed, failed; paused_at while paused) and the trigger that counts
// stored adds; a repeat EnqueueOnce stores no row, so fires none. The
// caller holds the write mutex.
func (q *Queue) initMeta() error {
for _, s := range []string{
`CREATE TABLE IF NOT EXISTS %[2]s (key TEXT PRIMARY KEY, value INTEGER NOT NULL)`,
// An older table starts from the rows it holds
`INSERT OR IGNORE INTO %[2]s (key, value) VALUES
('submitted', (SELECT COUNT(*) FROM %[1]s)),
('completed', (SELECT COUNT(*) FROM %[1]s WHERE status = 'completed')),
('failed', (SELECT COUNT(*) FROM %[1]s WHERE status IN ('failed', 'cancelled')))`,
`CREATE TRIGGER IF NOT EXISTS %[3]s AFTER INSERT ON %[1]s BEGIN
UPDATE %[2]s SET value = value + 1 WHERE key = 'submitted';
END`,
} {
if _, err := q.f.db.Exec(fmt.Sprintf(s, q.table, q.meta, quoteIdent(q.name+"__sqliteq_submitted"))); err != nil {
return err
}
}
return nil
}

// queueID is sqliteq:<h>:fifo|prio:<name>: <h> the first 8 hex of the
// SHA-256 of the file's resolved path, so every process on the file agrees,
// two files never collide, and no path shows; the name in lower case, as
// SQLite matches tables. sqliteq has no in-memory mode: every queue is in
// a file.
func queueID(path, name string, priority bool) string {
sum := sha256.Sum256([]byte(path))
kind := "fifo"
if priority {
kind = "prio"
}
return "sqliteq:" + hex.EncodeToString(sum[:4]) + ":" + kind + ":" + foldName(name)
}

// QueueID implements contract.QueueInspector (see queueID).
func (q *Queue) QueueID() string {
return q.id
}

// QueueStats implements contract.QueueInspector, in one statement (one
// snapshot). Counts: rows per status, cancelled under failed. Totals:
// adds stored and jobs finished (cancelled ones as failed; a waiting job
// dropped for a cancel is neither) since the last Purge; an older table
// starts from the rows it held. Paused is the file's, not the cache.
// ponytail: counts scan the table; index status if kept rows grow large.
func (q *Queue) QueueStats(ctx context.Context) (contract.QueueStats, error) {
if q.closed.Load() {
return contract.QueueStats{}, errClosed
}
var (
s contract.QueueStats
pausedAt sql.NullInt64
)
err := q.f.db.QueryRowContext(ctx, fmt.Sprintf(`SELECT
COUNT(*) FILTER (WHERE status = 'pending'),
COUNT(*) FILTER (WHERE status = 'processing'),
COUNT(*) FILTER (WHERE status = 'completed'),
COUNT(*) FILTER (WHERE status IN ('failed', 'cancelled')),
COALESCE((SELECT value FROM %[2]s WHERE key = 'submitted'), 0),
COALESCE((SELECT value FROM %[2]s WHERE key = 'completed'), 0),
COALESCE((SELECT value FROM %[2]s WHERE key = 'failed'), 0),
(SELECT value FROM %[2]s WHERE key = 'paused_at')
FROM %[1]s`, q.table, q.meta)).Scan(
&s.Counts.Pending, &s.Counts.Processing, &s.Counts.Completed, &s.Counts.Failed,
&s.Totals.Submitted, &s.Totals.Completed, &s.Totals.Failed, &pausedAt)
if err != nil {
return contract.QueueStats{}, err
}
s.Paused, s.PausedAt = pausedAt.Valid, pausedAt.Int64
return s, nil
}

// Jobs implements contract.QueueInspector: pending in hand-out order,
// processing by ack id, completed, failed (with cancelled, state
// cancelled) and cancelled newest finish first. Not counted by sqliteq:
// attempts are 0 pending, else 1; an ended row's start is lost. errors: one entry, from
// the job's outcome record, when it has one.
// ponytail: ended pages sort by updated_at unindexed; fine until kept rows
// grow large.
func (q *Queue) Jobs(ctx context.Context, state contract.JobState, offset, limit int) ([]contract.JobRecord, error) {
where, order := "", "id"
switch state {
case contract.JobPending:
where, order = "status = 'pending'", q.order
case contract.JobProcessing:
where = "status = 'processing'"
case contract.JobCompleted:
where, order = "status = 'completed'", "updated_at DESC, id DESC"
case contract.JobFailed:
where, order = "status IN ('failed', 'cancelled')", "updated_at DESC, id DESC"
case contract.JobCancelled:
where, order = "status = 'cancelled'", "updated_at DESC, id DESC"
default:
return nil, fmt.Errorf("sqliteq: unknown job state %q", state)
}
if q.closed.Load() {
return nil, errClosed
}
jobs := []contract.JobRecord{}
if limit <= 0 {
return jobs, nil
}

rows, err := q.f.db.QueryContext(ctx, fmt.Sprintf("SELECT id, data, job_id, status, created_at, updated_at FROM %s WHERE %s ORDER BY %s LIMIT ? OFFSET ?",
q.table, where, order), limit, max(0, offset))
if err != nil {
return nil, err
}
defer rows.Close()

var ended []int // failed and cancelled jobs with a Job ID
for rows.Next() {
var (
id int64
data []byte
once sql.NullString
status string
created, updated any
)
if err := rows.Scan(&id, &data, &once, &status, &created, &updated); err != nil {
return nil, err
}
j := contract.JobRecord{ID: jobIDOf(once, data), State: contract.JobState(status)}
j.Payload, j.EnqueuedAt = unwrap(data)
if j.EnqueuedAt == 0 {
j.EnqueuedAt = unixMilli(created)
}
switch state {
case contract.JobPending:
case contract.JobProcessing:
j.StartedAt, j.Attempts = unixMilli(updated), 1
default:
j.FinishedAt, j.Attempts = unixMilli(updated), 1
if j.ID != "" && state != contract.JobCompleted {
ended = append(ended, len(jobs))
}
}
// Spec 3.1: a handed-out job without an id goes by its ack id
if j.ID == "" && state != contract.JobPending {
j.ID = strconv.FormatInt(id, 10)
}
jobs = append(jobs, j)
}
if err := rows.Err(); err != nil {
return nil, err
}
if len(ended) > 0 {
if err := q.addErrors(ctx, jobs, ended); err != nil {
return nil, err
}
}
return jobs, nil
}

// addErrors gives each jobs[i] of ended its outcome record's error, when
// the record has its state; retention may have trimmed it.
func (q *Queue) addErrors(ctx context.Context, jobs []contract.JobRecord, ended []int) error {
args := make([]any, len(ended))
for i, at := range ended {
args[i] = jobs[at].ID
}
rows, err := q.f.db.QueryContext(ctx, fmt.Sprintf("SELECT job_id, state, error FROM %s WHERE job_id IN (%s) AND error IS NOT NULL",
q.jobs, strings.Repeat("?, ", len(args)-1)+"?"), args...)
if err != nil {
return err
}
defer rows.Close()

type outcome struct{ state, error string }
byID := map[string]outcome{}
for rows.Next() {
var id string
var o outcome
if err := rows.Scan(&id, &o.state, &o.error); err != nil {
return err
}
byID[id] = o
}
if err := rows.Err(); err != nil {
return err
}
for _, at := range ended {
j := &jobs[at]
if o, ok := byID[j.ID]; ok && o.state == string(j.State) {
j.Errors = []contract.JobError{{Attempt: 1, At: j.FinishedAt, Error: o.error}}
}
}
return nil
}

// unixMilli is a TIMESTAMP column read back as Unix ms; 0 when NULL or
// not a time.
func unixMilli(v any) int64 {
if t, ok := v.(time.Time); ok {
return t.UnixMilli()
}
return 0
}

// unwrap returns the payload and enqueue time of item: an envelope's data
// and enqueued_at, or for anything else the item itself if it is JSON, else
// the item as a JSON string. The bytes are kept as stored, never re-encoded.
func unwrap(item []byte) (json.RawMessage, int64) {
var env struct {
Status json.RawMessage `json:"status"`
Data json.RawMessage `json:"data"`
EnqueuedAt json.RawMessage `json:"enqueued_at"`
}
// An envelope has status and data (spec section 3). A field is nil only
// when its key is absent: a JSON null still sets it to "null".
if bytes.HasPrefix(bytes.TrimSpace(item), []byte("{")) && json.Unmarshal(item, &env) == nil && env.Status != nil && env.Data != nil {
var at int64
_ = json.Unmarshal(env.EnqueuedAt, &at)
return env.Data, at
}
if json.Valid(item) {
return item, 0
}
s, _ := json.Marshal(string(item))
return s, 0
}
Loading
Loading