diff --git a/CHANGELOG.md b/CHANGELOG.md index 8961d18..467165e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `Reject(ackID, reason)` on `Queue` and `PriorityQueue`, required by varmq's `IAcknowledgeable` since varmq#173: finishes an item that failed for good, never requeued. Deleted like an ack, or with `WithRemoveOnComplete(false)` kept with `status` `failed`. Same group commit as `Acknowledge`. The reason is the job's outcome error. - Job outcome (varmq spec 6.13): `WatchOutcome` (`contract.OutcomeWatcher`) and `AcknowledgeResult` (`contract.ResultAcknowledgeable`) on both queues. A job with a Job ID (`EnqueueOnce` id, else the item's JSON `id`) gets a record in `__sqliteq_jobs` when it ends, same transaction as the finish. Watches poll it by seq every 50 ms (new records only), from any queue object on the file, any process. Retention: completed 1000 / 24 h, failed and cancelled 10000 / 7 d; the count limit spares records under 1 min old. A polling queue object also sweeps retention and expired marks once a minute. - Job cancel (varmq spec 6.14): `CancelJob`, `OnCancel`, `AcknowledgeCancelled` (`contract.JobCanceller`, `CancelNotifier`, `CancelAcknowledgeable`). Marks in `__sqliteq_cancelled`, 24 h each. Every pop drops a marked waiting job (outcome `cancelled`, at most 1000 per transaction; a single-item pop goes on past that, a batch pop reports an item left); `OnCancel` polls new marks. `ErrJobEnded` when the record shows it ended. +- Queue inspection (`contract.QueueInspector`) on both queues, so varmq's REST API and vartop list sqliteq queues. `QueueID` is `sqliteq::fifo|prio:`, `` the first 8 hex of the SHA-256 of the file's resolved path, `` in lower case. `QueueStats`: counts per row status in one statement (cancelled under failed; completed and failed only with `WithRemoveOnComplete(false)`), totals submitted / completed / failed from new table `__sqliteq_meta` (an insert trigger counts adds; a finish counts in its transaction; an older table starts from its rows, so jobs deleted before the upgrade are missing; `Purge` resets them), and the pause. `Jobs`: pending in hand-out order, processing by ack id, kept completed, failed (with cancelled) and cancelled rows newest first, payload the envelope's `data` verbatim, errors from the outcome record. Attempts are not counted (0 pending, 1 after). +- Durable queue pause (varmq spec 6.7, 6.8): `Pause`, `Resume`, `Paused`, `PauseRefusals` (`varmq.QueuePauser`) and `OnPauseChange` (`varmq.PauseNotifier`). The flag is a row of `__sqliteq_meta`; every pop statement selects nothing while it exists, in every process on the file; it outlives restarts and `Purge`. A refused pop sets `Paused`; while set, the queue object polls the row every 50 ms to see a `Resume` from another process. - `EnqueueOnce` that stores an id drops its old record and mark (trigger). `Purge` clears records and marks too. `Queue.Close` drops watches and hooks. ### Breaking diff --git a/README.md b/README.md index 6058b8b..ca8f296 100644 --- a/README.md +++ b/README.md @@ -192,6 +192,16 @@ queue := worker.WithPersistentQueue(q) queue.Add("hello") ``` +### Inspection and pause + +Both queue types implement varmq's `QueueInspector`, `QueuePauser` and `PauseNotifier`, so a bound queue shows up in varmq's REST API (`GET /queues`, `GET /queues/{id}/jobs`, `PATCH /queues/{id}/actions/pause|resume`) and in vartop, as a redisq queue does. + +- **Queue ID**: `sqliteq::fifo:` or `sqliteq::prio:`. `` is the first 8 hex digits of the SHA-256 of the database file's resolved path, so every process on the file agrees, two files with one queue name never collide, and no path shows. `` is in lower case. sqliteq has no in-memory mode, so every queue has a file. A file reached through two different paths (another mount, say) gets two IDs. +- **Counts**: rows per `status`, read in one statement. Cancelled rows count as failed. Completed and failed rows exist only with `WithRemoveOnComplete(false)`; by default they are deleted, so those counts are 0. Kept rows have no retention. +- **Totals**: `submitted` counts stored adds (a repeat `EnqueueOnce` is not one), `completed` and `failed` count finishes (`AcknowledgeCancelled` counts as failed; a waiting job dropped for a cancel counts as neither). They live in `__sqliteq_meta`, start from the rows an older table holds when this version first opens it, and `Purge` resets them. Seeded totals count the rows the table holds then, so jobs deleted before the upgrade (the default `WithRemoveOnComplete(true)`) are missing from them. +- **Jobs**: pending in hand-out order, processing by ack id, completed, failed (with cancelled, `state` `cancelled`) and cancelled newest finish first. `payload` is the varmq envelope's `data`, or the stored item itself (as a JSON string when it is not JSON). A job without an id shows its ack id once handed out. sqliteq does not count attempts: they read 0 while pending and 1 after. A kept row loses its start time when it ends. A failed job's `errors` holds its reason when the job has an id and its outcome record is still kept. +- **Pause**: `Pause()` stores a `paused_at` row in `__sqliteq_meta`. Every pop selects nothing while it exists, so the pause reaches every process on the file at once, outlives restarts, and survives `Purge`; only `Resume()` ends it. Adds, acks and rejects go on. `Paused()` is this object's cache: a pop refused for the pause sets it, and while it is set the queue polls the row every 50 ms, so a `Resume` from another process wakes this one's workers within that time. + sqliteq does not import varmq. Batch pops and `EnqueueOnce` need varmq ≥ v1.10.0. From that release, a worker whose only source is the sqliteq queue claims several jobs in one statement through `DequeueBatchWithAckIds` (a worker bound to several queues claims one job per call), and a job added with an id (`WithJobId`, or one from `WithJobIdGenerator`) goes through `EnqueueOnce`. An older varmq calls neither: it claims one job per call and adds every job with `Enqueue`, which still works. ## Storage @@ -228,11 +238,11 @@ In the folder layout the queue name becomes a file name, so it may only contain Names are case-insensitive in both layouts: `Jobs` and `jobs` are the same queue, stored in `jobs.db` in the folder layout. -### One process per file +### Processes and files -Run one process per database file. sqliteq keeps a registry in the process, keyed by the file's real path (made absolute, with symlinks resolved). Every `New` in the process that reaches the same file shares one connection pool and one write lock, so two `New` calls on one path never write through two competing pools. Two `Queues` values can therefore hold the same queue, and you get the same handle from both. The file closes when the last `Queues` using it is closed. +sqliteq keeps a registry in the process, keyed by the file's real path (made absolute, with symlinks resolved). Every `New` in the process that reaches the same file shares one connection pool and one write lock, so two `New` calls on one path never write through two competing pools. Two `Queues` values can therefore hold the same queue, and you get the same handle from both. The file closes when the last `Queues` using it is closed. -Another process is not coordinated with this one. The first time a process opens a queue, it puts every item still marked as processing back to pending, so a second process opening a file that is in use would rerun the first process's running jobs. sqliteq does not take an OS lock to stop that. +Other processes can open the same file. SQLite's lock orders their writes, and what lives in the file reaches every process: a pause stops every process's pops at once, and a `Resume`, a cancel or a job outcome reaches the others within 50 ms (they poll). Recovery is the limit. Every process that opens a queue runs recovery on its first open of that queue, whatever it does with it: consumers, producers, observers (a process that only serves the REST API) and `Pause`/`Resume` callers alike. Recovery puts every item still marked as processing back to pending, so a process that opens the queue while another one runs its jobs makes those jobs run again. sqliteq does not take an OS lock to stop that. Sharing a file is safe only when every process opens the queue before any of them takes jobs, and no process opens it, or opens it again after a restart, later. On macOS, spell a path the same way every time: the registry key keeps the letter case you pass, so `Data` and `data` would be two entries for one folder. On Windows, Go resolves existing folders and files to their on-disk letter case, so different spellings of them share one entry. @@ -290,7 +300,7 @@ Each queue is a table with these columns: - `id`: autoincrement primary key. Items come out in `id` order, and a priority queue's in `priority`, then `id` order. The ack id `DequeueWithAckId` returns is this id in decimal. - `priority`: priority queues only; a lower number comes out first. Opening an older table as a priority queue adds it. - `data`: the stored item. -- `status`: `pending`, `processing` or `completed`. +- `status`: `pending`, `processing`, or on a kept row `completed`, `failed` or `cancelled`. - `ack`: 0 when an item is added, 1 once it is acknowledged or rejected and kept with `WithRemoveOnComplete(false)` (`status` `completed` or `failed`). On open, only `processing` rows with `ack` 0 go back to pending. - `ack_id`: kept from v0.2 so old tables need no rebuild; no longer used. - `job_id`: the id an item was added with through `EnqueueOnce`, unique among the rows that have one; NULL for `Enqueue`. Opening an older table adds it. @@ -299,7 +309,8 @@ Each queue is a table with these columns: On top of that table: - Two partial indexes: one covers the pending items in queue order, and a unique one covers `job_id` on the rows that have one, which `EnqueueOnce` relies on. -- A claim or a pop is one `UPDATE ... RETURNING` or `DELETE ... RETURNING` statement. +- A claim or a pop is one `UPDATE ... RETURNING` or `DELETE ... RETURNING` statement. It selects nothing while the queue is paused. +- `__sqliteq_meta` holds the totals and the pause; a trigger counts each stored add. - Writes to one file (adds, claims, acks, purges) take one lock in the process, so they wait for each other in Go rather than in SQLite's busy handler. `Len` and `Values` do not take it. - Concurrent `Enqueue` and `Acknowledge` calls on one file are committed together in one transaction. Each call still gets its own result, and if the shared transaction fails, each call is retried on its own so one bad call fails alone. diff --git a/inspect.go b/inspect.go new file mode 100644 index 0000000..f753d87 --- /dev/null +++ b/inspect.go @@ -0,0 +1,253 @@ +package sqliteq + +import ( + "bytes" + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "encoding/json" + "fmt" + "strconv" + "strings" + "time" + + "github.com/goptics/varmq/contract" +) + +// Queue inspection (varmq contract.QueueInspector; spec redis-storage.md 7) +// on SQLite: counts from the queue's table, totals from __sqliteq_meta. +// Completed and failed jobs are rows kept by WithRemoveOnComplete(false); +// by default ended rows are deleted, so those counts and pages are empty. + +var ( + _ contract.QueueInspector = (*Queue)(nil) + _ contract.QueueInspector = (*PriorityQueue)(nil) +) + +const metaSuffix = "__sqliteq_meta" + +// initMeta creates the meta table (key -> integer: the totals submitted, +// completed, failed; paused_at while paused) and the trigger that counts +// stored adds; a repeat EnqueueOnce stores no row, so fires none. The +// caller holds the write mutex. +func (q *Queue) initMeta() error { + for _, s := range []string{ + `CREATE TABLE IF NOT EXISTS %[2]s (key TEXT PRIMARY KEY, value INTEGER NOT NULL)`, + // An older table starts from the rows it holds + `INSERT OR IGNORE INTO %[2]s (key, value) VALUES + ('submitted', (SELECT COUNT(*) FROM %[1]s)), + ('completed', (SELECT COUNT(*) FROM %[1]s WHERE status = 'completed')), + ('failed', (SELECT COUNT(*) FROM %[1]s WHERE status IN ('failed', 'cancelled')))`, + `CREATE TRIGGER IF NOT EXISTS %[3]s AFTER INSERT ON %[1]s BEGIN + UPDATE %[2]s SET value = value + 1 WHERE key = 'submitted'; + END`, + } { + if _, err := q.f.db.Exec(fmt.Sprintf(s, q.table, q.meta, quoteIdent(q.name+"__sqliteq_submitted"))); err != nil { + return err + } + } + return nil +} + +// queueID is sqliteq::fifo|prio:: the first 8 hex of the +// SHA-256 of the file's resolved path, so every process on the file agrees, +// two files never collide, and no path shows; the name in lower case, as +// SQLite matches tables. sqliteq has no in-memory mode: every queue is in +// a file. +func queueID(path, name string, priority bool) string { + sum := sha256.Sum256([]byte(path)) + kind := "fifo" + if priority { + kind = "prio" + } + return "sqliteq:" + hex.EncodeToString(sum[:4]) + ":" + kind + ":" + foldName(name) +} + +// QueueID implements contract.QueueInspector (see queueID). +func (q *Queue) QueueID() string { + return q.id +} + +// QueueStats implements contract.QueueInspector, in one statement (one +// snapshot). Counts: rows per status, cancelled under failed. Totals: +// adds stored and jobs finished (cancelled ones as failed; a waiting job +// dropped for a cancel is neither) since the last Purge; an older table +// starts from the rows it held. Paused is the file's, not the cache. +// ponytail: counts scan the table; index status if kept rows grow large. +func (q *Queue) QueueStats(ctx context.Context) (contract.QueueStats, error) { + if q.closed.Load() { + return contract.QueueStats{}, errClosed + } + var ( + s contract.QueueStats + pausedAt sql.NullInt64 + ) + err := q.f.db.QueryRowContext(ctx, fmt.Sprintf(`SELECT + COUNT(*) FILTER (WHERE status = 'pending'), + COUNT(*) FILTER (WHERE status = 'processing'), + COUNT(*) FILTER (WHERE status = 'completed'), + COUNT(*) FILTER (WHERE status IN ('failed', 'cancelled')), + COALESCE((SELECT value FROM %[2]s WHERE key = 'submitted'), 0), + COALESCE((SELECT value FROM %[2]s WHERE key = 'completed'), 0), + COALESCE((SELECT value FROM %[2]s WHERE key = 'failed'), 0), + (SELECT value FROM %[2]s WHERE key = 'paused_at') + FROM %[1]s`, q.table, q.meta)).Scan( + &s.Counts.Pending, &s.Counts.Processing, &s.Counts.Completed, &s.Counts.Failed, + &s.Totals.Submitted, &s.Totals.Completed, &s.Totals.Failed, &pausedAt) + if err != nil { + return contract.QueueStats{}, err + } + s.Paused, s.PausedAt = pausedAt.Valid, pausedAt.Int64 + return s, nil +} + +// Jobs implements contract.QueueInspector: pending in hand-out order, +// processing by ack id, completed, failed (with cancelled, state +// cancelled) and cancelled newest finish first. Not counted by sqliteq: +// attempts are 0 pending, else 1; an ended row's start is lost. errors: one entry, from +// the job's outcome record, when it has one. +// ponytail: ended pages sort by updated_at unindexed; fine until kept rows +// grow large. +func (q *Queue) Jobs(ctx context.Context, state contract.JobState, offset, limit int) ([]contract.JobRecord, error) { + where, order := "", "id" + switch state { + case contract.JobPending: + where, order = "status = 'pending'", q.order + case contract.JobProcessing: + where = "status = 'processing'" + case contract.JobCompleted: + where, order = "status = 'completed'", "updated_at DESC, id DESC" + case contract.JobFailed: + where, order = "status IN ('failed', 'cancelled')", "updated_at DESC, id DESC" + case contract.JobCancelled: + where, order = "status = 'cancelled'", "updated_at DESC, id DESC" + default: + return nil, fmt.Errorf("sqliteq: unknown job state %q", state) + } + if q.closed.Load() { + return nil, errClosed + } + jobs := []contract.JobRecord{} + if limit <= 0 { + return jobs, nil + } + + rows, err := q.f.db.QueryContext(ctx, fmt.Sprintf("SELECT id, data, job_id, status, created_at, updated_at FROM %s WHERE %s ORDER BY %s LIMIT ? OFFSET ?", + q.table, where, order), limit, max(0, offset)) + if err != nil { + return nil, err + } + defer rows.Close() + + var ended []int // failed and cancelled jobs with a Job ID + for rows.Next() { + var ( + id int64 + data []byte + once sql.NullString + status string + created, updated any + ) + if err := rows.Scan(&id, &data, &once, &status, &created, &updated); err != nil { + return nil, err + } + j := contract.JobRecord{ID: jobIDOf(once, data), State: contract.JobState(status)} + j.Payload, j.EnqueuedAt = unwrap(data) + if j.EnqueuedAt == 0 { + j.EnqueuedAt = unixMilli(created) + } + switch state { + case contract.JobPending: + case contract.JobProcessing: + j.StartedAt, j.Attempts = unixMilli(updated), 1 + default: + j.FinishedAt, j.Attempts = unixMilli(updated), 1 + if j.ID != "" && state != contract.JobCompleted { + ended = append(ended, len(jobs)) + } + } + // Spec 3.1: a handed-out job without an id goes by its ack id + if j.ID == "" && state != contract.JobPending { + j.ID = strconv.FormatInt(id, 10) + } + jobs = append(jobs, j) + } + if err := rows.Err(); err != nil { + return nil, err + } + if len(ended) > 0 { + if err := q.addErrors(ctx, jobs, ended); err != nil { + return nil, err + } + } + return jobs, nil +} + +// addErrors gives each jobs[i] of ended its outcome record's error, when +// the record has its state; retention may have trimmed it. +func (q *Queue) addErrors(ctx context.Context, jobs []contract.JobRecord, ended []int) error { + args := make([]any, len(ended)) + for i, at := range ended { + args[i] = jobs[at].ID + } + rows, err := q.f.db.QueryContext(ctx, fmt.Sprintf("SELECT job_id, state, error FROM %s WHERE job_id IN (%s) AND error IS NOT NULL", + q.jobs, strings.Repeat("?, ", len(args)-1)+"?"), args...) + if err != nil { + return err + } + defer rows.Close() + + type outcome struct{ state, error string } + byID := map[string]outcome{} + for rows.Next() { + var id string + var o outcome + if err := rows.Scan(&id, &o.state, &o.error); err != nil { + return err + } + byID[id] = o + } + if err := rows.Err(); err != nil { + return err + } + for _, at := range ended { + j := &jobs[at] + if o, ok := byID[j.ID]; ok && o.state == string(j.State) { + j.Errors = []contract.JobError{{Attempt: 1, At: j.FinishedAt, Error: o.error}} + } + } + return nil +} + +// unixMilli is a TIMESTAMP column read back as Unix ms; 0 when NULL or +// not a time. +func unixMilli(v any) int64 { + if t, ok := v.(time.Time); ok { + return t.UnixMilli() + } + return 0 +} + +// unwrap returns the payload and enqueue time of item: an envelope's data +// and enqueued_at, or for anything else the item itself if it is JSON, else +// the item as a JSON string. The bytes are kept as stored, never re-encoded. +func unwrap(item []byte) (json.RawMessage, int64) { + var env struct { + Status json.RawMessage `json:"status"` + Data json.RawMessage `json:"data"` + EnqueuedAt json.RawMessage `json:"enqueued_at"` + } + // An envelope has status and data (spec section 3). A field is nil only + // when its key is absent: a JSON null still sets it to "null". + if bytes.HasPrefix(bytes.TrimSpace(item), []byte("{")) && json.Unmarshal(item, &env) == nil && env.Status != nil && env.Data != nil { + var at int64 + _ = json.Unmarshal(env.EnqueuedAt, &at) + return env.Data, at + } + if json.Valid(item) { + return item, 0 + } + s, _ := json.Marshal(string(item)) + return s, 0 +} diff --git a/outcome.go b/outcome.go index 643f2a3..d5589a0 100644 --- a/outcome.go +++ b/outcome.go @@ -172,7 +172,8 @@ type outcomeWatch struct { } // watchers are a queue object's outcome watches and OnCancel hooks, and the -// state of its one poll goroutine. mu is never held across a query. +// state of its one poll goroutine, which also polls the pause. mu is never +// held across a query. type watchers struct { mu sync.Mutex byID map[string]map[*outcomeWatch]struct{} @@ -267,8 +268,9 @@ func (q *Queue) WatchOutcome(id string, fn func(contract.JobOutcome)) (cancel fu } } -// poll reads new outcome records and cancel marks every pollEvery, and -// sweeps every sweepEvery, until Close or until nothing is watched. +// poll reads new outcome records, cancel marks and, while the cache says +// paused, the pause row every pollEvery, and sweeps every sweepEvery, +// until Close or until nothing is watched and the cache says running. func (q *Queue) poll() { t := time.NewTicker(pollEvery) defer t.Stop() @@ -282,7 +284,8 @@ func (q *Queue) poll() { } w.mu.Lock() - if len(w.byID) == 0 && len(w.hooks) == 0 { + paused := q.pz.paused.Load() + if len(w.byID) == 0 && len(w.hooks) == 0 && !paused { w.polling = false w.mu.Unlock() return @@ -297,6 +300,9 @@ func (q *Queue) poll() { if hooks { q.pollCancels(seq) } + if paused { + q.pollPause() + } if time.Since(swept) >= sweepEvery { swept = time.Now() q.sweep() diff --git a/pause.go b/pause.go new file mode 100644 index 0000000..05388a2 --- /dev/null +++ b/pause.go @@ -0,0 +1,150 @@ +package sqliteq + +import ( + "database/sql" + "errors" + "fmt" + "sync" + "sync/atomic" + "time" +) + +// Queue pause (varmq QueuePauser and PauseNotifier; spec redis-storage.md +// 6.7, 6.8) on SQLite. The flag is row paused_at (ms) of +// __sqliteq_meta: every pop statement selects nothing while it +// exists, so a Pause from any process on the file stops hand-outs at once, +// and it outlives restarts. Paused is this object's cache: set by its own +// Pause and Resume, by a pop the row refused, and, while it is true, by +// the poll, which reads the row every pollEvery, so a Resume from another +// process reaches it within pollEvery. + +// pauseState is a Queue's Paused cache and OnPauseChange hooks. mu makes +// each gen check and cache swap one step, so a stale read never overwrites +// a newer Pause or Resume of this object. +type pauseState struct { + paused atomic.Bool // written under mu + gen atomic.Uint64 // moves at each Pause and Resume, after its write + refusals atomic.Uint64 // pops refused for a pause + mu sync.Mutex + hooks map[*func(bool)]struct{} +} + +// Pause stops every pop of the queue, in every process on the file, until +// Resume. Adds, acks, rejects and recovery work as before; Purge keeps the +// pause. Idempotent: the first pause time is kept. It returns once the row +// is stored. +func (q *Queue) Pause() error { + return q.setPauseRow(true, fmt.Sprintf("INSERT OR IGNORE INTO %s (key, value) VALUES ('paused_at', ?)", q.meta), time.Now().UnixMilli()) +} + +// Resume undoes Pause. Idempotent. +func (q *Queue) Resume() error { + return q.setPauseRow(false, fmt.Sprintf("DELETE FROM %s WHERE key = 'paused_at'", q.meta)) +} + +// setPauseRow runs a Pause or Resume write and sets the cache to p. +func (q *Queue) setPauseRow(p bool, query string, args ...any) error { + if q.closed.Load() { + return errClosed + } + q.f.mu.Lock() + _, err := q.f.db.Exec(query, args...) + q.f.mu.Unlock() + if err != nil { + return err + } + q.setPaused(p, q.pz.gen.Add(1)) + return nil +} + +// Paused reports whether the queue is paused, as this object last saw it. +// No I/O. +func (q *Queue) Paused() bool { + return q.pz.paused.Load() +} + +// PauseRefusals counts this object's pops refused for a pause (varmq's +// optional QueuePauser method), so varmq tells a refusal racing a Resume +// from a failed pop. +func (q *Queue) PauseRefusals() uint64 { + return q.pz.refusals.Load() +} + +// OnPauseChange implements varmq.PauseNotifier: fn gets the cache each time +// this object sees it change (its own Pause or Resume, a refused pop, the +// poll), until cancel. fn runs on the goroutine that saw the change, never +// under a lock of this package, and must not block; racing changes can +// reach fn out of order, never with a value the cache did not hold. +func (q *Queue) OnPauseChange(fn func(paused bool)) (cancel func()) { + h := &fn + q.pz.mu.Lock() + if q.pz.hooks == nil { + q.pz.hooks = map[*func(bool)]struct{}{} + } + q.pz.hooks[h] = struct{}{} + q.pz.mu.Unlock() + + return func() { + q.pz.mu.Lock() + delete(q.pz.hooks, h) + q.pz.mu.Unlock() + } +} + +// setPaused sets the cache to p if gen is still g: the gen of the call +// that read p (a pop or the poll; a Pause or Resume since may be newer than +// p), or the one this object's own Pause or Resume moved it to. On a +// change, after mu, it starts the poll (paused) and calls the hooks. +func (q *Queue) setPaused(p bool, g uint64) { + q.pz.mu.Lock() + if q.pz.gen.Load() != g { + q.pz.mu.Unlock() + return + } + if q.pz.paused.Swap(p) == p { + q.pz.mu.Unlock() + return + } + fns := make([]func(bool), 0, len(q.pz.hooks)) + for h := range q.pz.hooks { + fns = append(fns, *h) + } + q.pz.mu.Unlock() + + if p { + w := &q.w + w.mu.Lock() + if !w.closed { + w.startLocked(q) + } + w.mu.Unlock() + } + for _, fn := range fns { + fn(p) + } +} + +// refused records a pop refused for a pause, by a call that began at gen g. +func (q *Queue) refused(g uint64) { + q.pz.refusals.Add(1) + q.setPaused(true, g) +} + +// pollPause reads the pause row for the poll. +func (q *Queue) pollPause() { + g := q.pz.gen.Load() + if paused, err := readPaused(q.pausedAt); err == nil { + q.setPaused(paused, g) + } +} + +// readPaused reports whether the pause row exists, through stmt +// (q.pausedAt, maybe in a tx). +func readPaused(stmt *sql.Stmt) (bool, error) { + var at int64 + err := stmt.QueryRow().Scan(&at) + if errors.Is(err, sql.ErrNoRows) { + return false, nil + } + return err == nil, err +} diff --git a/priority_queue_test.go b/priority_queue_test.go index a7cadcf..327f748 100644 --- a/priority_queue_test.go +++ b/priority_queue_test.go @@ -3,6 +3,7 @@ package sqliteq import ( "errors" "fmt" + "slices" "strconv" "testing" @@ -697,3 +698,39 @@ func TestPriorityOutcomeAndCancel(t *testing.T) { t.Errorf("Expected the re-added job first, got %s", v) } } + +// A priority queue lists pending jobs in hand-out order (priority, then +// insert), counts as Queue does, and pauses every pop. +func TestPriorityInspectAndPause(t *testing.T) { + qs := mustNew(t, t.TempDir()) + defer qs.Close() + pq, err := qs.NewPriorityQueue("pq") + if err != nil { + t.Fatal(err) + } + pq.Enqueue(job("low"), 5) + pq.Enqueue(job("high"), 1) + pq.EnqueueOnce(job("mid"), 3, "mid") + pq.Enqueue(job("high2"), 1) + + if got := ids(jobs(t, pq.Queue, contract.JobPending, 0, 10)); !slices.Equal(got, []string{"high", "high2", "mid", "low"}) { + t.Errorf("pending: got %q", got) + } + if got := ids(jobs(t, pq.Queue, contract.JobPending, 1, 2)); !slices.Equal(got, []string{"high2", "mid"}) { + t.Errorf("page: got %q", got) + } + if s := stats(t, pq.Queue); s.Counts.Pending != 4 || s.Totals.Submitted != 4 { + t.Errorf("stats: got %+v", s) + } + + if err := pq.Pause(); err != nil { + t.Fatal(err) + } + refusedPops(t, pq.Queue) + if err := pq.Resume(); err != nil { + t.Fatal(err) + } + if v, _ := pop(t, pq.Queue); v != string(job("high")) { + t.Errorf("got %s, want high", v) + } +} diff --git a/queue.go b/queue.go index be1fbdd..1ae55ed 100644 --- a/queue.go +++ b/queue.go @@ -19,6 +19,10 @@ import ( // remainingCap bounds the pending count a batch pop reports const remainingCap = 256 +// afterEmptyPop runs between an empty pop outside a tx and its read of the +// pause row; tests only. +var afterEmptyPop = func() {} + // Queue implements the Queue interface using SQLite as the storage backend type Queue struct { f *file @@ -29,12 +33,15 @@ type Queue struct { closed atomic.Bool jobs, cancelled string // quoted outcome and cancel mark tables - w watchers // outcome watches and OnCancel hooks + meta string // quoted totals and pause table + id string // QueueID + w watchers // outcome watches, OnCancel hooks, pause poll + pz pauseState // Paused cache and OnPauseChange hooks records atomic.Int64 // outcome records written, for trims // Prepared once per handle; they close with the file's pool insert, claim, pop, fin, count, countCapped, values *sql.Stmt - putJob, unmark, hasMark, isMarked *sql.Stmt + putJob, unmark, hasMark, isMarked, pausedAt, tally *sql.Stmt } // claimed is one row a claim moved to processing @@ -58,6 +65,8 @@ func newQueue(f *file, name string, priority bool, opts ...Option) (*Queue, erro removeOnComplete: true, // Default to removing completed items jobs: quoteIdent(name + jobsSuffix), cancelled: quoteIdent(name + cancelledSuffix), + meta: quoteIdent(name + metaSuffix), + id: queueID(f.key, name, priority), } q.w.stop = make(chan struct{}) if priority { @@ -79,6 +88,13 @@ func newQueue(f *file, name string, priority bool, opts ...Option) (*Queue, erro if err := q.prepare(); err != nil { return nil, fmt.Errorf("failed to prepare statements: %w", err) } + // A pause kept in the file holds this object from its first pop + if paused, err := readPaused(q.pausedAt); err != nil { + q.closeStmts() + return nil, fmt.Errorf("failed to read pause: %w", err) + } else if paused { + q.setPaused(true, q.pz.gen.Load()) + } return q, nil } @@ -100,6 +116,9 @@ func (q *Queue) initSchema() error { if err := q.initSidecars(); err != nil { return fmt.Errorf("failed to create outcome tables: %w", err) } + if err := q.initMeta(); err != nil { + return fmt.Errorf("failed to create meta table: %w", err) + } return nil } @@ -204,7 +223,8 @@ func indexColumns(db *sql.DB, index string) ([]string, error) { } // prepare prepares every statement of the queue once. Each read keeps -// status = 'pending' literal, or SQLite cannot use the partial index. +// status = 'pending' literal, or SQLite cannot use the partial index. Pops +// select nothing while the pause row exists. func (q *Queue) prepare() (err error) { defer func() { if err != nil { @@ -216,7 +236,8 @@ func (q *Queue) prepare() (err error) { if q.priority { cols, vals, returning = cols+", priority", vals+", ?", "id, priority, data, job_id" } - pending := fmt.Sprintf("SELECT id FROM %s WHERE status = 'pending' ORDER BY %s", q.table, q.order) + pending := fmt.Sprintf("SELECT id FROM %s WHERE status = 'pending' AND NOT EXISTS (SELECT 1 FROM %s WHERE key = 'paused_at') ORDER BY %s", + q.table, q.meta, q.order) // fin ends a processing row: deleted, or kept with its end as status fin := fmt.Sprintf("DELETE FROM %s WHERE id = ? AND status = 'processing' RETURNING job_id, data", q.table) @@ -241,6 +262,8 @@ func (q *Queue) prepare() (err error) { {&q.count, fmt.Sprintf("SELECT COUNT(*) FROM %s WHERE status = 'pending'", q.table)}, {&q.countCapped, fmt.Sprintf("SELECT COUNT(*) FROM (SELECT 1 FROM %s WHERE status = 'pending' LIMIT %d)", q.table, remainingCap)}, {&q.values, fmt.Sprintf("SELECT data FROM %s WHERE status = 'pending' ORDER BY %s", q.table, q.order)}, + {&q.pausedAt, fmt.Sprintf("SELECT value FROM %s WHERE key = 'paused_at'", q.meta)}, + {&q.tally, fmt.Sprintf("UPDATE %s SET value = value + 1 WHERE key = ?", q.meta)}, } { if *s.stmt, err = q.f.db.Prepare(s.query); err != nil { return err @@ -252,7 +275,7 @@ func (q *Queue) prepare() (err error) { // closeStmts closes the statements prepared so far func (q *Queue) closeStmts() { - for _, s := range []*sql.Stmt{q.insert, q.claim, q.pop, q.fin, q.count, q.countCapped, q.values, q.putJob, q.unmark, q.hasMark, q.isMarked} { + for _, s := range []*sql.Stmt{q.insert, q.claim, q.pop, q.fin, q.count, q.countCapped, q.values, q.putJob, q.unmark, q.hasMark, q.isMarked, q.pausedAt, q.tally} { if s != nil { s.Close() } @@ -363,13 +386,17 @@ func (q *Queue) enqueue(item any, priority int, id string) bool { } // claimN moves up to n pending rows to processing in one statement and -// returns them in queue order -func (q *Queue) claimN(n int) ([]claimed, bool, error) { +// returns them in queue order; refused reports a pause stopped it +func (q *Queue) claimN(n int) (out []claimed, capped, refused bool, err error) { + g := q.pz.gen.Load() q.f.mu.Lock() - out, capped, err := q.claimGated(n) + out, capped, refused, err = q.claimGated(n) q.f.mu.Unlock() - if err != nil { - return nil, false, err + if refused { + q.refused(g) + } + if err != nil || refused { + return nil, false, refused, err } // RETURNING order is unspecified @@ -379,23 +406,35 @@ func (q *Queue) claimN(n int) ([]claimed, bool, error) { } return cmp.Compare(a.id, b.id) }) - return out, capped, nil + return out, capped, false, nil } // claimGated claims up to n rows; with a live cancel mark on the file it // runs in one transaction that drops each marked row it claims (outcome // cancelled) and claims again, at most maxDrops drops; capped reports that -// it stopped there. The caller holds the write mutex. -func (q *Queue) claimGated(n int) (out []claimed, capped bool, err error) { +// it stopped there. An empty claim reads the pause row: refused when it is +// set. Outside a tx a Resume from another process can land between the two, +// so an empty claim found unpaused claims once more. The caller holds the +// write mutex. +// ponytail: a second Pause and Resume, both in the window again, still make +// one empty pop look like a failure. +func (q *Queue) claimGated(n int) (out []claimed, capped, refused bool, err error) { now := time.Now() if !q.anyMark(now) { - out, err = q.claimRows(q.claim, n) - return out, false, err + for try := range 2 { + if out, err = q.claimRows(q.claim, n); err != nil || len(out) > 0 { + return out, false, false, err + } + afterEmptyPop() + if refused, err = readPaused(q.pausedAt); err != nil || refused || try == 1 { + return nil, false, refused, err + } + } } tx, err := q.f.db.Begin() if err != nil { - return nil, false, err + return nil, false, false, err } defer tx.Rollback() @@ -406,7 +445,7 @@ func (q *Queue) claimGated(n int) (out []claimed, capped bool, err error) { // No more than the drops left: a batch never passes the cap rows, err := q.claimRows(claim, min(n-len(out), maxDrops-drops)) if err != nil { - return nil, false, err + return nil, false, false, err } if len(rows) == 0 { break @@ -415,19 +454,26 @@ func (q *Queue) claimGated(n int) (out []claimed, capped bool, err error) { id := jobIDOf(c.once, c.data) drop, err := marked(isMarked, id, now) if err != nil { - return nil, false, err + return nil, false, false, err } if !drop { out = append(out, c) continue } if _, err := q.end(tx, c.id, id, contract.JobCancelled, "cancelled", nil, now); err != nil { - return nil, false, err + return nil, false, false, err } drops++ } } - return out, len(out) < n && drops >= maxDrops, tx.Commit() + capped = len(out) < n && drops >= maxDrops + if len(out) == 0 && !capped { + // The write lock is held: the row read is the one the claim saw + if refused, err = readPaused(tx.Stmt(q.pausedAt)); err != nil { + return nil, false, false, err + } + } + return out, capped, refused, tx.Commit() } // claimRows runs a claim statement and reads every row it returns; the @@ -463,53 +509,76 @@ func (q *Queue) Dequeue() (any, bool) { return nil, false } + g := q.pz.gen.Load() q.f.mu.Lock() - defer q.f.mu.Unlock() + data, ok, refused := q.dequeue() + q.f.mu.Unlock() + if refused { + q.refused(g) + } + return data, ok +} +// dequeue is Dequeue under the write mutex; an empty pop reads the pause +// row, refused when it is set, and pops once more when it is not (see +// claimGated). +func (q *Queue) dequeue() (data []byte, ok, refused bool) { now := time.Now() if !q.anyMark(now) { - data, _, err := popRow(q.pop) - return data, err == nil + for try := range 2 { + data, _, err := popRow(q.pop) + if !errors.Is(err, sql.ErrNoRows) { + return data, err == nil, false + } + afterEmptyPop() + if refused, err = readPaused(q.pausedAt); err != nil || refused || try == 1 { + return nil, false, refused + } + } } for { - data, capped, err := q.popGated(now) - if err != nil || data != nil || !capped { - return data, err == nil && data != nil + data, capped, refused, err := q.popGated(now) + if err != nil || data != nil || refused || !capped { + return data, err == nil && data != nil, refused } } } // popGated pops in one transaction, dropping each marked item it pops // (outcome cancelled), at most maxDrops; capped reports it stopped there -// with nothing. The caller holds the write mutex. -func (q *Queue) popGated(now time.Time) (data []byte, capped bool, err error) { +// with nothing, refused that the pause row stopped it. The caller holds +// the write mutex. +func (q *Queue) popGated(now time.Time) (data []byte, capped, refused bool, err error) { tx, err := q.f.db.Begin() if err != nil { - return nil, false, err + return nil, false, false, err } defer tx.Rollback() pop, isMarked := tx.Stmt(q.pop), tx.Stmt(q.isMarked) for range maxDrops { data, once, err := popRow(pop) if errors.Is(err, sql.ErrNoRows) { - return nil, false, tx.Commit() // keep the drops + if refused, err = readPaused(tx.Stmt(q.pausedAt)); err != nil { + return nil, false, false, err + } + return nil, false, refused, tx.Commit() // keep the drops } if err != nil { - return nil, false, err + return nil, false, false, err } id := jobIDOf(once, data) drop, err := marked(isMarked, id, now) if err != nil { - return nil, false, err + return nil, false, false, err } if !drop { - return data, false, tx.Commit() + return data, false, false, tx.Commit() } if err := q.record(tx, id, contract.JobCancelled, "cancelled", nil, now); err != nil { - return nil, false, err + return nil, false, false, err } } - return nil, true, tx.Commit() + return nil, true, false, tx.Commit() } // popRow runs a pop statement: the deleted row's data and job_id, or @@ -529,8 +598,8 @@ func (q *Queue) DequeueWithAckId() (any, bool, string) { // A pop stopped at the drop cap has nothing yet: pop again, it can't // report items left for { - c, capped, err := q.claimN(1) - if err != nil || len(c) == 0 && !capped { + c, capped, refused, err := q.claimN(1) + if err != nil || refused || len(c) == 0 && !capped { return nil, false, "" } if len(c) > 0 { @@ -550,9 +619,9 @@ func (q *Queue) DequeueBatchWithAckIds(items []any, ackIDs []string) (n, remaini return 0, 0, nil } - c, capped, err := q.claimN(want) - if err != nil { - return 0, 0, err + c, capped, refused, err := q.claimN(want) + if err != nil || refused { + return 0, 0, err // paused: nothing left to pop until a Resume } // At the drop cap items may wait still: report one, so the caller pops again left := 0 @@ -614,7 +683,16 @@ func (q *Queue) finish(ackID string, state contract.JobState, reason string, res }() } // No row is an outcome, not an error: it must not fail the round - return q.end(tx, id, "", state, reason, result, now) + if ok, err = q.end(tx, id, "", state, reason, result, now); !ok || err != nil { + return ok, err + } + // Totals: cancelled counts as failed (spec 6.14) + total := "failed" + if state == contract.JobCompleted { + total = "completed" + } + _, err = tx.Stmt(q.tally).Exec(total) + return err == nil, err }) } @@ -680,8 +758,8 @@ func (q *Queue) Values() []any { return items } -// Purge removes all items from the queue, with its outcome records and -// cancel marks (spec 6.6, 6.14) +// Purge removes all items from the queue, with its outcome records, cancel +// marks and totals; a pause stays (spec 6.6, 6.7, 6.14) func (q *Queue) Purge() { q.f.mu.Lock() defer q.f.mu.Unlock() @@ -696,6 +774,9 @@ func (q *Queue) Purge() { return } } + if _, err := tx.Exec("UPDATE " + q.meta + " SET value = 0 WHERE key <> 'paused_at'"); err != nil { + return + } tx.Commit() } diff --git a/queue_test.go b/queue_test.go index 6f55614..a8ab137 100644 --- a/queue_test.go +++ b/queue_test.go @@ -2,12 +2,15 @@ package sqliteq import ( "context" + "crypto/sha256" "database/sql" + "encoding/hex" "errors" "fmt" "path/filepath" "slices" "strconv" + "strings" "sync" "sync/atomic" "testing" @@ -1677,3 +1680,604 @@ func TestUndecodableEnvelopeIsPlainJob(t *testing.T) { t.Errorf("Expected no records, got %d", n) } } + +// QueueID is sqliteq:<8 hex of the file's path>:fifo|prio:: one file agrees in every process, two files never collide. +func TestQueueID(t *testing.T) { + dir := t.TempDir() + qs := mustNew(t, dir) + defer qs.Close() + q, _ := qs.NewQueue("Jobs") + pq, _ := qs.NewPriorityQueue("Prio") + sum := sha256.Sum256([]byte(q.f.key)) + if want := "sqliteq:" + hex.EncodeToString(sum[:4]) + ":fifo:jobs"; q.QueueID() != want { + t.Errorf("got %q, want %q", q.QueueID(), want) + } + if id := pq.QueueID(); !strings.HasPrefix(id, "sqliteq:") || !strings.HasSuffix(id, ":prio:prio") { + t.Errorf("priority queue id %q", id) + } + if otherProcess(t, q).QueueID() != q.QueueID() { + t.Error("Expected one id for one file in every process") + } + + // Same table name, other file + single := mustNew(t, filepath.Join(dir, "single.db"), WithSingleFile()) + defer single.Close() + q2, _ := single.NewQueue("jobs") + if q2.QueueID() == q.QueueID() { + t.Error("Expected two files to have two ids") + } + if strings.Contains(q2.QueueID(), dir) { + t.Error("Expected no path in the id") + } +} + +// stats reads QueueStats, failing on an error. +func stats(t *testing.T, q *Queue) contract.QueueStats { + t.Helper() + s, err := q.QueueStats(context.Background()) + if err != nil { + t.Fatal(err) + } + return s +} + +// QueueStats counts rows per state (cancelled under failed), totals every +// add and finish, and survives no Purge; the default deletes ended rows. +func TestQueueStats(t *testing.T) { + for _, keep := range []bool{false, true} { + t.Run(fmt.Sprintf("keep=%t", keep), func(t *testing.T) { + q, qs := openQueue(t, WithRemoveOnComplete(!keep)) + defer qs.Close() + for _, id := range []string{"a", "b", "c", "d", "e"} { + q.Enqueue(job(id)) + } + q.EnqueueOnce(job("f"), "f") + q.EnqueueOnce(job("f"), "f") // repeat: not submitted + q.CancelJob("d") + + _, a := pop(t, q) + _, b := pop(t, q) + _, c := pop(t, q) + pop(t, q) // drops d (cancelled, in no total), hands out e + q.Acknowledge(a) + q.Reject(b, "boom") + q.AcknowledgeCancelled(c, "stop") + + kept := int64(0) + if keep { + kept = 1 + } + want := contract.QueueStats{ + Counts: contract.JobCounts{Pending: 1, Processing: 1, Completed: kept, Failed: 3 * kept}, + Totals: contract.JobTotals{Submitted: 6, Completed: 1, Failed: 2}, + } + if got := stats(t, q); got != want { + t.Errorf("got %+v, want %+v", got, want) + } + if got := stats(t, otherProcess(t, q)); got != want { + t.Errorf("other process: got %+v, want %+v", got, want) + } + + q.Purge() + if got := stats(t, q); got != (contract.QueueStats{}) { + t.Errorf("after Purge: got %+v", got) + } + }) + } +} + +// jobs reads one page of Jobs, failing on an error. +func jobs(t *testing.T, q *Queue, state contract.JobState, offset, limit int) []contract.JobRecord { + t.Helper() + j, err := q.Jobs(context.Background(), state, offset, limit) + if err != nil { + t.Fatal(err) + } + return j +} + +func ids(js []contract.JobRecord) []string { + out := make([]string, len(js)) + for i, j := range js { + out[i] = j.ID + } + return out +} + +// Jobs pages each state: pending in hand-out order, processing by ack id, +// completed and failed (with cancelled) newest first; payload is the +// envelope's data, verbatim. +func TestJobs(t *testing.T) { + q, qs := openQueue(t, WithRemoveOnComplete(false)) + defer qs.Close() + before := time.Now().UnixMilli() + q.Enqueue([]byte(`{"id":"a","status":"Created","data":{"n":1.000000000000001},"enqueued_at":7}`)) + q.Enqueue([]byte("plain")) + q.Enqueue([]byte(`[1,2]`)) + for _, id := range []string{"b", "c", "d", "e", "f"} { + q.Enqueue(job(id)) + } + if got := ids(jobs(t, q, contract.JobPending, 0, 100)); !slices.Equal(got, []string{"a", "", "", "b", "c", "d", "e", "f"}) { + t.Fatalf("pending: got %q", got) + } + page := jobs(t, q, contract.JobPending, 0, 3) + if string(page[0].Payload) != `{"n":1.000000000000001}` || page[0].EnqueuedAt != 7 || page[0].Attempts != 0 || page[0].StartedAt != 0 { + t.Errorf("envelope: got %+v", page[0]) + } + if string(page[1].Payload) != `"plain"` || page[1].EnqueuedAt < before { + t.Errorf("plain bytes: got %+v (enqueued_at from the row)", page[1]) + } + if string(page[2].Payload) != `[1,2]` { + t.Errorf("JSON item: got %s", page[2].Payload) + } + if got := ids(jobs(t, q, contract.JobPending, 6, 5)); !slices.Equal(got, []string{"e", "f"}) { + t.Errorf("offset 6: got %q", got) + } + + acks := map[string]string{} + for range 7 { + v, ack := pop(t, q) + acks[v] = ack + } + plainAck := acks["plain"] + q.Acknowledge(acks[`{"id":"a","status":"Created","data":{"n":1.000000000000001},"enqueued_at":7}`]) + q.Reject(acks["plain"], "no id") + q.Reject(acks[string(job("b"))], "boom") + q.AcknowledgeCancelled(acks[string(job("c"))], "stopped") + q.Acknowledge(acks[string(job("d"))]) + + // A running job without an id shows its ack id + proc := jobs(t, q, contract.JobProcessing, 0, 10) + if got := ids(proc); !slices.Equal(got, []string{acks["[1,2]"], "e"}) { + t.Errorf("processing: got %q", got) + } + for _, j := range proc { + if j.State != contract.JobProcessing || j.Attempts != 1 || j.StartedAt < before || j.FinishedAt != 0 { + t.Errorf("processing job: got %+v", j) + } + } + + if got := ids(jobs(t, q, contract.JobCompleted, 0, 10)); !slices.Equal(got, []string{"d", "a"}) { + t.Errorf("completed: got %q", got) + } + failed := jobs(t, q, contract.JobFailed, 0, 10) + if got := ids(failed); !slices.Equal(got, []string{"c", "b", plainAck}) { + t.Fatalf("failed: got %q", got) + } + if failed[0].State != contract.JobCancelled || len(failed[0].Errors) != 1 || failed[0].Errors[0].Error != "stopped" { + t.Errorf("cancelled: got %+v", failed[0]) + } + if b := failed[1]; b.State != contract.JobFailed || b.FinishedAt < before || b.Attempts != 1 || + !slices.Equal(b.Errors, []contract.JobError{{Attempt: 1, At: b.FinishedAt, Error: "boom"}}) { + t.Errorf("failed: got %+v", b) + } + if len(failed[2].Errors) != 0 { + t.Errorf("Expected no errors without an outcome record, got %+v", failed[2].Errors) + } + if got := ids(jobs(t, q, contract.JobFailed, 1, 1)); !slices.Equal(got, []string{"b"}) { + t.Errorf("failed page 2: got %q", got) + } + + if j := jobs(t, q, contract.JobPending, 0, 0); j == nil || len(j) != 0 { + t.Errorf("limit 0: got %#v", j) + } + if j := jobs(t, q, contract.JobPending, 0, -1); j == nil || len(j) != 0 { + t.Errorf("limit -1: got %#v", j) + } + if _, err := q.Jobs(context.Background(), "bogus", 0, 10); err == nil { + t.Error("Expected an unknown state to fail") + } +} + +// With the default WithRemoveOnComplete(true) ended jobs are not kept, so +// their pages are empty. +func TestJobsRemoveOnComplete(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + q.Enqueue(job("a")) + _, ack := pop(t, q) + q.Acknowledge(ack) + if j := jobs(t, q, contract.JobCompleted, 0, 10); len(j) != 0 { + t.Errorf("got %+v", j) + } +} + +// A closed queue answers stats and jobs with an error, as varmq lists it. +func TestInspectClosed(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + q.Close() + if _, err := q.QueueStats(context.Background()); err == nil { + t.Error("Expected QueueStats to fail on a closed queue") + } + if _, err := q.Jobs(context.Background(), contract.JobPending, 0, 1); err == nil { + t.Error("Expected Jobs to fail on a closed queue") + } +} + +func TestImplementsPause(t *testing.T) { + var ( + _ varmq.QueuePauser = (*Queue)(nil) + _ varmq.QueuePauser = (*PriorityQueue)(nil) + _ varmq.PauseNotifier = (*Queue)(nil) + _ varmq.PauseNotifier = (*PriorityQueue)(nil) + ) +} + +// refusedPops checks that every pop kind hands out nothing and reports a +// pause. +func refusedPops(t *testing.T, q *Queue) { + t.Helper() + before := q.PauseRefusals() + if _, ok := q.Dequeue(); ok { + t.Error("Dequeue handed out an item while paused") + } + if _, ok, _ := q.DequeueWithAckId(); ok { + t.Error("DequeueWithAckId handed out an item while paused") + } + items, acks := make([]any, 4), make([]string, 4) + if n, rem, err := q.DequeueBatchWithAckIds(items, acks); n != 0 || rem != 0 || err != nil { + t.Errorf("batch pop while paused: got %d %d %v", n, rem, err) + } + if got := q.PauseRefusals() - before; got != 3 { + t.Errorf("Expected 3 refusals, got %d", got) + } + if !q.Paused() { + t.Error("Expected a refused pop to leave Paused true") + } +} + +// Pause stops every pop, keeps its first time, and is kept in the file; +// Resume lets pops go on. Hooks hear each change once. +func TestPause(t *testing.T) { + for _, marked := range []bool{false, true} { + t.Run(fmt.Sprintf("marked=%t", marked), func(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + if marked { + q.CancelJob("other") // gated pops + } + var calls []bool + var mu sync.Mutex + stop := q.OnPauseChange(func(p bool) { + mu.Lock() + calls = append(calls, p) + mu.Unlock() + }) + defer stop() + enqueueN(t, q, 3) + + if q.Paused() { + t.Fatal("Expected a new queue to run") + } + if err := q.Pause(); err != nil { + t.Fatal(err) + } + first := stats(t, q) + if !q.Paused() || !first.Paused || first.PausedAt == 0 { + t.Fatalf("after Pause: Paused %t, stats %+v", q.Paused(), first) + } + time.Sleep(2 * time.Millisecond) + if err := q.Pause(); err != nil { + t.Fatal(err) + } + if s := stats(t, q); s.PausedAt != first.PausedAt { + t.Errorf("Expected a second Pause to keep %d, got %d", first.PausedAt, s.PausedAt) + } + refusedPops(t, q) + if !q.Enqueue([]byte("added while paused")) || q.Len() != 4 { + t.Error("Expected adds to land while paused") + } + + if err := q.Resume(); err != nil { + t.Fatal(err) + } + if err := q.Resume(); err != nil { + t.Fatal(err) + } + if q.Paused() || stats(t, q).Paused { + t.Error("Expected Resume to clear the pause") + } + if _, ok := q.Dequeue(); !ok { + t.Error("Expected pops to go on after Resume") + } + mu.Lock() + defer mu.Unlock() + if !slices.Equal(calls, []bool{true, false}) { + t.Errorf("hooks: got %v", calls) + } + }) + } +} + +// waitPaused waits for q.Paused to become want. +func waitPaused(t *testing.T, q *Queue, want bool) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for q.Paused() != want { + if time.Now().After(deadline) { + t.Fatalf("Paused stayed %t", !want) + } + time.Sleep(pollEvery / 5) + } +} + +// A pause from another process stops this one's pops at once; its Resume +// reaches this object's Paused and hooks by the poll. +func TestPauseAcrossProcesses(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + other := otherProcess(t, q) + resumed := make(chan struct{}, 1) + defer q.OnPauseChange(func(p bool) { + if !p { + resumed <- struct{}{} + } + })() + enqueueN(t, q, 2) + + if err := other.Pause(); err != nil { + t.Fatal(err) + } + if q.Paused() { + t.Fatal("Expected the cache to wait for a pop or the poll") + } + if !stats(t, q).Paused { + t.Error("Expected QueueStats to read the pause from the file") + } + refusedPops(t, q) + + if err := other.Resume(); err != nil { + t.Fatal(err) + } + waitPaused(t, q, false) + select { + case <-resumed: + case <-time.After(5 * time.Second): + t.Fatal("no resume hook") + } + if _, ok := q.Dequeue(); !ok { + t.Error("Expected pops to go on after the other process resumed") + } +} + +// A pause is kept in the file: it outlives the process, and Purge keeps it. +func TestPauseSurvivesReopenAndPurge(t *testing.T) { + dir := t.TempDir() + qs := mustNew(t, dir) + q, _ := qs.NewQueue("q") + q.Enqueue([]byte("x")) + if err := q.Pause(); err != nil { + t.Fatal(err) + } + at := stats(t, q).PausedAt + q.Purge() + if s := stats(t, q); s.PausedAt != at || s.Totals != (contract.JobTotals{}) { + t.Errorf("after Purge: got %+v, want paused_at %d and no totals", s, at) + } + q.Enqueue([]byte("y")) + qs.Close() + + qs = mustNew(t, dir) + defer qs.Close() + q, _ = qs.NewQueue("q") + if !q.Paused() || !stats(t, q).Paused { + t.Fatal("Expected the pause to outlive the process") + } + refusedPops(t, q) + q.Resume() + if v, ok := q.Dequeue(); !ok || string(v.([]byte)) != "y" { + t.Errorf("got %v %t, want y", v, ok) + } + q.Close() + if q.Pause() == nil || q.Resume() == nil { + t.Error("Expected Pause and Resume on a closed queue to fail") + } +} + +// varmq: a paused queue holds the worker in every process; a Resume from +// another process wakes it. +func TestVarmqPause(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + other := otherProcess(t, q) + + ran := make(chan int, 4) + w := varmq.NewWorker(func(j varmq.Job[int]) { ran <- j.Data() }) + defer w.Stop() + handle := w.WithPersistentQueue(q) + if err := handle.Pause(); err != nil { + t.Fatal(err) + } + if !handle.IsPaused() { + t.Fatal("Expected the handle to report the pause") + } + handle.Add(1) + select { + case v := <-ran: + t.Fatalf("job %d ran while paused", v) + case <-time.After(10 * pollEvery): + } + if err := other.Resume(); err != nil { + t.Fatal(err) + } + select { + case <-ran: + case <-time.After(5 * time.Second): + t.Fatal("job did not run after the other process resumed") + } +} + +// An older table's totals start from the rows it holds. +func TestTotalsSeededFromRows(t *testing.T) { + path := filepath.Join(t.TempDir(), "old.db") + writeV023File(t, path, "q", false) + qs := mustNew(t, path, WithSingleFile()) + defer qs.Close() + q, err := qs.NewQueue("q") + if err != nil { + t.Fatal(err) + } + var rows int64 + q.f.db.QueryRow(`SELECT COUNT(*) FROM "q"`).Scan(&rows) + if s := stats(t, q); rows == 0 || s.Totals.Submitted != uint64(rows) { + t.Errorf("got %+v, want submitted %d", s.Totals, rows) + } +} + +// A stopped hook gets no calls; a refusal read before this object's own +// Resume does not undo it. +func TestPauseHookStopAndStaleRefusal(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + var calls []bool + var mu sync.Mutex + defer q.OnPauseChange(func(p bool) { + mu.Lock() + calls = append(calls, p) + mu.Unlock() + })() + q.OnPauseChange(func(bool) { t.Error("A stopped hook was called") })() + + q.Pause() + g := q.pz.gen.Load() + q.Resume() + q.refused(g) + if q.Paused() { + t.Error("Expected a stale refusal to leave the Resume") + } + mu.Lock() + if !slices.Equal(calls, []bool{true, false}) { + t.Errorf("hooks: got %v", calls) + } + mu.Unlock() + + // Racing: a refusal whose gen check passes must not apply after a + // whole Resume ran + q2, qs2 := openQueue(t) + defer qs2.Close() + for i := range 20000 { + q2.Pause() + g := q2.pz.gen.Load() + done := make(chan struct{}) + go func() { + q2.refused(g) + close(done) + }() + q2.Resume() + <-done + if q2.Paused() { + t.Fatalf("round %d: a stale refusal undid the Resume", i) + } + } +} + +// A Resume from another process between an empty pop and its read of the +// pause row is not a failed pop: the pop goes again and hands out the item. +func TestResumeBetweenPopAndRead(t *testing.T) { + q, qs := openQueue(t) + defer qs.Close() + other := otherProcess(t, q) + t.Cleanup(func() { afterEmptyPop = func() {} }) + pops := []struct { + name string + pop func() bool + }{ + {"Dequeue", func() bool { _, ok := q.Dequeue(); return ok }}, + {"DequeueWithAckId", func() bool { _, ok, _ := q.DequeueWithAckId(); return ok }}, + {"DequeueBatchWithAckIds", func() bool { + n, _, _ := q.DequeueBatchWithAckIds(make([]any, 2), make([]string, 2)) + return n == 1 + }}, + } + for _, p := range pops { + q.Enqueue([]byte(p.name)) + if err := other.Pause(); err != nil { + t.Fatal(err) + } + afterEmptyPop = func() { + afterEmptyPop = func() {} + if err := other.Resume(); err != nil { + t.Error(err) + } + } + if !p.pop() { + t.Errorf("%s: Expected the item after the Resume", p.name) + } + } +} + +// Completed, failed and cancelled pages sort by finish, not by add. +func TestJobsNewestFinishFirst(t *testing.T) { + q, qs := openQueue(t, WithRemoveOnComplete(false)) + defer qs.Close() + for _, id := range []string{"a", "b", "c", "d", "e", "f"} { + q.Enqueue(job(id)) + } + acks := make([]string, 6) + for i := range acks { + _, acks[i] = pop(t, q) + } + q.Acknowledge(acks[1]) + q.Acknowledge(acks[0]) + q.Reject(acks[3], "x") + q.Reject(acks[2], "x") + q.AcknowledgeCancelled(acks[5], "x") + q.AcknowledgeCancelled(acks[4], "y") + if got := ids(jobs(t, q, contract.JobCompleted, 0, 10)); !slices.Equal(got, []string{"a", "b"}) { + t.Errorf("completed: got %q", got) + } + if got := ids(jobs(t, q, contract.JobFailed, 0, 10)); !slices.Equal(got, []string{"e", "f", "c", "d"}) { + t.Errorf("failed: got %q", got) + } + cancelled := jobs(t, q, contract.JobCancelled, 0, 10) + if got := ids(cancelled); !slices.Equal(got, []string{"e", "f"}) { + t.Fatalf("cancelled: got %q", got) + } + if e := cancelled[0]; e.State != contract.JobCancelled || len(e.Errors) != 1 || e.Errors[0].Error != "y" { + t.Errorf("cancelled job: got %+v", e) + } +} + +// Seeding counts kept ended rows too: cancelled as failed. +func TestTotalsSeededFromEndedRows(t *testing.T) { + q, qs := openQueue(t, WithRemoveOnComplete(false)) + defer qs.Close() + enqueueN(t, q, 4) + _, a := pop(t, q) + _, b := pop(t, q) + _, c := pop(t, q) + q.Acknowledge(a) + q.Reject(b, "boom") + q.AcknowledgeCancelled(c, "stop") + if _, err := q.f.db.Exec("DROP TABLE " + q.meta); err != nil { + t.Fatal(err) + } + want := contract.JobTotals{Submitted: 4, Completed: 1, Failed: 2} + if got := stats(t, otherProcess(t, q)).Totals; got != want { + t.Errorf("got %+v, want %+v", got, want) + } +} + +// A finish that ends no row (repeat, stale or unknown ack) counts nothing. +func TestTotalsIgnoreNoopFinish(t *testing.T) { + for _, keep := range []bool{false, true} { + t.Run(fmt.Sprintf("keep=%t", keep), func(t *testing.T) { + q, qs := openQueue(t, WithRemoveOnComplete(!keep)) + defer qs.Close() + q.Enqueue(job("a")) + _, ack := pop(t, q) + if !q.Acknowledge(ack) { + t.Fatal("Acknowledge failed") + } + if q.Acknowledge(ack) || q.Reject(ack, "x") || q.AcknowledgeCancelled(ack, "x") || + q.Acknowledge("garbage") || q.Reject("999", "x") { + t.Error("Expected a finish of no processing row to return false") + } + if got, want := stats(t, q).Totals, (contract.JobTotals{Submitted: 1, Completed: 1}); got != want { + t.Errorf("got %+v, want %+v", got, want) + } + }) + } +} diff --git a/registry.go b/registry.go index 5db083a..c04e8d1 100644 --- a/registry.go +++ b/registry.go @@ -184,8 +184,13 @@ func (f *file) open(name string, priority bool, opts []Option) (*handle, error) h.q = q } + // ponytail: recovery runs on every process's first open, also one that + // never takes jobs (a producer, observer or pauser), and requeues jobs + // another process is running. Upgrade path: owner ids plus heartbeats + // on processing rows, requeueing only rows of dead owners (redisq's model). if !f.recovered[key] { if err := h.queue().requeueNoAck(); err != nil { + h.queue().Close() // stops a pause poll return nil, fmt.Errorf("failed to requeue unacknowledged rows: %w", err) } f.recovered[key] = true