Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/sqliteq.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: "1.24.1"
go-version: "1.27.0"
cache: true

- name: Install dependencies
Expand Down
14 changes: 13 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `Reject(ackID, reason)` on `Queue` and `PriorityQueue`, required by varmq's `IAcknowledgeable` since varmq#173: finishes an item that failed for good, never requeued. Deleted like an ack, or with `WithRemoveOnComplete(false)` kept with `status` `failed`. Same group commit as `Acknowledge`. The reason is not stored.
- `Reject(ackID, reason)` on `Queue` and `PriorityQueue`, required by varmq's `IAcknowledgeable` since varmq#173: finishes an item that failed for good, never requeued. Deleted like an ack, or with `WithRemoveOnComplete(false)` kept with `status` `failed`. Same group commit as `Acknowledge`. The reason is the job's outcome error.
- Job outcome (varmq spec 6.13): `WatchOutcome` (`contract.OutcomeWatcher`) and `AcknowledgeResult` (`contract.ResultAcknowledgeable`) on both queues. A job with a Job ID (`EnqueueOnce` id, else the item's JSON `id`) gets a record in `<name>__sqliteq_jobs` when it ends, same transaction as the finish. Watches poll it by seq every 50 ms (new records only), from any queue object on the file, any process. Retention: completed 1000 / 24 h, failed and cancelled 10000 / 7 d; the count limit spares records under 1 min old. A polling queue object also sweeps retention and expired marks once a minute.
- Job cancel (varmq spec 6.14): `CancelJob`, `OnCancel`, `AcknowledgeCancelled` (`contract.JobCanceller`, `CancelNotifier`, `CancelAcknowledgeable`). Marks in `<name>__sqliteq_cancelled`, 24 h each. Every pop drops a marked waiting job (outcome `cancelled`, at most 1000 per transaction; a single-item pop goes on past that, a batch pop reports an item left); `OnCancel` polls new marks. `ErrJobEnded` when the record shows it ended.
- `EnqueueOnce` that stores an id drops its old record and mark (trigger). `Purge` clears records and marks too. `Queue.Close` drops watches and hooks.

### Breaking

- Requires Go 1.27 (was 1.24), for the varmq dependency.
- A queue name must not contain `__sqliteq_`, in both layouts: a queue's indexes, sidecar tables and trigger use it.

### Changed

- Transactions begin `IMMEDIATE`. A finish (`Acknowledge`, `Reject`) always runs in a transaction; a pop runs in one while any cancel mark is live.

## [0.3.0] - 2026-10-23

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
<a href="https://pkg.go.dev/github.com/goptics/sqliteq"><img src="https://img.shields.io/badge/go-pkg-00ADD8.svg?logo=go" alt="Go Reference" /></a>
<a href="https://github.com/goptics/sqliteq/actions/workflows/sqliteq.yml"><img src="https://github.com/goptics/sqliteq/actions/workflows/sqliteq.yml/badge.svg" alt="CI" /></a>
<a href="https://codecov.io/gh/goptics/sqliteq"><img src="https://codecov.io/gh/goptics/sqliteq/branch/main/graph/badge.svg" alt="Codecov" /></a>
<a href="https://golang.org/doc/devel/release.html"><img src="https://img.shields.io/badge/Go-1.24+-00ADD8?style=flat-square&logo=go" alt="Go Version" /></a>
<a href="https://golang.org/doc/devel/release.html"><img src="https://img.shields.io/badge/Go-1.27+-00ADD8?style=flat-square&logo=go" alt="Go Version" /></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square" alt="License" /></a>
</p>

Expand Down Expand Up @@ -49,7 +49,7 @@
go get github.com/goptics/sqliteq
```

Requires Go 1.24+ and cgo: sqliteq uses [mattn/go-sqlite3](https://github.com/mattn/go-sqlite3), so building it needs `CGO_ENABLED=1` and a C compiler.
Requires Go 1.27+ and cgo: sqliteq uses [mattn/go-sqlite3](https://github.com/mattn/go-sqlite3), so building it needs `CGO_ENABLED=1` and a C compiler.

## Quick Start

Expand Down
192 changes: 192 additions & 0 deletions cancel.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
package sqliteq

import (
"database/sql"
"errors"
"fmt"
"time"

"github.com/goptics/varmq/contract"
)

// Job cancel (varmq spec/redis-storage.md 1.7.0, 6.14), on SQLite.
// CancelJob writes a mark to <name>__sqliteq_cancelled (expires_at: ms).
// Every pop drops a marked waiting job with outcome cancelled; OnCancel
// learns of marks by polling seq; AcknowledgeCancelled finishes a running
// job that stopped for it. Any finish, and an EnqueueOnce that stores the
// id, clears the mark. sqliteq has no workflow store, so no item is exempt.

var (
_ contract.JobCanceller = (*Queue)(nil)
_ contract.JobCanceller = (*PriorityQueue)(nil)
_ contract.CancelNotifier = (*Queue)(nil)
_ contract.CancelNotifier = (*PriorityQueue)(nil)
_ contract.CancelAcknowledgeable = (*Queue)(nil)
_ contract.CancelAcknowledgeable = (*PriorityQueue)(nil)
)

const (
// cancelTTL is how long one mark lasts.
cancelTTL = 24 * time.Hour
// maxDrops caps the marked items one pop drops; at the cap it returns
// what it has and the next pop goes on.
maxDrops = 1000
)

var (
errEmptyJobID = errors.New("sqliteq: empty job id")
errClosed = errors.New("sqliteq: queue is closed")
)

// CancelJob implements contract.JobCanceller: it marks job id cancelled for
// cancelTTL. A waiting job is dropped by the pop that would hand it out
// (outcome cancelled); OnCancel callbacks of every queue object on the file
// learn of a running one, which ends only if its worker stops it.
// contract.ErrJobEnded when its outcome record shows it ended. An unknown
// id is marked too: a later Enqueue of it is dropped, unless EnqueueOnce
// stores it (that clears the mark).
func (q *Queue) CancelJob(id string) error {
if id == "" {
return errEmptyJobID
}
if q.closed.Load() {
return errClosed
}

now := time.Now().UnixMilli()
q.f.mu.Lock()
defer q.f.mu.Unlock()
tx, err := q.f.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()

var state string
switch err := tx.QueryRow(fmt.Sprintf("SELECT state FROM %s WHERE job_id = ?", q.jobs), id).Scan(&state); {
case err == nil:
return contract.ErrJobEnded // every record is an end
case !errors.Is(err, sql.ErrNoRows):
return err
}
if _, err := tx.Exec(fmt.Sprintf("DELETE FROM %s WHERE expires_at <= ?", q.cancelled), now); err != nil {
return err
}
// REPLACE deletes an older mark of id, so the new one takes a new seq
// and reaches OnCancel again
if _, err := tx.Exec(fmt.Sprintf("INSERT OR REPLACE INTO %s (job_id, expires_at) VALUES (?, ?)", q.cancelled),
id, now+cancelTTL.Milliseconds()); err != nil {
return err
}
return tx.Commit()
}

// AcknowledgeCancelled implements contract.CancelAcknowledgeable: Reject
// with outcome cancelled and reason as its error. A kept row
// (WithRemoveOnComplete(false)) gets status 'cancelled'.
func (q *Queue) AcknowledgeCancelled(ackID, reason string) bool {
return q.finish(ackID, contract.JobCancelled, reason, nil)
}

// OnCancel implements contract.CancelNotifier: fn gets the job ID of each
// cancel request made after the call, through any queue object on this
// database file, until stop. Marks are polled every pollEvery; each call
// runs on its own goroutine. A mark gone by the poll (the job finished or
// was re-added) is never reported. A closed queue, or a failed first read,
// reports nothing.
func (q *Queue) OnCancel(fn func(jobID string)) (stop func()) {
// Read before the lock: a mark made after this read has a higher seq
var last sql.NullInt64
if err := q.f.db.QueryRow(fmt.Sprintf("SELECT MAX(seq) FROM %s", q.cancelled)).Scan(&last); err != nil {
return func() {} // 0 would replay every live mark
}

h := &cancelHook{fn: fn, from: last.Int64}
w := &q.w
w.mu.Lock()
defer w.mu.Unlock()
if w.closed {
return func() {}
}
if len(w.hooks) == 0 {
w.lastSeq = h.from // no poll read marks for anyone since
}
if w.hooks == nil {
w.hooks = map[*cancelHook]struct{}{}
}
w.hooks[h] = struct{}{}
w.startLocked(q)

return func() {
w.mu.Lock()
defer w.mu.Unlock()
delete(w.hooks, h)
}
}

// cancelHook is one OnCancel callback; it gets marks with seq above from.
type cancelHook struct {
fn func(string)
from int64
}

// pollCancels reads the live marks newer than seq and passes each to the
// hooks live after the read whose from is below it, so a hook added during
// the read still gets its marks, and a stopped one gets none.
func (q *Queue) pollCancels(seq int64) {
rows, err := q.f.db.Query(fmt.Sprintf("SELECT seq, job_id FROM %s WHERE seq > ? AND expires_at > ? ORDER BY seq LIMIT %d",
q.cancelled, maxDrops), seq, time.Now().UnixMilli())
if err != nil {
return
}
type mark struct {
seq int64
id string
}
var marks []mark
for rows.Next() {
var m mark
if rows.Scan(&m.seq, &m.id) == nil {
marks = append(marks, m)
}
}
if rows.Close() != nil || rows.Err() != nil || len(marks) == 0 {
return // read again from the old seq
}

w := &q.w
w.mu.Lock()
defer w.mu.Unlock()
if w.closed {
return
}
w.lastSeq = max(w.lastSeq, marks[len(marks)-1].seq)
for h := range w.hooks {
for _, m := range marks {
if m.seq > h.from {
go h.fn(m.id)
}
}
}
}

// anyMark reports whether a live mark exists, so a pop without marks pays
// one read; an error counts as a mark (the gated pop then reports it).
func (q *Queue) anyMark(now time.Time) bool {
var n int
if err := q.hasMark.QueryRow(now.UnixMilli()).Scan(&n); err != nil {
return true
}
return n == 1
}

// marked reports whether job id has a live mark; isMarked is q.isMarked
// in the caller's tx.
func marked(isMarked *sql.Stmt, id string, now time.Time) (bool, error) {
if id == "" {
return false, nil
}
var n int
err := isMarked.QueryRow(id, now.UnixMilli()).Scan(&n)
return n == 1, err
}
3 changes: 2 additions & 1 deletion commit.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
// code (such as a driver.Valuer): a panic there would strand the round and
// leave its transaction open. Convert user values before building it.
type op struct {
run func(tx *sql.Tx) (bool, error) // one caller's statement(s); a nil tx means autocommit
run func(tx *sql.Tx) (bool, error) // one caller's statement(s); a nil tx means autocommit, or a tx of its own
done chan result // buffered 2: one lead wake-up and the outcome
}

Expand Down Expand Up @@ -97,6 +97,7 @@ func (b *batcher) lead() {
// transaction of its own, so one bad call fails alone. A round of one op runs
// with a nil tx, as one autocommit statement: a database/sql Tx costs a
// context and a goroutine, which a round with nothing to share does not repay.
// An op of several statements (a finish) opens its own tx then.
func (f *file) commitRound(ops []*op) {
f.mu.Lock()
defer f.mu.Unlock()
Expand Down
50 changes: 50 additions & 0 deletions commit_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -491,3 +491,53 @@ func TestGroupCommitAfterClose(t *testing.T) {
}
waitAll(t, &wg, 10*time.Second)
}

// Finishes that share a round write their outcomes in its transaction,
// and again each in its own when a bad call fails the round.
func TestGroupFinishOutcomes(t *testing.T) {
for _, fail := range []bool{false, true} {
t.Run(map[bool]string{false: "shared", true: "fallback"}[fail], func(t *testing.T) {
q, qs := openQueue(t)
defer qs.Close()
for _, id := range []string{"a", "b", "c", "d"} {
q.Enqueue(job(id))
}
items, acks := make([]any, 4), make([]string, 4)
if n, _, err := q.DequeueBatchWithAckIds(items, acks); err != nil || n != 4 {
t.Fatalf("got %d %v", n, err)
}
rest := []func() bool{
func() bool { return q.Reject(acks[1], "r") },
func() bool { return q.AcknowledgeResult(acks[2], []byte(`1`)) },
func() bool { return q.AcknowledgeCancelled(acks[3], "c") },
}
if fail {
rest = append(rest, func() bool { return q.Enqueue(nil) })
}
got := oneRound(t, q.f, func() bool { return q.Acknowledge(acks[0]) }, rest)
for i, ok := range got[:4] {
if !ok {
t.Errorf("finish %d failed", i)
}
}
want := map[string]string{"a": "completed||", "b": "failed|r|", "c": "completed||1", "d": "cancelled|c|"}
rows, err := q.f.db.Query("SELECT job_id, state, COALESCE(error, ''), COALESCE(CAST(result AS TEXT), '') FROM " + q.jobs)
if err != nil {
t.Fatal(err)
}
defer rows.Close()
n := 0
for rows.Next() {
var id, state, e, r string
rows.Scan(&id, &state, &e, &r)
if g := state + "|" + e + "|" + r; g != want[id] {
t.Errorf("%s: got %s, want %s", id, g, want[id])
}
n++
}
if n != 4 {
t.Errorf("Expected 4 records, got %d", n)
}
})
}
}
8 changes: 6 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
module github.com/goptics/sqliteq

go 1.24
go 1.27

require github.com/mattn/go-sqlite3 v1.14.28
require (
github.com/goptics/varmq v1.10.1-0.20261108161945-6bc553db29c6
github.com/goptics/varmq/contract v0.6.0
github.com/mattn/go-sqlite3 v1.14.28
)
12 changes: 12 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,2 +1,14 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/goptics/varmq v1.10.1-0.20261108161945-6bc553db29c6 h1:0GooGJC22Z/9WxAz0f8aVzWyPkBDJhHF8JI4oYTIW+w=
github.com/goptics/varmq v1.10.1-0.20261108161945-6bc553db29c6/go.mod h1:RLnozQPouu3sq2tHoCixCXSoCZMPjB90MNEHvlpFz+E=
github.com/goptics/varmq/contract v0.6.0 h1://rpDlzSwsqvPDS98siuPDN1LyyBLFhiYeAmB7MZfbQ=
github.com/goptics/varmq/contract v0.6.0/go.mod h1:55tOUywUsZhBgyo8oJhNmKMknETVHL2OuMrITztanXg=
github.com/mattn/go-sqlite3 v1.14.28 h1:ThEiQrnbtumT+QMknw63Befp/ce/nUPgBPMlRFEum7A=
github.com/mattn/go-sqlite3 v1.14.28/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
6 changes: 6 additions & 0 deletions migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,12 @@ func TestMigrateV02File(t *testing.T) {
if got := indexNames(t, db, name); fmt.Sprint(got) != fmt.Sprint(want) {
t.Errorf("Round %d: indexes = %v, want %v", round, got, want)
}
for _, side := range []string{name + jobsSuffix, name + cancelledSuffix, name + readdSuffix} {
var n int
if err := db.QueryRow("SELECT COUNT(*) FROM sqlite_master WHERE name = ?", side).Scan(&n); err != nil || n != 1 {
t.Errorf("Round %d: expected %s on the old file, got %d %v", round, side, n, err)
}
}

if round == 1 {
// The two processing rows are pending again after recovery
Expand Down
Loading
Loading