Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Workflows (VarMQ spec 6.9 to 6.12): `NewWorkflowStore(qs, ...WorkflowOption)` returns the `contract.WorkflowStore` of a store root. (#25)
- The store registers with `contract.RegisterWorkflowStore`; `Close` unregisters it and releases its file. (#25)
- `WithWorkflowResultLimit` (default 64 KiB, `ErrResultTooLarge`) and `WithWorkflowRetention` (default 7 days). (#25)
- `ScopeID()` on `Queue` and `PriorityQueue`: `sqliteq:` and the absolute store folder (symlinks resolved), or the file under `WithSingleFile`. (#25)
- A file open under one scope is refused under another; `WithSingleFile` refuses the name `__sqliteq_workflows.db`. (#25)
- `FinishTask` finds the queue for any client of the scope in this process. (#25)
- `WithSingleFile`: a push into a queue no client has open becomes an outbox row in the same transaction. (#25)
- A plain job cannot take a job ID that starts with `wf:`; a held ID keeps the push in the outbox. (#25)
- `Purge` ends purged workflow tasks without a registered store; one transaction under `WithSingleFile`, a stored intent in the folder layout. (#25)
- `Submit` refuses a dependency that names no task, and replaces an expired, unswept workflow of the same ID. (#25)
- Folder layout: the store keeps its tables in `__sqliteq_workflows.db`. `WithSingleFile`: tables in the same file. (#25)
- Tables `__sqliteq_wf_workflows`, `__sqliteq_wf_tasks`, `__sqliteq_wf_outbox`. (#25)
- `FinishTask` in three steps in the folder layout: record the end and write the outbox, push each row with `EnqueueOnce`, acknowledge the job. (#25)
- A recovery loop pushes leftover outbox rows at open and every second, in both layouts. (#25)
- `WithSingleFile`: the end, the pushes and the ack are one transaction. (#25)
- A redelivered job, or a push repeated after the job was removed, is safe: the pre-run check skips a task that already ended. (#25)
- `Workflows` (`contract.WorkflowInspector`) and `WorkflowKeys` (`contract.WorkflowKeyLister`) list workflows newest first. (#25)
- A cancel mark never drops a valid workflow item; the workflow decides its end. (#25)
- Task JSON whose keys differ only in case (`WF` next to `wf`) is a plain job. (#25)
- `Purge` ends each purged workflow task as cancelled, reason `queue purged`; its workflow goes on. (#25)
- `Reject(ackID, reason)` on `Queue` and `PriorityQueue`, for VarMQ's `IAcknowledgeable` (varmq#173). (#21)
- `Reject` finishes an item that failed for good: never requeued. (#21)
- Rejected item deleted like an ack; with `WithRemoveOnComplete(false)` kept with `status` `failed`. (#21)
Expand Down
62 changes: 61 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
- **Job outcomes**: `WatchOutcome` tells any process on the file when a job completes, fails or is cancelled, with the worker's result.
- **Job cancel**: `CancelJob` drops a waiting job, and `OnCancel` tells every process on the file to stop a running one.
- **Job records and inspection**: counts, totals and job lists for VarMQ's REST API and vartop.
- **[Workflows](#workflows)**: `NewWorkflowStore` keeps VarMQ workflows (a DAG of jobs across queues) in SQLite, so they survive a restart.
- **Durable queue pause**: `Pause` stops the pops of every process on the file, and the pause outlives restarts.
- **Automatic WAL checkpoints**, tunable with `WithCheckpointPages`.
- **Thread-safe**: share one queue object across goroutines.
Expand Down Expand Up @@ -197,7 +198,7 @@ sqliteq imports VarMQ's `contract` module, and it imports VarMQ itself only in t

A worker whose only source is the sqliteq queue claims several jobs in one statement through `DequeueBatchWithAckIds`. A worker bound to several queues claims one job per call. A job added with an ID (`WithJobId`, or one from `WithJobIdGenerator`) goes through `EnqueueOnce`.

sqliteq has no workflow store and no `JobReader`. So VarMQ's one-job read (`GET /queues/{id}/jobs/{jobId}`) answers 501 for a sqliteq queue.
sqliteq has no `JobReader`. So VarMQ's one-job read (`GET /queues/{id}/jobs/{jobId}`) answers 501 for a sqliteq queue. For workflows, see [Workflows](#workflows).

### Job outcome and cancel

Expand Down Expand Up @@ -255,6 +256,65 @@ Both queue types implement VarMQ's `QueuePauser` and `PauseNotifier`. So VarMQ's

`Paused()` reads this queue object's cache, and a pop refused for the pause sets it. While it is set, the queue object polls the row every 50 ms. So a `Resume` from another process wakes this one's workers within that time. `OnPauseChange(fn)` calls `fn` each time the cache changes, and `PauseRefusals()` counts the pops refused for a pause.

### Workflows

A workflow is a graph of VarMQ tasks. A task runs after the tasks it depends on. sqliteq keeps the workflow state in SQLite, so a workflow survives a restart and spans several queues of one store.

Create the store once and keep it open while workers run. The order of the store and the queues does not matter. `NewWorkflowStore` registers the store for the scope of its queues. VarMQ finds it from there. A process that only runs workers needs this one call too.

```go
qs, _ := sqliteq.New("./data")
defer qs.Close()

store, err := sqliteq.NewWorkflowStore(qs)
if err != nil {
log.Fatal(err)
}
defer store.Close() // before qs.Close

resize, _ := qs.NewQueue("resize")
publish, _ := qs.NewQueue("publish")

wf := varmq.NewWorkflow(store)
worker := varmq.NewErrWorker(func(j varmq.Job[string]) error { return nil })
defer worker.WaitAndStop()

a := wf.Add("resize", worker.WithQueue(resize), "photo.jpg")
wf.Add("publish", worker.WithQueue(publish), "photo.jpg").After(a)

wf.Submit(ctx)
record, _ := wf.Wait()
```

Options of `NewWorkflowStore`:

- `WithWorkflowResultLimit(n)` caps a task result at `n` bytes. The default is 64 KiB. A bigger result fails the task with `ErrResultTooLarge`.
- `WithWorkflowRetention(d)` sets how long an ended workflow is kept. The default is 7 days. A running workflow is kept until it ends.

Every queue of a workflow must have the same scope. The scope of a queue is `ScopeID()`: `sqliteq:` and the absolute path of the store folder, or of the file under `WithSingleFile`, with symlinks resolved. A file already open under one scope cannot be opened under another, for example as a single file and as a queue file of a folder. `NewQueue` returns an error then. `WithSingleFile` refuses the file name `__sqliteq_workflows.db`. One process holds one store for each scope. A second `NewWorkflowStore` for the same scope returns `contract.ErrScopeRegistered`.

The store implements `contract.WorkflowStore`, `WorkflowInspector` and `WorkflowKeyLister`. So VarMQ's REST API lists and cancels sqliteq workflows.

**Where the state lives.** In the folder layout the store uses its own file, `__sqliteq_workflows.db`. Under `WithSingleFile` it uses three tables of the store file. The tables are `__sqliteq_wf_workflows`, `__sqliteq_wf_tasks` and `__sqliteq_wf_outbox`. A queue name must not contain `__sqliteq_`, so a queue never takes these names.

**How a task ends.** In the folder layout, two files cannot share one transaction. So `FinishTask` works in three steps:

1. One transaction in the workflow file records the end and the result. It also writes each task the end releases into the outbox table.
2. Each outbox row is pushed into its queue with `EnqueueOnce` and the job ID `wf:<id>:<task>`. Then the row is deleted.
3. The job is acknowledged in its queue.

A recovery loop pushes leftover outbox rows when the store opens and every second while it is open. A crash between the steps is safe. A row pushed twice stores one job while the first job is still in the queue. Once a worker has finished and removed that job, a second push would queue it again. The pre-run check (`Runnable`) then skips it, because its task already ended. A job that was redelivered is skipped the same way. Under `WithSingleFile` the end, the pushes and the ack are one transaction. A push into a queue that no client in the process has open becomes an outbox row in that transaction, and the loop pushes it later.

**Limits.**

- The recovery loop cannot wake the workers of a process. A task pushed by the loop starts when a worker next polls its queue.
- The loop pushes only into queues that some client in this process has open. Rows for other queues wait, and the store logs each such queue once.
- A job ID that starts with `wf:` is for workflow items. A plain job cannot take one. If an older row already holds the ID, the push stays in the outbox and the store logs it once.
- The stored items of waiting tasks and the undo items stay in the workflow file until the task is pushed or the workflow expires.
- Folder layout: `Purge` stores an intent in the workflow file before it deletes the rows. If the process crashes between them, the recovery loop ends the purged tasks later. It applies an intent only once the job's row is gone from its queue, so a slow or aborted purge never cancels a live task.

A valid workflow item is never dropped by a cancel mark: the workflow decides how its task ends. A `Purge` of a queue ends each purged workflow task as cancelled with the reason `queue purged`. The workflow goes on. This works also when no store is open in the process, as long as the workflow tables exist. Under `WithSingleFile` the delete and the ends are one transaction. An item whose JSON has keys that differ only in case, such as `WF` next to `wf`, is a plain job.

### Storage

#### Layout on disk
Expand Down
13 changes: 12 additions & 1 deletion cancel.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ import (
// Every pop drops a marked waiting job with outcome cancelled; OnCancel
// learns of marks by polling seq; AcknowledgeCancelled finishes a running
// job that stopped for it. Any finish, and an EnqueueOnce that stores the
// id, clears the mark. sqliteq has no workflow store, so no item is exempt.
// id, clears the mark. A valid workflow item (workflowTag) whose job ID is its tag's is never dropped: the
// workflow decides its end, and its mark stays.

var (
_ contract.JobCanceller = (*Queue)(nil)
Expand Down Expand Up @@ -190,3 +191,13 @@ func marked(isMarked *sql.Stmt, id string, now time.Time) (bool, error) {
err := isMarked.QueryRow(id, now.UnixMilli()).Scan(&n)
return n == 1, err
}

// markedDrop is marked for a pop: a valid workflow item is never dropped.
func markedDrop(isMarked *sql.Stmt, id string, data []byte, now time.Time) (bool, error) {
drop, err := marked(isMarked, id, now)
if err != nil || !drop {
return false, err
}
tag, ok := workflowTag(data)
return !(ok && id == tag.jobID()), nil
}
6 changes: 6 additions & 0 deletions example/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ The example program provides two demonstrations:
- Adding items with different priorities (lower number = higher priority)
- Observing how items are dequeued in priority order

3. **Workflow Example** - Runs a two-step VarMQ workflow over two queues:
- Creating a workflow store with `NewWorkflowStore`
- Adding tasks to queues and ordering them with `After`
- Waiting for the workflow to complete

## Running the Examples

To run the examples:
Expand All @@ -34,6 +39,7 @@ SQLiteQ Examples
---------------
1. Regular Queue Example
2. Priority Queue Example
3. Workflow Example
Choose an example (default is 1):
```

Expand Down
3 changes: 3 additions & 0 deletions example/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ func main() {
fmt.Println("---------------")
fmt.Println("1. Regular Queue Example")
fmt.Println("2. Priority Queue Example")
fmt.Println("3. Workflow Example")

// Default to regular queue example
var choice string
Expand All @@ -101,6 +102,8 @@ func main() {
switch choice {
case "2":
priorityQueueExample()
case "3":
workflowExample()
default:
regularQueueExample()
}
Expand Down
70 changes: 70 additions & 0 deletions example/workflow_example.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
package main

import (
"context"
"fmt"
"log"
"os"
"time"

"github.com/goptics/sqliteq"
"github.com/goptics/varmq"
)

// workflowExample runs a two-step workflow over two queues. The state lives
// in __sqliteq_workflows.db next to the queue files.
func workflowExample() {
dir, err := os.MkdirTemp("", "sqliteq-workflow-*")
if err != nil {
log.Fatalf("Failed to create temp folder: %v", err)
}
defer os.RemoveAll(dir)

qs, err := sqliteq.New(dir)
if err != nil {
log.Fatalf("Failed to open queues: %v", err)
}
defer qs.Close()

resize, err := qs.NewQueue("resize")
if err != nil {
log.Fatalf("Failed to create queue: %v", err)
}
publish, err := qs.NewQueue("publish")
if err != nil {
log.Fatalf("Failed to create queue: %v", err)
}

// The store registers itself for the scope of the queues
store, err := sqliteq.NewWorkflowStore(qs)
if err != nil {
log.Fatalf("Failed to create workflow store: %v", err)
}
defer store.Close()

step := func(name string) func(varmq.Job[string]) error {
return func(j varmq.Job[string]) error {
fmt.Printf("%s %s\n", name, j.Data())
return nil
}
}
resizer := varmq.NewErrWorker(step("resize"))
defer resizer.WaitAndStop()
publisher := varmq.NewErrWorker(step("publish"))
defer publisher.WaitAndStop()

wf := varmq.NewWorkflow(store)
a := wf.Add("resize", resizer.WithQueue(resize), "photo.jpg")
wf.Add("publish", publisher.WithQueue(publish), "photo.jpg").After(a)

if err := wf.Submit(context.Background()); err != nil {
log.Fatalf("Failed to submit workflow: %v", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
record, err := wf.WaitContext(ctx)
if err != nil {
log.Fatalf("Workflow did not finish: %v", err)
}
fmt.Println("workflow", record.ID, record.State)
}
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/goptics/sqliteq
go 1.27

require (
github.com/goptics/varmq v1.10.1-0.20261108161945-6bc553db29c6
github.com/goptics/varmq/contract v0.6.0
github.com/goptics/varmq v1.10.1-0.20261109165008-c75ed52b3b0a
github.com/goptics/varmq/contract v0.8.0
github.com/mattn/go-sqlite3 v1.14.28
)
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/goptics/varmq v1.10.1-0.20261108161945-6bc553db29c6 h1:0GooGJC22Z/9WxAz0f8aVzWyPkBDJhHF8JI4oYTIW+w=
github.com/goptics/varmq v1.10.1-0.20261108161945-6bc553db29c6/go.mod h1:RLnozQPouu3sq2tHoCixCXSoCZMPjB90MNEHvlpFz+E=
github.com/goptics/varmq/contract v0.6.0 h1://rpDlzSwsqvPDS98siuPDN1LyyBLFhiYeAmB7MZfbQ=
github.com/goptics/varmq/contract v0.6.0/go.mod h1:55tOUywUsZhBgyo8oJhNmKMknETVHL2OuMrITztanXg=
github.com/goptics/varmq v1.10.1-0.20261109165008-c75ed52b3b0a h1:+UjHIuTdKROtu6zLVk8b/efeP6uTmQqIy7TgshIHx6U=
github.com/goptics/varmq v1.10.1-0.20261109165008-c75ed52b3b0a/go.mod h1:4hso4XnUSTh5sjr/h2//MYh0FPGRvebLCx5AaoL4qII=
github.com/goptics/varmq/contract v0.8.0 h1:U87RzQITqs3TPNuzNXLCl9Fp9ebUa6Eiw3cDNd2cb5Q=
github.com/goptics/varmq/contract v0.8.0/go.mod h1:55tOUywUsZhBgyo8oJhNmKMknETVHL2OuMrITztanXg=
github.com/mattn/go-sqlite3 v1.14.28 h1:ThEiQrnbtumT+QMknw63Befp/ce/nUPgBPMlRFEum7A=
github.com/mattn/go-sqlite3 v1.14.28/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
Expand Down
70 changes: 64 additions & 6 deletions layout_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ import (
"strings"
"sync"
"testing"

"github.com/goptics/varmq/contract"
)

// mustNew opens a store at path and fails the test on error.
Expand Down Expand Up @@ -413,12 +415,12 @@ func TestSingleFileAndFolderOneEntry(t *testing.T) {
if err != nil {
t.Fatalf("Failed to create queue: %v", err)
}
b, err := single.NewQueue("JOBS")
if err != nil {
t.Fatalf("Failed to create queue: %v", err)
// One file under two roots would have two scopes: refused
if _, err := single.NewQueue("JOBS"); err == nil || a == nil {
t.Errorf("Expected an error for a file open under another scope, got %v", err)
}
if a != b {
t.Error("Expected the folder and single-file stores to share the handle")
if _, err := New(filepath.Join(dir, workflowsFile), WithSingleFile()); err == nil {
t.Error("Expected the workflow file name refused for WithSingleFile")
}
}

Expand Down Expand Up @@ -653,7 +655,7 @@ func TestSymlinkedFileOneEntry(t *testing.T) {
for _, store := range []struct {
path string
opts []StoreOption
}{{real, []StoreOption{WithSingleFile()}}, {link, []StoreOption{WithSingleFile()}}, {folder, nil}} {
}{{real, []StoreOption{WithSingleFile()}}, {link, []StoreOption{WithSingleFile()}}} {
qs := mustNew(t, store.path, store.opts...)
defer qs.Close()
q, err := qs.NewQueue("jobs")
Expand All @@ -668,6 +670,13 @@ func TestSymlinkedFileOneEntry(t *testing.T) {
t.Errorf("Path %d reached a different file entry", i+1)
}
}

// The same file under a folder root would have another scope: refused
qs := mustNew(t, folder)
defer qs.Close()
if _, err := qs.NewQueue("jobs"); err == nil {
t.Error("Expected a folder symlinking to an open single file refused")
}
}

func TestReopenAfterCloseNoRecovery(t *testing.T) {
Expand Down Expand Up @@ -698,3 +707,52 @@ func TestReopenAfterCloseNoRecovery(t *testing.T) {
t.Error("Acknowledge on the closed handle failed after the reopen")
}
}

// The scope is the canonical store root: a symlinked folder, and a queue
// file that is a symlink inside it, give the queues and the store one scope.
func TestScopeThroughSymlinks(t *testing.T) {
real := t.TempDir()
link := filepath.Join(t.TempDir(), "link")
if err := os.Symlink(real, link); err != nil {
t.Skipf("Cannot create a symlink: %v", err)
}
target := filepath.Join(t.TempDir(), "elsewhere.db")
if err := os.WriteFile(target, nil, 0o600); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target, filepath.Join(real, "a.db")); err != nil {
t.Skipf("Cannot create a symlink: %v", err)
}

qs := mustNew(t, link)
defer qs.Close()
a, err := qs.NewQueue("a")
if err != nil {
t.Fatal(err)
}
b, err := qs.NewQueue("b")
if err != nil {
t.Fatal(err)
}
st, err := NewWorkflowStore(qs)
if err != nil {
t.Fatal(err)
}
defer st.Close()

want, err := filepath.EvalSymlinks(real)
if err != nil {
t.Fatal(err)
}
want = "sqliteq:" + want
if a.ScopeID() != want || b.ScopeID() != want || st.ScopeID() != want {
t.Fatalf("Expected scope %q, got %q %q %q", want, a.ScopeID(), b.ScopeID(), st.ScopeID())
}
if got, ok := contract.LookupWorkflowStore(a.ScopeID()); !ok || got != contract.WorkflowStore(st) {
t.Error("Expected the queue's scope to find the store")
}
plan := planWF("w1", planTask("w1", "x", a), planTask("w1", "y", b, "x"))
if err := st.Submit(context.Background(), plan); err != nil {
t.Fatalf("Expected Submit across the symlinked queues to succeed, got %v", err)
}
}
Loading
Loading