Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `WithSingleFile`: the end, the pushes and the ack are one transaction. (#25)
- A redelivered job, or a push repeated after the job was removed, is safe: the pre-run check skips a task that already ended. (#25)
- `Workflows` (`contract.WorkflowInspector`) and `WorkflowKeys` (`contract.WorkflowKeyLister`) list workflows newest first. (#25)
- `WorkflowCounts` (`contract.WorkflowCounter`, contract v0.9.0): stored workflows per state, same rows as `WorkflowKeys`.
- `HostLocal` marker (`contract.HostLocal`): VarMQ puts host name into the store's ID in `GET /workflow-counts`, so same path on two hosts = two stores.
- Task times on `TaskRecord` and its `Undo`: `EnqueuedAt` at push (written to queue or outbox; a push that waits in the outbox keeps it), `StartedAt` when `Runnable` last answered true, `FinishedAt` at end. Unix ms, store clock.
- `Runnable` answers from a lock-free read: a foreign write lock gives no error. A true answer then stamps `StartedAt`, only on a still-pending task; a failed stamp is logged, answer stays true. Unknown workflow: `ErrWorkflowNotFound`, no row made.
- `Purge` keeps the jobs (logs the cause) when the file has workflow tables but no store is ready (check, open or init failed): no task vanishes under a workflow left `running`.
- `__sqliteq_wf_tasks` columns `enqueued_at`, `started_at`, `finished_at`, `undo_enqueued_at`, `undo_started_at`, `undo_finished_at` (INTEGER, default 0). Added at open to older tables; old rows read 0.
- Index `__sqliteq_wf_workflows_state` on `(state, created_at DESC, id)`.
- A cancel mark never drops a valid workflow item; the workflow decides its end. (#25)
- Task JSON whose keys differ only in case (`WF` next to `wf`) is a plain job. (#25)
- `Purge` ends each purged workflow task as cancelled, reason `queue purged`; its workflow goes on. (#25)
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ Options of `NewWorkflowStore`:

Every queue of a workflow must have the same scope. The scope of a queue is `ScopeID()`: `sqliteq:` and the absolute path of the store folder, or of the file under `WithSingleFile`, with symlinks resolved. A file already open under one scope cannot be opened under another, for example as a single file and as a queue file of a folder. `NewQueue` returns an error then. `WithSingleFile` refuses the file name `__sqliteq_workflows.db`. One process holds one store for each scope. A second `NewWorkflowStore` for the same scope returns `contract.ErrScopeRegistered`.

The store implements `contract.WorkflowStore`, `WorkflowInspector` and `WorkflowKeyLister`. So VarMQ's REST API lists and cancels sqliteq workflows.
The store implements `contract.WorkflowStore`, `WorkflowInspector`, `WorkflowKeyLister` and `WorkflowCounter`. So VarMQ's REST API lists, counts and cancels sqliteq workflows. Each task record carries its times: enqueued, started, finished (Unix ms, 0 when not reached).

**Where the state lives.** In the folder layout the store uses its own file, `__sqliteq_workflows.db`. Under `WithSingleFile` it uses three tables of the store file. The tables are `__sqliteq_wf_workflows`, `__sqliteq_wf_tasks` and `__sqliteq_wf_outbox`. A queue name must not contain `__sqliteq_`, so a queue never takes these names.

Expand Down
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/goptics/sqliteq
go 1.27

require (
github.com/goptics/varmq v1.10.1-0.20261109165008-c75ed52b3b0a
github.com/goptics/varmq/contract v0.8.0
github.com/goptics/varmq v1.10.1-0.20261110052717-8057d16e4b55
github.com/goptics/varmq/contract v0.9.0
github.com/mattn/go-sqlite3 v1.14.28
)
8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/goptics/varmq v1.10.1-0.20261109165008-c75ed52b3b0a h1:+UjHIuTdKROtu6zLVk8b/efeP6uTmQqIy7TgshIHx6U=
github.com/goptics/varmq v1.10.1-0.20261109165008-c75ed52b3b0a/go.mod h1:4hso4XnUSTh5sjr/h2//MYh0FPGRvebLCx5AaoL4qII=
github.com/goptics/varmq/contract v0.8.0 h1:U87RzQITqs3TPNuzNXLCl9Fp9ebUa6Eiw3cDNd2cb5Q=
github.com/goptics/varmq/contract v0.8.0/go.mod h1:55tOUywUsZhBgyo8oJhNmKMknETVHL2OuMrITztanXg=
github.com/goptics/varmq v1.10.1-0.20261110052717-8057d16e4b55 h1:aRBQi8UhqtTVfl20meNmvHq+v8SKGLc22wZTfhnh7p8=
github.com/goptics/varmq v1.10.1-0.20261110052717-8057d16e4b55/go.mod h1:wYDkTj4w50s08bqsWnsEg/pooFyBQrPHGzht31CDkE0=
github.com/goptics/varmq/contract v0.9.0 h1:Z/0pJNi6cviSIwavCyJn43fCK3ZLk19IXc3D7BQfBXE=
github.com/goptics/varmq/contract v0.9.0/go.mod h1:55tOUywUsZhBgyo8oJhNmKMknETVHL2OuMrITztanXg=
github.com/mattn/go-sqlite3 v1.14.28 h1:ThEiQrnbtumT+QMknw63Befp/ce/nUPgBPMlRFEum7A=
github.com/mattn/go-sqlite3 v1.14.28/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
Expand Down
198 changes: 198 additions & 0 deletions migration_test.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
package sqliteq

import (
"context"
"database/sql"
"fmt"
"maps"
"path/filepath"
"sort"
"testing"

"github.com/goptics/varmq/contract"
)

// The v0.2.3 schema, pasted from tag v0.2.3's queue.go initTable and
Expand Down Expand Up @@ -247,3 +251,197 @@ func TestPendingIndexNameFreeForQueues(t *testing.T) {
})
}
}

// The workflow tables as #25 (contract v0.8.0) made them: no task times, no
// state index. Pasted from that commit's workflow.go init.
const v08WorkflowTables = `
CREATE TABLE "__sqliteq_wf_workflows" (
id TEXT PRIMARY KEY,
hash TEXT NOT NULL,
state TEXT NOT NULL,
created_at INTEGER NOT NULL,
finished_at INTEGER,
fail_fast INTEGER NOT NULL,
undo_on_cancel INTEGER NOT NULL,
failed INTEGER NOT NULL DEFAULT 0,
cancelled INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX "__sqliteq_wf_workflows_created" ON "__sqliteq_wf_workflows" (created_at DESC, id);
CREATE INDEX "__sqliteq_wf_workflows_finished" ON "__sqliteq_wf_workflows" (finished_at) WHERE finished_at IS NOT NULL;
CREATE TABLE "__sqliteq_wf_tasks" (
workflow_id TEXT NOT NULL,
name TEXT NOT NULL,
ord INTEGER NOT NULL,
state TEXT NOT NULL,
reason TEXT NOT NULL DEFAULT '',
result BLOB,
open INTEGER NOT NULL DEFAULT 0,
deps TEXT NOT NULL,
always INTEGER NOT NULL,
queue_id TEXT NOT NULL,
job_id TEXT NOT NULL,
priority INTEGER NOT NULL DEFAULT 0,
item BLOB,
has_undo INTEGER NOT NULL DEFAULT 0,
undo_queue_id TEXT NOT NULL DEFAULT '',
undo_job_id TEXT NOT NULL DEFAULT '',
undo_item BLOB,
undo_state TEXT NOT NULL DEFAULT '',
undo_open INTEGER NOT NULL DEFAULT 0,
undo_reason TEXT NOT NULL DEFAULT '',
PRIMARY KEY (workflow_id, name)
) WITHOUT ROWID;
CREATE TABLE "__sqliteq_wf_outbox" (
seq INTEGER PRIMARY KEY AUTOINCREMENT,
workflow_id TEXT NOT NULL,
queue_id TEXT NOT NULL,
job_id TEXT NOT NULL,
item BLOB NOT NULL,
priority INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL
);
CREATE TABLE "__sqliteq_wf_purged" (
seq INTEGER PRIMARY KEY AUTOINCREMENT,
queue_id TEXT NOT NULL,
workflow_id TEXT NOT NULL,
task TEXT NOT NULL,
undo INTEGER NOT NULL,
created_at INTEGER NOT NULL
);
CREATE INDEX "__sqliteq_wf_outbox_workflow" ON "__sqliteq_wf_outbox" (workflow_id);
`

// writeV08Workflow builds the v0.8.0 workflow tables in the file at path,
// with running workflow "old" on queue queueID: a completed, b pushed, c
// waiting for b.
func writeV08Workflow(t *testing.T, path, queueID string) {
t.Helper()
db, err := sql.Open("sqlite3", path)
if err != nil {
t.Fatalf("Failed to open database: %v", err)
}
defer db.Close()

exec := func(query string, args ...any) {
t.Helper()
if _, err := db.Exec(query, args...); err != nil {
t.Fatalf("Fixture statement failed: %v\n%s", err, query)
}
}
exec(v08WorkflowTables)
exec(`INSERT INTO "__sqliteq_wf_workflows" (id, hash, state, created_at, fail_fast, undo_on_cancel) VALUES ('old', 'h', 'running', 1000, 0, 0)`)
for i, r := range []struct {
name, state, deps string
open int
item []byte
}{
{"a", "completed", `[]`, 0, nil},
{"b", "pending", `["a"]`, 0, nil},
{"c", "waiting", `["b"]`, 1, wfItem("old", "c", false)},
} {
exec(`INSERT INTO "__sqliteq_wf_tasks" (workflow_id, name, ord, state, open, deps, always, queue_id, job_id, item)
VALUES ('old', ?, ?, ?, ?, ?, 0, ?, ?, ?)`, r.name, i, r.state, r.open, r.deps, queueID, "wf:old:"+r.name, r.item)
}
}

// Workflow tables of contract v0.8.0 open: the task time columns are added
// once, old rows read 0, new pushes, starts and ends are stamped. A Purge
// with no store open migrates the tables too.
func TestMigrateWorkflowTaskTimes(t *testing.T) {
ctx := context.Background()
open := func(t *testing.T, single bool) (Queues, *Queue) {
t.Helper()
dir := t.TempDir()
root, wfPath, opts := dir, filepath.Join(dir, workflowsFile), []StoreOption(nil)
if single {
root = filepath.Join(dir, "all.db")
wfPath, opts = root, []StoreOption{WithSingleFile()}
}
qs := mustNew(t, root, opts...)
t.Cleanup(func() { qs.Close() })
a, err := qs.NewQueue("a")
if err != nil {
t.Fatal(err)
}
writeV08Workflow(t, wfPath, a.QueueID())
if !a.EnqueueOnce(wfItem("old", "b", false), "wf:old:b") {
t.Fatal("Failed to queue the old workflow's task")
}
return qs, a
}
store := func(t *testing.T, qs Queues) *WorkflowStore {
t.Helper()
st, err := NewWorkflowStore(qs)
if err != nil {
t.Fatalf("Failed to open the v0.8.0 workflow tables: %v", err)
}
t.Cleanup(func() { st.Close() })
for _, col := range taskTimeCols {
if ok, err := hasColumn(st.f.db, "__sqliteq_wf_tasks", col); err != nil || !ok {
t.Errorf("Expected column %s, got %t %v", col, ok, err)
}
}
return st
}

t.Run("open", func(t *testing.T) {
eachLayout(t, func(t *testing.T, single bool) {
qs, a := open(t, single)
st := store(t, qs)
r, err := st.Record(ctx, "old")
if err != nil {
t.Fatal(err)
}
if got := taskStates(r); got != "a=completed b=pending c=waiting" {
t.Errorf("got %s", got)
}
for _, tr := range r.Tasks {
if times(tr) != [3]int64{} {
t.Errorf("Expected no times on old task %s, got %v", tr.Name, times(tr))
}
}
counts, err := st.WorkflowCounts(ctx)
if err != nil || !maps.Equal(counts, map[contract.WorkflowState]int64{contract.WorkflowRunning: 1}) {
t.Errorf("counts %v %v", counts, err)
}
st.Close()

// Reopened: nothing to add; the old workflow goes on with times
st = store(t, qs)
if ok, err := st.Runnable(ctx, "old", "b", false); err != nil || !ok {
t.Fatalf("Runnable = %t %v", ok, err)
}
_, ack := take(t, a)
if _, ok := st.FinishTask(a.QueueID(), ack, done("old", "b", "")); !ok {
t.Fatal("FinishTask failed")
}
if r, err = st.Record(ctx, "old"); err != nil {
t.Fatal(err)
}
ta, tb, tc := r.Tasks[0], r.Tasks[1], r.Tasks[2]
if times(ta) != [3]int64{} || tb.EnqueuedAt != 0 || tb.StartedAt == 0 || tb.FinishedAt == 0 || tc.EnqueuedAt == 0 {
t.Errorf("Expected old times 0 and new ones set, got a %v b %v c %v", times(ta), times(tb), times(tc))
}
if got := taskStates(r); got != "a=completed b=completed c=pending" || a.Len() != 1 {
t.Errorf("got %s, a %d", got, a.Len())
}
})
})

t.Run("purge", func(t *testing.T) {
eachLayout(t, func(t *testing.T, single bool) {
qs, a := open(t, single)
a.Purge() // no store open: the on-demand one works on the old tables
r, err := store(t, qs).Record(ctx, "old")
if err != nil {
t.Fatal(err)
}
if got := taskStates(r); got != "a=completed b=cancelled c=cancelled" || r.State != contract.WorkflowFailed {
t.Errorf("got %s %s", got, r.State)
}
if r.Tasks[1].FinishedAt == 0 || r.Tasks[2].FinishedAt == 0 || r.Tasks[1].EnqueuedAt != 0 {
t.Errorf("Expected the purge's ends stamped, got b %v c %v", times(r.Tasks[1]), times(r.Tasks[2]))
}
})
})
}
8 changes: 7 additions & 1 deletion queue.go
Original file line number Diff line number Diff line change
Expand Up @@ -867,9 +867,15 @@ func (q *Queue) Values() []any {
// demand when the file has workflow tables. Under WithSingleFile the delete
// and the ends are one transaction. Folder layout: an intent is stored in
// the workflow file first, and the recovery loop applies one a crash left.
// Workflow tables there but no store ready (check, open or init failed):
// nothing is deleted, the cause is logged; a later Purge can retry.
func (q *Queue) Purge() {
ws, done := q.f.purgeStore()
ws, done, err := q.f.purgeStore()
defer done()
if err != nil {
log.Printf("sqliteq: purge of %s: jobs kept, workflow store not ready: %v", q.name, err)
return
}
if ws == nil {
q.purge(nil)
return
Expand Down
Loading
Loading