Docs · Features · Quick Start · Usage · How It Works · Guarantees · Comparison · Contributing
A typed worker pool and job queue for Go. Each worker is a concurrent pool that pulls jobs from any mix of in-memory queues, priority queues, shared storage (Redis, SQLite) and topics.
- High performance: lock-free appends and batched dispatch. See benchmarks.
- Queue types: standard and priority, in memory or in shared storage (Redis, SQLite) that survives restarts and serves many processes.
- Worker types:
NewWorker(no return value),NewErrWorker(error only),NewResultWorker(result and error). - Concurrency control: resize the pool at runtime, pause, resume, restart.
- Multi queue binding: one worker, many queues, a pluggable selection strategy.
- Topic fan-out: publish once, every subscribed worker gets its own job.
- Workflows: a DAG of jobs over queues with typed results,
Alwaysand sagaUndo. - Jobs: deduplicate by ID, cancel, or wait for a result, from any process on shared storage. Jobs.
- Queue pause: pause a queue for every worker in every process. Queues.
- Function helpers: submit plain
func()values, like pond or ants. Function helpers. - REST API and Prometheus metrics: inspect and control workers, queues, jobs and workflows over HTTP. REST API.
- Dashboard: vartop shows every process of a service in one place, with history, and acts on workers, queues, jobs and workflows.
- Extensible: plug in your own storage with
IQueueorIPriorityQueue. Adapters.
go get github.com/goptics/varmqpackage main
import (
"fmt"
"time"
"github.com/goptics/varmq"
)
func main() {
worker := varmq.NewWorker(func(j varmq.Job[int]) {
fmt.Printf("Processing %d\n", j.Data())
time.Sleep(1 * time.Second)
}) // default concurrency is 1
defer worker.WaitUntilIdle()
queue := worker.BindQueue()
queue.Add(1)
queue.Add(2)
}Runnable programs live in examples. Every topic below has a full page in the docs.
worker := varmq.NewWorker(func(j varmq.Job[int]) {
fmt.Printf("Processing %d\n", j.Data())
time.Sleep(500 * time.Millisecond)
}, 10) // concurrency 10, or 0 for one per CPU
defer worker.WaitUntilIdle()
queue := worker.BindQueue()
for i := range 100 {
queue.Add(i)
}Lower number = higher priority. A higher-priority job added later is dispatched first.
queue := worker.BindPriorityQueue()
for i := range 10 {
queue.Add(i, i%2) // even jobs first
}NewResultWorker returns a result and an error per job. NewErrWorker returns only the error.
worker := varmq.NewResultWorker(func(j varmq.Job[string]) (int, error) {
if j.Data() == "error" {
return 0, errors.New("error occurred")
}
return len(j.Data()), nil
})
defer worker.WaitUntilIdle()
queue := worker.BindQueue()
// Add returns at once; Result blocks until that job is done
if job, ok := queue.Add("The length of this string is 31"); ok {
result, err := job.Result()
fmt.Println("Result:", result, err) // Result: 31 <nil>
}Bind several queues to one worker. varmq.WithStrategy picks the next queue: Priority (default, bind order), RoundRobin, MaxLen or MinLen.
worker := varmq.NewWorker(func(j varmq.Job[string]) {
fmt.Println("Processing:", j.Data())
}, varmq.WithStrategy(varmq.RoundRobin))
defer worker.WaitUntilIdle()
q1 := worker.BindQueue()
q2 := worker.BindQueue()
pq := worker.BindPriorityQueue()With varmq.RoundRobin, empty queues are skipped:
Publish once and every subscribed worker gets its own job. A late subscriber sees only what is published after it subscribed.
topic := varmq.NewTopic[int]()
email.WithTopic(topic)
stock.WithTopic(topic)
for order := range 4 {
topic.Publish(order) // both workers get all 4 orders
}Same WithQueue / WithPriorityQueue. Jobs go out as JSON, survive restarts, and any process bound to the same queue runs them. Results come back to whichever process waits.
rq := redisq.New(url)
// consumer process
w := varmq.NewResultWorker(func(j varmq.Job[int]) (string, error) { ... })
w.WithQueue(rq.NewQueue("thumbs"))
// producer process: no worker
q := varmq.NewResultQueue[int, string](rq.NewQueue("thumbs"))
job, ok := q.Add(64)
if !ok {
return // storage refused the job
}
s, err := job.Result() // waits, whichever process ran itAdapters: ⚡ redisq · 🗃️ sqliteq · 🐘 PostgreSQL (upcoming). Check the versions table first.
Examples: SQLite, Redis, one process, Redis, producer and consumer. Docs: Persistence.
A DAG of jobs over queues, with typed results, Always() and saga Undo. Needs Go 1.27.
wf := varmq.NewWorkflow(varmq.NewWorkflowStore(), varmq.WithWorkflowID("upload-1"))
up := wf.Add("upload", uploadQ, "photo.png")
s := wf.Add("thumb-s", thumbQ, 64).After(up)
l := wf.Add("thumb-l", thumbQ, 512).After(up)
wf.Add("email", mailQ, "ann@example.com").
After(s, l).
Always().
Undo(unsendQ, "ann@example.com")
wf.Submit(ctx) // idempotent per ID
rec, _ := wf.WaitContext(ctx) // rec.State: completed, failed, compensated, ...Example: examples/workflow. Docs: Workflows.
mux := http.NewServeMux()
mux.Handle("/varmq/", varmq.Handler("/varmq"))Warning
The REST API has no authentication. It can read job payloads, stop workers and cancel jobs. Protect it before you expose it.
Routes and responses: REST API, spec/openapi.yaml.
A process can dial out to a vartop hub instead of serving the REST API on a port. It redials with backoff and returns when ctx ends.
go varmq.ConnectHub(ctx, "https://vartop.internal", token,
varmq.WithHubErrHandler(func(err error) { log.Printf("hub: %v", err) }),
)Warning
Whoever answers at the hub URL gets everything that Handler allows. Use TLS. WithHubReadOnly() and WithHubNoPayloads() limit what the hub can do and see.
Options, trust and sessions: Dashboard (vartop). Protocol: spec/hub.md.
A single event loop sleeps until something signals it: Add() putting a job in the queue, or a worker finishing one. On each signal it hands jobs to workers while the queue has work and a worker slot is free, then goes back to waiting.
The signal channel holds one pending signal, so extra signals from many Add calls are dropped. One wake-up is enough. Concurrency is a limit, not a fixed pool size: goroutines start while jobs wait and exit as the queue drains.
With many short jobs, the event loop becomes the bottleneck. So a worker can take a batch of jobs from an in-memory queue or topic and run them in order. Below, 8 cheap jobs with concurrency 2: the first worker takes 4, a second worker is started for 2, and the last 2 go out one at a time. Priority queues and multi-queue workers do not batch. Shared storage (redisq, sqliteq) that is the worker's only source instead pops one job per free slot in one call, up to 64, and starts each job at once.
Batch size rules and shared-storage dispatch: How it works.
| Where | Guarantee | Notes |
|---|---|---|
| In-memory queue | At most once | Jobs live in the process and are lost if it exits. |
| redisq, sqliteq | At least once | A job comes back if its consumer dies or its acknowledgment is lost. Make handlers safe to run twice. A custom adapter sets its own guarantee and can lose jobs. |
Add with WithJobId on redisq or sqliteq |
Stored once per ID | A retried Add with the same ID is stored once. The job can still run twice. |
| Topic | At most once, per subscription | Each subscribed worker gets every payload published after it subscribed. |
Cancel is cooperative: a running job stops only if its handler watches j.Context(). VarMQ does not retry a failed job. Details: Persistence.
VarMQ is compared with ants, pond v2, conc, workerpool, errgroup, gopool and two hand-rolled channel pools as a worker pool. It is also compared with Redis and SQLite job queues as shared storage. The numbers come from the benchmark repository.
| Feature | VarMQ | ants | pond | conc | workerpool | errgroup | gopool |
|---|---|---|---|---|---|---|---|
| Generic (type-safe) API | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| Tasks return results and errors | ✅ | ❌ | ✅ | ✅ | ❌ | ❌² | ❌ |
| Task groups (submit many, wait once) | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Priority queue | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Persistent queues (survive restarts) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Distributed queues (across instances) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Several queues bound to one pool | ✅ | ❌ | ✅⁴ | ❌ | ❌ | ❌ | ❌ |
| Topic fan-out (publish once, many pools) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Bounded queue, reject when full | ✅ | ✅ | ✅ | ❌ | ❌ | ❌³ | ❌ |
| Pause and resume | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Restart after stop | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| Resize pool at runtime | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
| Cancel work via context | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ |
| Panic recovery | ✅ | ✅ | ✅ | ❌¹ | ❌ | ❌ | ✅ |
| Task counters (completed, failed, ...) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| Built-in REST API | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
Checked against each package's source, October 2026. ¹ conc re-raises a task's panic from Wait by design. ² errgroup returns the first error, no results. ³ errgroup has no queue; TryGo only refuses a task while every slot is busy. ⁴ pond subpools: each has its own queue and runs on the parent pool's workers.
Run of 2026-10-25 on an i7-13700 (24 threads), Go 1.26, varmq v1.10.0. Every pool gets 50K workers and runs 15M tasks (sleep 10 ms, a short CPU loop, and a mix), sent by 1 to 1M producers. Time to run all tasks and memory per 1M tasks, lower is better; throughput, higher is better.
| Pool | 4 CPUs (s) | 8 CPUs (s) | 24 CPUs (s) | Throughput (M/s) | Memory (MB) |
|---|---|---|---|---|---|
| VarMQ | 5.07 | 3.54 | 3.85 | 4.89 | 115 |
| BufferedPool⁵ | 5.43 | 5.41 | 8.49 | 2.73 | 23 |
| errgroup | 6.38 | 5.69 | 9.23 | 2.29 | 111 |
| ants | 6.74 | 6.51 | 8.00 | 2.22 | 15 |
| gopool | 7.13 | 7.48 | 11.00 | 1.92 | 52 |
| GoroutinePool⁵ | 7.24 | 6.96 | 12.85 | 1.90 | 14 |
| conc | 8.45 | 8.61 | 17.38 | 1.56 | 15 |
| workerpool | 10.20 | 11.58 | 16.42 | 1.35 | 28 |
| pond v2 | 20.67 | 22.23 | 42.38 | 0.60 | 318 |
⁵ Hand-rolled: fixed workers reading a channel, unbuffered (GoroutinePool) or with 1M slots (BufferedPool). VarMQ uses more memory because Add never blocks, so it holds every waiting task as a job.
- Redis: VarMQ with redisq against Asynq, rmq, taskq and gocraft/work, 100 producers: up to 64K jobs/s and the lowest p50 latency (0.058 ms). gocraft/work leads with one instance and 10 workers, rmq with one producer.
↗️ Live chart - SQLite: VarMQ with sqliteq against River, goqite and backlite: 8.6K jobs/s mixed and 14.7K drain, against goqite's 3K.
↗️ Live chart
Every row, the groups VarMQ does not win, and micro-benchmarks: Benchmarks.
For the complete package documentation, types, and method signatures, browse the GoDoc.
Issues and pull requests are welcome. To set up and run the checks locally:
make init # install lefthook git hooks and tidy modules
make test # go test -race ./...
make bench # micro-benchmarksThe Redis examples need a local Redis: docker compose -f examples/compose.yml up -d. Commit messages follow Conventional Commits (feat:, fix:, docs:, ...).
This project has a Code of Conduct. By participating, you agree to its terms.