Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 35 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,51 +1,50 @@
# VarMQ Benchmarks
# Benchmarks

[![Live charts](https://img.shields.io/badge/%F0%9F%93%8A-Live_Charts-blue?style=for-the-badge)](https://benchmarks.goptics.org)

Benchmark suites for [VarMQ](https://github.com/goptics/varmq) against other Go worker pools and message queues. Raw `go test -bench` output is committed under `<suite>/bench/`; the charts at <https://benchmarks.goptics.org> are built from those files.
Benchmarks of Go worker pools and job queues: in-process worker pools, Redis-backed queues and SQLite-backed queues. Raw `go test -bench` output is committed under `<suite>/bench/`, and the charts at <https://benchmarks.goptics.org> are built from it.

## Suites

| Suite | Compares | Charts | Details |
|-------|----------|--------|---------|
| Worker pool | VarMQ vs Pond, Ants, WorkerPool, Conc, ErrGroup, Gopool and two channel baselines | [24](https://benchmarks.goptics.org/?id=pool-24cpu), [8](https://benchmarks.goptics.org/?id=pool-8cpu), [4 CPUs](https://benchmarks.goptics.org/?id=pool-4cpu) | [pool/README.md](pool/README.md) |
| Message queue, Redis | VarMQ (redisq) vs Asynq, rmq, taskq, gocraft/work, with one and four consumer processes | [redis](https://benchmarks.goptics.org/?id=redis) | [redis/README.md](redis/README.md) |
| Message queue, SQLite | VarMQ (sqliteq) vs goqite, backlite, River, at WAL checkpoint 1000 and 10000 pages | [sqlite](https://benchmarks.goptics.org/?id=sqlite) | [sqlite/README.md](sqlite/README.md) |

## Results

Pool run of 2026-10-24, Redis and SQLite runs of 2026-10-25, all `-benchtime=2x`. Full tables in the suite READMEs.

- Pool: VarMQ is the fastest subject in 40, 36 and 20 of the 55 rows at 24, 8 and 4 CPUs. It is slower than the buffered-channel baseline on the one-worker `Loop10` rows, except `1Mu-1t`, where the two are level.
- Redis: with the `Noop` task, VarMQ is first in 2 of the 6 one-process rows and VarMQDistributed in 2 of the 6 four-process rows, all at 100 producers; rmq leads every single-producer row. Pickup latency p50: taskq 0.03876 ms, VarMQDistributed 0.06492 ms, VarMQ 0.09407 ms, rmq 1.464 ms, Asynq 979.7 ms.
- SQLite: VarMQ (sqliteq) is first in all 8 scenario and checkpoint groups: on `Mixed1Q` at ckpt1000 it runs 3.1x goqite, 4.1x River and 23x backlite, and ten queues in separate files reach 107,459 jobs/s at ckpt10000.
| Suite | What runs | Libraries | Charts | Details |
|-------|-----------|-----------|--------|---------|
| Worker pool | 9 subjects (7 libraries plus two channel baselines); 5 workload shapes, 4 tasks, 4 pool sizes, 1M tasks per op; recorded at GOMAXPROCS 24, 8 and 4 | [ants](https://github.com/panjf2000/ants), [conc](https://github.com/sourcegraph/conc), errgroup ([x/sync](https://github.com/golang/sync)), gopool ([bytedance/gopkg](https://github.com/bytedance/gopkg)), [pond](https://github.com/alitto/pond), [VarMQ](https://github.com/goptics/varmq), [workerpool](https://github.com/gammazero/workerpool) | [pool-24cpu](https://benchmarks.goptics.org/?id=pool-24cpu), [pool-8cpu](https://benchmarks.goptics.org/?id=pool-8cpu), [pool-4cpu](https://benchmarks.goptics.org/?id=pool-4cpu) | [pool/README.md](pool/README.md) |
| Message queue, Redis | 6 subjects from 5 libraries; 100,000 jobs per op through 1 or 4 library instances in one process, plus enqueue-only and enqueue-to-handler latency | [Asynq](https://github.com/hibiken/asynq), [gocraft/work](https://github.com/gocraft/work), [rmq](https://github.com/adjust/rmq), [taskq](https://github.com/vmihailenco/taskq), [VarMQ](https://github.com/goptics/varmq) with [redisq](https://github.com/goptics/redisq) | [redis](https://benchmarks.goptics.org/?id=redis) | [redis/README.md](redis/README.md) |
| Message queue, SQLite | 5 subjects from 4 libraries; 4 scenarios at WAL checkpoint 1000 and 10000 pages | [backlite](https://github.com/mikestefanello/backlite), [goqite](https://github.com/maragudk/goqite), [River](https://github.com/riverqueue/river), [VarMQ](https://github.com/goptics/varmq) with [sqliteq](https://github.com/goptics/sqliteq) | [sqlite](https://benchmarks.goptics.org/?id=sqlite) | [sqlite/README.md](sqlite/README.md) |

## Environment

Intel Core i7-13700 (24 threads), linux/amd64, Go 1.26.2. Pool suite at GOMAXPROCS 24, 8 and 4. Redis 7.2 (Debian image) pinned to CPU 14, client on cores 0-13 and 16-23. SQLite databases on ext4 over NVMe.

| Library | Version |
|---------|---------|
| varmq | 415ad73 (unreleased, `replace` in each go.mod) |
| redisq | v1.5.0 |
| sqliteq | v0.3.0 |
| pond | v2.7.2 |
| ants | v2.12.1 |
| workerpool | v1.2.2 |
| conc | v0.3.0 |
| x/sync (errgroup) | v0.24.0 |
| bytedance/gopkg (gopool) | v0.1.4 |
| asynq | v0.26.0 |
| rmq | v5.2.0 |
| taskq | v3.2.9 |
| gocraft/work | v0.5.1 |
| goqite | v0.4.0 |
| backlite | v0.8.0 |
| river | v0.49.0 |
Intel Core i7-13700 (24 threads), linux/amd64, Go 1.26, SQLite databases on ext4 over NVMe.

The pool suite sets GOMAXPROCS to 24, 8 and 4 with no core pinning. The SQLite suite runs on all 24 threads. Redis 7.2 (Debian image) is pinned to CPU 14 and the Redis suite's client to CPUs 0-13 and 16-23 (22 threads), so Redis has a whole physical core.

| Library | Version | Suite |
|---------|---------|-------|
| [varmq](https://github.com/goptics/varmq) | v1.10.0 | pool, redis, sqlite |
| [ants](https://github.com/panjf2000/ants) | v2.12.1 | pool |
| [workerpool](https://github.com/gammazero/workerpool) | v1.2.2 | pool |
| [conc](https://github.com/sourcegraph/conc) | v0.3.0 | pool |
| [x/sync](https://github.com/golang/sync) (errgroup) | v0.24.0 | pool |
| [bytedance/gopkg](https://github.com/bytedance/gopkg) (gopool) | v0.1.4 | pool |
| [pond](https://github.com/alitto/pond) | v2.7.2 | pool |
| [redisq](https://github.com/goptics/redisq) | v1.5.0 | redis |
| [Asynq](https://github.com/hibiken/asynq) | v0.26.0 | redis |
| [rmq](https://github.com/adjust/rmq) | v5.2.0 | redis |
| [taskq](https://github.com/vmihailenco/taskq) | v3.2.9 | redis |
| [gocraft/work](https://github.com/gocraft/work) | v0.5.1 | redis |
| [go-redis](https://github.com/redis/go-redis) | v9.23.0 (taskq uses v8.11.5) | redis |
| [redigo](https://github.com/gomodule/redigo) | v1.9.3 | redis |
| [sqliteq](https://github.com/goptics/sqliteq) | v0.3.0 | sqlite |
| [goqite](https://github.com/maragudk/goqite) | v0.4.0 | sqlite |
| [backlite](https://github.com/mikestefanello/backlite) | v0.8.0 | sqlite |
| [River](https://github.com/riverqueue/river) with its [SQLite driver](https://github.com/riverqueue/river/tree/master/riverdriver/riversqlite) | v0.49.0 | sqlite |
| [go-sqlite3](https://github.com/mattn/go-sqlite3) | v1.14.28 | sqlite |

## Running

Needs Go 1.26+ and [Task](https://taskfile.dev); Docker for the Redis suite; a C compiler for the SQLite suite; [vizb](https://github.com/goptics/vizb) and `jq` for `task site:preview`.
Needs Go 1.26+, [Task](https://taskfile.dev), Docker for the Redis suite, a C compiler for the SQLite suite, and [vizb](https://github.com/goptics/vizb) and `jq` for `task site:preview`.

varmq v1.10.0 is not tagged yet; each go.mod builds it from a checkout of [goptics/varmq](https://github.com/goptics/varmq) at `../../varmq` through a `replace` directive.

```bash
task --list # every task
Expand Down
8 changes: 4 additions & 4 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ vars:
DATA: worker/public/data
DATA_URL: https://benchmarks.goptics.org/dataset
BAR: vizb bar -p /n/y/x -T s -M MB --round -l --border-radius 8,8,0,0
# C1, Enqueue and Latency groups race VarMQ's persistent queue against these;
# VarMQDistributed only appears in the C4 groups. Keep in step with redis/subjects.go.
# Subjects shown in the C1, Enqueue and Latency groups; VarMQDistributed only
# appears in the C4 groups. Keep in step with redis/subjects.go.
SUBJECTS: '(VarMQ|Asynq|Rmq|Taskq|GocraftWork)'
# catalog order = dashboard order; listed literally (a glob sorts 4cpu before 24cpu)
JSONS: >-
Expand All @@ -32,13 +32,13 @@ tasks:
- '{{.BAR}} --stack pool/bench/pool-4cpu.txt -n "Worker Pool (4 CPUs)" --id pool-4cpu -d "Same matrix at GOMAXPROCS 4" -o {{.DATA}}/pool-4cpu.json'
- >-
{{.BAR}} --stack redis/bench/redis.txt -n "Message Queue Redis" --id redis
-d "VarMQ (redisq) vs Asynq, rmq, taskq, gocraft/work on Redis 7.2. C1 groups: one consumer process; C4: four. Plus enqueue-only and pickup latency"
-d "Asynq, gocraft/work, rmq, taskq and VarMQ (redisq) on Redis 7.2. C1 groups: one library instance; C4: four instances in the same process. Plus enqueue-only and enqueue-to-handler latency"
-f '^(Benchmark)?((Durable/[^/]+C1|Enqueue/Enqueue|Latency/[^/]+)/[^/]+/{{.SUBJECTS}}$|Durable/[^/]+C4/)'
--better p50-ms:low,p99-ms:low,max-ms:low
-o {{.DATA}}/redis.json
- >-
{{.BAR}} --stack sqlite/bench/sqlite.txt -n "Message Queue SQLite" --id sqlite
-d "VarMQ (sqliteq) vs goqite, backlite, River on SQLite: one queue, drain, priority, ten queues; WAL checkpoint 1000 vs 10000 pages"
-d "backlite, goqite, River and VarMQ (sqliteq) on SQLite: one queue, drain, priority, ten queues; WAL checkpoint 1000 vs 10000 pages"
--better walMB:low -o {{.DATA}}/sqlite.json
- jq -se 'all(.data | length > 0)' {{.JSONS}}
- jq -s 'map({id, name})' {{.JSONS}} > {{.DATA}}/_catalog.json
Expand Down
79 changes: 44 additions & 35 deletions pool/README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,35 @@
# Worker pool benchmarks

Nine in-process Go worker pools driven by one harness, 1M tasks per op, recorded at 24, 8 and 4 CPUs: [pool-24cpu](https://benchmarks.goptics.org/?id=pool-24cpu), [pool-8cpu](https://benchmarks.goptics.org/?id=pool-8cpu), [pool-4cpu](https://benchmarks.goptics.org/?id=pool-4cpu).
Nine Go worker pools run by one harness, 1M tasks per op: [ants](https://github.com/panjf2000/ants), [bytedance/gopkg](https://github.com/bytedance/gopkg) gopool, [conc](https://github.com/sourcegraph/conc), [pond](https://github.com/alitto/pond), [varmq](https://github.com/goptics/varmq), [workerpool](https://github.com/gammazero/workerpool), [x/sync](https://github.com/golang/sync) errgroup and two channel baselines.
Charts at 24, 8 and 4 CPUs: [pool-24cpu](https://benchmarks.goptics.org/?id=pool-24cpu), [pool-8cpu](https://benchmarks.goptics.org/?id=pool-8cpu), [pool-4cpu](https://benchmarks.goptics.org/?id=pool-4cpu).

## Results

Run of 2026-10-24, varmq 415ad73 (unreleased, `replace` in go.mod), `-benchtime=2x`, 495 rows per file. Each file has 55 task/size/workload rows (11 groups like `Loop10W50K` x 5 workloads); the table counts the rows in which each subject has the lowest `ns/op`, and no other subject is fastest in any row.
Run of 2026-10-24, varmq v1.10.0, `-benchtime=2x`, 495 rows per file. Each file has 55 task/size/workload rows (11 groups such as `Loop10W50K` x 5 workloads), each with 9 subjects. The table counts the rows in which each subject has the lowest `ns/op`; no other subject is fastest in any row.

| CPUs | Rows | VarMQ | BufferedPool | ErrGroup | Ants |
|------|------|-------|--------------|----------|------|
| 24 | 55 | 40 | 13 | 2 | 0 |
| 8 | 55 | 36 | 13 | 6 | 0 |
| 4 | 55 | 20 | 20 | 11 | 4 |
| CPUs | Rows | VarMQ | Ants | ErrGroup | BufferedPool |
|------|------|-------|------|----------|--------------|
| 24 | 55 | 40 | 0 | 2 | 13 |
| 8 | 55 | 36 | 0 | 6 | 13 |
| 4 | 55 | 20 | 4 | 11 | 20 |

On the five one-worker `Loop10W1` rows of each file, VarMQ's `ns/op` divided by BufferedPool's is 3.26x to 4.22x on the four workloads other than `1Mu-1t` (4.22x on `1u-1Mt` at 24 CPUs) and level on `1Mu-1t` (1.04x at 24 CPUs, 0.96x at 8, 1.08x at 4).
On the 15 one-worker `Loop10W1` rows (5 workloads x 3 files), BufferedPool is fastest in 14. On the four workloads other than `1Mu-1t`, the next subject (Gopool on `1u-1Mt`, VarMQ on the other three) takes 2.02x to 3.81x BufferedPool's `ns/op`. On `1Mu-1t`, BufferedPool and VarMQ are the two fastest, and the slower of them takes 1.04x, 1.04x and 1.08x the faster one's `ns/op` at 24, 8 and 4 CPUs. BufferedPool is fastest at 24 and 4 CPUs, VarMQ at 8.

## Subjects

| Name | Version | What it is |
|------|---------|------------|
| VarMQ | 415ad73 (unreleased, `replace` in go.mod) | [goptics/varmq](https://github.com/goptics/varmq), worker bound to a queue |
| Pond | v2.7.2 | [alitto/pond/v2](https://github.com/alitto/pond) |
| Ants | v2.12.1 | [panjf2000/ants/v2](https://github.com/panjf2000/ants) |
| WorkerPool | v1.2.2 | [gammazero/workerpool](https://github.com/gammazero/workerpool) |
| Conc | v0.3.0 | [sourcegraph/conc](https://github.com/sourcegraph/conc) `pool` with a goroutine cap |
| ErrGroup | v0.24.0 | [x/sync/errgroup](https://pkg.go.dev/golang.org/x/sync/errgroup): one goroutine per task behind a semaphore, not a pool |
| Gopool | v0.1.4 | [bytedance/gopkg](https://github.com/bytedance/gopkg) `util/gopool`: submit never blocks; soft worker cap |
| GoroutinePool | baseline | N goroutines reading from an unbuffered channel |
| BufferedPool | baseline | Same, but the channel holds 1M tasks |
| Name | Version | What it is | Submit when the pool is full | Teardown |
|------|---------|------------|------------------------------|----------|
| VarMQ | v1.10.0 | [varmq](https://github.com/goptics/varmq) worker bound to an in-memory queue (`BindQueue`) | Never blocks: `q.Add` appends to an unbounded queue and signals the worker with a non-blocking send | `WaitAndStop`: waits until no job is queued or running, then signals its workers to stop and returns without waiting for them to exit |
| Ants | v2.12.1 | [ants](https://github.com/panjf2000/ants) `NewPool` with default options | `Submit` blocks until a worker is free | `Release`, returns without waiting for its workers to exit |
| WorkerPool | v1.2.2 | [workerpool](https://github.com/gammazero/workerpool) | Never blocks: `Submit` enqueues the task | `StopWait`, waits for its workers |
| Conc | v0.3.0 | [conc](https://github.com/sourcegraph/conc) `pool` with a goroutine cap | `p.Go` blocks at the cap | `Wait`, waits for its goroutines |
| ErrGroup | v0.24.0 | [x/sync](https://github.com/golang/sync) `errgroup` with `SetLimit`: one goroutine per task, not a pool | `g.Go` blocks at the limit | `Wait`, waits for its goroutines |
| Gopool | v0.1.4 | [bytedance/gopkg](https://github.com/bytedance/gopkg) `util/gopool`, soft worker cap | Never blocks: unbounded task list | None; idle workers exit on their own |
| GoroutinePool | baseline | N goroutines reading one unbuffered channel | Blocks: the send waits for a worker to receive | Closes the channel, waits for all N workers to exit |
| BufferedPool | baseline | Same, with a channel that holds 1M tasks | Never blocks: the channel holds all 1M | Closes the channel, waits for all N workers to exit |
| Pond | v2.7.2 | [pond](https://github.com/alitto/pond) v2 `NewPool` | Never blocks: `DefaultQueueSize` is `Unbounded` | `StopAndWait`, waits for its workers |

Ants, Gopool and VarMQ do not wait for their workers to exit in teardown, so they skip that cost inside the timed region.

All subjects live in [subjects.go](subjects.go).

Expand All @@ -40,12 +43,16 @@ All subjects live in [subjects.go](subjects.go).
| `10Ku-100t` | 10,000 | 100 |
| `1Mu-1t` | 1,000,000 | 1 |

| Task | Behaviour |
|------|-----------|
| `Sleep10ms` | I/O-like: sleep 10 ms |
| `Loop10` | CPU-like: 10 iterations of trivial math |
| `Sleep5msLoop5` | Mixed: 5 iterations then sleep 5 ms |
| `Loop10K` | CPU-like: 10,000 iterations, about 1 µs. Pool of 1 only. |
A producer is a goroutine. The harness spawns the producers one after another in a loop, with no start barrier. Each producer submits its share of tasks and exits. Producers are not joined; the run waits for task completions. `1Mu-1t` therefore spawns one million goroutines inside the timed region, each submitting one task.

| Task | What it runs |
|------|--------------|
| `Sleep10ms` | `time.Sleep(10ms)`: a timer park, not I/O |
| `Loop10` | A counted loop of 10 iterations |
| `Sleep5msLoop5` | A 5-iteration loop, then a 5 ms sleep |
| `Loop10K` | A counted loop of 10,000 iterations; pool of one only |

The loop body is `_ = i * i`, so the compiler removes the multiplication and the CPU tasks are empty counted loops. `Loop10K` takes about 1 µs on this machine: BufferedPool's `Loop10KW1` rows come to 1.02 to 1.18 µs per task.

| Size | Tasks |
|------|-------|
Expand All @@ -54,27 +61,29 @@ All subjects live in [subjects.go](subjects.go).
| 300K workers | `Sleep10ms`, `Loop10`, `Sleep5msLoop5` |
| 1 worker | `Loop10`, `Loop10K` |

W1 runs CPU tasks only, since one worker sleeping 10 ms per task would take about three hours for 1M tasks.
GoroutinePool and BufferedPool start all N workers when the pool is built, so 300K goroutines start inside the timed region at `W300K`. BufferedPool's channel buffer holds all 1M tasks. W1 runs CPU tasks only, since one worker sleeping 10 ms per task would need about 2.8 hours.

The matrix is recorded at 24, 8 and 4 CPUs (`-cpu`), one dataset each.
The matrix is recorded with `-cpu` 24, 8 and 4, which sets GOMAXPROCS without pinning cores; rows carry a `-24`, `-8` or `-4` suffix.

## Method

A subject is a factory that, given `maxWorkers`, returns `submit` and `teardown`. Producers are goroutines: a `100u-10Kt` run starts 100 goroutines, each submitting 10,000 tasks. One op is build the pool, submit 1M tasks, wait for all to finish, tear down, with construction and shutdown inside the timed region.
The timed region of one op covers building the pool, spawning the producers, every submit, the wait until all 1M tasks have completed, and teardown. Completions are counted by the harness's own WaitGroup around each task, not by any pool signal.

- `ns/op`: wall time of that whole region
- `ops/s`: the same measurement as a rate, 1M x `b.N` / elapsed
- `B/op` and `allocs/op`: allocations of the whole process during the op, including any per-task closure an adapter allocates

- `ns/op`: wall time to push all 1M tasks through the pool and drain it
- `ops/s`: tasks completed per second (custom metric, higher is better)
- `B/op` and `allocs/op`: memory cost
`-benchtime=2x` means two ops per row, with no warm-up op.

## Running

| Task | What it does |
|------|--------------|
| `task pool:test` | `go vet` and `go test` |
| `task pool:bench` | Every benchmark once (`-benchtime=1x`) |
| `task pool:txt` | Full matrix at 24, 8 and 4 CPUs into `bench/pool-24cpu.txt`, `bench/pool-8cpu.txt` and `bench/pool-4cpu.txt` (hours) |
| `task pool:test` | `go vet ./...` and `go test ./...` |
| `task pool:bench` | Every benchmark once: `-benchtime=1x -benchmem -timeout 0` |
| `task pool:txt` | Full matrix with `-benchtime=2x -benchmem -timeout 0` at `-cpu` 24, 8 and 4 into `bench/pool-24cpu.txt`, `bench/pool-8cpu.txt` and `bench/pool-4cpu.txt`; the committed files took 943.059 s, 724.893 s and 711.582 s |

Rows are named `BenchmarkTasks/<Task>W<size>/<workload>/<pool>`, with `W` for workers and sizes in thousands ending in `K`: `BenchmarkTasks/Loop10W50K/100u-10Kt/VarMQ` is the `Loop10` task on 50,000 workers, 100 producers x 10,000 tasks each, subject VarMQ. One slice, the pool of one worker:
Rows are named `BenchmarkTasks/<Task>W<size>/<workload>/<pool>-<cpus>`, with `W` for workers and sizes in thousands ending in `K`. `BenchmarkTasks/Loop10W50K/100u-10Kt/VarMQ-24` is the `Loop10` task on 50,000 workers, 100 producers x 10,000 tasks each, subject VarMQ, at `-cpu 24`. One slice, the pool of one worker:

```bash
go test -run xxx -bench 'Tasks/W1$' -benchtime=1x
Expand Down
2 changes: 1 addition & 1 deletion pool/Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ tasks:
bench:
desc: Every benchmark once
cmds:
- go test -run xxx -bench . -benchtime=1x -benchmem
- go test -run xxx -bench . -benchtime=1x -benchmem -timeout 0
txt:
desc: Full matrix at 24, 8 and 4 CPUs into bench/ (hours)
cmds:
Expand Down
4 changes: 1 addition & 3 deletions pool/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ require (
github.com/alitto/pond/v2 v2.7.2
github.com/bytedance/gopkg v0.1.4
github.com/gammazero/workerpool v1.2.2
github.com/goptics/varmq v1.9.0
github.com/goptics/varmq v1.10.0
github.com/panjf2000/ants/v2 v2.12.1
github.com/sourcegraph/conc v0.3.0
golang.org/x/sync v0.24.0
Expand All @@ -18,6 +18,4 @@ require (
go.uber.org/multierr v1.9.0 // indirect
)

// replace github.com/goptics/varmq => ../varmq

replace github.com/goptics/varmq => ../../varmq
Loading
Loading