Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/site.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: site
on:
push:
branches: [main]
paths: ["*/bench/*.txt", "Taskfile.yml", "worker/**", ".github/workflows/site.yml"]
pull_request:
paths: ["*/bench/*.txt", "Taskfile.yml", "worker/**", ".github/workflows/site.yml"]
workflow_dispatch:

concurrency:
group: site
cancel-in-progress: false

jobs:
site:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- if: hashFiles('pool/bench/*.txt', 'redis/bench/*.txt', 'sqlite/bench/*.txt') == ''
run: echo "::notice::no bench txt on this ref yet, nothing to build"
- if: hashFiles('pool/bench/*.txt', 'redis/bench/*.txt', 'sqlite/bench/*.txt') != ''
run: |
sh -c "$(curl -fsSL https://taskfile.dev/install.sh)" -- -d -b "$HOME/.local/bin" v3.50.0
VIZB_RELEASE_TAG=v0.23.0 bash -c "$(curl -fsSL https://vizb.goptics.org/install.sh)"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- if: hashFiles('pool/bench/*.txt', 'redis/bench/*.txt', 'sqlite/bench/*.txt') != ''
run: task site:build
- if: hashFiles('pool/bench/*.txt', 'redis/bench/*.txt', 'sqlite/bench/*.txt') != '' && github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
uses: cloudflare/wrangler-action@v3
with:
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
workingDirectory: worker
wranglerVersion: "4"
command: deploy
22 changes: 22 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, built with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Build directory
bin/
dist/
worker/public/
worker/.wrangler/
worker/node_modules/

# Profile files
*.prof
103 changes: 103 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
version: "3"

includes:
pool: { taskfile: pool/Taskfile.yml, dir: pool }
redis: { taskfile: redis/Taskfile.yml, dir: redis }
sqlite: { taskfile: sqlite/Taskfile.yml, dir: sqlite }

vars:
PUBLIC: worker/public
DATA: worker/public/data
DATA_URL: https://benchmarks.goptics.org/dataset
BAR: vizb bar -p /n/y/x -T s -M MB --round -l --border-radius 8,8,0,0
# keep in step with redis/subjects.go: a subject missing here is dropped from redis-persistent
SUBJECTS: '(VarMQ|Asynq|Rmq|Taskq|GocraftWork)'
# catalog order = dashboard order; listed literally (a glob sorts 4cpu before 24cpu)
JSONS: >-
worker/public/data/pool-24cpu.json worker/public/data/pool-8cpu.json worker/public/data/pool-4cpu.json
worker/public/data/redis-persistent.json worker/public/data/redis-distributed.json
worker/public/data/sqlite-persistent.json

tasks:
default:
cmds:
- task --list

site:json:
desc: Convert every bench txt to dataset JSON plus the catalog
cmds:
- rm -rf {{.DATA}} && mkdir -p {{.DATA}}
- '{{.BAR}} --stack pool/bench/pool-24cpu.txt -n "Pool (24 CPUs)" --id pool-24cpu -d "9 Go worker pools, 5 workload shapes, 1M tasks, GOMAXPROCS 24" -o {{.DATA}}/pool-24cpu.json'
- '{{.BAR}} --stack pool/bench/pool-8cpu.txt -n "Pool (8 CPUs)" --id pool-8cpu -d "Same matrix at GOMAXPROCS 8" -o {{.DATA}}/pool-8cpu.json'
- '{{.BAR}} --stack pool/bench/pool-4cpu.txt -n "Pool (4 CPUs)" --id pool-4cpu -d "Same matrix at GOMAXPROCS 4" -o {{.DATA}}/pool-4cpu.json'
- >-
{{.BAR}} redis/bench/redis.txt -n "Redis (Persistent)" --id redis-persistent
-d "One consumer process: VarMQ persistent queue vs Asynq, rmq, taskq, gocraft/work; plus enqueue-only and pickup latency"
-f '^(Benchmark)?(Durable/[^/]+C1|Enqueue/Enqueue|Latency/[^/]+)/[^/]+/{{.SUBJECTS}}$'
--better p50-ms:low,p99-ms:low,max-ms:low
-o {{.DATA}}/redis-persistent.json
- >-
{{.BAR}} redis/bench/redis.txt -n "Redis (Distributed)" --id redis-distributed
-d "Four consumer processes: VarMQ distributed queue vs Asynq, rmq, taskq, gocraft/work"
-f '^(Benchmark)?Durable/[^/]+C4/'
-o {{.DATA}}/redis-distributed.json
- >-
{{.BAR}} sqlite/bench/sqlite.txt -n "Sqlite (Persistent)" --id sqlite-persistent
-d "sqliteq through a VarMQ worker: one queue, drain, priority, ten queues; WAL checkpoint 1000 vs 10000 pages"
--better walMB:low -o {{.DATA}}/sqlite-persistent.json
- jq -se 'all(.data | length > 0)' {{.JSONS}}
- jq -s 'map({id, name})' {{.JSONS}} > {{.DATA}}/_catalog.json

site:html:
desc: Build the lazy-catalog page against DATA_URL
cmds:
- mkdir -p {{.PUBLIC}}
- vizb ui -c bar --data-url {{.DATA_URL}} -o {{.PUBLIC}}/index.html

site:build:
desc: json + html, what CI runs before wrangler deploy
cmds:
- task: site:json
- task: site:html
vars: { DATA_URL: '{{.DATA_URL}}' }

site:preview:
desc: Same six datasets embedded in one HTML, no Worker
cmds:
- task: site:json
- mkdir -p dist
- jq -s . {{.JSONS}} > dist/all.json
- vizb ui -c bar dist/all.json -o dist/index.html
- cmd: xdg-open dist/index.html
platforms: [linux]
- cmd: open dist/index.html
platforms: [darwin]

site:dev:
desc: Build against localhost:8787 and run wrangler dev
cmds:
- task: site:build
vars: { DATA_URL: http://localhost:8787/dataset }
- cd worker && npx -y wrangler@4 dev

site:deploy:
desc: Production build then deploy (CI does the same)
cmds:
- task: site:build
- cd worker && npx -y wrangler@4 deploy

report:
desc: Run every suite (hours), then open the local preview
cmds:
- task: redis:up
- task: pool:txt
- task: redis:txt
- task: redis:down
- task: sqlite:txt
- task: site:preview

clean:
desc: Remove dist, the built site and local build artifacts
cmds:
- rm -rf dist {{.PUBLIC}} worker/.wrangler
- find . -name '*.prof' -o -name '*.test' | xargs rm -f
161 changes: 161 additions & 0 deletions pool/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# Pool benchmarks

Nine Go worker pools driven by one harness across five workload shapes and four task types, 1M tasks per run. VarMQ is raced against Pond, Ants, WorkerPool, Conc, ErrGroup, Gopool and two hand-rolled channel baselines.

This module is `package main` with its own `go.mod`. Run everything from this directory, or from the repo root as `task pool:<name>`.

## Run

| Task | What it does |
|------|--------------|
| `task pool:test` | `go vet` and `go test` |
| `task pool:bench` | Every benchmark once (`-benchtime=1x`) |
| `task pool:txt` | Full matrix at 24, 8 and 4 CPUs into `bench/pool-24cpu.txt`, `bench/pool-8cpu.txt` and `bench/pool-4cpu.txt` (hours) |

One slice, for example the pool of one worker (about 40 s):

```bash
go test -run xxx -bench 'Tasks/W1$' -benchtime=1x
```

Loop10 task, 50K workers, 100 producers, only the baselines:

```bash
go test -run xxx -bench 'Tasks/Loop10W50K/100u-10Kt/(GoroutinePool|BufferedPool)$' -benchtime=1x
```

One pool size, here 100K workers (also `W50K`, `W300K`):

```bash
go test -run xxx -bench 'Tasks/W100K$' -benchtime=1x
```

## Results

`bench/pool-24cpu.txt`, `pool-8cpu.txt` and `pool-4cpu.txt` are one run from
2026-10-24: varmq 415ad73 (unreleased, through the `replace` directive in
`go.mod`), `-benchtime=2x`, 495 rows each. Counting the fastest pool per row
over the 55 rows of each file (11 task and size groups x 5 workloads): VarMQ is
first in 40 rows at 24 CPUs, 36 at 8 and 20 at 4; BufferedPool takes 13, 13
and 20; ErrGroup 2, 6 and 11; Ants 4 at 4 CPUs. The one-worker `Loop10W1`
rows are the consistent loss, where VarMQ runs at about a quarter of
BufferedPool: with no parallelism the hand-off cost is all there is.

## Subjects

| Name | What it is |
|------|------------|
| **VarMQ** | [goptics/varmq](https://github.com/goptics/varmq), worker bound to a queue |
| **Pond** | [alitto/pond](https://github.com/alitto/pond) v2 |
| **Ants** | [panjf2000/ants](https://github.com/panjf2000/ants) v2 |
| **WorkerPool** | [gammazero/workerpool](https://github.com/gammazero/workerpool) |
| **Conc** | [sourcegraph/conc](https://github.com/sourcegraph/conc) `pool` with a goroutine cap |
| **ErrGroup** | [x/sync/errgroup](https://pkg.go.dev/golang.org/x/sync/errgroup) with `SetLimit`. Not a pool: a new goroutine per task behind a semaphore, the stdlib-adjacent default many projects reach for |
| **Gopool** | [bytedance/gopkg](https://github.com/bytedance/gopkg) `util/gopool`. Tasks go to an unbounded list, so `submit` never blocks; idle workers exit on their own. The worker cap is soft and can overshoot slightly under contention |
| **GoroutinePool** | N goroutines reading from an **unbuffered** channel. Baseline. |
| **BufferedPool** | Same, but the channel holds 1M tasks. Baseline. |

All subjects live in [subjects.go](subjects.go).

## Dimensions

Every subject is run across the full product of these axes, except that a pool of one worker runs only the CPU tasks (see **Pool size**):

**Workload shape** (producers x tasks per producer, always 1M tasks total)

| Name | Producers | Tasks each |
|------|-----------|------------|
| `1u-1Mt` | 1 | 1,000,000 |
| `100u-10Kt` | 100 | 10,000 |
| `1Ku-1Kt` | 1,000 | 1,000 |
| `10Ku-100t` | 10,000 | 100 |
| `1Mu-1t` | 1,000,000 | 1 |

**Task type**

| Name | Behaviour |
|------|-----------|
| `Sleep10ms` | I/O-like: sleep 10 ms |
| `Loop10` | CPU-like: 10 iterations of trivial math |
| `Sleep5msLoop5` | Mixed: 5 iterations then sleep 5 ms |
| `Loop10K` | CPU-like: 10,000 iterations, about 1 µs. Pool of 1 only. |

**Pool size**

| Size | Tasks | What it shows |
|------|-------|---------------|
| 50K workers | `Sleep10ms`, `Loop10`, `Sleep5msLoop5` | How the pool scales when work can run in parallel |
| 100K workers | `Sleep10ms`, `Loop10`, `Sleep5msLoop5` | The same, with twice the goroutines to spawn and schedule |
| 300K workers | `Sleep10ms`, `Loop10`, `Sleep5msLoop5` | The same, at six times the 50K size |
| 1 worker | `Loop10`, `Loop10K` | What each pool costs per task: submit, hand-off and wake-up, with no parallelism to hide it |

A pool of one runs no sleeping tasks: every pool would show the same `time.Sleep`, and one worker sleeping 10 ms per task needs almost three hours for 1M tasks. Sizes and their tasks live in `poolSizes` in [subjects.go](subjects.go).

**CPU count**: results are recorded for the machine default (24 cores on the reference machine), 8 cores and 4 cores (`-cpu` flag).

## Benchmark names

Every row is named `BenchmarkTasks/<Task>W<size>/<workload>/<pool>`, where `W` stands for workers and sizes in thousands end in `K` (`W50K`, `W100K`, `W300K`, `W1`), in the same style as the workload names:

| Row | Task | Pool size | Workload | Pool |
|-----|------|-----------|----------|------|
| `BenchmarkTasks/Loop10W50K/100u-10Kt/VarMQ` | `Loop10` | 50,000 workers | 100 producers x 10K tasks | VarMQ |
| `BenchmarkTasks/Sleep10msW300K/1Mu-1t/Pond` | `Sleep10ms` | 300,000 workers | 1M producers x 1 task | Pond |
| `BenchmarkTasks/Loop10KW1/1u-1Mt/Ants` | `Loop10K` | 1 worker | 1 producer x 1M tasks | Ants |

## Metrics

Each benchmark reports the standard Go numbers plus throughput:

- `ns/op`: wall time to push all 1M tasks through the pool and drain it
- `ops/s`: tasks completed per second (custom metric, higher is better)
- `B/op` and `allocs/op`: memory cost

On a `W1` row there is no parallelism, so `ns/op` divided by 1,000,000 is the time per task: the task itself plus everything the pool adds to it.

## How it works

```
subjects.go types.go benchmark_test.go
┌──────────────┐ ┌──────────────┐ ┌──────────────────────────────┐
│ VarMQ │ │ poolSubmit │ │ for each pool size │
│ Pond │ ──▶ │ poolTeardown │ ──▶│ for each of its task types │
│ Ants │ │ workload │ │ for each workload │
│ WorkerPool │ │ TaskConfig │ │ for each subject │
│ Conc │ │ │ │ │
│ ErrGroup │ └──────────────┘ │ spawn producers │
│ Gopool │ │ submit, wait, teardown│
│ GoroutinePool│ │ │
│ BufferedPool │ │ │
└──────────────┘ └──────────────────────────────┘
```

1. **A subject is a factory.** Given `maxWorkers`, it returns `submit(task)` and `teardown()`. Nothing else leaks into the harness.
2. **Producers are goroutines.** For a `100u-10Kt` run the harness starts 100 goroutines, each calling `submit` 10,000 times. A `WaitGroup` counts task completions.
3. **One iteration = full lifecycle.** Build the pool, push 1M tasks, wait for all to finish, tear the pool down. Pool construction and shutdown are deliberately inside the timed region.
4. **Results go to text, text goes to charts.** `task pool:txt` saves `go test -bench` output to `bench/pool-24cpu.txt`, `bench/pool-8cpu.txt` and `bench/pool-4cpu.txt`. The root `task site:build` turns each file into a dataset with [Vizb](https://github.com/goptics/vizb).

## Adding a pool

Append an entry to `subjects` in [subjects.go](subjects.go):

```go
{
name: "MyPool",
factory: func(maxWorkers int) (poolSubmit, poolTeardown) {
p := mypool.New(maxWorkers)
return p.Submit, p.StopAndWait
},
},
```

That is the whole integration. The harness picks it up on the next run. The factory is called with every pool size in `poolSizes`, including `1`, so it must work with a single worker.

## Reading the charts

- Chart groups are named after the task and pool size, for example `Loop10W50K` or `Loop10W1`. Inside a group, bars are grouped by workload shape, then by subject.
- The `W50K`, `W100K` and `W300K` groups show how each pool scales with parallel work; comparing the same group across sizes shows what more workers cost or buy. The `W1` groups show what each pool costs per task, with nothing running in parallel to hide it.
- Hover a bar for exact `ns/op`, `ops/s` and memory values.
- The 24, 8 and 4 CPU runs are separate datasets (`pool-24cpu`, `pool-8cpu`, `pool-4cpu`) in the dataset switcher, so you can see how each pool scales down.
- Compare each library against **GoroutinePool** first. The gap to that baseline is the real overhead of the library.
- **ErrGroup** shows what skipping a pool costs: no worker reuse, one goroutine spawned per task. A pool that loses to it adds overhead without saving any.
18 changes: 18 additions & 0 deletions pool/Taskfile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
version: "3"
tasks:
test:
desc: Vet and compile
cmds:
- go vet ./...
- go test ./...
bench:
desc: Every benchmark once
cmds:
- go test -run xxx -bench . -benchtime=1x -benchmem
txt:
desc: Full matrix at 24, 8 and 4 CPUs into bench/ (hours)
cmds:
- mkdir -p bench
- set -o pipefail; go test -run xxx -bench . -benchtime=2x -benchmem -cpu=24 -timeout 0 | tee bench/pool-24cpu.txt
- set -o pipefail; go test -run xxx -bench . -benchtime=2x -benchmem -cpu=8 -timeout 0 | tee bench/pool-8cpu.txt
- set -o pipefail; go test -run xxx -bench . -benchtime=2x -benchmem -cpu=4 -timeout 0 | tee bench/pool-4cpu.txt
Loading
Loading