thread-pool

A header-only C++20 thread pool built on std::jthread and std::stop_token.

Tasks are submitted as arbitrary callables, executed on a fixed set of worker threads, and their results delivered through std::future. Exceptions thrown inside a task are captured and rethrown on the caller's side. Cancellation is cooperative, modelled on .NET's CancellationToken.

#include <tp/thread_pool.hpp>

tp::ThreadPool pool(4);
auto fut = pool.submit([](int a, int b) { return a + b; }, 2, 3);
int result = fut.get();   // 5

Design

Three headers, no external dependencies, nothing to link:

Header	Provides	Used by the pool?
`tp/thread_pool.hpp`	`ThreadPool`, `TaskCancelled`	—
`tp/task.hpp`	`Task` — move-only type-erased callable	yes
`tp/task_queue.hpp`	`TaskQueue<T>` — standalone blocking queue	no

TaskQueue<T> is a general-purpose blocking queue that is not used by ThreadPool; the pool holds its own queue inline so it can share one mutex and condition variable with the workers. TaskQueue is kept because it is useful on its own (and independently tested), not because the pool depends on it.

Design constraints worth stating up front:

Fixed worker count. No dynamic thread injection. Set at construction.
Single shared queue. No per-worker queues, no work stealing. See Performance notes for why that is not currently the bottleneck.
Non-copyable, non-movable. Workers capture this; moving a live pool would dangle.

Components

graph TD
    subgraph pool["tp::ThreadPool"]
        Q["std::queue&lt;Task&gt;<br/>guarded by mtx_"]
        CV["std::condition_variable_any<br/>cv_"]
        W1["worker 0<br/>std::jthread"]
        W2["worker 1<br/>std::jthread"]
        W3["worker N-1<br/>std::jthread"]
    end

    Caller["caller thread<br/>submit()"] -->|"emplace + notify_one"| Q
    Caller -.->|"returns"| Fut["std::future&lt;R&gt;"]

    Q --- CV
    CV -->|"wakes one"| W1
    CV --> W2
    CV --> W3

    W1 -->|"pop, invoke"| Run["task()"]
    Run -->|"set_value / set_exception"| Fut

    style Q fill:#e8f0fe,stroke:#4285f4,color:#000
    style CV fill:#fef7e0,stroke:#f9ab00,color:#000
    style Fut fill:#e6f4ea,stroke:#34a853,color:#000
    style Run fill:#fce8e6,stroke:#ea4335,color:#000

Everything — the queue, the closed flag, and the workers' wait predicate — is guarded by a single mutex. The condition variable is _any rather than the plain std::condition_variable because only that overload accepts a std::stop_token in wait(), which is what makes shutdown work without a sentinel task.

How a task flows through the pool

sequenceDiagram
    participant C as Caller
    participant P as ThreadPool
    participant Q as tasks_ queue
    participant W as Worker thread
    participant F as std::future

    C->>P: submit(f, args...)
    P->>P: create std::promise&lt;R&gt;
    P->>F: p.get_future()
    Note over P: promise is moved INTO the task<br/>(Task is move-only, so no shared_ptr)
    P->>Q: lock, emplace(Task), unlock
    P->>W: cv_.notify_one()
    P-->>C: return future

    W->>W: wake from cv_.wait
    W->>Q: lock, pop task, unlock
    Note over W: mutex released BEFORE running,<br/>so other workers are not blocked
    W->>W: task()

    alt task returns normally
        W->>F: p.set_value(result)
    else task throws
        W->>F: p.set_exception(current_exception())
    end

    C->>F: fut.get()
    F-->>C: result, or rethrows

Two details that matter:

notify_one happens after the mutex is released. Otherwise the woken worker would immediately block trying to acquire the lock the notifier still holds.
The task runs with the mutex released. A long-running task does not stall submitters or other workers.

Worker lifecycle

Each worker runs this loop until stop is requested and the queue is drained:

stateDiagram-v2
    [*] --> Waiting: worker starts

    Waiting --> Waiting: spurious wakeup, predicate false
    Waiting --> Popping: queue non-empty
    Waiting --> Exiting: stop requested AND queue empty

    Popping --> Running: pop task, release mutex
    Running --> Waiting: task complete

    Exiting --> [*]

    note right of Waiting
        cv_.wait(lock, st, pred)
        pred = !tasks_.empty()
        The stop_token ends the wait
        even with no notify.
    end note

    note right of Running
        Exceptions never escape:
        submit() wraps the call in
        try/catch and stores them
        in the promise.
    end note

The key invariant: a worker only exits when the queue is empty. A stop request alone is not enough. That is what guarantees queued work is not dropped at shutdown.

Shutdown

The destructor is where the ordering is subtle, and where this pool previously had a use-after-free bug.

flowchart TD
    D["~ThreadPool()"] --> RS["request_stop() on every worker"]
    RS --> NA["cv_.notify_all()"]
    NA --> J["join every worker<br/>EXPLICITLY, in the destructor body"]
    J --> M["members destroyed:<br/>cv_, mtx_, tasks_, then workers_"]
    M --> Done["done"]

    J -.->|"workers finish<br/>draining the queue"| Drain["remaining tasks run<br/>to completion"]
    Drain -.-> J

    style J fill:#fce8e6,stroke:#ea4335,stroke-width:3px,color:#000
    style M fill:#e8f0fe,stroke:#4285f4,color:#000

Why the explicit join is load-bearing. C++ destroys members in reverse declaration order. workers_ is declared first, so it is destroyed last — meaning tasks_, mtx_ and cv_ would already be gone while the workers were still inside worker_loop() using them. Relying on ~jthread to join is therefore a use-after-free: ASan reports it as a double-free of a queued task, freed once by a worker and once by ~queue.

The fix is the explicit join in the destructor body, before any other member dies. Do not remove it. tests/test_stress.cpp contains the regression guards.

Guarantees at shutdown:

Every task accepted by submit() runs to completion. Nothing is dropped.
Every returned std::future is satisfied — even if the pool is destroyed before the caller calls get().
The destructor must not run concurrently with submit(). That is the caller's responsibility.

Cancellation

submit_cancellable() takes a std::stop_token and passes it to the callable as its first argument. This mirrors .NET's CancellationToken overloads.

flowchart TD
    S["submit_cancellable(token, f, args...)"] --> Q["task queued"]
    Q --> W["worker picks it up"]
    W --> C{"token.stop_requested()?"}

    C -->|"yes"| Cancel["task NEVER runs<br/>future gets tp::TaskCancelled"]
    C -->|"no"| Run["invoke f(token, args...)"]

    Run --> Poll{"task polls<br/>st.stop_requested()"}
    Poll -->|"yes"| Early["task returns early<br/>future gets that value"]
    Poll -->|"never checks"| Full["task runs to completion<br/>future gets its result"]

    Cancel --> End["future ready"]
    Early --> End
    Full --> End

    style Cancel fill:#fce8e6,stroke:#ea4335,color:#000
    style Early fill:#fef7e0,stroke:#f9ab00,color:#000
    style Full fill:#e6f4ea,stroke:#34a853,color:#000

There are two distinct checkpoints:

When cancelled	Outcome
While still queued	Task is discarded, future holds `tp::TaskCancelled`
While running, task polls the token	Task returns early; its return value is delivered normally
While running, task ignores the token	Task completes; cancellation has no effect

Cancellation is cooperative. Nothing preempts a running task — same semantics as .NET, where OperationCanceledException is only raised by code that actually checks the token.

std::stop_source src;

auto fut = pool.submit_cancellable(src.get_token(),
    [](std::stop_token st, int n) {
        for (int i = 0; i < n; ++i) {
            if (st.stop_requested()) return i;   // cooperative exit
            do_work(i);
        }
        return n;
    }, 1000);

src.request_stop();

try {
    int done = fut.get();          // how far it got
} catch (const tp::TaskCancelled&) {
    // never started
}

One stop_source can cancel a whole batch of tasks, which is the usual .NET pattern.

Bulk dispatch

Submitting one task per element is the wrong shape for fine-grained work: each element pays a full dispatch — lock, enqueue, notify, wake a thread — which costs far more than the work itself. for_each() submits one task per chunk instead, amortising that cost across many elements.

flowchart LR
    R["range of N elements"] --> Split["split into chunks<br/>default: N / (workers x 4)"]

    Split --> C0["chunk 0"]
    Split --> C1["chunk 1"]
    Split --> C2["chunk 2"]
    Split --> CN["chunk k-1"]

    C0 --> Caller["CALLING thread<br/>runs chunk 0 itself"]
    C1 --> W["worker threads"]
    C2 --> W
    CN --> W

    Caller --> Join["wait for all chunks"]
    W --> Join
    Join --> Done["for_each returns"]

    style Caller fill:#fef7e0,stroke:#f9ab00,color:#000
    style Join fill:#e8f0fe,stroke:#4285f4,color:#000
    style Done fill:#e6f4ea,stroke:#34a853,color:#000

Measured, 10,000 elements of light work, 24-core machine:

Approach	Throughput	vs serial
One `submit()` per element	32.0 K items/s	0.002x — far worse
Serial loop, no pool	15.7 M items/s	1x
`for_each()`	40.2 M items/s	2.6x

for_each is ~1,255x faster than one task per element. That gap is the wakeup cost, not the work — see Performance notes.

Chunk size defaults to N / (workers × 4): a few chunks per worker, so uneven per-element cost can still be balanced without over-dispatching. The measured optimum for the case above is 100–500 elements per chunk, and the default lands at ~104. Pass an explicit chunk_size to override.

Chunk size	Throughput
1	26.8 M items/s (dispatch-bound)
10	33.2 M items/s
100	41.1 M items/s
500	37.5 M items/s
2,500	24.4 M items/s
10,000 (one chunk)	15.7 M items/s (= serial)

Two behaviours worth knowing:

The calling thread runs chunk 0. A pool of N workers gives N+1 threads of parallelism, and the caller does not sit idle. submit_bulk() does not do this, since its caller is expected to go do something else.
An exception aborts the rest of its own chunk, not just that element. for_each waits for every chunk to finish before rethrowing, so no task is left running against a range the caller may destroy. If several chunks throw, one exception is rethrown and the rest are dropped — the same contract as Parallel.For.

Why not `std::function`

The pool stores tasks in tp::Task, a purpose-built move-only type-erased callable, rather than std::function<void()>. Two reasons:

1. std::function requires a copy-constructible target. A task capturing a std::unique_ptr or a std::promise does not compile — the error surfaces deep in <functional> as "std::function target must be copy-constructible". Tasks are moved into a queue and moved out again; they are never copied, so the requirement buys nothing. Working around it means boxing the promise in a shared_ptr, which costs an extra allocation on every submit.

2. Small-buffer optimisation. libstdc++'s std::function inlines only 16 bytes. tp::Task inlines 64, which fits the common case of a promise plus a small argument tuple with no heap allocation at all.

Measured on the actual submit path (lambda capturing a promise plus two ints):

Storage	Allocations per task
`std::function` + `shared_ptr<promise>`	4
`tp::Task` + promise by value	2

The two remaining allocations belong to std::promise's shared state, not to the task. C++23 spells this type std::move_only_function; tp::Task is the C++20 stand-in.

graph LR
    subgraph task["tp::Task — 80 bytes"]
        Buf["buf_[64]<br/>inline storage"]
        VT["vtable_<br/>invoke / move / destroy"]
    end

    Small["small callable<br/>≤ 64 bytes,<br/>nothrow-movable"] -->|"placement new"| Buf
    Large["large or over-aligned<br/>or throwing move"] -->|"heap, pointer stored"| Buf

    style Buf fill:#e8f0fe,stroke:#4285f4,color:#000
    style VT fill:#fef7e0,stroke:#f9ab00,color:#000

API reference

`tp::ThreadPool`

Member	Description
`ThreadPool(size_t n = hardware_concurrency())`	Starts `n` workers. `n == 0` becomes 1.
`submit(f, args...) -> future<R>`	Enqueues a callable. Arguments are captured by value (moved if rvalues). Move-only callables and arguments are supported.
`submit_cancellable(token, f, args...) -> future<R>`	As above; `f` additionally receives `token` as its first argument.
`for_each(first, last, f, chunk_size = 0)`	Applies `f` to every element in parallel and blocks until done. The calling thread runs one chunk. The analogue of `Parallel.For`.
`submit_bulk(first, last, f, chunk_size = 0) -> future<void>`	Non-blocking form of `for_each`. The caller does not participate.
`size() const -> size_t`	Worker count. Constant for the pool's lifetime.
`pending() const -> size_t`	Tasks queued but not yet picked up. A snapshot.

submit() and pending() are safe to call concurrently from any thread, including from inside a running task. The destructor is not.

Submitting from inside a task is supported, but a task that blocks on a future fulfilled by the same pool needs more than one worker, or it deadlocks.

`tp::TaskCancelled`

Derives from std::runtime_error. Stored in the future of a task that was cancelled before it started. The analogue of .NET's OperationCanceledException.

`tp::Task`

Move-only type-erased void() callable with a 64-byte inline buffer. Usable standalone.

Member	Description
`Task()`	Empty.
`Task(F&& f)`	Wraps any callable invocable as `f()`.
`operator()()`	Invokes it. Undefined if empty — check first.
`explicit operator bool() const`	Whether it holds a callable.
`kBufferSize`	Inline capacity, 64 bytes. A performance knob, not a limit.

`tp::TaskQueue<T>`

Standalone thread-safe FIFO queue with blocking and non-blocking pop. T need only be movable. Not used by ThreadPool.

Member	Description
`push(T)`	Enqueues, wakes one consumer. Throws `std::runtime_error` if closed.
`try_pop() -> optional<T>`	Never blocks. `nullopt` if empty.
`wait_pop(stop_token) -> optional<T>`	Blocks while empty. `nullopt` only when empty and closed or stopped.
`close()`	Wakes every consumer. Idempotent. Queued items remain poppable.
`size()`, `empty()`	Snapshots; may be stale on return.

Items queued before close() are still returned, so nothing accepted is lost.

Building

Requires C++20 with <stop_token> and std::jthread support: GCC 11+, Clang 17+ (with libstdc++), or MSVC 19.30+.

The library itself is header-only — copy include/tp/ and add it to your include path, or use CMake:

add_subdirectory(thread-pool)
target_link_libraries(my_target PRIVATE tp::tp)

Presets

Preset	Build type	Purpose
`release`	Release	Optimised, benchmarks enabled
`asan`	Debug	AddressSanitizer + UBSan
`tsan`	Debug	ThreadSanitizer

cmake --preset release
cmake --build build/release -j

The first configure downloads Catch2 and Google Benchmark via FetchContent, both pinned to exact tags.

Options

Option	Default	Effect
`TP_BUILD_TESTS`	`ON`	Build the test suite
`TP_BUILD_BENCH`	`OFF`	Build the benchmarks (`ON` in the `release` preset)

Testing

89 test cases across six files:

File	Covers
`test_task.cpp`	`Task` — inline vs heap storage, move semantics, destroy-exactly-once
`test_task_queue.cpp`	`TaskQueue` — FIFO order, close semantics, blocking, stop tokens
`test_thread_pool.cpp`	`ThreadPool` — results, void tasks, exceptions, callable kinds, `pending()`, shutdown
`test_cancellation.cpp`	`submit_cancellable` — cancel-while-queued, cooperative exit, batch cancellation
`test_bulk.cpp`	`for_each` / `submit_bulk` — chunk arithmetic, uneven splits, caller participation, exception propagation
`test_stress.cpp`	Multi-producer/consumer, submit hammering, repeated construction/destruction

ctest --test-dir build/release -j4          # all
ctest --test-dir build/release -E stress    # skip the slow ones
./build/release/tests/tp_tests "[cancel]"   # by Catch2 tag

Run under both sanitizers before trusting a change to the shutdown path:

cmake --preset asan && cmake --build build/asan -j && ctest --test-dir build/asan
cmake --preset tsan && cmake --build build/tsan -j && ctest --test-dir build/tsan

The suite passes clean under Release, ASan and TSan.

Benchmarks

cmake --preset release && cmake --build build/release -j
./build/release/bench/tp_bench
./build/release/bench/tp_bench --benchmark_filter=Granularity
./build/release/bench/tp_bench --benchmark_format=json > results.json

Benchmark	Question it answers
`BM_SubmitAndWait`	End-to-end cost of one task
`BM_SubmitOnly`	Producer-side cost alone
`BM_PromiseAllocationOnly`	How much of that is the promise
`BM_ThroughputTrivial`	Sustained throughput, futures collected
`BM_ThroughputNoFutures`	Same, fire-and-forget
`BM_GranularityPooled` / `Serial`	At what task size does the pool beat a plain loop
`BM_Bulk*`	Batched dispatch vs one task per element vs serial
`BM_BulkChunkSize`	Where the chunk-size sweet spot is
`BM_ContendedSubmit`	Does the single mutex serialise producers
`BM_QueuePushPop` / `ProducerConsumer`	`TaskQueue` in isolation

Performance notes

Measured cost decomposition of a single task, on the development machine:

Component	Cost	Share
Condition variable wakeup	~43,700 ns	95%
`std::promise` + `future`	~65 ns	0.14%
Mutex + queue push/pop	~8.6 ns	0.02%
`Task` construction + invoke	~0.2 ns	~0%

Read this carefully before optimising anything. The dispatch cost is dominated almost entirely by the thread wakeup. Replacing std::future with a custom atomic-based future would target 0.14% of the cost; adding work-stealing queues would target 0.02%. Neither is worth doing on this evidence.

That ~43µs wakeup is also not representative of Linux. The development machine is WSL2, where scheduling latency is pathological:

Mechanism	Round-trip
`std::condition_variable`	48,728 ns
Raw `futex` syscall	48,046 ns
Spin, no syscall	76 ns

A raw futex costs the same as the condition variable, so libstdc++ adds nothing; and a bare sleep(0) on the same machine takes ~74µs where it should take ~50ns. On native Linux, expect low-single-digit microseconds instead. Re-measure on your target platform before drawing conclusions.

Practical consequence, from BM_GranularityPooled vs BM_GranularitySerial: submit() only beats a serial loop once each task does a non-trivial amount of work. For fine-grained work the fix is to dispatch less often — which is what for_each() does, and it is worth ~1,255x over one task per element.

Limitations and future work

Known and deliberate:

No .then() / when_all / when_any. std::future has no continuation support, so composing tasks means blocking on get(). Adding these needs a custom future type.
No work stealing. One shared queue. BM_ContendedSubmit is currently flat from 1 to 16 threads, so producer contention is not the bottleneck — revisit only if a benchmark says otherwise.
No dynamic sizing. Fixed worker count, no thread injection on blocking. For CPU-bound work a fixed pool sized to the core count is usually better anyway.
No task priorities.
submit() must not race the destructor. Not enforced.

Not planned: async/await semantics, which need C++20 coroutines and a fundamentally different design. The direction the standard is heading for this problem is std::execution (P2300, senders/receivers) in C++26.

Name		Name	Last commit message	Last commit date
.vscode		.vscode
bench		bench
examples		examples
include/tp		include/tp
tests		tests
.clang-format		.clang-format
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
CMakePresets.json		CMakePresets.json
README.md		README.md

Navigation Menu

Repository navigation

About

Repository files navigation

thread-pool

Contents

Design

Components

How a task flows through the pool

Worker lifecycle

Shutdown

Cancellation

Bulk dispatch

Why not `std::function`

API reference

`tp::ThreadPool`

`tp::TaskCancelled`

`tp::Task`

`tp::TaskQueue<T>`

Building

Presets

Options

Testing

Benchmarks

Performance notes

Limitations and future work

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

thread-pool

Contents

Design

Components

How a task flows through the pool

Worker lifecycle

Shutdown

Cancellation

Bulk dispatch

Why not std::function

API reference

tp::ThreadPool

tp::TaskCancelled

tp::Task

tp::TaskQueue<T>

Building

Presets

Options

Testing

Benchmarks

Performance notes

Limitations and future work

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Why not `std::function`

`tp::ThreadPool`

`tp::TaskCancelled`

`tp::Task`

`tp::TaskQueue<T>`