Skip to content

Scale the default CPU thread count with the machine - #2315

Merged
felladrin merged 1 commit into
mainfrom
fix/cap-default-cpu-threads
Aug 8, 2026
Merged

Scale the default CPU thread count with the machine#2315
felladrin merged 1 commit into
mainfrom
fix/cap-default-cpu-threads

Conversation

@felladrin

@felladrin felladrin commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Description

Currently, cpuThreads defaults to hardwareConcurrency - 2, which oversubscribes on anything but a small machine. navigator.hardwareConcurrency reports logical processors, so half of it approximates the physical core count on the SMT CPUs most users have, and that is where wllama's throughput peaks.

Measured on a 16-core/32-thread box, in Chromium, with SmolLM2-135M Q4_K_M, a 660-token prompt and the WASM CPU backend (n_gpu_layers: 0), reading wllama's own timings and peak RSS across the browser process tree:

threads prompt tok/s generation tok/s peak RSS
6 59.8 34.9 999 MB
14 81.4 47.5 1008 MB
16 68.3 57.5 1106 MB
22 67.7 34.0 1073 MB
30 53.7 16.5 1535 MB

The peak sits at 16, the physical core count. At 30 threads (what this machine gets today) generation is about 3.5x slower and peak memory is ~39% higher. In a separate run where n_threads matched hardwareConcurrency exactly, leaving no core for the main thread, it degraded much further, to 0.43 tok/s.

The default now scales with the machine rather than using a fixed number, so it lands on the physical core count at any size:

logical processors before after
2 1 1
4 2 2
8 6 4
16 14 8
32 30 16

The setting stays user-editable, so anyone who wants more threads can still raise it. The input description already warned that "a value that is too high may cause the app to hang"; this stops the default from being that value.

The thread count now lives in getDefaultCpuThreads() so it can be tested without stubbing navigator.

How to test

  1. Run npx vitest run client/modules/settings.test.ts. The new getDefaultCpuThreads tests cover the scaling, the rounding on an odd processor count, and the "never oversubscribe" invariant.
  2. Clear the settings key from localStorage, reload, and open Menu => AI Settings. "CPU threads to use" should read half your logical processor count (the value navigator.hardwareConcurrency reports in the console).

Note: the benchmark numbers come from a standalone Playwright harness rather than from this repository, so they are not reproducible from a script here. The regression is measurable in the app by setting cpuThreads to hardwareConcurrency - 2 and then to half, on a machine with many cores, and comparing generation speed.

Only the measurements at 16 and 30 threads were repeated; the other rows are single runs, and the machine was not fully idle.

Currently, `cpuThreads` defaults to `hardwareConcurrency - 2`, which
oversubscribes on anything but a small machine. `navigator.hardwareConcurrency`
reports logical processors, so half of it approximates the physical core count
on the SMT CPUs most users have, and that is where wllama's throughput peaks.

Measured on a 16-core/32-thread box (SmolLM2-135M Q4_K_M, 660-token prompt,
WASM CPU backend): 30 threads generated at 16.5 tok/s and peaked at 1535 MB,
against 57.5 tok/s and 1106 MB at 16 threads. The peak sits at the physical
core count.

The default now scales with the machine instead of using a fixed ceiling, and
the setting stays user-editable.
@felladrin
felladrin force-pushed the fix/cap-default-cpu-threads branch from 6150877 to 7a7dd46 Compare August 8, 2026 02:10
@felladrin felladrin changed the title Cap the default CPU thread count for Wllama Scale the default CPU thread count with the machine Aug 8, 2026
@felladrin
felladrin marked this pull request as ready for review August 8, 2026 02:14
@felladrin
felladrin merged commit 8e669d0 into main Aug 8, 2026
7 checks passed
@felladrin
felladrin deleted the fix/cap-default-cpu-threads branch August 8, 2026 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant