Scale the default CPU thread count with the machine - #2315
Merged
Conversation
Currently, `cpuThreads` defaults to `hardwareConcurrency - 2`, which oversubscribes on anything but a small machine. `navigator.hardwareConcurrency` reports logical processors, so half of it approximates the physical core count on the SMT CPUs most users have, and that is where wllama's throughput peaks. Measured on a 16-core/32-thread box (SmolLM2-135M Q4_K_M, 660-token prompt, WASM CPU backend): 30 threads generated at 16.5 tok/s and peaked at 1535 MB, against 57.5 tok/s and 1106 MB at 16 threads. The peak sits at the physical core count. The default now scales with the machine instead of using a fixed ceiling, and the setting stays user-editable.
felladrin
force-pushed
the
fix/cap-default-cpu-threads
branch
from
August 8, 2026 02:10
6150877 to
7a7dd46
Compare
felladrin
marked this pull request as ready for review
August 8, 2026 02:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Currently,
cpuThreadsdefaults tohardwareConcurrency - 2, which oversubscribes on anything but a small machine.navigator.hardwareConcurrencyreports logical processors, so half of it approximates the physical core count on the SMT CPUs most users have, and that is where wllama's throughput peaks.Measured on a 16-core/32-thread box, in Chromium, with SmolLM2-135M Q4_K_M, a 660-token prompt and the WASM CPU backend (
n_gpu_layers: 0), reading wllama's owntimingsand peak RSS across the browser process tree:The peak sits at 16, the physical core count. At 30 threads (what this machine gets today) generation is about 3.5x slower and peak memory is ~39% higher. In a separate run where
n_threadsmatchedhardwareConcurrencyexactly, leaving no core for the main thread, it degraded much further, to 0.43 tok/s.The default now scales with the machine rather than using a fixed number, so it lands on the physical core count at any size:
The setting stays user-editable, so anyone who wants more threads can still raise it. The input description already warned that "a value that is too high may cause the app to hang"; this stops the default from being that value.
The thread count now lives in
getDefaultCpuThreads()so it can be tested without stubbingnavigator.How to test
npx vitest run client/modules/settings.test.ts. The newgetDefaultCpuThreadstests cover the scaling, the rounding on an odd processor count, and the "never oversubscribe" invariant.settingskey from localStorage, reload, and open Menu => AI Settings. "CPU threads to use" should read half your logical processor count (the valuenavigator.hardwareConcurrencyreports in the console).Note: the benchmark numbers come from a standalone Playwright harness rather than from this repository, so they are not reproducible from a script here. The regression is measurable in the app by setting
cpuThreadstohardwareConcurrency - 2and then to half, on a machine with many cores, and comparing generation speed.Only the measurements at 16 and 30 threads were repeated; the other rows are single runs, and the machine was not fully idle.