From 1dca3d5226711ab30da4e185d51c1ce0dd5eef60 Mon Sep 17 00:00:00 2001 From: Paul Zeiger Date: Tue, 22 Sep 2026 06:07:50 +0000 Subject: [PATCH 1/2] Document Numba/OMP_NUM_THREADS oversubscription alongside FFT threading The real-space multislice algorithm and a few other CPU-only code paths use Numba's own thread pool, controlled by OMP_NUM_THREADS (or NUMBA_NUM_THREADS directly) rather than the fftw.threads/mkl.threads knobs already documented here, but subject to the same oversubscription risk against Dask's own chunk-level parallelism. Measured: these kernels are memory-bandwidth-bound and scale poorly past a handful of threads even for one isolated computation, and the per-kernel thread count stops mattering altogether once Dask is running many chunks concurrently. Co-Authored-By: Claude Sonnet 5 --- docs/user_guide/appendix/performance_tips.ipynb | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/user_guide/appendix/performance_tips.ipynb b/docs/user_guide/appendix/performance_tips.ipynb index 199b1c30..0fcab4d3 100644 --- a/docs/user_guide/appendix/performance_tips.ipynb +++ b/docs/user_guide/appendix/performance_tips.ipynb @@ -278,6 +278,15 @@ "number of workers times the FFT threads at or below your physical core count. FFTW also exposes its planner through\n", "`fftw.planning_effort`; more patient planning can pay off for long-running simulations that reuse one grid size.\n", "\n", + "The same principle applies to Numba, which a few CPU-only code paths use directly — the\n", + "real-space multislice algorithm among them. Numba's thread pool follows `OMP_NUM_THREADS` if it\n", + "is set, or every visible core otherwise, and `NUMBA_NUM_THREADS` overrides that independently of\n", + "the other libraries above. These kernels are memory-bandwidth-bound and scale poorly past a\n", + "handful of threads even in isolation, so a large value rarely helps a single computation —\n", + "benchmark before assuming your full core count is best. Once Dask is already running many chunks\n", + "concurrently, the per-kernel thread count barely matters at all: with enough independent tasks\n", + "saturating a node, giving each one its own extra threads has nothing left to parallelize into.\n", + "\n", "### Use \"good\" numbers of `gpts`\n", "\n", "FFT implementations are most efficient for array sizes that factor into small primes (2, 3, 5 and 7). Since the exact\n", From b17f3a44abf02ac15ed188e84378df451088526d Mon Sep 17 00:00:00 2001 From: Paul Zeiger Date: Tue, 22 Sep 2026 06:30:43 +0000 Subject: [PATCH 2/2] Reflect that capping can actively help, not merely be free Confirmed independently on a second machine (10-core/20-thread Xeon, 20 concurrent Dask-like callers): when worker count exceeds physical core count, capping the per-kernel thread count measured 15-18% faster, not merely indistinguishable, since a lower-core-count-per- worker-count machine leaves no spare capacity for a kernel's own internal threads to exploit even in principle. Co-Authored-By: Claude Sonnet 5 --- docs/user_guide/appendix/performance_tips.ipynb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/user_guide/appendix/performance_tips.ipynb b/docs/user_guide/appendix/performance_tips.ipynb index 0fcab4d3..7a3ddc98 100644 --- a/docs/user_guide/appendix/performance_tips.ipynb +++ b/docs/user_guide/appendix/performance_tips.ipynb @@ -284,8 +284,11 @@ "the other libraries above. These kernels are memory-bandwidth-bound and scale poorly past a\n", "handful of threads even in isolation, so a large value rarely helps a single computation —\n", "benchmark before assuming your full core count is best. Once Dask is already running many chunks\n", - "concurrently, the per-kernel thread count barely matters at all: with enough independent tasks\n", - "saturating a node, giving each one its own extra threads has nothing left to parallelize into.\n", + "concurrently, keeping this low costs nothing at worst, and can measurably help: when the number\n", + "of concurrent workers already exceeds your physical core count — common once a worker pool is\n", + "sized to logical rather than physical cores — each kernel call claiming its own extra threads adds\n", + "contention with nothing left to actually parallelize into, and can cost real throughput rather\n", + "than merely fail to gain any.\n", "\n", "### Use \"good\" numbers of `gpts`\n", "\n",