Cap onnxruntime intra-op threads on Apple Silicon to stop the embedding CPU peg

[scottyallen]

Problem

Embedding is done one exchange at a time (batch=1) throughout indexing, sync, and the re-embed migration. At that size onnxruntime-node's default intra-op thread pool fans each tiny int8 matmul across every core. On Apple Silicon this pegs ~5 cores for almost no throughput gain, and it makes a bulk re-embed (e.g. the model migration) hog the whole machine. It's especially visible when several Claude Code sessions/worktrees trigger syncs, or when the machine is otherwise busy and the pool oversubscribes and fights itself.

Fix

Cap the embedding session's thread pools when we know parallelism won't help:

• intraOpNumThreads: 2 and interOpNumThreads: 1, applied only on darwin && arm64.
• Gated as a strict no-op elsewhere: when no cap resolves, the pipeline() call passes no session_options at all, so Linux/Intel/CI are byte-for-byte unchanged.
• Env override EPISODIC_MEMORY_EMBED_THREADS: a positive integer sets the cap; 0 / negative / non-numeric restores onnxruntime's default (off-switch). Parsed defensively.

Same model, same q8 CPU execution provider, so embeddings are bit-identical to the uncapped output — no re-index is triggered.

Measurements (M2 Max, bge-small-en-v1.5 q8, batch=1, N=200)

Config	per-item wall	cores (cpu/wall)	vs stored q8 vectors	re-index
Uncapped default (current)	11.3 ms	4.67	baseline	-
Capped intra=2 / inter=1	10.6 ms	1.78	cosine 1.000000 (exact)	no

Same-or-faster wall time while using ~2.6x fewer cores. The gap widens under system load, because capping stops the pool from oversubscribing.

Alternatives evaluated and rejected

• CoreML EP (device: 'coreml') on the q8 model: ~9x slower (~105 ms/item), never actually leaves the CPU (still ~5 cores — the int8 graph partitions poorly), and produces different vectors (would force a re-index). Hard no.
• WebGPU fp32 (dawn): genuinely offloads (~0.78 cores) but only at fp32, whose vectors differ from the stored q8 ones (full re-index) for no wall-clock gain, and it's experimental.

The thread cap is the only option that reduces CPU with zero behavioral change to the embeddings.

Tests

• New unit test for resolveIntraOpThreads() covering positive override, 0, non-numeric, and the platform default.
• Full suite green (211/211).

🤖 Generated with Claude Code