Skip to content

perf(cpu): eliminate hot path index math + add SIMD threshold (#81, #90) - #150

Merged
kolkov merged 2 commits into
mainfrom
perf/cpu-hotpaths-sprint2
Aug 4, 2026
Merged

kolkov merged 2 commits into
mainfrom
perf/cpu-hotpaths-sprint2

Conversation

@kolkov

@kolkov kolkov commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #81 (all 8/8 items now complete) and #90 (SIMD min slice threshold).

CPU hot paths (#81) — 3 remaining items fixed

  • Reduce: replaced per-element N-D coordinate decomposition with innerOuter block iteration (same pattern as Gather/Chunk fixes)
  • Scatter-add: replaced per-element make([]int, ndim) allocation with pre-allocated buffer — allocs O(N) → O(1)
  • Batch broadcast matmul: precompute batch offset table — O(1) lookup vs O(ndim) per batch element

SIMD threshold (#90)

  • Added simdMinLen = 32 guard to all 47 SIMD dispatch points
  • Below 32 elements, scalar loop runs — avoids function-pointer overhead exceeding SIMD savings
  • Threshold covers 2 full AVX-256 vectors (2x8) and 2 AVX-512 half-widths (2x16)
  • Benchmark + unit test suite added

Test plan

  • go build ./...
  • go test -short ./internal/backend/cpu/... — 337 tests pass
  • golangci-lint run — 0 issues
  • Benchmark suite: 9 hot path + 24 threshold benchmarks added

Closes #81
Closes #90

kolkov added 2 commits August 4, 2026 12:38
…atch matmul (#81)

Reduce: replace coordinate decomposition with innerOuter block iteration.
Scatter-add: replace per-element make([]int) with pre-allocated buffer.
BatchMatMul: precompute batch offsets table, O(1) lookup vs O(ndim)/element.

ScatterAdd allocs: O(N) → O(1). All 337 cpu tests pass.
Add simdMinLen=32 guard to all SIMD dispatch points (47 sites across
element-wise, activation, sign, and sum ops). Below 32 elements, scalar
loop runs — avoids function-pointer call overhead exceeding SIMD savings.

Threshold empirically determined: covers 2 full AVX-256 vectors (2x8)
and 2 AVX-512 half-widths (2x16). Benchmark + unit test suite added.
@kolkov
kolkov merged commit d52fe1c into main Aug 4, 2026
11 checks passed
@kolkov
kolkov deleted the perf/cpu-hotpaths-sprint2 branch August 4, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant