Exact CPU attention and per-row sites (plan slices S1–S5)

[dansupergameprogrammer]

Before: the CPU forward pass handled five steps one element or one key at a time, using the full exact integer computation even where a row-level form gives the same answer faster:

• the per-row sites (SwiGLU sigmoid, residual landing, RMSNorm divide);
• attention's probability-times-value step;
• the requant funnel's element loop;
• the softmax row;
• on the QK-norm path, the Q31 attention score.

After: each of the five runs per row, and every tier stays bit-identical to v1.9.0. The savings below are per token, measured at engine level on a shared 4-vCPU cloud Xeon with synthetic weights. They are not end-to-end figures for any consumer. S1–S4 are scaled to Qwen2.5-0.5B depth (24 layers, 14 heads), and S5 to Qwen3-0.6B depth (28 layers, 16 heads).

• S1, per-row tables: 256-entry tables for the sigmoid, landing and norm sites, at rows of 512 or more. Saves 0.80 ms/token.
• S2, prob·V on int16 multiply-add: AVX2 and AVX-512BW, head_dim % 16 == 0, and only rows that pass the int16 condition. On AVX2, prefill saves 0.82 / 3.36 / 6.75 ms/token at T = 128 / 512 / 1,024, and decode saves 3.92 ms/token at context 300. The kernel is 16–21x faster per call.
• S3, requant row leaf in 64-bit lanes: 4 lanes on AVX2, 8 on AVX-512BW. Saves 2.53 ms/token on AVX2 (2.66 on AVX-512).
• S4, guarded softmax fast path: integer estimates with exact corrections, with the shipped body as the fallback. On AVX2, prefill saves 0.10 / 0.46 / 0.91 ms/token at T = 128 / 512 / 1,024, and decode saves 0.53 ms/token.
• S5, Q31 score row (QkQ31ScoreRow):
  • Scope: the Qwen3 QK-norm path only. Qwen2.5 models do not take this path.
  • Method: every key of a query head in one call. Each channel's product splits exactly into three 16-bit pieces, which 16-bit multiply-add sums.
  • Guard: head_dim ≤ 512, and every ratio in [0, 2^32).
  • Savings: on AVX2 the attention score cost per token falls from 11.5 to 0.47 ms at T = 128 and from 97.6 to 3.23 ms at T = 1,024. On AVX-512 it falls from 9.5 to 0.47 and from 76.1 to 3.1.
  • A one-key row on AVX-512 is about 0.06 µs slower.
  • On the Zen 2 box (AVX2), the score row measured 0.588 vs 10.264 ms/token at 128 tokens and 3.63 vs 94.07 at 1,024, with zero mismatches.

Bit-identity to v1.9.0

• Digest: the axis digest's GLOBAL hash is f740f833… on all ten legs: auto and forced scalar/SSE2/AVX2/AVX-512, each on GCC 13.3 and Clang 18.1. S5 appends its rows to the c32_attention section, so every other section equals S4's. Every leg also equals S5's red run, where the row entry still ran the v1.9.0 per-key loop.
• Golden pins, generated from the v1.9.0 tag: S1 8836d5eb…, S2 b0d1a6cd…, S3 3e3abed7…, S4 2e47ea3c…, S5 daea9a39…. The QK-norm fixture that drives both layer loops is 336b8d41….
• Save blobs and decoded tokens match the base in every row of each slice's blob protocol.
• Those records are in docs/attention-rowsites/s{1..5}/.

How

• Each slice adds a guard that admits only rows where the fast form is provably exact. Every other row goes to the v1.9.0 code, and per-tier taken/skipped counters let tests confirm which path ran.
• The SIMD bodies have internal linkage and dispatch through the existing tier selection.
• No C API change. gpu_1p0.h, the Gate A header parity check and the C ABI verb list are untouched. Five internal C++ declarations join installed headers, following the pattern already there (GemmPath/DispatchGemmPath, RequantTokenCodeWide, QkQ31ScoreForTier): SitesKernel, SelectSitesKernel and DispatchSitesKernel in matmul.h, RequantRowWide in intmath.h, and QkQ31ScoreRow in forward_sites.h. None is exported, and each header comment says so.
• The linkage and isolation checkers cover the new bodies. S5 adds the three forward_sites.cpp objects to the linkage job.
• The fp-free scan passes on both compilers, with its allow-lists unchanged and 0 REJECT:
  • GCC: 574 / 580 / 582 ACCEPT on auto / forced AVX2 / forced AVX-512.
  • Clang: 503 / 508 / 509.
• On MSVC and clang-cl, the AVX-512 fast paths stay off by default behind SUPERSLM_SITES_AVX512_MSVC=0 until they have run there. The forced AVX-512 Windows legs build with the switch set to 1.
• Deviations from the plan are recorded in docs/attention-rowsites-s{1..5}-progress.md.

Verified on Linux (GCC 13.3 and Clang 18.1, Release):

• The auto and forced SSE2/AVX2/AVX-512 suites pass with 0 failures on both compilers, including the 11.1(d) artifact cell. Every binary prints all six pinned hashes.
• The digests on all ten legs are identical, and the fp-free scan passes on all six libraries.
• pytest tests/ci/ (with tests/t2296-fp-free-open-red-suite) matches main: 1,070 passed, with the same 4 environmental failures in the t2296 suite on both.
• These all pass: check_gpu_guard_status_parity.py, check_present_tense_defect_comments.py, check_ci_claims.py, the linkage checker (on GCC objects, as CI runs it), the isolation ctest entries, the forward-leaf checker and the GPU census check.
• CI on GitHub: green on 0242af8.

Code review: done on the Zen 2 box, with no bugs found. Its two findings are fixed here. The no-new-ABI promise now names the internal declarations (above). The two generated golden-pin headers are pinned to LF in .gitattributes, so a Windows checkout matches their generators byte for byte.

Fixes to existing files

• Two commits update the nine line citations in gpu_layer_loop_guards.def. They move by 106 lines after S1–S4's additions to forward_sites.cpp, then by 225 more after S5's. The parity checker confirms each citation lands on its rejecting return.
• The first of those commits also rewords one progress-file row that tripped check_ci_claims.py.

Coverage floors (re-pinned at the owner's word)

• Hosted runners may lack AVX-512, and on those runners the new AVX-512 bodies cannot run.
• src/matmul.cpp and src/intmath.cpp are therefore pinned to 69.23076923076923 and 83.47107438016529. CI run 36584892310 recorded those values on a runner with no AVX-512 (the forced AVX-512 suite exited on SIGILL). A floor pinned from such a run holds on either kind of runner.
• include/superslm/layer_marshal.h is not re-pinned. It had dropped because the 11.1(d) cell links PreflightScanWscFolds but skips without its artifact. A committed cell, TestPreflightScanWscFolds, now runs that function on every build, and the file measures 45.83% against its 40.0 floor.
• The coverage leg now also prints the recorded floors to its log.

Still owed:

• A real Qwen3 artifact through the S5 kernel. A QK-norm artifact is still refused at map time on this host, so S5's evidence is the pinned QK-norm fixture plus a one-layer forward at Qwen3-0.6B width.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XfuJLAZ6EFSt7mB85oNm68