Pack line by line when a sliver would read one element per cache line

[lkdvos]

Pack line by line when a sliver would read one element per cache line

Problem

TCCG intensli_7, C[e,c,b,f,a] = A[a,b,c,d,e] * B[d,f]: the M composite follows C (e,c,b,a), so a pack-A sliver reads MR values of e (A stride dim^4) over kc steps of d (dim^3). Every element is its own cache line, and that line's other elements (A's unit-stride a, the slowest M coordinate) are needed only dim^3 M rows later. At power-of-two dims the lines touched in between alias in a few sets at every cache level and none survives: pack A costs one memory latency per element (Rome dim16 58.8 ns/elem, dim32 148; Cascade Lake dim24 45, dim32 55). Padding every axis by one element gives 1.4 ns/elem, which confirms the diagnosis.

Fix

• Plan (src/planning/pack_split.jl). _pack_split runs once per operand in _plan_contract. A group splits when all of these hold:

  • its sliver axis steps the operand by at least a line;
  • a later axis is unit-stride;
  • the per-sliver walk's reuse window, (free extent ahead of that axis) x (lines one coordinate touches per K block), exceeds the core's L2 share (twice that for complex).

  A split plan enumerates the free composite with one line's worth (L) of the unit-stride axis right after a prefix of E coordinates. The result is a 12-byte PackSplit stored in the plan (mpack/npack). mc/nc become whole E x L groups and may use the budget the blocking reserved for the requested kc when the K extent clamps it. The packed format, the K order and the kernels do not change.

• Path (src/execution/barrier.jl). _NestPath{UNPACKED_B, AFF, SPLIT} carries the split flags. _select_path(plan) reads them from the plan, so execution just follows the path. The path table grows from 128 to 512 singletons. A split group's ramp flags are false.

• Execution (src/execution/execute.jl, src/packing/transposed.jl). Only the split path enumerates _split_group and packs each macro block with _pack_block_transposed!, which reads every line once and scatters it into L slivers' panels. The K step is innermost when K steps stay within a page. The walker is @noinline: inlined, it slows the nest's micro-kernel loop (unsplit ccsd_t_3_dim16 was 1.17-1.24x slower in an earlier in-process A/B). Unsplit plans compile exactly main's nest.

• Compile latency:

  • The planner's path hint predicts "no split". A split plan that comes through TensorOperations therefore crosses the existing dispatch barrier to its split path. _hint_holds still folds for automatic kernels.
  • The structural test is inlined.
  • The window test is out of line and shared across eltypes.
  • The block-extent search sits behind a dynamic call and is compiled only when a plan reaches it. As a side effect, planning a split plan through tensorcontract! allocates 96 B per call (boxed arguments). This happens only for contractions whose pack walk misses L2 (milliseconds and up). execute! stays at 0 B.

When a group splits

A group splits only if the natural walk can neither keep its lines nor stream them:

• K steps within a page, complex eltype: never. Each sliver element's lines are then a stream the prefetchers follow, so a split only saves refetching them. With 4 elements per line, that saving does not pay for the planar scatter. This regressed 1.15-1.36 on znver2/znver4 at d1ff0e0.
• Otherwise: the reuse window must exceed the core's L2 share when the sliver pitch is a whole number of pages, because the gathered lines then fold into a few sets. If the pitch is not a page multiple, the window must exceed L2 plus the core's L3 share. The threshold is doubled for complex.

Which plans change (deterministic)

Plans were dumped for 852 :contract suite plans per profile, using the benchmark nodes' cache sizes. Every plan that does not split is identical to main.

• icelake-server / znver4 (29 splits each):
  • Float64 and Float32: intensli_3/4_dim32 and intensli_6/7/8_dim16/24/32, plus ccsd_7_dim32 for Float32.
  • ComplexF64 and ComplexF32: intensli_7_dim16/24/32.
• znver2 (36 splits): the same, plus:
  • Float64: ccsd_7_dim32 and intensli_3/4_dim24;
  • ComplexF64: intensli_3_dim32 and intensli_7_dim12;
  • ComplexF32: ccsd_7_dim32 and intensli_3_dim32.

Measurements

Same-node A/B on one core per node class (icelake = Xeon Ice Lake, genoa = EPYC Zen 4, rome = EPYC Zen 2): one allocation runs main, branch, branch, main over the :contract synthetic and TCCG cases (TCCG dims 12-32), Float64 and ComplexF64. Values are branch/main QuasiStrided time, best of the two rounds (< 1 = faster); cases with every ratio within 5% are omitted. No correctness mismatches against StridedBLAS.

Float64

case	icelake	genoa	rome
intensli_3_dim32	0.67 (4.3 → 2.8 ms)	0.47 (6.4 → 3.0 ms)	0.32 (12.3 → 3.9 ms)
intensli_4_dim24	0.94	1.00	1.10 (0.85 → 0.93 ms)
intensli_4_dim32	0.75 (4.3 → 3.2 ms)	0.86	0.97
intensli_6_dim16	0.54 (4.0 → 2.1 ms)	0.96	1.02
intensli_6_dim24	0.67 (38 → 25 ms)	0.85	0.97
intensli_6_dim32	0.37 (387 → 144 ms)	0.45 (243 → 109 ms)	1.03
intensli_7_dim16	0.65 (4.0 → 2.6 ms)	0.34 (7.6 → 2.6 ms)	0.04 (79 → 3.4 ms)
intensli_7_dim24	0.93	0.75 (48 → 36 ms)	0.97
intensli_7_dim32	0.58 (353 → 204 ms)	0.56 (548 → 305 ms)	0.07 (5.23 s → 348 ms)
intensli_8_dim16	0.58 (4.2 → 2.4 ms)	0.85	0.92
intensli_8_dim24	0.78 (39 → 30 ms)	0.72 (31 → 22 ms)	0.94
intensli_8_dim32	0.38 (385 → 148 ms)	0.49 (246 → 121 ms)	1.05

ComplexF64

case	icelake	genoa	rome
intensli_7_dim16	0.87 (6.1 → 5.3 ms)	0.87 (8.6 → 7.5 ms)	0.27 (56 → 15 ms)
intensli_7_dim24	0.86 (100 → 86 ms)	1.09 (103 → 113 ms)	1.16 (121 → 139 ms)
intensli_7_dim32	0.84 (481 → 402 ms)	0.79 (672 → 529 ms)	0.20 (3.31 s → 668 ms)

geomean, all cases ≥ 0.1 ms	icelake	genoa	rome
Float64	0.935	0.947	0.931
ComplexF64	0.995	0.998	0.976

Known remaining losses: ComplexF64 intensli_7_dim24 on genoa and rome, and Float64 intensli_4_dim24 on rome still split and are 1.09-1.16x slower. Nothing in the cache profile separates them from wins with the same layout on the other nodes, so they are left rather than special-cased. The first node round (d1ff0e0, split rule without the complex and L3 conditions) lost 1.11-1.36x on genoa/rome for ComplexF64 intensli_4/6/8 and Float64 dim-12 cases; this rule no longer splits those (now 0.96-1.02).

Compile latency and call floor (local, indicative):

• First call (probe_ttfx.jl, 3 interleaved runs): main Float64 3.1-3.2 s, ComplexF64 2.1-2.3 s; branch 3.2-3.3 s and 2.2 s.
• Per-call floor (probe_call_floor.jl): 0 B/call in every case, timings within noise of main.

Tests

The new tests pin the new behaviour:

• the planner picks the split path for intensli_7 at dim 16 (a 4 MB reuse window) and not at dim 4;
• contractions through forced splits match an engine-free reference. They cover four eltypes; the default, 1m and fmaddsub packed formats; conj; beta = 0 over NaN and beta != 0; tails; A-only and A+B splits; and both walk orders;
• the split path is 0 B/call;
• tensorcontract! through a split plan matches StridedNative.

The full suite passes.

🤖 Generated with Claude Code