Repository navigation
Performance and simplification: per-call floor, scrambled layouts, AVX-512 GEMM peak, degenerate shapes, compile latency - #10
Merged
Merged
Conversation
…roup `_classify_labels` listed the contracted labels in `indA` order and the K `AxisGroup` kept it. The packers step through K in the composite's enumeration order, so that order decides how BOTH packs walk memory, and `indA` order is wrong for the upstream suite's `contract_scrambled` layout, C[a,e] = A[a,b,c,d] * B[d,c,b,e]: B's unit-stride axis `d` became the slowest K coordinate, so `pack_b!` jumped 96^2 elements per K step and touched a new cache line for every element it read. Measured on the suite case dim96_1_3_1_contract_scrambled (Float64, Cascade Lake ccqlin038, pinned, benchmark/probes/probe_stage_breakdown.jl): pack B was 92% of a 4.56 s contraction at 49 ns per B element (pack A: 1.3 ns/elem). With the K order sorted by B's strides the same contraction takes 0.60 s (3.6 -> 27.2 GF/s; StridedBLAS 13.0), pack B 1.2 ns/elem. Rule (`_order_contract_labels`, src/planning/labels.jl): stable-sort the K labels by |stride| in whichever operand has the smaller minimum K stride (K is that operand's inner group and has no contiguous free axis to hide behind); ties by the larger free extent, then A. Layouts whose K strides are already ascending in the chosen operand -- every gemm_ready/a_permuted/ b_permuted suite layout -- keep exactly the previous order (checked: dim48_1_3_1_b_permuted plans the same K group before and after). The swapped orientation reuses the same order with the maps exchanged. Tests: planner pins for scrambled/gemm_ready/b_permuted/both-inner/tie/ singleton/single-label/swapped cases, and a correctness sweep (all four dtypes, alpha/beta, conj, both orientations, `execute!` and the tilewise oracle) against the loop reference. Also adds three probes: probe_stage_breakdown.jl (an instrumented copy of `_execute_nest!` timing pack B / pack A / kernel+store, with an optional hand-made M-axis split), probe_case_ab.jl (interleaved StridedBLAS vs QuasiStrided timing of one suite-style case through tensorcontract!), and probe_msplit_prototype.jl (M enumeration-order variants for the TCCG intensli cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e option benchmark/submit_scrambled_verify.sh: one job per node class (icelake, genoa, rome) running the bench_to_suite.jl subset the rule touches (every synthetic layout at dims 32 and 96, TCCG at 8/16/24) and, for the Rome-only intensli_7_dim16 slowdown (18x slower than intensli_6_dim16 on znver2, 2.6x on znver4, absent on Ice Lake and Cascade Lake), the stage breakdown of intensli_7/intensli_6 at dims 16 and 8 as planned and under two hand-made M enumeration orders. probe_stage_breakdown.jl gains `--shape MR,NR,W` to plan with another ISA's menu shape (e.g. the AVX2 Float64 tile 8,6,4) on this host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The large-GEMM plateau (QS 65-69 GF/s vs OpenBLAS ~101 at 3969^3 Float64 on Cascade Lake and Ice Lake; parity on Zen4/Zen2) is the (16,6,8) microkernel being FRONT-END bound on Intel: its K step is 2 A loads + 6 vbroadcastsd + 12 register FMAs + 5 loop-overhead instructions = 25 fused-domain uops per 6 FMA-cycles, above Skylake's 4-wide allocation. Measured with perf stat on L1-resident packed panels: 2.98 IPC, 23.8 flops/cycle (74% of 32). Doubling the A vectors per column halves the broadcasts and loop overhead per FMA: (32,6,8) is 38 instructions per 12 FMA-cycles and measures 29.1 flops/cycle (91%; OpenBLAS dgemm on the same core: 28.7). Zen4 is not affected because its 512-bit FMAs are double-pumped (12 cycles per K step, ~2 uops/cycle) and it dispatches 6-wide. - `_rule_mv`: MV = 4 for the real method on :avx512, 2 everywhere else (24 accumulators + 4 A + 2 scratch = 30 of 32 registers; AVX2's 16 hold exactly the MV = 2 tile). Complex methods keep MV = 2. - Menus gain (32,6,8) / (64,6,16); the MV = 2 shapes stay as the short-M step-down and `_demote_for_run` targets. - `_extent_shape`: the MV = 4 shape steps down to its MV = 2 sibling below one tall tile, and below two when the half tile pads strictly less (Qm in (MR, 3MR/2]); a padding comparison, not a throughput model. Engine, 3969^3 Float64, ccqlin038 (interleaved, same process): (16,6,8) 73.8 -> (32,6,8) 87.1 GF/s, OpenBLAS 105.7. bench_to_suite subset: dim63_2_2_2 all five layouts 1.22-1.31x, dim32_2_2_2 1.26-1.32x, dim24_2_2_2 1.14-1.30x; ComplexF64 unchanged (0.94-1.02x); no correctness mismatches; per execute! allocation unchanged (336 B, size-independent). benchmark/bench_large_gemm_peak.jl + submit_large_gemm_peak.sh: the cross-node validation (kernel-only ceiling per menu shape, engine per shape vs OpenBLAS, the step-down boundary, an (mc, kc) arm) for icelake / genoa / rome, plus the suite subset. The job also prints its CPU confinement and BLAS thread count before timing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…namic dispatch
The TensorOperations adapter paid a fixed ~1.1 us (Float64) / ~5.5 us
(ComplexF64) per tensorcontract! call on tiny problems -- 4-8x / 13-23x
StridedBLAS on the upstream suite's batched dim 4-16 cases, 8-15x on the
tiny blas-equivalent ones. Profiled on ccqlin038 (Cascade Lake, Julia
1.12.7), the 8x8x8 GEMM: execute! was 0.23 / 0.41 us; the rest was planning.
Root causes, in order of size:
* Every plan re-derived the default kernel shape and blocking row from
the TargetProfile through `Val(profile.isa)` -- a dynamic dispatch on a
runtime Symbol, chained through the override row, the rule, the fitted
shape, the small-M rule and the blocking model. _default_kernel(C64,8,8)
cost 1.2 us and plan_contract called it TWICE (the swapped-orientation
candidate, dead for complex T); default_blocking(kernel) another 0.3-0.4
us. Float64's were 75 + 95 ns.
* The M/N/K label lists were Vector{Int}s (five allocations, ~370 B) and
the composite ranks therefore runtime values, so the _plan_contract
barrier was a dynamic dispatch that boxed every non-isbits argument
(~800 B more) -- even for Float64, whose kernel Union alone would have
union-split.
* For a complex T the kernel Union spans the whole menu (ten members), so
even with everything else concrete the barrier stayed a jl_apply_generic
that missed the method cache far more often than not (profiled: ~15% of
the call in subtype.c), and execute!(plan) was a second such dispatch
over a Union-typed, boxed plan.
* _qs_task_workspace's lazy-build branch was inferred as the abstract
ContractWorkspace, making the adapter's own call into the planner
dynamic too (three boxed StridedViews + tuples + executor per call).
Changes:
* src/planning/defaults.jl (new): ResolvedDefaults, cached per (profile,
eltype) in one slot per supported eltype, keyed by === on the immutable
TargetProfile (so a profile written after load, as forced_isa_runner.jl
does, takes effect without a hook). _default_kernel and
default_blocking(kernel) move here and read the cache; _default_shape
returns the (shape, method) choice as plain values.
* labels.jl / pair_group.jl: label lists are NTuple{D,Int} with D fixed by
the label tuples' lengths (_group_ranks: NA = M+K, NB = N+K, NC = M+N);
_build_pair_group is a single static method. Allocation-free.
* plan.jl: plan_contract's body is _planned(f, ...) with a continuation
applied inside the barrier; the kernel is resolved for an automatic
choice through _with_menu_kernel (kernel_selection.jl), a generated
if-ladder over the closed menu that calls the continuation with a
CONCRETE kernel in every arm -- no menu-wide Union is ever dispatched
on. Everything the barrier needs travels in one _PlanRequest. The
swapped-orientation candidate is resolved only for real T.
* tensoroperations.jl: the default-allocator tensorcontract! passes an
executor as the continuation (plan built and run in one typed frame),
and _qs_task_workspace is concretely typed on both branches.
Semantics are unchanged: same kernel choice, same blocking, same swap and
run-length demotion (pinned by the existing planning tests, updated only
for the Vector -> Tuple representation). Measured (same host, medians of
15 x 2000 calls through tensorcontract!(..., QuasiStridedBackend()), pooled
workspace):
Float64 8x8x8 1.10 us / 1232 B -> ~0.5 us / 0 B (StridedBLAS 0.15)
Float64 4x4 outer 0.79 us / 1072 B -> ~0.3 us / 0 B (StridedBLAS 0.22)
ComplexF64 8x8x8 5.50 us / 2032 B -> 0.67 us / 0 B (StridedBLAS 0.43)
ComplexF64 4x4 outer 5.06 us / 1776 B -> 0.43 us / 0 B (StridedBLAS 0.33)
ComplexF64 16^3 6.90 us / 2032 B -> 1.98 us / 0 B (StridedBLAS 1.5-2.0)
plan_contract itself: 0.62 -> ~0.22 us (Float64), 3.58 -> 0.24 us
(ComplexF64). Full test suite green (137144 passed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The FMAddSub kernel is what the AVX-512 small-M demotion selects for every
complex contraction with Qm < 24, so its store sits on the per-call floor
of exactly the tiny problems where the floor matters. Its only store was
the scalar interleaved reader (1m's), ~3 ns per element through
_axpby_tile!: 46% of an 8x8x8 ComplexF64 execute! (0.19 of 0.41 us).
The accumulator is already in Complex{R}'s native [re, im, ...] order, so
the fast path needs no interleave shuffle: one W-real accumulator vector is
W/2 consecutive destination elements, and alpha*R + beta*C is computed
lane-wise with addsub / fmaddsub on the same Base Complex expression trees
_axpby_tile! reaches (bitwise equal on the vectorized lanes; the straddling
row tail goes through _axpby_at!). Gated by _complex_vector_eligible, the
same predicate as planar's vector store; everything else keeps the scalar
reader.
Tests: new test/microkernels/test_fmaddsub_store_fastpath.jl (values
bitwise vs an independent reference and within tolerance vs the scalar
reader over every menu shape and edge extent, alpha/beta contracts, guard
band, allocation-free); the existing reader test now uses scattered rows
so it still exercises the scalar path. Full suite green (181764 passed).
Measured on ccqlin038 (Cascade Lake, Julia 1.12.7), interleaved, two rounds,
whole tensorcontract!(..., QuasiStridedBackend()) calls, scalar-only store
vs this one (benchmark/probes/probe_call_floor.jl, medians of 15 x 2000):
ComplexF64 8x8x8 0.68 -> 0.53 us (StridedBLAS 0.43)
ComplexF64 16x16x16 1.97 -> 1.35 us (StridedBLAS 1.46)
ComplexF64 dim-4 outer 1.45 -> 0.84 us (StridedBLAS 1.00)
Also adds the two probes used for this work: probe_call_floor.jl (public
API, runs against any checkout's benchmark env, for A/B) and
probe_call_breakdown.jl (prefix / workspace / planning / execute! split).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12ae2b4 sorted K by stride in whichever operand had the smaller minimum K stride. That regresses TRG's plaquette contraction, A(24^5)[x,k1,y,z,k2] * B(24^3)[k2,w,k1]: B holds the unit-stride K axis, so the rule sorted by B and made A's fastest K step 24^4 elements (2.6 MB) -- a page-crossing, unprefetchable walk over the 8M-element operand -- to save refetches of a 110 KB B that lives in L2 whatever the order. Measured on ccqlin038 (Cascade Lake, pinned, benchmark/probes/probe_network_korder.jl, which now times indA order / 12ae2b4's rule / this model interleaved): trg_plaquette chi=24 Float64, pairwise: 12ae2b4 rule 1.69x slower than indA order; cost model keeps indA order (0.996x). Network: 1.23-1.29x vs 1.02x. trg_plaquette chi=32 ComplexF64, pairwise: 1.31x vs 0.996x; network 1.10x vs 1.00x. `_order_contract_labels` now picks among indA order, A-sorted and B-sorted by `_k_order_cost` summed over both operands: n * walk * amplification, where walk = 3 when the order's fastest K axis steps more than a page, and amplification = elements per line along the operand's smallest-stride axis u when u is a K axis that is not the fastest, the slivers are not whole lines, and the lines in flight before u advances exceed the core's L2 share (`_l2_core_bytes`, from the detected cache profile). Ties keep indA order. Which suite cases change (benchmark/probes/probe_korder_changes.jl, now with --sources, run at the benchmarked leg dims): only contract_scrambled layouts -- dim16/24 {0_2_3,3_2_3}, dim32 {2_2_2,1_3_1,0_2_3}, dim63/96 {2_2_2,1_3_1}. dim96_1_3_1_contract_scrambled plans the same K group as 12ae2b4 (0.60 s, was 4.56 s). No TCCG, batched, gemm_ready, a/b_permuted or both_permuted case changes; small scrambled cases whose operands fit L2 (dim8, dim16_1_3_1) keep indA order. Tests: the planner pins now drive `_order_contract_labels` with an explicit L2 size (flip at l2 = 0, keep at 1 MB), pin the TRG shape to indA order, check the plan-level flip and swapped orientation at an L2-exceeding size, and run the correctness sweep at that size so the reordered path is the one executed. Full suite green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntensli_7 cause submit_scrambled_verify.sh part 1 now runs every synthetic size the K-order cost model reorders (16/32/63/96) and the :network cases (TRG is the shape it must keep; its decision reads the core's L2 share, 512 KB on Rome). Part 2 runs intensli_7/intensli_6 stage breakdowns under variants that separate the candidate causes of the Rome-only intensli_7_dim16 slowdown: dims 12/24 (non-power-of-two controls), --pad 1 (dim-16 extents with no power-of-two stride), --nothp 1 (4 KB pages only), --msplit-pos 2 (pack A reuses each A line from L1), and dim 32 with and without padding. probe_stage_breakdown.jl gains --pad P (A and C stored with the first axis padded to dim+P in the parent) and --nothp 1 (A and C in an anonymous mmap with MADV_NOHUGEPAGE). On ccqlin038 (Cascade Lake, AVX2 tile --shape 8,6,4) neither moves intensli_7_dim16 much (pack A 2.84 -> 2.87 ns/elem padded, 3.52 without THP), so a large effect on Rome would localise the cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e it
The Rome-only intensli_7_dim16 slowdown has a twin on ccqlin038 (Cascade
Lake): intensli_7 at dims 24 and 32, C[e,c,b,f,a] = A[a,b,c,d,e] * B[d,f].
The M composite follows C (e,c,b,a), so a pack-A sliver gathers 8 values of
e (A stride dim^4) over kc steps of d (A stride dim^3), one element per
cache line, and a line's other elements are needed only a full M pass later.
When those strides are multiples of 4 KB the whole sliver panel aliases in
set-indexed structures and pack A collapses to about one memory latency
per element. Padding every axis of A (no 4 KB-multiple stride, same walk)
removes it:
ccqlin038, --shape 8,6,4, pinned (probe_stage_breakdown.jl):
intensli_7 dim24 0.421 s, pack A 52.3 ns/elem
intensli_7 dim24 --pad 1 0.082 s, pack A 6.3 ns/elem
pack-A walk alone (probe_pack_order.jl, ns/elem, aligned -> padded):
dim16 2.83 -> 2.47, dim20 5.34 -> 5.21, dim24 46.3 -> 6.95,
dim28 11.4 -> 9.8, dim32 52.0 -> 10.1
dim20 and dim28 (strides not 4 KB multiples) never collapse. Which aligned
dims trip depends on the machine (dim16 on Rome, 24/32 here), and neither
THP (--nothp 1: 2.84 -> 3.52 ns/elem at dim16) nor walking the block K-outer
(helps dim24 46 -> 21, hurts dim20/28) is the lever.
probe_pack_order.jl times the pack-A walk alone (precomputed offsets, plain
loads) in the shipped sliver order and K-outer order; probe_stage_breakdown
--pad now pads every axis (padding only the first left dim^3*(dim+1)*8 a
4 KB multiple at dim 32). The verification job runs both at dims
16/20/24/28/32 so Rome and Genoa confirm or refute the aliasing diagnosis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in src/planning/plan.jl, the swapped-orientation block: kept
perf/call-overhead's structure (`_plan_request` / `_plan_with_kernel`, the
kernel resolved after the orientation is fixed) and built the swapped K
group from the scrambled branch's `korder` instead of `klabels`, so the
swapped orientation reuses the chosen K order with the maps exchanged.
The scrambled branch's K-order cost model (src/planning/labels.jl) is
still written against Vector{Int} label lists, which call-overhead turned
into NTuples; the port follows in the next commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
perf/scrambled-strides wrote `_order_contract_labels` / `_choose_k_order`
/ `_k_order_cost` against Vector{Int} klabels/morder/norder;
perf/call-overhead made those lists statically sized tuples so planning
allocates nothing and the group ranks are compile-time constants. After
the merge the planner called the Vector methods with tuples. This ports
the model without changing a single decision:
* `_sort_labels_by_stride` is `_order_free_labels` (the same stable,
allocation-free insertion sort) over another operand.
* `_choose_k_order` gathers each operand's facts once, in `klabels`
order (`_KOperand`: K extents, |strides|, element count/size, whole-line
slivers), and represents the three candidates (indA order, A-sorted,
B-sorted) as permutation tuples of 1:DK, so costing one is plain tuple
indexing instead of a `findfirst` per axis visited. Same candidates,
same comparison order, same strict `<` (ties keep indA order). The
result is an NTuple{DK,Int}: the K group type stays concrete.
* Zero or one K label returns at a compile-time branch before anything is
computed; when both sorted candidates equal indA order (every
gemm_ready/a_permuted/b_permuted layout) it returns before any cost is
evaluated or the L2 share read.
* `_k_order_cost` tests `share == 1` as `2*su*S > 64` (equivalent: `u` is
never a singleton) so its runtime division is paid only on the return
that uses the value.
* The core's L2 share is resolved once per (profile, eltype) into
`ResolvedDefaults.l2_core` (`_l2_core_bytes(profile)`, read from the
detected cache profile) and read lazily via `_l2_core_bytes(T)`; the
label-keyed `_k_order_cost` and `_l2_core_bytes()` stay for probes.
Checked: every K-order decision for the upstream suite (all :contract
sources at synthetic dims 4..96, TCCG 8/16/24, batched 4..64, four dtypes,
planning only; plus every pairwise contraction of the :network mps/ctmrg/
trg cases <= 2 GiB, Float64/ComplexF64) is identical to
perf/scrambled-strides, K group strides included (958 decisions; only
contract_scrambled layouts reorder: dim16/24 {0_2_3,3_2_3}, dim32
{2_2_2,1_3_1,0_2_3}, dim48 {2_2_2,1_3_1,0_2_3}, dim63/96 {2_2_2,1_3_1};
TRG and every gemm_ready/permuted/TCCG/batched case keep indA order). A new
test replays the scrambled branch's label-keyed model verbatim against the
port on 500 random layouts (187 of them reorder), and another pins the
result as an inferred, allocation-free NTuple.
Per call (ccqlin038, pinned, interleaved against perf/call-overhead,
probe_call_floor.jl, medians of 4 runs): 0 B/call everywhere; Float64
8x8x8 523 vs 511 ns, d4 outer 535 vs 542; ComplexF64 8x8x8 517 vs 540,
d4 outer 848 vs 855 (all single-K: no model work). Multi-K tiny calls pay
the model: plan_contract +15-20 ns for a 2-label gemm_ready d4 case, +55-85
ns for a 3-label scrambled d4 case (of ~0.3 us planning, ~1.3-1.9 us
per call).
probe_network_korder.jl's override follows the tuple signature.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
perf/large-gemm-peak (developed against 3a00fbd) changed `_default_kernel(T, Qm, Qn)` in src/planning/kernel_selection.jl, which perf/call-overhead had moved to src/planning/defaults.jl as the cached `_default_shape` / `_default_kernel`. Resolution: * kernel_selection.jl header: large-gemm's MV = 4 rule and three demotions, pointing at defaults.jl for the per-call entry points, plus call-overhead's paragraph on the cache. * `_extent_shape(profile, T, method, Qm)` keeps its signature (the tests drive it with synthetic profiles) and now delegates to a shape-taking form, `_extent_shape(shape, T, method, Qm)` -- pure, inlined, no `Val(profile.isa)` dispatch. * `_default_shape` applies that form to the cached `ResolvedDefaults.shape` before the `Qm < mr` demotion, exactly where large-gemm's `_default_kernel` applied it; the MV = 4 shapes reach the kernel through `_with_menu_kernel`'s ladder, which is generated over `kernel_shapes`, and the blocking row through `_real_blocking_row`, which reads `_derived_shape` as before. Checked against a build of perf/large-gemm-peak on this host (ccqlin038, AVX-512): `_default_kernel(T, Qm, Qn)` and `default_blocking` agree for all four dtypes, Qm = 0..300, Qn in {1, 8, 256}; full plans (kernel, blocking, orientation, M/N/K groups) agree for every :contract suite case at the dims used by the K-order check, except the K group of the 56 contract_scrambled plans the K-order model reorders. Full suite green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed cost of one micro-tile dominates small contractions, and two
things inflated it: the real and planar `store_tile!` were not inlined into
the tile loop, and every per-tile/per-pack prologue carried inline
`throw(ArgumentError("...$m..."))` checks whose string interpolation pulled
formatting code into the hot function.
* `store_tile!` for the real (SIMD) and planar kernels, and
`unsafe_pack_a!`/`unsafe_pack_b!`, are now `@inline`. The FMAddSub
store stays out of line: its store was scalar only (~64 `_axpby_tile!`
calls at `(8,8)`) and inlining it measured slower (ComplexF64 16^3,
tiles 1210 -> 1596 ns).
* The prologue checks call `@noinline` helpers (`_throw_tile_extent`,
`_throw_negative_kc`, src/microkernels/interface.jl) instead of
throwing inline; a GUARDRAIL comment there keeps it that way.
Measured on ccqlin038 (Cascade Lake, Julia 1.12.7): the 16x6 Float64 tile's
fixed cost went from ~37 to ~26 ns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The register kernels consume B one broadcast scalar at a time, so B needs
no layout to be read unpacked. `_execute_nest!` now skips the B pack and
hands the kernel an `UnpackedBView` (per N sliver: NR column base offsets
plus the panel's K axis; padding columns alias the last valid one, so every
address is inside the hoisted B bounds check) whenever `_use_unpacked_b`
says so: an eligible kernel (SIMD, planar, FMAddSub -- not 1m or scalar),
Qm <= 256, and a K composite that is an affine ramp with B step +-1.
The kernels read B through `_b_step_load` / `_b_step_load2`, whose default
methods are the packed-panel formulas (dispatch on the panel type, so the
packed paths compile to the same code as before). Results are bitwise
identical to the packed path.
Why: at small M the O(nr*kc) scalar gather per (jc, pc) block is not repaid
by M-sliver reuse. Measured on ccqlin038 (Cascade Lake, Julia 1.12.7),
`execute!` only, interleaved medians, packed -> unpacked (OpenBLAS mul!):
Float64 16^3 482 -> 361 ns (171) 32^3 1.93 -> 1.36 us (0.72)
64^3 9.8 -> 8.4 us (5.8) 128^3 64.6 -> 55.0 us (62.4)
ComplexF64 16^3 1.56 -> 1.65 us (1.35, within noise) 32^3 6.9 -> 5.8 us (6.3)
64^3 32.8 -> 29.1 us (33.7) 128^3 242 -> 215 us (219)
Strided-K B (B stored N-fastest) stays packed: 1.2-2.1x losses there.
`_UNPACKED_B_MODE` (:auto/:always/:never) overrides the rule for tests and
benchmarks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a degenerate free extent the register microkernel is the wrong tool:
it vectorizes along M, so at M == 1 every A vector is 1/mr full, and with
the roles swapped (N == 1, or the planner putting the matrix operand's free
composite in M) the matrix operand goes through a pack that gathers its
K-major rows one element at a time -- that gather dominated the suite's
`C[cde] = A[ab] B[abcde]` cases.
`execute!` now tries `_try_execute_dot!` first when Qm == 1 or Qn == 1. It
applies when the MATRIX operand's K map is the unit ramp (the vector
operand may have any K layout: it is gathered once per K block, with its
conjugation folded in), its storage is a DenseVector, and Qk >= one vector.
It computes NB outputs at a time (8 real, 4 complex), W-wide FMAs along
K, the K tail as one masked step, one horizontal reduction per output;
complex uses two accumulator planes against the gathered vector and its
pair-swapped copy (no shuffles in the loop). A conjugated matrix operand
becomes a flipped gather plus an output conj. The gather buffer is the
plan's packed-A workspace; every address is bounds-checked per block
before use. `_DOT_MODE` overrides the choice for tests and benchmarks.
Measured on ccqlin038 (Cascade Lake, Julia 1.12.7), `execute!` only,
interleaved medians, nest -> dot (OpenBLAS gemv):
Float64 1x36x216 4.9 -> 1.9 us (1.0) 1x64x512 18.9 -> 4.2 us (3.4)
1x144x1728 176 -> 81 us (76) 1x256x4096 728 -> 331 us (323)
ComplexF64 1x36x216 12.1 -> 2.6 us (2.4) 1x64x512 41.7 -> 7.6 us (8.6)
1x144x1728 304 -> 157 us (152) 1x256x4096 1359 -> 680 us (679)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through the nest a K = 1 contraction is all fixed cost: every micro-tile runs one K step and a full store, cld(M,mr)*cld(N,nr) of them (176 tiles for a 128x128 Float64 outer product at 16x6), for an output a plain streaming write covers in a fraction of the time. `execute!` now tries `_try_execute_outer!` when Qk == 1: for a real T whose M composite is the unit ramp in both A and C, DenseVector A and C storage and Qm >= one vector, each C column is `alpha * (A .* B[n]) + beta * C`, W rows at a time with a scalar row tail -- no packing, no tiles. The arithmetic is the nest's term for term (bitwise equal except on a signed zero). N and B may have any layout. Complex is left to the nest: it is already within ~10% of OpenBLAS from 63x63 up. `_OUTER_MODE` overrides the choice for tests and benchmarks. Measured on ccqlin038 (Cascade Lake, Julia 1.12.7), Float64, `execute!` only, interleaved medians, nest -> outer (OpenBLAS): 16x16 319 -> 227 ns (84) 63x63 2.40 -> 1.16 us (0.81) 128x128 6.15 -> 2.70 us (3.07) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-B paths Times `execute!` with each path's mode override set to :never (the nest) and :auto, interleaved rep by rep, plus OpenBLAS on the equivalent dense matrices; the numbers in the three path commits come from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inlined into `_execute_nest!`, `_micro_tiles_unpacked_b!` grew the nest for every plan, including packed-B ones: ~+4% on TCCG ccsd_2 dim16 Float64 (K = 256, B packed) on ccqlin038, 3-tree interleaved comparison. Out of line (one call per (jc, pc, ic) block) the packed plans are back to the pre-unpacked-B time and the unpacked 16^3/32^3 matmuls are unchanged (~305 ns / ~1.2-1.3 us). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs bench_degenerate.jl and the bench_to_suite.jl :contract subset the unpacked-B/dot/outer paths target, with submit_bench_to_suite.sh's julia resolution (rocky8 reservation fallback). Not submitted from here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in src/microkernels/fmaddsub.jl (store_tile! docstring only; neither side inlines it): keep call-overhead's description of the vector and scalar store paths, plus degenerate's note on why it stays out of line, marked as measured before the vector store existed. Full suite on this tree: 189763/189763 (Julia 1.12.7). Known issue, not yet fixed: first-call latency and test time regress sharply (see the follow-up). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…spatch barriers
The first `@tensor backend = QuasiStridedBackend()` call in a fresh process
had grown from ~3 s (main) to 28 s (Float64) / 87 s (ComplexF64) on
integration/perf, and the test suite from ~14 to ~121 min
(benchmark/probes/probe_ttfx.jl, ccqlin038, Julia 1.12.7). SnoopCompile on
the first ComplexF64 call: 4146 method instances inferred, 85 s of
inference, 40 instances of `execute!`/`_execute_nest!` (the unrolled kernel
ladder's 10 menu kernels x 4 conj/identity transform pairs, every arm
compiled although one runs), 320 of `_dot_nest!` (40 plans x 4 lane widths
x 2 matrix sides) and 40 of `_micro_tiles_unpacked_b!`. Float64: 6 plans
(3 kernels x 2 M/N orientations), 26 s of inference, of which the nest alone
was 9 s (1.5 s per instance, ~15x main's: every `Union` sliver axis split
into up to four inlined copies of the tile at each pack and tile call).
Fix: every runtime choice is made BEFORE a dynamic call and passed through
it as a singleton, so inference stops at the call and only the callee
actually reached is compiled. The call boxes nothing
(src/execution/barrier.jl): static choices cross as singletons (`Val` of the
shape, the method, the transforms, a path marker), the operand storages as
themselves (already heap objects), and everything else in a typed slot
(`Base.RefValue{P}`) owned by the workspace and reused across calls. No slot
holds an operand array, so a pooled workspace retains no user data.
* Kernel barrier (`_plan_with_kernel`, src/planning/plan.jl): the
automatic `(shape, method)` -- now including the run-length demotion,
applied on shapes (`_demote_shape_for_run`) -- is turned into a
`Val(shape)` by a menu ladder (`_menu_val`) and the plan is built in
`_plan_resolved`, specialised on that one kernel and transform pair.
Replaces `_with_menu_kernel`/`_plan_demoted`.
* Path barrier (`execute!`, src/execution/execute.jl): `_select_path`
picks the dot path (`_DotPath{matB, W}`), the outer path (`_OuterPath{W}`)
or the nest (`_NestPath{unpacked_b, aff}`) -- the same rules, in the same
order, as before -- and `_execute_path!` runs only that one. The TO
adapter's continuation (`_Execute`) predicts the path before the kernel
barrier (`_path_hint`) so a TensorOperations call crosses ONE barrier;
the prediction is checked against the plan (`_hint_holds`) before use.
`_try_execute_dot!`/`_try_execute_outer!` are gone; `_dot_applicable`,
`_outer_applicable`, `_use_unpacked_b` keep their meaning.
* Static sliver axis types (`_NestPath`'s `aff`): a map that is an affine
ramp makes every block descriptor regular, so its sliver axis is always
an `AffineAxis` (`_axis_of(..., Val(true))`) and the `PtrScatterAxis` arm
is not compiled. Also for the half-packed nest (`_HalfPackPath`).
* Tile loops out of line (`_micro_tiles_packed_b!`, `_half_packed_micro_tiles!`
now `@noinline`, like `_micro_tiles_unpacked_b!`): inference/LLVM cost is
superlinear in function size; test_halfpack.jl inference 207 -> 152 s.
* `ContractWorkspace` is a `mutable struct` with `const` fields: one heap
object, so plans, slots and the task pool hold a pointer instead of
copying 21 GC-tracked fields. This alone took the 8x8x8 call through
`tensorcontract!` from 0.47 to 0.38 us (Float64) and 0.54 to 0.45 us
(ComplexF64).
* `_default_shape` returns a concrete `RealMethod` for real `T` (the
FMAddSub small-M demotion is complex-only), and `_path_hint` takes the
method's unpacked-B eligibility as a `Bool`: a hint call union-split on
the method was emitted out of line and boxed the request (256 B/call).
Measured, ccqlin038 (Cascade Lake), Julia 1.12.7, cores 4-7:
probe_ttfx.jl integration/perf -> this (main)
using 0.6 s -> 0.5 s
first Float64 call 28.0 s -> 2.7-2.9 s (3.0 s)
first ComplexF64 call 87.0 s -> 2.0 s (3.0 s)
first K=1 outer 10.1 s -> 0.5 s (2.0 s)
SnoopCompile, first call alone (inferred MIs total / in QuasiStrided):
Float64 1062 / 686 -> 521 / 254 (inference 26.8 -> 2.4 s)
ComplexF64 4146 / 2895 -> 806 / 265 (inference 85.3 -> 2.7 s)
(the first call of a process, whichever eltype, also infers ~1300 more
instances of TensorOperations/Base code, ~1.3 s, as on main.)
Package precompile 3.2 -> 3.4 s, pkgimage 1.88 -> 2.18 MiB.
Pkg.test(): 121 min -> 15.1 min, all 189763 pass
probe_call_floor.jl (median ns/call, 3 interleaved runs each, 0 B/call
both):
Float64 8x8x8 465 -> 370 16^3 643 -> 510 d4 outer 465 -> 351
ComplexF64 8x8x8 522 -> 462 16^3 1355 -> 1171 d4 outer 816 -> 714
Warm throughput (same host, interleaved): dim63_2_2_2_gemm_ready Float64
80-84 -> 84-85 GF/s; dim96_1_3_1_contract_scrambled 0.422 -> 0.424 s;
benchmark/bench_degenerate.jl outer/dot/unpacked-B paths unchanged
within run-to-run noise.
Julia 1.13.0: full suite passes (10.5 min). Julia 1.10.12: the same 62
failures and 1 error as integration/perf itself (all in tests: allocation
assertions not skipped below 1.11, a test depending on a skipped call's side
effect, `sort(::Tuple)`), fixed separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
integration/perf (and the previous commit, unchanged) failed 62 tests and
errored one on Julia 1.10.12, all in the tests themselves:
* test_plan_contract.jl: `sort(::Tuple)` is Julia 1.11+; sort a
collected vector (the default sort is stable) and rebuild the tuple.
* test_unpackedb.jl: the steady-state allocation check is skipped below
1.11, and a skipped `@test` does not evaluate its expression, so the
product check after it saw an unwritten C. Measure outside the `@test`.
* test_plan_contract.jl (K order, 32 B/call) and
test_fmaddsub_store_fastpath.jl (FMAddSub store, 80 B/call): allocation
assertions that allocate on 1.10; skipped below 1.11 like the suite's
other allocation assertions.
Runic also re-indented one `for` header in test_plan_contract.jl.
Pkg.test(), ccqlin038: Julia 1.10.12 184058 pass / 183 broken (skipped)
in 11.8 min; Julia 1.12.7 189763 pass in 15.1 min.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Layout only; the FormatCheck workflow runs Runic over the whole tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI (lts, ubuntu-latest) failed "CPUID probe is an independent fallback": `_isa_rank(probe) <= _isa_rank(table)` evaluated `2 <= 1`, i.e. the live CPUID probe read AVX-512 while `_UARCH_ISA[Sys.CPU_NAME]` said AVX2. Julia 1.10's CPU detector (LLVM 15) has no `znver4` -- the name is absent from its libjulia-internal, present from 1.11 on -- so it names AMD Zen 4 and Zen 5 hosts "znver3". GitHub's ubuntu runners include Zen 4 EPYCs; the same test passed on Julia 1 (1.13.1), which names the CPU correctly. The TEST was wrong, not the code: the table entry is right for genuine Zen 3 (AVX2 only), and `_detect_isa` already takes the narrower of name and probe, so such a host safely gets AVX2 shapes. The assertion exists to catch a table entry too narrow for its microarchitecture, which a stale name is not. It is now skipped only for "znver3" below Julia 1.11 (still enforced there on 1.11+ and for every other name), and a new assertion checks for every listed name that `_detect_isa` returns the narrower of the two. The "runs on this host" testset also logs the detected profile, CPU name and probe, so a host-dependent failure can be read off the CI log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS CI (both Julia versions) failed "K order: plan_contract flips contract_scrambled at an L2-exceeding size": the plan kept indA order (40, 40, 5) instead of reordering to (5, 40, 40). The case's footprint was fixed at b*c*e*64 B = 40*40*40*64 = 4.096 MB, commented "beyond every measured core's L2 share". GitHub's macos-26-arm64 runner is a 3-vCPU virtual M1 whose sysctl reports hw.perflevel0.l2cachesize = 12582912 and hw.perflevel0.cpusperl2 = 3, so `_l2_core_bytes` = 4 MiB = 4.194 MB: the lines fit, and the model correctly keeps indA order. The TEST encoded an x86 L2 size; the detection and the model are right. `_cache_topology_darwin` reads the cluster L2 and its sharing count, and `_l2_core_bytes` divides it (a real M1/M2 with 4 cores per cluster gives 3/4 MB); undetected L2 falls back to 1 MB. `_KO_BIG`'s e extent is now derived from the detected share (never below the old 40, so x86 hosts run the unchanged case), and the flip test asserts that premise. The correctness testset sharing `_KO_BIG` thereby also exercises the reordered K on every host, as its "(reordered K)" comment intends. test/forced_isa_runner.jl gains optional QS_FAKE_L1D / QS_FAKE_L2 / QS_FAKE_L2_SHARING knobs (unset: the all-zero levels it used before) to model such a host. Verified on x86 by installing the runner's profile (:neon, 16 B, 32 regs, L1d 128 KB, L2 12 MB / 3): the old test reproduces exactly the six CI failures, the new one passes there, with sharing 4 (real M1) and with sharing 0 (undetected: 12 MB share, e = 123). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS CI failed `!_outer_eligible(plan_contract(_outer_maker(T, W - 1, 9,
1)()...))` for Float64. W = `_dot_lanewidth(Float64)` = 2 on NEON (16 B
vectors), the same W `_outer_applicable` uses, so both agree on "fills one
vector". But M = W - 1 = 1 makes C a 1 x 9 matrix whose N axis is
unit-stride too, so the planner swaps the roles (as for :cpermC) and the
printed plan has M = 9 (mgroup lengths (9,)), unit-stride in both B and C:
a genuine outer product, correctly eligible. On AVX-512/AVX2 M = 7/3 is not
a singleton, so no swap -- the test encoded W > 2.
The TEST was wrong. The below-a-vector check now uses a (W-1) x (W-1) C
(no long N to swap to), and the (W-1) x 9 case asserts the bound against the
plan's actual M. Correctness of the swapped path on a length that W does
not divide (vector body plus scalar tail) is added to the "matches the nest,
the oracle and a reference" testset: C = 1 x 9, all four alpha/beta
regimes. Verified on x86 under a forced NEON profile (vector_bytes = 16,
so the path runs at W = 2 with Vec{2}): passes there and on the native
AVX-512 profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s slivers PR #10's MV = 4 AVX-512 rule regressed ccsd_t_2/4 and ao2mo_2 (dim 16/24, Float64) by 1.5-2.5x. Mechanism (plan dumps, benchmark/probes/ probe_mv4_plans.jl; stage breakdown, probe_stage_breakdown.jl): C's only unit-stride run in these cases is 16 or 24 long and lies on the N side as given. At MV = 2 the swap decision (_prefer_swap: run_n >= mr_swapped) cleared it with mr = 16 and fed that run to M; with mr = 32 it declined, so the plan kept M on a stride-16 C axis and every tile went through the scattered store (ccsd_t_2_dim16: kernel+store 97% of the time, 9.5 vs 23.5 GF/s). ao2mo_2 is the same, the as-is orientation having stepped down to MR = 16 on Qm = 16 while the swapped one was compared at 32. _store_shape: the MV = 4 shape steps down to its MV = 2 sibling when C's leading unit run along M is at least one half tile but does not make every tall sliver unit-stride (Qm != run, run % MR != 0). It is applied per orientation inside _default_shape (which gains a run argument, default Qm = no layout known), so _candidate_mrs feeds the swap the tile each side will really run, and again in _plan_with_kernel with the chosen side's run. Static per plan, no allocation, the shape still crosses the kernel barrier as one Val. Measured (probe_mv4_store_sweep.jl, named kernels, Cascade Lake): a broken tall sliver loses to a whole half-height one by up to 5x at shallow K and is still behind at Qk = 256-1024 (Float64 run 16: 0.95-0.98; Float32 run 32: 0.80 at 256, 0.98 at 512); where run is MR-aligned MV = 4 wins at every Qk measured. probe_mv4_regress.jl, ccqlin038 (us, fix / main): ccsd_t_2 dim16 24167 / 27760, ccsd_t_4 dim16 22603 / 26830, ao2mo_2 dim16 81 / 99, ccsd_t_2 dim24 350602 / 483285, ao2mo_2 dim24 771 / 802 -- the plans are now identical to MV = 2's for all five. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the upstream-suite driver, the general engine benchmarks, the two acceptance probes (probe_ttfx.jl, probe_call_floor.jl) and one generic Slurm submit script (submit.sh) in place of the per-experiment ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ches bench_kernels.jl times every menu kernel (all complex methods), the default plan and OpenBLAS per dtype and shape, plus the L1-resident tile ceiling, and reports the complex efficiency ratio from the default rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add FMAddSubKernel and the automatic dot/outer/unpacked-B paths, fix the ncon example and the unqualified engine calls, point to bench_to_suite.jl, drop measurement and history notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither was reachable from plan_contract or contract!: the direct scalar loop never beat execute!, and unpacked B already covers small M. Drops their sources, tests, includes and the half-pack dispatch table, and trims the barrier prose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measurement anecdotes and history go; comments keep only the reasons for non-obvious choices. The plan-level _dot_applicable/_outer_applicable/ _use_unpacked_b wrappers and _has_oracle had test callers only. Code paths, dispatch barriers and the mode switches are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One brute-force reference, one matmul fixture maker with operand layouts, and one fresh-plan runner serve every path test. The fixed multi-block driver cases fold into the randomized real/complex agreement loop, which now also checks the oracle. Paths are selected through _select_path, and forced onto the nest through the mode switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ents Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… MV = 4 fixes On Genoa (znver4) the MV = 4 tile buys nothing where it was meant to: PR #10 head vs main, bench_to_suite jobs 7124412 / 7112089, dim32/dim63_2_2_2 (all layouts) run at 0.97-0.99x main's time, consistent with the MV = 4 rationale (the MV = 2 tile is front-end bound on Intel; a double-pumped Zen4 core is not). The same comparison regresses there, and only there, mps_*_D64 (1.69-1.73x), dim12/dim16_2_2_2_gemm_ready (1.24-1.33x) and trg_plaquette_chi16/24 (1.15-1.27x), all of which improve on Ice Lake (0.51-0.97x). Those plans have MR-aligned C runs (mps D64: 64/768/4096), so the C-run step-down does not reach them; on Cascade Lake MV = 4 wins on the same per-plan shapes (probe_mv4_store_sweep.jl at mps D64's 4096x24x6 / 768x64x128: 1.09-1.14x). So the fix for genoa is not the store rule: `_profile_mv` returns MV = 2 for cpu_name znver4 (and, unmeasured, znver5), i.e. main's selection there. Intel hosts are unchanged. Not node-verified from here (no job submission). benchmark/submit_mv4_fix.sh runs probe_mv4_plans.jl in the four selections (mv2 = main, mv4,nostore = PR head, mv4, fix) on the regressed cases, three interleaved rounds, plus bench_large_gemm_peak.jl (engine per named shape) on the node. probe_mv4_plans.jl: combinable --mode flags, --cases all / --reps for a whole-suite plan diff, runs against main's tree too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k ~ 200) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Docstrings only on the public kernel constructors; internal functions get at most a short comment. Measurement tables, host/date anecdotes and development history removed. - Merge the identical 1m and fmaddsub scalar tile readers into one `_store_tile_lanepair!`. - One constructor check `_check_vector_shape` (width, evenness, divisibility) and one generator-time accumulator check `_check_acc`. - Drop the test-only `planar/onem/fmaddsub_register_pressure` and the two-method form of `_execute_tile_prologue!`; inline the LLVM type-name helpers of `_fmaddsub`. Emitted native code of accumulate, store_tile!, and the driver's micro-tile barriers is unchanged for every checked shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matches the base's structure exactly for the per-call path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`mk_contract` (test_scalar_kernel.jl) checks the shared kernel contract with test-local packers and accumulator readers: accumulate vs A*B, kc == 0 and split kc, every alpha/beta branch with NaN old C at beta == 0, partial tiles, nonfinite padding lanes, scattered/negative-stride/strided/SubArray/scattered- column destinations, rejected inputs, and zero allocations on scattered and dense destinations for every shipped shape. The two store fast-path files share `mk_store_fastpath`. Kernel files keep only their own edge cases (lane primitives, instruction selection, 1m's inner kernel, InterleavedFormat packing, selection and blocking) and one end-to-end conj run each. Dropped: register-pressure formula tests, trivial accessor checks, adjacency and cross-kernel agreement sweeps covered by the reference comparison. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fpack mention Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…val) `_a_step_offset` existed only so `UnpackedAView` (src/execution/halfpack.jl) could read A in place. With halfpack deleted in the execution area, the K step calls `packed_a_offset` directly; the emitted native code is unchanged. On this branch alone, halfpack.jl's `_a_step_offset` method is now dead, so test/execution/test_halfpack.jl fails until that deletion is merged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pers Removes the inert _unbroken_fraction guard, the kernel-form _demote_for_run, the profile-form _extent_shape/_small_m_shape, the label-list _prefer_swap, the label-keyed _k_order_cost, _sort_labels_by_stride, the l2 variant of _order_contract_labels and the Val-keyed default_blocking methods. Shares one menu ladder between _kernel_from_shape and _menu_val, and sorts free labels through _sort_perm. Plan decisions are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A same-node A/B on genoa showed MV = 4 is faster on the mps networks and ties MV = 2 on large GEMMs, so AVX-512 CPUs no longer special-case AMD. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One import list in runtests.jl for the shared test scope, instead of three partial ones. Fold the fmaddsub store fast-path test into the planar one, whose helper it uses. avecs_per_column had no callers left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lkdvos
changed the title
Restores the exception dropped in eb4987e. Two independent genoa full-suite runs showed the 32x6 tile slower on mps D64, 12-16 GEMMs, ctmrg chi100 and batched dim 128, while large GEMMs tie; the earlier same-node mps A/B was unreliable because those timings are bimodal per process. Plans are now identical to the pre-simplification tree on every profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
Final node runs (full upstream suite, single core, vs the pre-PR baseline)Jobs 7126245 (icelake), 7126251 (genoa), 7126252 (rome) on QuasiStrided time, new / baseline (geomean; < 1 = faster):
QuasiStrided / StridedBLAS time, baseline → now (geomean):
Remaining outliers: rome ComplexF64 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates the performance work found by re-running the upstream TensorOperationsBenchmarks suite against StridedBLAS on icelake, genoa and rome, then simplifies the whole codebase before merge.
Performance changes
contract_scrambledup to 3.7× slower (pack B walked K with huge strides)Measured on nodes (first round of this PR vs the pre-PR baseline, QuasiStrided time new/old): batched 0.27–0.58, synthetic 0.61–0.73, TCCG 0.70–0.88 on icelake/genoa/rome. Icelake large Float64 GEMM 65.6 → 95.7 GF/s (OpenBLAS 101.8). A short-K Float64 regression from the taller tile (
ccsd_t_2/4,ao2mo_2at dim 16/24, 1.5–2.5×) was traced to the swap decision and fixed; a same-node A/B confirms the affected cases are back to the 16×6 tile's time on icelake and genoa, with the large-GEMM gain kept.Simplification
Behaviour-preserving; comments now only explain unexpected choices, with no machine/run/history references.
src/test/benchmark/execute_direct!andexecute_half_packed!(opt-in, never selected, not public) and dead helpers.benchmark/pruned to the suite driver and plots, a shared harness,bench_kernels.jl,bench_blocking_model.jl,bench_degenerate.jl, the first-call and call-floor probes, and one genericsubmit.sh.Checks
Follow-ups (separate PRs)
perf/pack-alias-4k(4 KB-aliasing collapse when packing, up to 18× on rome) andperf/avx2-complex(spill-free AVX2 complex kernels, 1.09–1.52× on rome).🤖 Generated with Claude Code