Benchmark fused SM103 DP4A small-M INT8 scaled matmul

[heiheiha798]

Summary

This issue tracks a measurement-gated B300 experiment for the small-M CUDA bitsandbytes::int8_scaled_mm path used by Linear8bitLt.

Today the CUDA path materializes an INT32 output through cuBLASLt and then launches a second kernel to apply row/column scales, optional bias, and FP16 conversion. For token decode (M=1) and small batch/beam (M<=4), the extra launch, BLASLt setup, and 4*M*N-byte INT32 intermediate can be a disproportionate part of every 8-bit linear layer.

The hypothesis is that an SM103-only W8A8 DP4A GEMV kernel can compute the exact INT32 dot product and emit the scaled FP16 result in one launch, improving recurring LLM.int8 decode without changing public schemas, quantization state, or the exact INT32 matmul API.

This is an unmeasured hypothesis, not a performance claim.

Proposed bounded method

• Add one CUDA kernel in a dedicated source file. One warp owns one output feature for one activation row; lanes traverse contiguous K in packed four-byte chunks using __dp4a, then perform an integer warp reduction.
• Lane 0 applies the same CUDA epilogue as the existing kdequant_mm_int32_fp16 path: fmaf(dot * row_stat * col_stat, 6.200012e-05f, bias), followed by one FP16 conversion.
• Use a fixed, measured CTA warp count sufficient to expose transformer-scale N across B300's 148 SMs. Do not add persistent weight repacking, workspace, shape-specific kernel families, or serialization state.
• Register a CUDA implementation of bitsandbytes::int8_scaled_mm and select the fused path only when all of these hold:
  • exact runtime compute capability 10.3;
  • effective flattened M <= 4;
  • contiguous row-major INT8 A and B;
  • K % 4 == 0;
  • FP32 row/column statistics;
  • output dtype is FP16 (or omitted, preserving the existing FP16 default);
  • bias is absent or FP16.
• Every architecture, dtype, layout, shape, validation/error, and bias case outside that contract must execute the current int8_linear_matmul plus int8_mm_dequant composition unchanged.
• Preserve bitsandbytes::int8_linear_matmul, including its exact INT32 output/API, unchanged.

The initial B300 feasibility matrix is terminal. If DP4A does not win coherently across the fixed primary matrix, record a no-go rather than introducing per-shape rescue rules, weight transforms, tensor-core kernels, or a new dispatch framework.

Expected scope

• CMakeLists.txt: add one CUDA-only source.
• New csrc/gemm_int8_simt.cu and csrc/gemm_int8_simt.cuh: fused kernel and launcher.
• csrc/pythonInterface.cpp or a narrowly exported symbol from the new translation unit: one private launcher entry point.
• bitsandbytes/backends/cuda/ops.py: ctypes setup, CUDA registration, exact guard, and existing-composite fallback.
• tests/test_ops.py: parity, dispatch/fallback, stream, and edge coverage.
• One focused benchmarking/int8/ B300 harness if the existing drivers cannot isolate the operation.

No public API, schema, FakeTensor contract, quantization format, persistent state, or correction-GEMM change is expected.

Correctness oracle

For the selected path, require bitwise FP16 equality with the existing CUDA composition int8_mm_dequant(int8_linear_matmul(...)):

• M in {1,2,3,4} and flattened 2-D/3-D activation shapes.
• Representative and boundary K/N, including non-multiple N tails; K divisible by 4 on the selected path.
• Full-range/random/zero/extreme INT8 operands.
• Zero, finite, small, and large FP32 row/column scales.
• Bias absent and FP16 bias present.
• Repeated launches and a non-default current stream.

Independently require the integer dot to equal torch.matmul(A.int(), B.int().t()) for tested transformer dimensions, remaining within the non-overflow INT32 domain, and compare final output to the existing repository reference tolerance as a secondary oracle.

Fallback controls must prove existing behavior for M=5/8/16, K not divisible by 4, noncontiguous inputs, non-FP16 output, unsupported scale/bias dtype, non-SM103 devices/mocks, invalid/empty dimensions, and all existing error contracts. Preserve torch.compile and opcheck behavior because the public schema and fake registration are unchanged.

Run focused LLM.int8 tests, relevant test_ops/test_functional/Linear8bitLt coverage, official-compatible and native SM103 builds as available, full pre-commit, and compute-sanitizer memcheck/racecheck/synccheck on full/tail/extreme cases. All CUDA compilation and GPU execution must run through Slurm.

B300 feasibility and performance gates

Use isolated baseline/candidate builds from the same upstream commit in one B300 allocation, with resident precreated inputs/outputs, CUDA-event timing, at least 20 warmups, and at least seven interleaved timed batches. Report all batch samples plus median and p10/p90.

Phase 0: terminal raw gate

Compare fused versus the current two-operation composition for:

• M in {1,2,4};
• (K,N) in {(3584,3584), (4096,4096), (4096,11008), (4096,14336), (8192,28672)} subject to allocation memory;
• bias absent/present;
• M in {8,16} as verified fallback controls.

Proceed only if correctness is bitwise, the M=1..4 primary matrix has at least 1.10x geometric-mean speedup, no primary cell is below 0.98x, and every fallback control remains baseline-identical. A win confined to one shape or a result where B300 IMMA/cuBLASLt remains faster is a no-go.

Public-path gate

• Benchmark steady-state Linear8bitLt and a synthetic decoder stack with representative attention/MLP projections at threshold 0 and threshold 6 with realistic outlier columns.
• Exclude quantization/state construction and allocation from timing.
• Require at least 1.08x for the pure-INT8 stack and 1.03x for the default thresholded stack, with no memory growth or primary regression.
• Run the existing generation benchmark only if a model artifact is already locally available; model download is not required.

Profile one M=1 and one M=4 cell with the available profiler to prove that one fused launch replaces cuBLASLt plus dequantization, no hidden synchronization occurs, and bandwidth/occupancy support the claimed mechanism.

Risks and non-goals

• B300's tuned IMMA/cuBLASLt path may beat DP4A despite removal of the second launch and INT32 intermediate.
• Threshold-6 outlier discovery/correction can dilute base-path gains.
• Packed signed loads, N tails, and INT32 overflow boundaries require strict validation.
• Open upstream PRs touching adjacent INT8 dispatch and BLASLt handling may require a normal rebase, but this experiment must not copy or stack those branches.
• Do not add BF16 semantics, weight repacking, tensor-core/WGMMA implementations, descriptor caching, persistent workspace, correction-path fusion, shape-by-shape dispatch, or changes to int8_linear_matmul.

No current upstream issue or PR was found that implements a native CUDA W8A8 DP4A GEMV or fused small-M int8_scaled_mm path. Existing upstream performance reports motivate the user path but do not propose this kernel.

[heiheiha798]

Cycle 19 result: aborted during final validation

This cycle is PR_ABORTED, not a correctness or performance NO-GO. The single bounded validation retry was exhausted by focused test/setup defects before the required sanitizer, complete benchmark, and profiler gates could run. No branch was pushed and no draft PR was created.

Source state

• Baseline: 95f9af309d4d5793847169c39288dcd3fcbdf564
• Local branch: b300/sm103-int8-dp4a-small-m
• Final unpushed head: 062600a80b9468f2aca9c69a64afa7b59965d8a7
• The implementation remained within the issue scope: one private SM103 DP4A kernel, exact guarded CUDA routing, focused tests, and a tracked benchmark CLI.

Job outcomes

5218 built isolated baseline and candidate libraries with the official CUDA 13 x64-compatible target list and verified the candidate library/private symbol. It then stopped in focused pytest:

• six new cases attempted an independent CUDA torch.matmul over INT32 inputs, but this PyTorch build reports addmm_cuda as unsupported for INT32;
• the focused eligible-dispatch assertion also returned false in that test process;
• sanitizer, complete benchmark, Nsight Systems, and NCU did not run.

5219 was the one authorized replacement at exact head 062600a80b9468f2aca9c69a64afa7b59965d8a7. It again completed both isolated official-compatible builds and verified the loaded candidate library/private symbol. The test-only correction moved the small independent integer-dot oracle to CPU and isolated the cached device-property lookup. Results before the stop were:

• all six existing int8_scaled_mm cases passed;
• all six selected M<=4 DP4A cases passed the exact CPU INT32 dot oracle;
• those cases explicitly selected the SM103 route and passed raw-bit equality against the existing CUDA composition;
• non-default-stream execution and input/stat/bias immutability checks passed;
• the selected fullgraph compile case passed.

The remaining dispatch/fallback test then failed before its controls because _gpu_dispatch_props was a plain function without the test's expected .cache_clear() attribute at that boundary. This is a focused test-state defect. Under the fixed one-replacement rule, it was not repaired or rerun.

Evidence boundary

The earlier forced Phase-0 pilot showed promising isolated-kernel timing, but final validation never reached the required sanitizer, complete automatic-dispatch benchmark, Linear8bitLt/decoder-stack gate, Nsight Systems, or NCU stages. Therefore this cycle makes no production correctness, public-path performance, or merge-readiness conclusion.

Persistent raw evidence:

• benchmark-evidence/cycle19/slurm-5218-candidate.out
• benchmark-evidence/cycle19/slurm-5219-candidate.out
• benchmark-evidence/cycle19/phase0-job-5217.jsonl
• benchmark-evidence/cycle19/slurm-5217-phase0.out

The issue remains open for future independent investigation. No additional retry or scope expansion was performed.