Investigate SM103 SIMT-vs-MMA dispatch for fused 4-bit GEMM

[heiheiha798]

Problem

The fused CUDA 4-bit GEMM has a second dispatch layer in csrc/gemm_4bit.cu that chooses between its existing SIMT and MMA implementations. The current wide_n_simt rule covers SM89, SM120, and SM121 at M <= 6, but not SM103. SM103 is otherwise classified as an HBM architecture, so some small-M, wide-N calls can select MMA without direct B300 calibration.

This issue treats a better SM103 rule as a hypothesis to measure, not as an assumed performance win. It is independent of fork issue #1 / draft PR #2: the investigation and any acceptance claim must be based on current upstream/main, not on the unmerged Python-level dispatch change from cycle 1.

Scope

• Baseline: bitsandbytes-foundation/bitsandbytes@95f9af309d4d5793847169c39288dcd3fcbdf564.
• Measure the two existing internal implementations independently on B300, then compare them with current upstream automatic dispatch.
• Only if a coherent decision-changing region is demonstrated, add the narrowest SM103-only predicate or threshold to the existing dispatch in csrc/gemm_4bit.cu.
• Add a reproducible benchmark under benchmarking/ and focused dispatch/correctness coverage using existing repository patterns.
• Experimental forcing may use private benchmark/build scaffolding, but must not become a public API or require a new kernel.

The user-facing acceptance set must contain shapes where current upstream actually enters the fused kernel. Forced-path measurements on shapes that upstream routes to dequantize + F.linear may be controls, but cannot establish the merge benefit by themselves.

Correctness oracle

For representative target and adjacent control cells:

• compare forced SIMT, forced MMA where K % 64 == 0, and automatic public torch.ops.bitsandbytes.gemm_4bit against bitsandbytes.functional.dequantize_4bit followed by torch.nn.functional.linear;
• cover FP16 and BF16, NF4 and FP4, nested quantization on/off, bias on/off, and representative block sizes 32/64/128/256 using repository tolerances plus shape/dtype/finite checks;
• retain non-aligned K as an existing-SIMT legality/control case; do not force the MMA path where its alignment requirement is not met;
• run the relevant existing CUDA 4-bit tests and full pre-commit suite. Avoid extracting a new generic dispatch abstraction solely for testing if a focused decision enumeration and public-path tests are sufficient.

B300 benchmark plan

Use one B300 Slurm allocation and record job/node/GPU, driver/CUDA/PyTorch/bitsandbytes versions, exact commit/build identity, commands, environment, and raw JSONL/log paths.

• Target M={4,5,6}; use M={3,7,8} as adjacent controls.
• Include current-upstream fused-path shapes: for M=5-6, prioritize K < N at one or more 148-SM waves and shapes at or above the existing three-wave rule; for M=4, include the first cells above the current one-wave HBM SIMT clause.
• Sweep immediately below/at/above 148 and 296 MMA output-tile waves, plus realistic projections such as (N,K)=(11008,4096), (14336,4096), and a wide vocabulary projection if memory permits. Smaller square shapes may remain controls only when the public upstream selector does not enter the fused kernel.
• Interleave forced SIMT, forced MMA, and upstream automatic dispatch within the same allocation. Use at least 20 warmups and 7 rounds of 100 CUDA-event samples per variant; report median, p10/p90, round medians, complete samples, and correctness results rather than a best run.

Acceptance / no-go

Proceed to a source change and draft PR only if the data show a simple SM103-local region where:

• one internal path wins consistently across repeated rounds by more than the measured noise (use the repository's 5% decision band as the default materiality threshold);
• candidate automatic dispatch matches that winner;
• adjacent/control cells show no meaningful regression; and
• correctness and relevant tests pass.

If FP16/BF16 or neighboring shapes do not support one maintainable predicate, or the gain is within noise, record a no-go on this issue rather than add a heuristic or expand scope.

Non-goals

No new kernel or kernel-math change; no Python top-level custom-vs-fallback edit; no SM100 or other-architecture change; no CUDA wheel/build-target change; no public API; no dependency on the cycle-1 branch; no architecture generalization beyond evidence from B300.