Evaluate lane-pair nested-scale broadcast in CUDA SIMT 4-bit GEMM

[heiheiha798]

Hypothesis

The CUDA SIMT 4-bit GEMM currently reconstructs the same nested block scale twice for each adjacent lane pair when blocksize == 64.

Each lane handles 32 consecutive K values per outer iteration. With a 64-value quantization block, lanes (2i, 2i + 1) therefore use the same blk_idx. In the nested-statistics path, both lanes independently load the same uint8 scale index, codebook entry, second-level absmax, and offset. Params4bit and Linear4bit use blocksize 64 and compressed statistics by default, so this path is relevant to repeated small-M 4-bit inference.

This issue tracks a measurement-gated experiment. It does not assert a speedup in advance.

Proposed scope

• Change only the CUDA branch of csrc/gemm_4bit_simt.cu.
• For absmax_8bit != nullptr && blocksize == 64, let the even lane reconstruct scale_f and broadcast its exact float bits to the adjacent odd lane with a converged full-warp shuffle.
• Keep every lane participating in the shuffle, including inactive lanes in the final K group.
• Preserve the existing code verbatim for non-nested statistics, other blocksizes, and the preprocessor-separated HIP path.
• Do not change the public API or ABI, quantization format, GEMM launch geometry, Python dispatch, SIMT-vs-MMA selection, or add architecture/shape-specific fallback rules.

The lane-pair invariant depends on the existing custom-op alignment requirement: inner_k = group * 1024 + lane * 32, and public dispatch rejects K values not divisible by blocksize.

Correctness gate

Build isolated baseline and candidate libraries from upstream/main commit 95f9af309d4d5793847169c39288dcd3fcbdf564 with identical settings. Record and assert the actual native library loaded by each benchmark/test process.

Require raw-bit equality of the complete baseline and candidate outputs for:

• FP4 and NF4;
• FP16, BF16, and FP32 inputs/outputs;
• M = 1, 2, and 3, plus forced-SIMT M = 4 and 8 controls;
• bias absent and present;
• nested statistics with blocksize 64;
• K = 64, 128, 960, 1024, 1088, 4096, 4160, and 11008;
• N around partial output tiles (1, 3, 4, 63, 64, 65) and representative large dimensions.

Also verify that packed weights, activations, bias, nested indices/codebook/state2 absmax, and offset remain unchanged. Compare against the repository dequantize-plus-F.linear reference with existing tolerances, run the relevant gemm_4bit, matmul_4bit, compressed-statistics, bias, and offset-dtype tests, and repeat launches.

Run focused CUDA compute-sanitizer synccheck and racecheck cases. The structural requirement is that the full-mask shuffle is outside lane-active divergence and is reached by every lane in each live warp. Build the official CUDA 13 x64-compatible target list and compile the HIP path if the toolchain is available; disclose unavailable HIP validation rather than claiming it.

B300 benchmark gate

Use one B300/SM103 allocation with clean, isolated baseline and candidate builds. Primary deployed-code evidence must use the official CUDA 13 x64 target input 75;80;86;89;90;100;120; a native SM103 build may be reported only as a separately labeled corroborating control. Record the exact commit, resolved targets, loaded-library paths, node/GPU/driver/CUDA/PyTorch identities, commands, environment, job ID, and raw JSONL/log locations.

Interleave baseline and candidate with CUDA events, at least 20 warmups and at least seven timed batches with enough inner iterations for stable medians. Report every batch, median, p10, and p90.

Primary direct-op cells are nested blocksize-64 FP4/NF4 across FP16/BF16/FP32, M = 1/2/3, and realistic (N, K) families such as (4096,4096), (11008,4096), (4096,11008), (14336,4096), and (8192,8192), plus selected small/tail controls. Verify the SIMT implementation actually executes.

Controls must include non-nested blocksize 64, nested blocksize 128/256, and forced-SIMT M = 4/8. Measure a public bitsandbytes.matmul_4bit or Linear4bit forward path with default compressed statistics on realistic M = 1/2/3 shapes, excluding allocation and quantization from steady-state timing.

Proceed to a production change only if all correctness and sanitizer checks pass, at least two realistic direct nested blocksize-64 cells improve repeatably by at least 5% with dispersion below the effect, at least one realistic public-path cell improves repeatably by at least 3%, and no adjacent primary or source-equivalent control cell regresses materially (5% ceiling). Otherwise record a no-go without adding crossover heuristics, dispatch rules, prefetch changes, or a broader kernel rewrite.

Risks and non-goals

The duplicate reads may already coalesce or hit cache, while the shuffle adds an instruction and dependency. A no-gain outcome is plausible. The experiment must not be broadened to the HIP prefetch path, MMA kernels, blocksize-specific dispatch, other architectures, or the aborted multi-warp quantization work from issue #12.

[heiheiha798]

B300 result: NO-GO

The architecture-independent lane-pair reconstruction change is synchronization-safe and bitwise-correct in this experiment, but it does not provide a coherent performance win. It regresses adjacent FP16/BF16 primary cells by 6-12%, so there is no basis for a production change or draft PR.

Evidence provenance

• Baseline commit and clean branch head: 95f9af309d4d5793847169c39288dcd3fcbdf564 on b300/sm103-simt-nested-scale-broadcast.
• Job 5039 is setup-failure evidence only. It passed the B300 identity gate and built both isolated libraries, then the private harness failed before its first correctness record because its byte comparator did not handle the scalar nested offset.
• Job 5040 is the authoritative run. It used one NVIDIA B300 SXM6 AC (CC 10.3, 148 SMs), PyTorch 2.13.0+cu130, CUDA toolkit 13.0.88, and driver 580.126.09.
• Both variants were extracted independently from the baseline and forced through the existing SIMT dispatcher. Only the candidate tree reconstructed the nested block-64 scale in the even lane and broadcast it to the adjacent lane. Their native entrypoints were distinct.
• Primary deployed-code input 75;80;86;89;90;100;120 resolved in both builds to 75-real;80-real;86-real;89-real;90-real;100-real;120. The benchmark asserted that bitsandbytes loaded the isolated candidate library.
• Baseline/candidate build times were 59.56/59.53 seconds. Libraries were 3,922,032/3,942,512 bytes (+20,480 bytes for this fat build).
• Raw artifacts: benchmark-evidence/cycle10/slurm-5040-simt-scale-pair.out, benchmark-evidence/cycle10/sm103-simt-scale-pair-job-5040.jsonl, and the two compute-sanitizer-*-job-5040.log files.

Exact job command:

python benchmark-evidence/cycle10/sm103_simt_nested_scale_pair.py \
  --output benchmark-evidence/cycle10/sm103-simt-scale-pair-job-5040.jsonl \
  --build-root "$SLURM_TMPDIR/bnb-simt-scale-pair-5040/benchmark-build" \
  --compute-capability '75;80;86;89;90;100;120' \
  --warmup 20 --repetitions 50 --rounds 7

Each timing cell alternated baseline/candidate order, used 20 warmups per round and seven rounds, and retained every CUDA-event sample plus every round median in JSONL.

Correctness and synchronization

• 288/288 nested block-64 cases were raw-bit equal for the complete baseline/candidate output. Coverage included FP4/NF4, FP16/BF16/FP32, M=1/2/3, bias off/on, K=64/128/960/1024/1088/4096/4160/11008, partial N tiles, and realistic shapes.
• Activations, packed weights, bias, nested indices/codebook/state2 absmax, and offset remained raw-bit unchanged. All reference comparisons were finite and passed.
• All 72 primary direct, 36 direct control, and 24 public-path measurement records were also raw-bit equal.
• Focused candidate compute-sanitizer --tool racecheck covered full/tail NF4/FP4 probes over 40 launches and reported 0 hazards, 0 errors, and 0 warnings.
• Focused candidate compute-sanitizer --tool synccheck covered the same converged full-mask shuffle probes and reported 0 errors.
• ROCm/hipcc was unavailable on both the login and Slurm environments, so no HIP compile/runtime claim is made. The experimental source edit was confined to the CUDA preprocessor branch.

Performance

Representative median latency in milliseconds (candidate/baseline ratio; lower is better):

Path / shape	Dtype / M	Format	Baseline	Candidate	Ratio
direct 4096x4096	FP32 / 3	NF4	0.076736	0.072011	0.9384
direct 4096x4096	FP32 / 3	FP4	0.076736	0.071989	0.9381
direct 4096x11008	FP32 / 3	NF4	0.197579	0.183264	0.9275
direct 4096x11008	FP32 / 3	FP4	0.197579	0.183275	0.9276
direct 4096x11008	FP16 / 2	NF4	0.033717	0.035744	1.0601
direct 4096x11008	BF16 / 2	FP4	0.033717	0.035755	1.0604
direct 11008x4096	FP16 / 2	NF4	0.033739	0.037771	1.1195
direct 11008x4096	BF16 / 2	FP4	0.033728	0.037781	1.1202
public 4096x4096	FP32 / 3	NF4	0.076736	0.071979	0.9380
public 11008x4096	FP16 / 3	NF4	0.044683	0.045995	1.0294

Across the 72 primary nested/block-64 direct cells:

• Only 4 cells cleared the 5% gain threshold: FP32 M=3 on 4096x4096 and 4096x11008, for both NF4 and FP4. Their seven round ratios were stable (0.9275-0.9424).
• Eight adjacent primary cells regressed by more than 5%: FP16/BF16 M=2 on 4096x11008 (1.0601-1.0604) and 11008x4096 (1.1195-1.1202), for both formats. Those regressions were likewise stable across rounds.
• FP16/BF16 groups were generally parity-to-slower rather than showing the FP32/M=3 effect.

Only 2 of 24 public-path cells cleared 3%, both FP32 M=3 at 4096x4096 (0.9380-0.9385). The public 11008x4096 M=3 controls instead regressed by up to 3.06%.

Decision

This fails the required coherent-region and no-adjacent-regression gates. Restricting the change to the isolated FP32/M=3/shape cells would require the dtype/shape dispatch matrix explicitly excluded by this issue. Native-SM103 corroboration, repository test expansion, full pre-commit, and production edits were therefore skipped after the primary official-compatible performance gate failed.

No tracked source change, commit, push, or draft PR was created. The issue remains open as negative calibration evidence.

salvage_hint=NONE