Evaluate multi-warp CTAs for SM103 4-bit quantization

[heiheiha798]

Problem

The default CUDA FP4/NF4 quantization path uses block size 64. On current
main, that path launches kQuantizeBlockwiseSmall<T, 64, ...> with one
32-thread warp per CTA, and that warp processes one 64-value quantization
block. Large weight tensors therefore launch one one-warp CTA for every 64
values.

The kernel already separates work into independent logical quantization
groups through THREADS_PER_QB, qb_id, per-group WarpReduce storage, and
per-group absmax storage. A multi-warp CTA could process several independent
64-value blocks while preserving the current per-block reduction order and
packed output layout. On B300/SM103, this may reduce CTA scheduling overhead
and avoid the residency ceiling of one-warp CTAs. This is a measurement-gated
hypothesis about model quantization and conversion time, not a performance
claim and not a decode-time GEMM optimization.

Scope

• Baseline:
  bitsandbytes-foundation/bitsandbytes@95f9af309d4d5793847169c39288dcd3fcbdf564.
• Experiment with the existing one-warp launch and fixed four-/eight-warp CTA
  variants for the existing FP4/NF4 kQuantizeBlockwiseSmall path at block
  sizes 64 and 32. Experimental forcing remains private benchmark scaffolding.
• If evidence is positive, select one coherent CTA width only on exact runtime
  compute capability 10.3. All non-SM103 CUDA devices and HIP must retain the
  current one-wave launch and behavior.
• If the selected width loses only below a clear crossover, allow at most one
  evidence-backed num_blocks threshold under the same exact-SM103 guard.
  Do not add dtype-, quant-type-, or shape-specific dispatch rules.
• Keep the current quantization arithmetic, logical-warp reduction order,
  quantile maps, absmax layout, packed format, public API/schema, and CUDA
  wheel target lists unchanged.
• Implement device detection narrowly and cache it per current device using
  repository conventions. Include host dispatch overhead in public-path
  timing; do not introduce a general architecture-dispatch framework.

Expected production files are limited to csrc/kernels.cu,
csrc/kernels.cuh, and csrc/ops.cu, plus focused tests and a reproducible
benchmark if the experiment proceeds. This work must start independently from
the baseline above and must not depend on any unmerged fork branch.

Correctness and portability oracle

Build isolated baseline and forced variants from the same source, compiler,
flags, and target list. From identical deterministic inputs, require bitwise
equality with baseline for both packed bytes and float32 absmax values across:

• FP4 and NF4; FP16, BF16, and FP32 inputs; block sizes 64 and 32.
• Sizes 1, 15, 16, 17, 31, 32, 33, 63, 64, 65, immediately below/at/above
  each candidate CTA tile, and realistic model weight dimensions.
• All-zero values, signed zero, repeated maxima, alternating signs, random
  finite and mixed-magnitude values, and non-divisible final CTA groups.
• Repeated launches and allocating/public functional calls with
  compress_statistics=False and True; input buffers must remain unchanged.

Run the relevant Test4bitBlockwiseQuantOps and
TestQuantize4BitFunctional coverage, compressed-statistics and roundtrip
tests, focused CUDA compute-sanitizer --tool racecheck and synccheck cases
for full and partial multi-warp CTAs, an official-compatible CUDA build, a HIP
compile check, and full pre-commit. The HIP path must remain one-wave; if a HIP
toolchain is unavailable, disclose that limitation rather than claiming HIP
validation.

B300 benchmark and acceptance

Use one B300/SM103 Slurm allocation and isolated clean builds. Primary deployed
evidence must use the official CUDA 13 x64 compatible target input
75;80;86;89;90;100;120; native 103 may be reported only as a separately
labeled corroborating control. Record the exact revisions, resolved target
list, node/GPU/driver/CUDA/PyTorch/Python identity, commands, environment,
build time, library-size delta, and raw JSONL/log paths.

Interleave the one-/four-/eight-warp variants with CUDA events, at least 20
warmups and seven timed batches, enough repetitions for stable medians, and
report all round samples plus median and p10/p90. The primary direct matrix is
FP4/NF4 x FP16/BF16/FP32 x block size 64; block size 32 is a control. Include
CTA-tail boundaries, 64K and 1M elements, and representative weights such as
4096x4096, 11008x4096, 14336x4096, and 8192x8192 where memory permits.

Also time bitsandbytes.functional.quantize_4bit on realistic resident weight
tensors at default block size 64 with compressed statistics off and on. Exclude
allocation and input creation from kernel timing, but include the exact-SM103
host selection overhead in public-path timing.

Proceed only if one CTA width:

• is bitwise correct and sanitizer-clean;
• improves at least two realistic large blocksize-64 direct cells by at least
  5%, with interleaved dispersion below the effect;
• produces a repeatable at least 3% gain in realistic public
  functional.quantize_4bit cells; and
• causes no greater-than-5% regression in adjacent large cells and no material
  regression in the selected small/tail region.

If no single width and at most one simple block-count crossover satisfy these
conditions, record a no-go. A direct-only, native-103-only, isolated-shape, or
dtype/format-specific win is not sufficient.

Non-goals and risks

This issue does not change dequantization, 4-bit GEMM dispatch, SIMT/MMA
selection, quantization formats, public APIs, other architectures, or HIP wave
geometry. It does not implement the separate K=2..5 API from upstream PR
bitsandbytes-foundation#1858.

The main risks are multi-warp CUB storage/tail participation, shared-memory or
register pressure, binary-size growth from extra template instantiations, and
small-tensor regressions. Quantization is primarily startup/conversion work, so
kernel speedup that disappears in the public path is a no-go.

[heiheiha798]

Calibration attempt result: aborted before performance measurement

This investigation did not produce a performance conclusion or a production candidate. The bounded calibration retry budget was exhausted by two private-harness failures, so the cycle is being recorded as aborted rather than as a kernel-performance no-go.

Both runs used the independent upstream baseline 95f9af309d4d5793847169c39288dcd3fcbdf564 on a single NVIDIA B300 SXM6 AC (runtime CC 10.3, 148 SMs), with isolated 1-, 4-, and 8-warp source trees built for the official CUDA 13 x64-compatible target input 75;80;86;89;90;100;120.

Job 5010

• Passed the B300/SM103 identity gate and built all three isolated variants.
• Completed 300 direct correctness records across FP4/NF4, FP16/BF16/FP32, block sizes 32/64, small/tail/realistic sizes, and deterministic input patterns.
• All completed records reported raw-bit-equal packed output and float32 absmax, with unchanged input.
• Failed before performance and sanitizer work because the private ctypes loader omitted the existing six-argument declarations for the unchanged General8 symbols used by compress_statistics=True. The resulting pointer marshalling error caused an illegal memory access. This is a harness setup failure, not kernel evidence.

Raw local evidence:

• benchmark-evidence/cycle9/slurm-5010-quant4-multiwarp.out
• benchmark-evidence/cycle9/sm103-quant4-multiwarp-job-5010.jsonl

Job 5017

• Repeated the same identity gate, isolated official-compatible builds, and 300 direct raw-bit correctness records after adding only the missing ctypes declarations.
• Again, every completed direct record was bitwise equal with unchanged input.
• Completed four uncompressed public functional.quantize_4bit correctness records (FP4/FP16 at 65, 513, 4097, and 1,048,576 elements), including the provided output-buffer control, with raw-bit-equal output/state and unchanged input.
• Reached the first compressed-statistics result without the prior illegal access, then failed in the private result comparator: PyTorch rejects changing a 0-D float QuantState.offset directly to a byte view (self.dim() cannot be 0). The run therefore stopped before performance and sanitizer measurement.

Raw local evidence:

• benchmark-evidence/cycle9/slurm-5017-quant4-multiwarp.out
• benchmark-evidence/cycle9/sm103-quant4-multiwarp-job-5017.jsonl

Disposition

No tracked source change, commit, pushed branch, or draft PR was created. No direct or public-path latency, dispersion, sanitizer, or acceptance-gate result exists, so these runs do not support either adopting or rejecting the multi-warp kernel hypothesis on performance grounds. A future investigation would first need to make the scalar raw-byte comparison shape-safe and then rerun the full predefined matrix under a new authorization.

The issue remains open as an unvalidated hypothesis.