Retry tiled LLM.int8 outlier detection with an empty-mask fast path

[heiheiha798]

Salvage context

This is the single permitted salvage attempt for issue #14, following its B300 no-go result. It starts independently from the latest upstream main; the source experiment branch is evidence only and must not be continued, rebased, or stacked.

The source experiment replaced eager O(rows * K) LLM.int8 outlier discovery with a private tiled FP16 CUDA detector that writes an O(K) int32 column mask. On B300, the detector's core mechanism was exact and fast whenever outliers existed:

• 320 baseline/candidate comparisons were exact for quantized rows, row statistics, ordered CUDA int64 outlier columns, and unchanged inputs.
• All 80 one/five/sparse/dense direct cells favored the detector by at least 23.6%; their pattern-median speedups were 1.46x, 1.47x, 1.46x, and 1.43x.
• Threshold-enabled Linear8bitLt improved 1.21x for decode, 1.17x for a batched case, and 1.09x for prefill; a resident decode stack improved 1.06x.
• Prefill peak allocation fell from about 67.1 MB to 16.8 MB.

The complete gate failed for one localized reason: the candidate always called dynamic argwhere(mask), even when the mask was empty. Every direct regression over 3% was a no-outlier cell (15 of 20); the no-outlier median was 0.925x and the worst cell was 0.802x. The resident prefill stack was also flat at 0.9965x.

Bounded hypothesis

Keep the measured detector unchanged at 256 columns per CTA and a fixed 32-row tile, but have that same kernel also produce one O(1) device found flag:

1. Initialize the int32[K] mask and one device int32 found scalar to zero.
2. When a detector thread finds an outlier, set its column flag and set found to one with race-free, idempotent device operations.
3. After the detector, evaluate only the O(1) found scalar in Python.
4. If false, construct the existing empty CUDA int64 tensor without calling argwhere.
5. If true, call the existing argwhere(mask).view(-1) path.

This deliberately reintroduces one host-visible scalar synchronization, but it does not restore the eager O(rows * K) abs, boolean mask, or global reduction. The experiment must determine whether avoiding empty argwhere fixes no-outlier regressions without erasing the measured nonempty and public-path gains.

Exact scope

• Use the same private FP16 tiled detector and O(K) mask design from the source experiment.
• Add only the O(1) found output and empty-mask branch.
• Keep row tile 32 and 256 columns per CTA fixed. Do not retry row tiles, add architecture/shape/data thresholds, or introduce a dispatch table.
• Use an idempotent atomic or equally local race-free aggregation inside the same kernel. Do not add a separate reduction kernel.
• Keep threshold == 0, kInt8VectorQuant, quantization math, row statistics, multi-row output zeroing, mixed matmul, schemas, autograd, default backend, and public API unchanged.
• Keep the current upstream FP16 scope. Do not copy or stack the unmerged BF16 work in upstream PR Add BF16 support for int8_vectorwise_quant / LLM.int8 activation quant bitsandbytes-foundation/bitsandbytes#1985.
• Preserve HIP/ROCm source and build compatibility; B300 supplies the performance decision, not permission to break shared sources.

Expected production surface remains limited to the existing CUDA backend integration points: csrc/kernels.cu / .cuh, csrc/ops.cu / .cuh, csrc/pythonInterface.cpp, bitsandbytes/backends/cuda/ops.py, and focused tests.

No fusion, public schema, wider INT8 redesign, new architecture, dtype expansion, general fallback framework, or second bounded fix is in scope.

Correctness gates

Build isolated baseline and candidate variants from the same latest upstream commit. Require exact torch.equal results for quantized rows, row statistics, and ordered outlier columns, including the empty CUDA int64 dtype/device/rank contract. Inputs must remain raw-bit unchanged.

Repeat the source matrix, including:

• rows around the 32-row tile through 2048;
• K around 256-column boundaries plus 3584, 4096, 8192, and 14336;
• thresholds 0.1, 2.0, 3.0, and 6.0;
• no, one, repeated-column, five, sparse, and dense outliers;
• exact positive/negative threshold values, immediate-below values, first/last rows and columns, NaN, and Inf;
• actual candidate public-op provenance and repeated launches.

threshold == 0 must still return outlier_cols is None. Threshold casting must match current A.abs() >= threshold semantics. Run the focused vectorwise/COO tests, threshold-enabled Linear8bitLt tests, applicable fullgraph/dynamic-output compile tests, supported CUDA builds, shared-source compatibility checks, and full pre-commit.

B300 validation and unchanged acceptance gate

Use one B300 allocation and separately extracted baseline/candidate builds with identical software and the official CUDA 13 x64-compatible target input 75;80;86;89;90;100;120. Assert the exact loaded libraries and distinct native entry points. Interleave at least 30 host-wall samples per cell after at least 200 warmups, synchronizing after calls so dynamic-output cost is included.

Measure the original direct matrix over rows {1, 8, 64, 512, 2048}, K {3584, 4096, 8192, 14336}, and no/one/five/1%-sparse/dense patterns. Keep no-outlier, dense, and repeated-column cases as veto cells: the scalar branch can add host latency, and dense inputs can contend on the shared found flag.

Profile decode and prefill to show that the candidate still removes the O(rows * K) eager intermediates and global reduction. Report the scalar synchronization honestly, along with launches, allocation, bandwidth, occupancy, and atomic behavior. The source detector used more temporary memory than baseline for decode because int32[K] exceeded the tiny rows=1 eager intermediates; do not hide or redefine that tradeoff.

Required public evidence remains:

• threshold-enabled Linear8bitLt for decode and batched/prefill;
• the same reproducible resident synthetic stack for decode and prefill, with setup and weight quantization excluded.

Proceed to a draft PR only if every correctness/build gate passes and all original performance gates hold:

• at least four representative direct cells improve by at least 10%;
• no primary direct cell regresses by more than 3%;
• Linear8bitLt improves by at least 5% in decode and in at least one batched/prefill case, with no primary Linear cell regressing by more than 2%;
• both resident-stack regimes improve by at least 3%, with no regime regressing by more than 2%;
• profiling confirms the intended O(rows * K) work and temporary removal.

If the scalar synchronization erases the nonempty/public gains, dense flag contention causes a veto regression, either stack misses its gate, or any correctness/build check fails, record a final no-go. This salvage cycle receives no further retry or derivative.

Technical story and risks

The substantive result remains the tiled native detector that replaces several eager full-tensor passes and materially reduces prefill allocation. The found flag is a bounded completion of that kernel/system method, not a standalone micro-optimization.

Primary risks are explicit: the host-visible scalar check may consume the current 9.2% prefill-layer margin, atomics to one flag may hurt dense inputs, and decode still has low detector occupancy and a larger mask allocation than the eager baseline. The unchanged gates decide whether the completed design is broadly useful enough for upstream.

[heiheiha798]

Final result: NO-GO on B300

The single permitted salvage attempt completed in Slurm job 5056. The bounded found-scalar fast path preserves exact results and most of the source detector's gains, but it still fails two unchanged acceptance gates. No production candidate, commit, branch push, or draft PR was produced.

Environment and provenance

• Baseline and experiment source: clean upstream/main commit 95f9af309d4d5793847169c39288dcd3fcbdf564 on the independent local branch b300/int8-outlier-mask-empty-fastpath.
• Job/node/GPU: Slurm 5056, dev-slurm, NVIDIA B300 SXM6 AC, runtime CC 10.3, 148 SMs.
• Software: driver 580.126.09, CUDA 13.0.88 toolchain, PyTorch 2.13.0+cu130, Python 3.12.3.
• Build input: official CUDA 13 x64-compatible targets 75;80;86;89;90;100;120. Both isolated builds resolved them to 75-real;80-real;86-real;89-real;90-real;100-real;120.
• Baseline and candidate were extracted into separate fresh source trees. The benchmark asserted distinct private entry points and that the candidate public op loaded /tmp/bnb-int8-outlier-found-5056/benchmark-build/source-candidate/bitsandbytes/libbitsandbytes_cuda130.so.
• Build times were 59.15 s baseline and 59.72 s candidate. Library sizes were 3,922,064 and 3,926,736 bytes, a 4,672-byte increase.
• Timing was synchronized host wall time, with variants alternated after 200 warmups and 30 samples per cell. This includes the dynamic-output and scalar synchronization costs.

The job command was:

sbatch benchmark-evidence/cycle12/slurm_sm103_int8_outlier_found.sh

The benchmark invocation recorded by the job was:

python benchmark-evidence/cycle12/sm103_int8_outlier_found.py \
  --output benchmark-evidence/cycle12/sm103-int8-outlier-found-job-5056.jsonl \
  --build-root /tmp/bnb-int8-outlier-found-5056/benchmark-build \
  --compute-capability '75;80;86;89;90;100;120' \
  --warmup 200 \
  --samples 30

Raw workspace evidence:

• benchmark-evidence/cycle12/slurm-5056-int8-outlier-found.out
• benchmark-evidence/cycle12/sm103-int8-outlier-found-job-5056.jsonl
• benchmark-evidence/cycle12/sm103-int8-outlier-found-job-5056-decode.ncu.csv
• benchmark-evidence/cycle12/sm103-int8-outlier-found-job-5056-prefill.ncu.csv

The raw log ends with SM103_INT8_OUTLIER_FOUND_CALIBRATION=COMPLETE. Slurm accounting was unavailable because the local accounting service refused its connection, so the raw terminal marker is the completion record.

Correctness and mechanism

All 340 exact correctness records passed. Baseline and candidate matched with torch.equal for quantized rows, row statistics, and ordered CUDA int64 outlier columns, including the empty tensor contract. Input storage remained raw-bit unchanged. Coverage included the source boundary/threshold/NaN/Inf matrix plus repeated-column and repeated-launch controls, and the actual candidate public-op library provenance was asserted.

The candidate kept the fixed 256-column CTA, row tile 32, int32[K] mask, and one int32 found scalar. Torch profiling confirmed that candidate discovery still removes aten::abs and aten::any; it performs the disclosed one-scalar host synchronization and calls argwhere only for a nonempty mask.

• Decode profile, rows=1/K=3584: peak allocation increased from 14,848 to 20,992 bytes because the O(K) mask remains larger than this tiny eager input.
• Prefill profile, rows=2048/K=8192: peak allocation fell from 67,125,760 to 16,843,264 bytes while the O(rows*K) eager intermediates remained absent.
• NCU decode sparse: 14 CTAs, 256 threads/CTA, 6.240 us kernel duration, 10.16% achieved occupancy.
• NCU prefill dense contention control: 2,048 CTAs, 256 threads/CTA, 48.640 us kernel duration, 82.87% achieved occupancy. Dense and repeated-column timing controls did not produce the veto regressions; most retained material wins.

No HIP/ROCm toolchain was available (HIP_TOOLCHAIN=UNAVAILABLE), so this experiment makes no HIP compile or runtime validation claim.

Performance result

The candidate retained broad direct wins: 26 representative cells improved by at least 10%. However, the hard direct gate required zero primary regression above 3%, and four no-outlier cells failed it:

rows	K	baseline median (ms)	candidate median (ms)	baseline/candidate
64	4,096	0.07055	0.07316	0.9643x
64	8,192	0.05678	0.06065	0.9361x
64	14,336	0.05903	0.06234	0.9469x
512	3,584	0.07318	0.07581	0.9654x

Public-path medians were:

workload	baseline/candidate	gate
Linear8bitLt decode	1.0976x	pass, >=5%
Linear8bitLt batched	1.0880x	pass, >=5%
Linear8bitLt prefill	1.0308x	below 5%, but batched satisfies the combined batched/prefill requirement
resident synthetic stack decode	1.0787x	pass, >=3%
resident synthetic stack prefill	1.0259x	fail, below 3%

There were no Linear8bitLt or resident-stack regressions beyond their 2% veto limit. Nevertheless, the four direct no-outlier regressions and the resident prefill stack miss are independently terminal under the fixed contract.

Disposition

The scalar fast path solves most of the empty-mask penalty from issue #14, but not consistently enough to meet the no-regression gate, and its recurrent prefill benefit remains below the required 3%. A new shape/data heuristic, a second synchronization strategy, or another bounded fix would violate the one-attempt salvage contract and would make the production policy more complex than the evidence supports.

This result is therefore final NO-GO for salvage id bitsandbytes:source-cycle-11:int8-outlier-mask-empty-fastpath. The issue remains open as an evidence record. There is no production edit or draft PR, and no recursive salvage candidate.