[cuda] tolerance-based gain tie-break in best-split reductions

[maxwbuckley]

Summary

CPU's best-split finder scans bins ascending and uses strict >, so on a true gain plateau (multiple bins with mathematically equal split gain) the lowest-index bin wins. CUDA's parallel reduction has the same intent, but its histogram aggregation introduces ~1e-15 relative FP noise from the order of atomic adds. That noise flips which bin from the plateau has the slightly-higher numerical gain on CUDA, so the parallel argmax adopts a different bin than CPU's exact computation.

The threshold-encoding mismatch is cosmetic at round 1 — training data routes identically — but compounds through score updates and surfaces as structural tree divergence around round 3 in cases like reg_bagging and multi_dense from the parity sweep. It also affects TreeSHAP and out-of-sample prediction, which depend on the encoded threshold value rather than the routing decision.

The fix

ReduceBestGainWarp and ReduceBestGainBlock in cuda_best_split_finder.cu now use a small relative tolerance (1e-12, ~5000× fp64 epsilon) when comparing gains. Two gains within the tolerance band are treated as tied, and the reduction prefers the lower thread index — which corresponds to the lower bin index, matching CPU's "first bin in scan order wins" behaviour.

The tolerance is sized to absorb the observed ~1e-15 reduction-order noise without crossing any genuine gain difference seen in practice. The condition for adopting the other thread's value becomes:

const double tol = fmax(fabs(gain), fabs(other_gain)) * 1e-12;
const bool other_better = (other_gain > gain + tol)
                       || (fabs(other_gain - gain) <= tol && other_thread_index < thread_index);

Impact (CPU/CUDA parity sweep, 53 cases)

The sweep applies tolerances RAW_TOL = 1e-5 for predictions, THRESH_TOL = 1e-6 for tree thresholds, LEAF_TOL = 1e-5 for leaf values.

metric	before	after
Predictions exactly bit-identical (max|Δ| == 0)	8/53	8/53
Predictions within 1e-5	37/53	40/53
Tree structure within 1e-6/1e-5 tolerance	5/53	38/53
Both	5/53	38/53

The dominant effect is on tree structure: 33 more cases now have CPU and CUDA picking the same bin from the gain plateau, so the encoded threshold values agree (within THRESH_TOL). The fix does not change the bit-identical-predictions count — remaining FP-epsilon-magnitude prediction differences come from parallel-reduction noise in leaf-level grad/hess sums, which is a separate phenomenon untouched by this PR.

Three cases that previously diverged structurally now match within FP epsilon (rather than 1e-1 magnitude):

case	before max|Δ|	after max|Δ|
reg_bagging	3.87e-1	4.4e-16
reg_pred_contrib	4.83e-2	1.1e-16
reg_oos	4.49e-1	1.4e-17

The 13 "cosmetic threshold-encoding" cases identified in the prior parity investigation (e.g. reg_dense, reg_huber, bin_dense, multi_softmax_lr, multi_w, reg_dart, reg_goss, bin_goss, rank_xendcg and others) all move from "predictions match at FP epsilon, trees encode different thresholds" to "predictions still match at FP epsilon, trees match within THRESH_TOL."

The remaining 13 prediction-divergent cases have unrelated root causes (weighted percentile open from prior PRs, quantized training covered by Felix's PR #1, LambdaRank atomicAdd_block ordering, hybrid CPU objectives like gamma/tweedie/fair).

Test plan

• Parametrized regression test added in tests/python_package_test/test_dual.py::test_cuda_split_gain_tie_break_matches_cpu covering bagging, plain dense, max_depth, and lambda_l2 configurations. Each was known to produce a different bin from a gain plateau prior to this fix.
• Test gated on TASK=cuda per the existing convention.
• All 4 cases pass on RTX 5090 / CUDA 13.2 / sm_120.
• No regressions on the 5 cases that were already clean.
• Verified that 13 cosmetic-threshold cases from the broader parity sweep now produce trees that agree within THRESH_TOL.

🤖 Generated with Claude Code

[BelixRogner]

Thanks Max — and Claude Code. Sharp diagnosis: CPU's sequential scan with strict > gives a deterministic "first bin in scan order wins" on a true gain plateau, and CUDA's parallel reduction picks up ~1e-15 ULP noise from atomic-add order in histogram aggregation that flips which bin from the plateau has the slightly-higher numerical gain. Cosmetic at round 1, but compounds via score updates into structural tree divergence by round 3 — and affects TreeSHAP / OOS predictions since they encode the threshold itself.

The fix is correct in structure: OtherIsBetterWithTieBreak compares with relative 1e-12 tolerance (~5000× fp64 epsilon, well above the ~1e-15 noise floor, well below any plausible real gain difference) and tie-breaks to the lower thread index. With the monotone thread→bin mapping in this reduction, that recovers CPU's "first bin wins" semantics. Applied symmetrically to ReduceBestGainWarp and ReduceBestGainBlock.

Impact (tree-structure match 5/53 → 38/53, three structurally-divergent cases drop from 0.05–0.4 max|Δ| to fp64-epsilon) is the right kind of result.

Tests cover the failure modes; gated correctly on TASK=cuda. Comment block on the tolerance choice will help future maintainers.

Merging once CI lint completes (still in flight at time of comment).

[BelixRogner]

Quick rebase nudge — #7 and #8 just landed on master and touched files this PR also modifies (cuda_algorithms.hpp / cuda_best_split_finder.cu / etc.), so this branch now shows a merge conflict on GitHub. One more git merge master && git push round should clear it; CI is otherwise green (the apparent failures yesterday were all environmental — dask socket flakes, a cancelled job rolling up, and a Boost-headers wheel-build issue, none of which were touching the actual PR content).

Ready to merge as soon as the conflict's resolved.