[cuda] implement cost-effective gradient boosting on the CUDA tree learner

[maxwbuckley]

Problem: the delta

When training with device_type="cuda", the cost-effective gradient boosting parameters (cegb_tradeoff, cegb_penalty_split, cegb_penalty_feature_coupled, cegb_penalty_feature_lazy) were silently ignored. CEGB lives only in the CPU serial learner (CostEfficientGradientBoosting), which subtracts a per-split cost from each candidate's gain; the CUDA kernels had no penalty plumbing, and the CUDA learner had no "stop when best penalized gain ≤ 0" condition.

Measured delta before this PR (400×6 synthetic data, learning_rate=0.1, 5 rounds, gpu_use_dp=true):

case	per-tree leaves CPU	per-tree leaves CUDA	max pred delta
cegb_penalty_split=0.1	[8, 10, 4, 6, 6]	[31, 31, 31, 31, 31]	0.31
cegb_penalty_split=1.0	[1] (growth stops)	[31, 31, 31, 31, 31]	0.91
cegb_penalty_split=5.0	[1] (growth stops)	[31, 31, 31, 31, 31]	0.91
cegb_tradeoff=0.5, penalty_split=1.0	[2]	[31, 31, 31, 31, 31]	0.83

CPU prunes aggressively (down to a stump when the penalty exceeds all gains); CUDA grew full 31-leaf trees regardless — the penalties had zero effect.

Fix

Implements CEGB in the CUDA best-split finder:

• Split + coupled penalties: the cost delta tradeoff · penalty_split · num_data_in_leaf + tradeoff · coupled[real_fidx] (coupled term only while a feature is unused anywhere in the model) is subtracted from each task's finalized gain in the dispatch kernels — before cross-feature/cross-leaf comparison, exactly where CPU applies new_split.gain -= cegb_->DeltaGain(...).
• Feature-usage tracking: per-task coupled penalties live in a device array; the host mirrors CPU's is_feature_used_in_split_ (accumulated over the whole model, not per tree) and zeroes a feature's penalty entries the first time it's used in any split.
• Stop condition: FindBestFromAllSplitsKernel now requires gain > 0 to select a split, mirroring CPU's if (best_leaf_SplitInfo.gain <= 0.0) break. Without CEGB this is a no-op (valid splits always have positive gain); with CEGB the penalty can push every gain negative and growth must stop — this was the missing piece that let CUDA keep splitting at gain ≤ 0.
• cegb_penalty_feature_lazy: needs per-(row, feature) cost tracking (a device bitset + per-leaf scan); not implemented — now rejected with a clear Log::Fatal instead of being silently ignored.

The first commit's blanket guard is replaced: split and coupled penalties work on CUDA; only the lazy penalty is still rejected.

Result: the delta after the fix

Same measurement, post-fix:

case	per-tree leaves CPU	per-tree leaves CUDA	match	max pred delta
cegb_penalty_split=0.1	[8, 10, 4, 6, 6]	[8, 10, 4, 6, 6]	✓	0.0
cegb_penalty_split=1.0	[1]	[1]	✓	0.0
cegb_penalty_split=5.0	[1]	[1]	✓	0.0
coupled=[5,5,5,5,5,5]	[31, ...]	[31, ...]	✓	2.2e-16
coupled=[0.1, 0.1, 5, 5, 5, 5]	[31, ...]	[31, ...]	✓	2.2e-16
cegb_tradeoff=0.5, penalty_split=1.0	[2]	[2]	✓	0.0

Identical tree shapes and feature selection on every case; predictions match bit-for-bit (≤ 1 fp64 ULP).

Tests (in test_dual.py, gated on TASK=cuda)

• test_cuda_cegb_matches_cpu — 6 parametrized CEGB configs, asserting per-tree leaf counts, feature sets, and predictions (atol=1e-10) all match CPU. All fail on the unpatched build (CUDA grows 31-leaf trees); all pass with the fix.
• test_cuda_cegb_lazy_penalty_raises — the unimplemented lazy penalty raises a clear error.

Full test_dual.py suite (61 tests) passes.

Known limitation

CPU's UpdateLeafBestSplits retroactively re-evaluates cached splits in other leaves when a feature first becomes used (the coupled penalty disappears for those cached candidates). CUDA caches one best split per leaf and cannot recover a runner-up without a per-(leaf, feature) buffer; in adversarial coupled-penalty configurations this second-order effect could pick a different (still valid) split order. Not observed in any test configuration; documented for completeness.

🤖 Generated with Claude Code

[BelixRogner]

Thank you, Max — and thank you, Claude Code (independently 🙂). Verdict: SOLID, no blocking bug for the supported/tested usage.

Reviewed against CostEfficientGradientBoosting::DeltaGain and the CPU subtraction site in serial_tree_learner.cpp:

• Cost-delta tradeoff·penalty_split·num_data_in_leaf + tradeoff·coupled[real_fidx] matches CPU exactly, subtracted on the finalized per-task gain before the cross-feature/cross-leaf reductions — same place as new_split.gain -= cegb_->DeltaGain(...).
• Feature-usage tracking is genuinely model-global (only assigned in SetCEGB/Init, deliberately not reset in BeforeTrain), matching CPU's if (!init_) semantics.
• The gain > 0 stop condition is strict and a true no-op without CEGB — verified valid splits always carry strictly-positive stored gain (current_gain > min_gain_shift), so it can't drop a legitimate split even at min_gain_to_split=0.
• cegb_penalty_feature_lazy → clear Log::Fatal. num_data_in_leaf is the correct per-leaf count at evaluation time.

Two non-blocking notes (uncommon paths, worth a docstring or follow-up, not a merge blocker):

• reset_parameter mid-training won't change cegb_* on CUDA — penalties are snapshotted at Init; ResetConfig doesn't re-run SetCEGB. CPU reads them live.
• The kernels use global_num_data_in_*_leaf; identical to local for single-GPU (your tested path), but would differ from CPU per-row penalty scale under multi-GPU.

Your documented "retroactive re-eval" limitation is real and honestly stated — the tests just don't engineer a case that triggers a promotion. Fine to leave as-is.

Merge blocker is only the required ruff format check + a git merge master (these five overlap). macOS red = dask socket flake.

P.S. — you proved bit-for-bit leaf-count and feature-set parity across 6 CEGB configs and then shipped unformatted test code. 😄 The penalty for skipping ruff format is, fittingly, cost-effective: one command.