[cuda] implement forced splits (forcedsplits_filename) on the CUDA tree learner

[maxwbuckley]

Problem: the delta

When training with device_type="cuda", forcedsplits_filename was silently ignored. ForceSplits exists only in SerialTreeLearner::Train; the CUDA learner overrides Train and never consults the forced-split JSON.

Measured delta before this PR (400×6 synthetic data, 30 rounds, gpu_use_dp=true; "roots" = the set of root split features across all 30 trees):

forced-split case	forced root feature	CPU tree roots	CUDA tree roots	max pred delta
root only (feat 2 @ 0.5)	2	{2} ✓	{0, 1} ✗	0.49
root + nested children	2	{2} ✓	{0, 1} ✗	0.91
mixed features (root 1, children 0/3)	1	{1} ✓	{0, 1} ✗	0.74

CPU honors the forced structure in every tree; CUDA splits on whatever has the best gain — the user's forced structure is silently discarded.

Fix

Implements forced splits on the CUDA tree learner, mirroring SerialTreeLearner::ForceSplits:

• ComputeForcedSplitKernel — computes split info for a given (feature, bin threshold) from the leaf's histogram: the CUDA analogue of FeatureHistogram::GatherInfoForThresholdNumerical. Single-threaded with the same right-to-left bin accumulation order as the CPU loop, so the arithmetic is bit-identical.
• ForceSplitsCUDA — a BFS pre-pass over the forced-split JSON before the main split loop, using the CPU's compute-before-apply structure (forceSplitMap pattern): each iteration constructs histograms for the active (smaller, larger) leaf pair, runs the regular best-split search, syncs host best-split arrays with the device cache, computes forced split infos for pending JSON nodes, then applies one forced split.
• ApplySplit — the main loop's split-application block, extracted and shared with the pre-pass.
• Unsupported configurations (quantized gradients, multi-GPU, categorical features) fall back with a clear warning.

The first commit's Log::Fatal guard is removed: forced splits now work on CUDA.

Result: the delta after the fix

Same measurement, post-fix — across 4 forced-split shapes × {8, 31} leaves × 2 seeds × {1, 30} rounds (32 configs):

forced-split case	CUDA tree roots	tree-0 structure vs CPU	max pred delta
root only	{forced feature} ✓	identical (features/gains/counts/leaf values)	≤ 8.9e-16
root + nested children	{forced feature} ✓	identical	≤ 8.9e-16
mixed features	{forced feature} ✓	identical	≤ 8.9e-16
3-deep chain	{forced feature} ✓	identical	≤ 8.9e-16

Every tree honors the forced structure, the first tree's structure matches CPU node-for-node, and predictions match at FP epsilon (≤ 1 fp64 ULP) in all 32 configurations.

(Only difference vs CPU: the displayed real-valued threshold for the same bin boundary can differ — e.g. 0.4433 vs 0.4365 for the same bin — a known threshold-encoding cosmetic difference that does not affect predictions.)

Bring-up notes for reviewers

Three bugs were found and fixed during development — they explain the design:

1. Forced split infos must be computed before applying any split of the same BFS level (CPU's forceSplitMap pattern), because applying a split re-targets the active (smaller, larger) leaf pair.
2. Every forced leaf needs a regular best-split search during the pre-pass, or the main loop never considers it for further growth (the remaining num_leaves budget would be spent only under the last forced split).
3. The host-side best-split arrays must be synced with the device-side cache for pre-pass-searched leaves (SyncLeafBestSplitToHost); otherwise the main loop applies splits with mismatched host (feature/threshold) and device (gain/sums) state, corrupting trees.

Tests (in test_dual.py, gated on TASK=cuda)

Test	Cases	What it pins down
test_cuda_forced_splits_honored	8	every tree's root is the forced feature — fails on the old build
test_cuda_forced_splits_match_cpu	16	tree-0 structural equality + prediction parity (atol=1e-10) over 30 rounds

Full test_dual.py suite (78 tests) passes.

🤖 Generated with Claude Code

[BelixRogner]

Thank you, Max — and thank you, Claude Code (separately 🙂). Really strong work here, but this is the one of the five that needs a fix before merge.

Deep review against SerialTreeLearner::ForceSplits / GatherInfoForThresholdNumerical. The good news: the BFS pre-pass structure is faithful, the right-to-left bin accumulation in ComputeForcedSplitKernel is bit-identical, and all three documented bring-up bugs (compute-before-apply, every-forced-leaf-searched, host/device sync via SyncLeafBestSplitToHost) are genuinely handled, not just described. Verdict: MINOR-CONCERNS — two things:

1. Latent bug: ComputeForcedSplitKernel sets the split fields but never assigns out->inner_feature_index (nor out->leaf_index). The slot comes from Resize'd, uninitialized device memory, so cuda_tree.cu:91 (split_feature_inner[new_node_index] = cuda_split_info->inner_feature_index) writes garbage into split_feature_inner_ for every forced node. It's latent — your tests pass because predict()/dump_model() use split_feature_ (the real index, set correctly) — but any in-session consumer of split_feature_inner() / Tree::PredictByMap on a CUDA forced-split tree reads garbage. One-line fix in the kernel: out->inner_feature_index = task->inner_feature_index; (and set out->leaf_index for parity).

2. path_smooth divergence (unguarded): the forced kernel hardcodes GetLeafGain<true,false> / CalculateSplittedLeafOutput<true,false> i.e. USE_SMOOTHING=false, while the main finder uses use_smoothing_. With path_smooth>0, forced-node outputs/gains are computed without smoothing while the rest of the tree uses it — a silent CPU divergence with no warning. Please either guard path_smooth>0 with a warning, or template smoothing like the main finder. (The hardcoded USE_L1=true is fine — identity when lambda_l1=0.)

Acceptable-as-scoped: categorical forced splits abort forcing on CUDA (vs applied on CPU) — it's warned and documented as numerical-only, so no objection.

Test gap worth closing while you're in here: a path_smooth>0 case (would catch #2) and something that reads split_feature_inner/PredictByMap (would catch #1).

And: ruff format + git merge master for the lint/conflict. macOS red is the dask socket flake.

P.S. — bit-identical accumulation order down to the __double2int_rn rounding, three self-found bugs in the writeup… and the linter caught you anyway. 😄 Friends don't let friends skip ruff format.