fix(experimental.pp.hvg): scope per-batch clip threshold in pearson_residuals HVG

[Arshammik]

Hi,

In _highly_variable_pearson_residuals, the clip parameter is reassigned inside the per-batch loop when it comes in as None: if clip is None: clip = np.sqrt(n). After the first iteration, clip is no longer None, so every subsequent batch silently reuses the first batch's threshold instead of computing its own sqrt(n_batch).

This matters because Lause, Berens & Kobak (2021) define the clip as sqrt(N) over the cells entering the residual computation, and with batch_key each batch is its own residual computation. For a 5000/200 split, the small batch gets sqrt(5000) ≈ 70.7 instead of sqrt(200) ≈ 14.1, so residuals that should be clipped in the small batch are not, its per-gene variances are inflated, and HVG ranking shifts. There is also a quieter reproducibility consequence: because np.unique sorts the labels, the threshold that leaks into all batches is whichever one is alphabetically first, so renaming "A"↔"B" on the same data changes the HVG output.

The fix is a one-line scoping change — a loop-local clip_batch that is recomputed each iteration from the current batch's n. The user-facing clip argument is untouched, so any caller that passed an explicit value still gets exactly that value applied to every batch.

Two tests in tests/test_highly_variable_genes.py cover this. The first monkeypatches _calculate_res_sparse to capture the clip value passed on each batch and asserts two distinct values appear for a 5000/200 split; the second swaps batch labels A↔B on identical data and asserts residual_variances is unchanged. Both fail on main and pass on this branch, and the rest of the pearson-residual suite is green.

rapids-singlecell ports this routine verbatim and inherits the same bug; I have a parallel fix queued there that will reference this PR number once assigned.

Thanks,
Arsham

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.61%. Comparing base (4ba31e4) to head (402b167).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4138   +/-   ##
=======================================
  Coverage   79.61%   79.61%           
=======================================
  Files         120      120           
  Lines       12786    12787    +1     
=======================================
+ Hits        10179    10181    +2     
+ Misses       2607     2606    -1     

Flag	Coverage Δ
hatch-test.low-vers	78.86% <100.00%> (+<0.01%)	⬆️
hatch-test.pre	79.47% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...c/scanpy/experimental/pp/_highly_variable_genes.py	94.28% <100.00%> (+0.05%)	⬆️

... and 1 file with indirect coverage changes

[Zethson]

Thanks! Could you please ensure that the pre-commit checks pass?

[Arshammik]

@Zethson confirming that the pre-commit checks have passed.

[Zethson]

@jlause do you have any thoughts on this matter?