In-situ muon efficiency method for the Z dilepton and W analyses

[davidwalter2]

Draft — opened for visibility while the scope is still being split. Not ready to merge as-is, see Scope caveats below.

What this adds

An in-situ determination of the muon ID / HLT / Iso data-MC scale factors, as an alternative to the external tag-and-probe SFs. The three efficiency steps are floated as Chebyshev polynomials in (pT, uT), decorrelated per probe-eta bin, as unconstrained nuisances (not NOIs), so the scale factors are determined by the fit itself rather than taken from an external measurement.

The Z dilepton analysis is split into four categories — nominal, failIso, failHLT, failID — which become four fit channels sharing the same coefficient nuisances, and that cross-category constraint is what determines the efficiencies. Reco and tracking SFs stay external.

The passing-leg scale factor is the polynomial directly, f_pass = 1 + P, which makes it MC-independent; the MC efficiency enters only the failing-leg factor f_fail = (1 - eMC(1+P))/(1 - eMC). Each analysis therefore supplies its own effMC. The method is implemented for both the dilepton (two-leg tag-and-probe) and the W single-muon (single-leg) topologies through a shared C++ base. The coefficients are common to W and Z, so the two can be fitted simultaneously with the scale factors fully correlated between them.

The exact likelihood is linearised around a central point and iterated: make_insitu_effSF.py accumulates theta_new = theta_prev + delta * n_hat, fed back via --insituSFFile.

Workflow

1. mz_dilepton.py --makeInsituEffMC --makeUTQuantileHists — probe spectra and uT quantile inputs
2. scripts/corrections/make_insitu_effMC.py — per-step MC efficiencies
3. mz_dilepton.py --insituEffMCFile ... — templates and coefficient response tensors
4. setupRabbit.py --muonInsituEfficiency with the four --baseNames
5. fit, then make_insitu_effSF.py to produce the next iteration's central coefficients

Conditioning of the coefficients

Chebyshev polynomials are orthogonal under the weight 1/sqrt(1-x^2), which is nothing like the probe density, so the coefficients come out strongly correlated — correlation condition number 6.1e4 over the 2112-coefficient block in the 2D fit, 1.3e5 in the 4D one. That is what made the fit slow: trust-krylov's inner solve needs ~sqrt(kappa) Hessian-vector products, and steps get rejected. Measured on the full-statistics 2D fit: 145 minutes, with 20 of 78 iterations returning a bit-identical loss and the worst single iteration burning 27 minutes to make no progress.

This branch previously carried an offline fix (make_insitu_basis.py plus --insituBasisFile) that recombined the coefficients into a basis orthonormal under the probe density. That has been removed in favour of preconditioning the fit parameters inside the fitter, WMass/rabbit#150, which builds the transform from the model's own Hessian, refreshes it as the fit moves, and needs no regenerated inputs upstream. The offline version also required the same basis file to be used across the W and Z histmakers by hand, since the nuisances are correlated between them — a consistency requirement nothing checked.

For reference, on the same full-statistics 2D input: unpreconditioned 145.5 min, the offline basis 179 min, preconditioning in rabbit 14.3 min. All three converge to genuine minima; the preconditioned fits reach EDM ~1e-16 against ~1e-11 unpreconditioned.

Scope caveats

• The branch carries content merged in from the unmerged 260216_totalPhaseSpace branch (make_pixel_corrections.py, setup.sh, .gitignore). Those are not part of this feature and should be split out or land separately.
• The in-situ mode is opt-in throughout: without --insituEffMCFile the histmakers follow the existing external tag-and-probe path unchanged.

🤖 Generated with Claude Code