Merge upstream master into GhentAnalysis/master (0.3 release cycle)

[JulesVandenbroeck]

Overview

This PR tracks all changes in columnflow/columnflow:master
that are not yet present in GhentAnalysis/master. It is a documentation-only / dummy PR —
its purpose is to give an overview of what an eventual merge from upstream would bring in.

The base branch (GhentAnalysis/master_merge_from_upstream) is identical to GhentAnalysis/master
at the time of this PR's creation.

---

Migration Guide — Required Changes for Existing Analyses

The following changes are required in any existing analysis to run with the updated columnflow after this PR. All other changes listed above are additive and backwards-compatible.

---

1. Update patch_bundle_repo_exclude_files in columnflow_patches.py

Why it breaks: BundleRepo.exclude_files now stores absolute paths (built internally via _cf_path() / _repo_path() helpers) instead of relative paths. The old patch re-prefixed every existing entry with the relative path from the analysis base to the CF base — this now produces invalid doubled paths and breaks remote job bundling.

Replace the old function body:

# OLD — breaks with the updated columnflow
def patch_bundle_repo_exclude_files():
    from columnflow.tasks.framework.remote import BundleRepo

    cf_rel = os.path.relpath(os.environ["CF_BASE"], os.environ["MYANALYSIS_BASE"])
    exclude_files = [os.path.join(cf_rel, path) for path in BundleRepo.exclude_files]
    exclude_files.extend([
        "docs", "tests", "data", "assets", ".law", ".setups", ".data", ".github",
    ])
    BundleRepo.exclude_files[:] = exclude_files

    logger.debug("patched exclude_files of cf.BundleRepo")

with:

# NEW — only append analysis-specific extra exclusions
def patch_bundle_repo_exclude_files():
    from columnflow.tasks.framework.remote import BundleRepo

    # add additional files to exclude
    BundleRepo.exclude_files += ["docs"]

    logger.debug("patched exclude_files of cf.BundleRepo")

The standard exclusions (tests, data, .setups, .data, .github, etc.) are now included for both the CF repo and the analysis repo by default. Only append paths that are specific to your analysis and not already covered. Also remove import os at the top of columnflow_patches.py if it is no longer used elsewhere.

---

2. Add **kwargs to all TAF hook function signatures

Why it breaks: The framework now passes keyword arguments when invoking TAF hook functions. Any hook function without **kwargs that calls super() — or is used as a base via bases= — will raise a TypeError.

# OLD — missing **kwargs, breaks when super() is involved
@my_producer.init
def my_producer_init(self: Producer) -> None:
    ...

@my_producer.requires
def my_producer_requires(self: Producer, task, reqs) -> None:
    ...

# NEW — add **kwargs to every hook signature
@my_producer.init
def my_producer_init(self: Producer, **kwargs) -> None:
    ...

@my_producer.requires
def my_producer_requires(self: Producer, task, reqs, **kwargs) -> None:
    ...

Apply to all @my_taf.init, @my_taf.requires, @my_taf.setup, and @my_taf.teardown functions across calibrators, selectors, and producers.

---

3. Call super() in TAF hooks when using bases=

Why it breaks: If an analysis TAF uses bases=(some_base_taf,) and defines its own hook functions, it must explicitly call the base TAF's hook at the start of each overridden hook. Without the call, the base TAF's initialization, requirement registration, and setup logic (e.g. external file loading, normalization weight table building) is silently skipped.

@my_producer.init
def my_producer_init(self: Producer, **kwargs) -> None:
    super(my_producer, self).init_func(**kwargs)
    # ... analysis-specific init ...

@my_producer.requires
def my_producer_requires(self: Producer, task, reqs, **kwargs) -> None:
    super(my_producer, self).requires_func(task=task, reqs=reqs, **kwargs)
    # ... analysis-specific requirements ...

@my_producer.setup
def my_producer_setup(self, task, reqs, inputs, reader_targets, **kwargs) -> None:
    super(my_producer, self).setup_func(
        task=task, reqs=reqs, inputs=inputs, reader_targets=reader_targets, **kwargs,
    )
    # ... analysis-specific setup ...

This applies to calibrators, selectors, and producers that extend a built-in columnflow TAF via bases=.

4. Small changes

• rucio_report_access boolean should be added to law.cfg for all fs

---

New Features

Framework & Core

• Refactoring for 0.3 release (Refactoring for 0.3 release columnflow/columnflow#628) — major restructuring of the core framework
• Accelerate loading time and imports (Accelerate loading time and imports columnflow/columnflow#737) — faster startup by deferring heavy imports
• Make default_remote_claw_sandbox configurable via law.cfg — remote sandbox now settable per-project
• Add dfs lookup pattern negation — negate patterns in DFS column lookups
• Add tmp dir checks, add cf_setup_post_install hook — post-install hook for custom setup steps
• Allow skipping parts of post setup — fine-grained control over setup steps
• Improve python version handling in setup — more robust Python version detection
• Report rucio access (Report access of central LFNs to rucio bookkeeping server columnflow/columnflow#803) — log Rucio site access during dataset queries
• Add req helpers to mixins (Add same-task requirement helpers to mixins. columnflow/columnflow#738) — convenience methods for adding task requirements in mixins
• Port parameters to WrapperTask's (Port parameters to WrapperTask's columnflow/columnflow#784) — wrapper tasks now expose the same parameters as their wrapped tasks
• Control row group merging in MergeReducedEvents — configurable row-group size in Parquet output
• Make merging chunk size configurable (Make merging chunk size configurable. columnflow/columnflow#707) — tune memory/IO trade-off in merge steps
• Improve reduced events merging — smarter chunk layout in merged Parquet files
• Add transfer_produced_columns util — helper to copy produced columns between events arrays
• Hook column union — column union now participates in hook callbacks
• Forward remote claw sandbox — remote sandbox propagated through task dependency chain

Producers & Calibrators

• Generalize normalization weight producer (Generalize normalization weight producer. columnflow/columnflow#718) — handles more generic dataset configurations
• Extend DY weight application to use btag multiplicity (Extend dy weight application to use btag multiplicity. columnflow/columnflow#739) — DY weights now depend on b-tag count
• Make dy_weight producer compatible with both central and custom DY sf (Update dy_weight producer columnflow/columnflow#770) — unified interface for official/custom scale factors
• Adapt DY weight producer for custom weights (adapting dy weight producer for custom weights columnflow/columnflow#724) — pass custom weights directly to producer
• Make jet collection used in DY weights more flexible (dy_weights columnflow/columnflow#696) — configurable jet collection for DY reweighting
• Adding b-tag producer for working point based SFs (Adding b-tag producer for working point based SFs columnflow/columnflow#777) — new dedicated b-tag WP scale-factor producer
• Merge WP bins (Allow merging wp bins in btag_wp_weights producer columnflow/columnflow#792) — merge b-tag working-point bins in scale-factor evaluation
• Add btag count selector (Add btag count selector. columnflow/columnflow#775) — ready-made selector for b-tag multiplicity cuts
• Enable support for CMS electron weight producer v5 — updated to CMS EGM v5 correctionlib schema
• Refactor and fix MET phi calibration (Fix met phi calibrator columnflow/columnflow#725) — cleaner MET-phi corrector with updated format
• Update met_phi Calibrator to new format (update met_phi Calibrator to new format columnflow/columnflow#719)
• Add muon_sr calibrator (Add muon scale and resolution calibrator. columnflow/columnflow#754) — new calibrator for muon scale and resolution corrections
• [cms] Refactor egamma calibrators (Fix and refactor egamma calibrators. columnflow/columnflow#745) — cleaner EGamma calibration interface
• Efficiency information from muon correctionlib files (efficiency information from muon correctionlib files columnflow/columnflow#791) — expose muon efficiency from correctionlib
• Remove year from intrinsic btag weight names (Remove year from intrinsic btag weight names. columnflow/columnflow#726) — year-agnostic naming convention
• Externalize coffea from_root handling — coffea ROOT-file reading decoupled from core IO
• Update sandboxes, avoid dask_awkward in IO (Update sandboxes, avoid dask_awkward in IO columnflow/columnflow#735) — columnar sandbox no longer requires dask-awkward
• Add option for hook to update dataset_selection_stats in norm weight producer — hook-based stats update
• Allow evaluating multiple working points with single electron_weights Producer (allow evaluating multiple working points with single electron_weights Producer columnflow/columnflow#694)
• Streamline chunked IO (Streamline chunked io. columnflow/columnflow#789) — unified chunk reader/writer API
• Add simple column selection to UniteColumns — filter columns when uniting datasets
• Extend tes versions — support for newer tau energy scale correction versions
• Add t->w->tau children in gen_top_lookup — tau daughters now tracked in top-quark gen lookup
• Implement data_per_era tag to jec config (Fix/jec 2023 columnflow/columnflow#700) — JEC keys now selectable per data-taking era
• Allow strings in load_correction_set — pass string file paths directly to correctionlib loader

Selectors & Categorization

• Improve categorizer calls (Improve categorizer calls in category_ids producer. columnflow/columnflow#702) — cleaner internal dispatch for categorizer TAFs
• Add parent_mode flag to create_category_combinations, add change tracking (Add parent_mode flag to create_category_combinations, add change tracking. columnflow/columnflow#736)
• Inherit selector shifts in cf_default reducer (Inherit selector shifts in cf_default reducer. columnflow/columnflow#761) — shifts declared in selector automatically available in reducer
• Sort configs by ids for multi-config representation (Sort configs by ids for the multi config representation columnflow/columnflow#713) — deterministic ordering in multi-config setups
• Check and remove overlapping processes (check and remove overlaping processes columnflow/columnflow#712) — automatic deduplication of overlapping process definitions
• Pad with nominal if shift source missing in config (pad with nominal if shift source missing in config columnflow/columnflow#715) — graceful fallback for missing shift sources

Histogramming & Plotting

• Multi-variable plots (Multi-variable plots columnflow/columnflow#769) — plot multiple variables in a single task invocation
• Histogram user multiconfig (Feature/histogram user multiconfig columnflow/columnflow#709) — histogram tasks now support multiple config objects simultaneously
• Update hist axis labels during histogram merging (update hist axis labels during histogram merging columnflow/columnflow#705) — axis metadata preserved across histogram merges
• Add category id cache in config_util (Add category id cache. columnflow/columnflow#787) — faster category lookups in large configs
• Add subplots_cfg in plot_all (add subplots_cfg in plot_all columnflow/columnflow#742) — subplot configuration passed through plot_all loops
• Add plot function for efficiency plots (plotting enhancements columnflow/columnflow#723) — dedicated helper for efficiency curve plotting
• Allow defining initial values for categorical hist axes — pre-populate string/int categorical axes
• Improve leaf category handling in histograms (Improve leaf category handling in histograms columnflow/columnflow#783) — correct aggregation over leaf categories
• Add some customization hooks for merge histograms — pre/post-merge hooks for histogram processing
• Remove hard-coded nominal shift source usage (Remove hard-coded nominal shift source usage. columnflow/columnflow#762) — nominal shift sourced from config, not hardcoded
• More flexibility in plot_all loop — pass extra kwargs through plot_all
• Make plotting faster using non-interactive backend — use Agg backend to avoid GUI overhead
• Combinatorics protection during histogram filling (Combinatorics protection during histogram filling. columnflow/columnflow#796) — guard against event x object combinatoric explosions
• Reduce processes in plot tasks to those present in datasets — no phantom processes in legend
• Use dict-of-keys sparse matrices for vectorized lookups (memory intensive sparse matrix in normalization_weights producer columnflow/columnflow#763) — major performance improvement in SF lookups
• Add pilot option to MergeShiftedHistograms — run merge in pilot/test mode
• Warn about flow content for FlowStrategy.move — alert when under/overflow is non-zero after moving
• Fix data uncertainties for density (fix data uncertainties for density columnflow/columnflow#773) — correct uncertainty propagation for density-normalized histograms
• Forward known values to hist hooks (Forward known values to hist hooks. columnflow/columnflow#727) — hook callbacks receive pre-computed axis values
• Improve proxy handling in remote jobs (docs: add Lara813 as a contributor for code columnflow/columnflow#765)

Inference / Datacards

• Sequentialize and optimize datacard writing (Sequentialize and optimize datacard writing. columnflow/columnflow#728) — faster, memory-efficient datacard production
• Refactor datacard parameter transformations (Refactor datacard parameter transformations. columnflow/columnflow#732) — cleaner transformation API
• Add flip_(smaller|larger)_if_one_sided transformations (Add transformations for flipping rate effects. columnflow/columnflow#734) — new transformations for one-sided systematics
• Add ParameterTransformation for ratifying + envelope if one-sided (add ParameterTransformation for ratifying + envelope if one-sided columnflow/columnflow#731)
• Allow removing negative contributions per process (allow removing negative contributions per process columnflow/columnflow#730) — option to zero-out negative process yields
• Allow block modification before writing datacards — user hook before final datacard output
• Add rounding warning to datacard for small effects — warns when systematics round to zero
• Make datacard writer class configurable in task — swap in custom writer subclass via config
• Merge workflow reqs of different variables in CreateDatacards (merge workflow reqs of different variables in CreateDatacards columnflow/columnflow#689) — single merged requirement for all variable branches
• Inference model caching (inference model caching columnflow/columnflow#714) — avoid re-building inference models on each task invocation
• Asymmetric systematic uncertainties (Fix/asymmetric syst unc columnflow/columnflow#710) — support for asymmetric systematics in datacards
• Allow multiple processes per dataset in datacard writer (allow multiple processes per dataset in datacard writer columnflow/columnflow#733)
• Move empty-process skipping from histogram loading to datacard writing

Tasks & Infrastructure

• Add BundleExternalFilesWrapper — wrapper task for bundling external files in parallel
• Persistent local files of BundleExternalFiles (Persistent local files of BundleExternalFiles. columnflow/columnflow#752) — cached external files survive task re-runs
• Add --merge-columns option to cf_inspect tool — inspect merged-column content in cf_inspect
• Simplify requiring producers (Simplify requiring producers. columnflow/columnflow#756) — shorter API for declaring producer dependencies
• Rename max-runtime to {htcondor,slurm}-runtime (Rename max-runtime -> {htcondor,slurm}-runtime. columnflow/columnflow#755) — runtime flag split per scheduler
• Add variable_repr to control paths (Add variabble_repr to control paths. columnflow/columnflow#751) — custom string representation for variable objects
• Add gen_dy_lookup and gen_higgs_lookup — generator-level Drell-Yan and Higgs particle lookups
• Refactor generator-level top and top decay product lookup (Refactor generator-level top and top decay product lookup columnflow/columnflow#741)
• Add objects for interacting with CMS CAT meta data (Add objects for interacting with CMS CAT meta data. columnflow/columnflow#740) — interface to CMS CAT POG JSON metadata
• Allow multiple nano inputs in first tasks (Allow multiple nano inputs in first tasks. columnflow/columnflow#788) — GetDatasetLFNs and friends accept multiple NanoAOD inputs
• Add embed_with_mask helper — embed sub-array values back into parent array using a mask
• Apply jet_mask to events (Apply jet_mask to events in btag_wp_weights. columnflow/columnflow#780) — jet mask correctly applied before further processing
• Add confirm message to cf_remove_tmp — interactive confirmation before removing tmp files
• Allow patterns in get_shifts_from_sources — brace/glob patterns in shift source lookup
• Accept lists in ExternalFile subpaths / Allow single subpath in ExternalFile — flexible path specification
• Add ML model task pinning (Add ml model task pinning. columnflow/columnflow#703) — pin specific ML model versions in task parameters
• Add sourcing of CMS setup to ensure scram is available (added sourcing of cms setup to ensure scram is available columnflow/columnflow#699)
• Optionally bypass branch-level plot requirements (Accelerate branch-level plot requirements. columnflow/columnflow#716)
• Add option to skip auto categories in track_category_changes
• Pass-through workflow requirements in CreateHistograms (Pass-through workflow requirements in CreateHistograms. columnflow/columnflow#757)

---

Bug Fixes

Histograms & Plots

• Hotfix rewriting of outputs in MergeHistograms
• Hotfix shift id-to-name conversion in histogram post-processing
• Hotfix empty histogram issue in inference base task (two separate fixes)
• Hotfix nominal shift in PlotShiftedVariablesPerShift1D
• Hotfix hist validation in datacard writer
• Hotfix sci-style axis label notation in plots / fix horizontal y-offset label position
• Hotfix single shift selection in plotting
• Hotfix save_div in plot scale factor
• Hotfix multi-axis filling with discrete variables
• Bug fix in fill_hist function (bug fix in fill_hist function columnflow/columnflow#768)
• Fix variance of fake data in datacard writer
• Fix plotting process shift map (Fix/plotting process shift map columnflow/columnflow#760)
• Hotfix bad import in plot utils
• Avoid using flat_np_view for value assignment (Avoid using flat_np_view for value assignment. columnflow/columnflow#759)
• Use safe concatenation (Use safe concatenation. columnflow/columnflow#758)
• Hotfix multi-config lookup via patterns
• Hotfix process object selection for multi-config datacards
• Hotfix variable shape type in combine datacard writer
• Fix skipping data in CreateDatacards
• Hotfix datacard writing when variables are missing in config
• Hotfix parameter group cleaning in inference model

Producers & Calibrators

• Hotfix kwargs for array function super calls
• Hotfix super() calls in all task array functions
• Hotfix TAF class attribute inheritance and TAF instance method defaults
• Hotfix calibrator/producer requirement handling
• Hotfix btag_wp_weights producer: clamp pt and eta efficiency map axes
• Hotfix ExternalFile dataclass: drop None values
• Hotfix sorted_ak_to_root helper for nested arrays
• Hotfix CMS jet veto map usage; correct bit masks (correct bit masks jet veto map columnflow/columnflow#778)
• Hotfix CMS muon calibrator: remove outdated rnd_gen argument
• Hotfix abs eta in CMS muon weight producer
• Hotfix CMS tau energy scale variations
• [cms] Hotfix varied energy errors in EGM calibration
• [cms] Hotfix egamma calibrator: use same random numbers for all smearing variations
• [cms] Hotfix tau energy calibration: skip e-fake mask
• Hotfix TEC calibrator: add back charge column
• Hotfix inclusive dataset attribute and lookup in norm weight producer
• Hotfix norm weight logging
• Hotfix combined jets calibrator
• Hotfix validation check in stitched normalization weight production
• Hotfix missing xsecs for stitched weight producer
• Hotfix nbtags variable in DY weight producer
• Fix handling of non_zero_mask in murf_envelope (fix handling of non_zero_mask in murf_envelope columnflow/columnflow#704)
• Fix JER application on JEC variations (Fix to jer application on jec variations columnflow/columnflow#665)
• Add exception when era aux is missing (Fix/jec 2023 columnflow/columnflow#700)
• Fix scope issue in seed producer
• Hotfix depth limit of gen particles
• Hotfix saving of columns in gen_particle lookups
• Hotfix higgs gen lookup: consider effective gluon/photon decays
• Hotfix typo in gen_top lookup
• Skip string columns in finiteness checks (Producing string columns columnflow/columnflow#743)

Tasks & Workflow

• Hotfix task_namespace of Wrapper tasks
• Hotfix producer group resolution in ProduceColumnsWrapper; fix brace expansion
• Hotfix config object resolution and order (deterministic and group lookup)
• Hotfix process selection when only datasets are given
• Hotfix default version injection into tasks with same family
• Hotfix version resolution, pinning, and lookup
• Hotfix forwarding of known_shifts for instance caching
• Hotfix required producers/calibrators for workflows
• Hotfix attributes added by TAF decorators
• Hotfix: allow brace patterns in TAF shifts
• Fix duplicate producers and requirements
• Fix workflow parameter passthrough in pilot mode (Fix workflow parameter passthrough in pilot mode. columnflow/columnflow#785)
• Fix bundle unpacking (Fix ExternalFiles bundle unpacking. columnflow/columnflow#786)
• Fix total size logged by MergeReductionStats
• Fix reduction chunk size control
• Fix task key lookup (Fix task pinning columnflow/columnflow#697)
• Hotfix requirement order for consistent output removal
• Hotfix consistent branch reqs between MergeReducedEvents and MergeSelectionStats
• Raise explicit error in reduction on option type masks

IO & Environment

• Hotfix ChunkedParquetReader
• Multiple fixes regarding empty files or almost-empty chunks (mutliple fixes regarding empty files or (almost) empty chunks columnflow/columnflow#750)
• Hotfix reduction to skip empty chunks
• Fix mamba setup
• Hotfix htcondor/conda software setup in remote jobs
• Fix missing datasets in MultiConfig
• Hotfix CAT metadata update check for missing POG dirs
• Hotfix category flattening
• Correct JSON file extension for stats file
• Do not consider empty axis as "missing" (histogram axis fix)
• Fix opening root files in cf_inspect / after coffea update (Fix cf_inspect script after coffea update. columnflow/columnflow#753)
• Log broken parquet file paths
• Fix warning message in BundleExternalFiles task
• Hotfix repo bundling: add missing user config
• Failsafe setting of htcondor requested memory
• Apply blinding threshold before process scaling
• Hotfix electron weight producer with nested working points

---

Small Changes

Dependency & Submodule Updates

• Multiple law, order, scinum, and boost-histogram submodule/version updates
• Update default CMSSW versions in CMSSW sandboxes

Code Quality & Cleanup

• Multiple typo fixes across the codebase
• Minor code cleanups and consistency changes
• Cleanup of e/mu id producers; remove unneeded columns in CMS TEC calibrator
• Consistent handling of kwargs in teardown functions
• Improve treepath detection in cf_inspect; optimize UniteColumns ROOT compression
• Improve tmp file check and flag file handling in venv setup
• Use return code in cf_remove_tmp; add local directory check
• [cms] Add note on TEC-to-MET propagation
• Store jec level as attribute on jec correctors
• Bump version in __version__ file
• Improve readability of make_jme_keys

Logging & Verbosity

• More verbose errors for parquet metadata failures, norm weight missing cross sections, memory issues during filling, and broken histogram files
• Warn about flow content for FlowStrategy.move
• [cms] Update log in CheckCATUpdates task; add URL to log

Documentation

• docs: add LennertGriesing, Lara813, aalvesan, Bogdan-Wiederspan, LuSchaller as contributors

Tests

• Adjust unit tests for new behaviour and deterministic config object ordering

---

This PR was auto-generated to track upstream changes. It is not intended to be merged directly.