Add CRAB workflow support for remote LAW tasks

[kandrosov]

Summary

Adds a first-class CRAB backend for FLAF LAW tasks (via law.contrib.cms.CrabWorkflow), so the full analysis chain (AnaTuple → HistTuple → histograms → plots) can run on CRAB with the same remote I/O (fs_*) model as HTCondor+bundles.

Analysis products are not staged out through CRAB storage; workers write/read via FLAF remote filesystems. File-list / merge-plan tasks remain local by design.

Key changes

• CrabWorkflow in run_tools/law_customizations.py (submit, MyProxy gate, user_custom staging, log URLs)
• Task mixins: AnaTuple* / Analysis tasks inherit CrabWorkflow where HTCondor already applies
• Bootstrap / CMSSW: relocate ProjectRename safely; force CMSSW_BASE=FLAF_CMSSW_BASE; materialize absolute AFS/CVMFS symlinks in the cmssw bundle
• Rucio: set RUCIO_ACCOUNT on pilots (not the pilot USER)
• user_custom: ship to remote workers; resolve staged/hashed paths
• Setup: honor top-level variables: when the active histTuple_flavor list is empty
• CRAB write-check site derived from fs_default; site lists from global.yaml crab:; memory 2 GB * n_cpus
• Docs: new docs/workflow/crab.md + updates for arguments / user_custom

Testing

Full crab CI (run_ci_test.sh … crab, TestModel, --test 1000, Run3_2022EE) PASS for:

• HH_bbtautau
• HH_bbWW
• H_mumu

MyProxy: CRAB hash credential (crab createmyproxy / workspace setup_crab_myproxy.sh), ≥5 days remaining.

Companion PRs

• cms-flaf/HH_bbtautau — law.cfg crab sandbox + HHbtag path fix
• cms-flaf/HH_bbWW — law.cfg crab sandbox
• cms-flaf/H_mumu — law.cfg crab sandbox

Merge this FLAF PR first, then analysis PRs (and a follow-up submodule advance if desired).

Notes / known non-blockers

• Occasional CRAB hang on heavy AnaTuple branches (e.g. VBFH): kill + resubmit recovered in CI
• HistMerger may still expand full AnalysisCache producers even with --branches 0 (wall-time cost only)

[Copilot]

Pull request overview

This PR adds a first-class CRAB (WLCG) backend for FLAF's LAW tasks, alongside the existing HTCondor backend, so the full analysis chain can run anywhere on the CMS grid using the same FLAF remote-I/O model (fs_default/gfal + stageout_logs.sh) rather than CRAB's own stageout. It refactors the shared HTCondor bundle/render logic into reusable helpers, introduces a CrabWorkflow mixin, and hardens the bootstrap/CMSSW relocation for AFS-less workers.

Changes:

• Add CrabWorkflow, FLAFCrabJobFileFactory, and a MyProxy-gating workflow proxy in run_tools/law_customizations.py; mix CrabWorkflow into the AnaTuple/Analysis tasks; extract shared bundle/bootstrap/render helpers.
• Bootstrap/CMSSW hardening (bootstrap.sh, RunKit/includeCMSSWlibs.py, Common/Setup.py): safe ProjectRename, force CMSSW_BASE=FLAF_CMSSW_BASE, materialize external symlinks in the cmssw bundle, resolve staged user_custom paths, honor top-level variables:, and set RUCIO_ACCOUNT from the submitter.
• CRAB-aware log naming (stageout_logs.sh), HistProducerFromNTuple ImplicitMT/materialization fix, a grid-access probe in the hello-world test task, and new/updated docs.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
run_tools/law_customizations.py	Core CrabWorkflow integration + refactor of bundle/bootstrap/render helpers
bootstrap.sh	Relocate CMSSW safely on workers, force CMSSW_BASE, set RUCIO_ACCOUNT
RunKit/includeCMSSWlibs.py	Prefer FLAF_CMSSW_BASE over CMSSW_BASE with existence check
run_tools/stageout_logs.sh	Distinct remote log name per CRAB job number
Common/Setup.py	resolve_user_custom_path, top-level variables: handling, force CMSSW_BASE
Analysis/HistProducerFromNTuple.py	Materialize RDF hists via GetValue(); disable ImplicitMT
Analysis/tasks.py	Add CrabWorkflow mixin, crab_output_directory, pass --user-custom
AnaProd/tasks.py	Add CrabWorkflow mixin to AnaTuple tasks
Analysis/AnalysisCacheProducer.py	Add help text to --user-custom
test/hello_world_task.py	Grid/X509/Rucio access probe for CRAB validation
mkdocs.yml, docs/workflow/*.md, docs/configuration/user-custom.md	Documentation for CRAB and variables:

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109

[cms-flaf-bot]

pipeline#15527414 started

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109

[cms-flaf-bot]

pipeline#15527835 started

[cms-flaf-bot]

pipeline#15527835 failed

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109
• backend=github

[cms-flaf-bot]

GitHub Actions integration workflow dispatched for HH_bbWW, HH_bbtautau, H_mumu

[cms-flaf-bot]

❌ GitHub Actions integration pipeline failed.

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109

[cms-flaf-bot]

pipeline#15528712 started

[cms-flaf-bot]

pipeline#15528712 failed

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109

[cms-flaf-bot]

pipeline#15536685 started

[cms-flaf-bot]

pipeline#15536685 passed

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Common/Setup.py:443

• The comment states that when the active flavor's variable list is empty, the top-level variables: list is used as the active "plot/full-res set". However, the else branch below only reassigns self.histTuple_plot_vars = list(user_vars); self.histTuple_fullres_vars is left as the (empty) flavor value and is never populated from user_vars. Since histTuple_fullres_vars is consumed downstream (e.g. Analysis/HistTupleProducer.py:86 as fullres_variables), this branch will produce no full-resolution variables despite the comment. Either update the comment to say only the plot set is driven by user_vars, or also assign histTuple_fullres_vars if full-resolution histograms are intended here.

        # - When the flavor list is empty (e.g. H_mumu default): use the list as the
        #   active plot/full-res set so user_custom alone can drive a short CI chain
        #   without requiring histTuple_flavor: CI.

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109

[cms-flaf-bot]

pipeline#15538117 started

[cms-flaf-bot]

pipeline#15538117 passed

[kandrosov]

@cms-flaf-bot please test

• Enable CRAB workflow defaults and fix HHbtag path on remote workers HH_bbtautau#80
• Enable CRAB workflow defaults and docs link H_mumu#67
• Enable CRAB workflow defaults and docs link HH_bbWW#109

[cms-flaf-bot]

pipeline#15539896 started

[cms-flaf-bot]

pipeline#15539896 passed

[kandrosov]

Verified on lxplus that the ci-like workflow via crab works for all analyses