Conversation
…al PoC)
- Add rucio: and xrootd: config blocks to snakemake.yml with JLab
defaults and commented BNL alternatives
- Root Snakefile: add _rucio_upload_enabled flag, _rucio_output() helper,
and shared _RUCIO_DID_PREFIX = rucio://{scope}/VALIDATION/benchmarks/detector/{job_id}
- Root Snakefile: add configurable _xrootd_base_url; fix get_remote_path()
to support Rucio reads
- barrel_ecal Snakefile: integrate Rucio storage outputs directly into the
four producing rules (sim, particles_analysis, pi0_analysis,
pion_rejection_analysis); no separate publish step
- Add profiles/rucio/config.yaml with storage-rucio-upload-rse setting
- Document required upstream snakemake-storage-plugin-rucio fixes in
snakemake-storage-plugin-rucio-required-changes.md
- README: document Rucio upload configuration
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an initial (“PoC”) integration of Snakemake’s Rucio storage plugin to upload benchmark outputs (focused on barrel_ecal), and makes XRootD base URL configurable to support both XRootD and Rucio “read” modes.
Changes:
- Introduces
xrootd:andrucio:config blocks insnakemake.ymlplus aprofiles/rucioSnakemake profile. - Adds Rucio upload helpers (
_rucio_upload_enabled,_rucio_output,_RUCIO_DID_PREFIX) and expands remote-path handling forremote_provider: rucio. - Updates
benchmarks/barrel_ecal/Snakefileproducing rules to stage-copy selected outputs into Rucio storage targets, and documents usage inREADME.md.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
snakemake.yml |
Adds XRootD and Rucio configuration blocks (currently enabling Rucio host by default). |
profiles/rucio/config.yaml |
Adds a Snakemake profile intended to configure Rucio upload RSE. |
Snakefile |
Adds Rucio upload toggle/helper and makes XRootD base URL configurable; supports remote_provider: rucio. |
benchmarks/barrel_ecal/Snakefile |
Adds Rucio storage outputs and post-step cp staging for upload in 4 producing rules. |
README.md |
Documents how to run uploads via the Rucio profile. |
Comments suppressed due to low confidence (1)
benchmarks/barrel_ecal/Snakefile:214
- Same pattern here:
root ...and the subsequentcpare separate commands without explicit error chaining. Chaining with&&(or setting-e) will ensure the upload-staging copy only happens if the analysis step succeeds.
cd {wildcards.DETECTOR_CONFIG}
root -l -b -q '{input.script}+g("../{input.electron}", "../{input.piminus}")'
cd - > /dev/null
exec cp {output.pion_rej_json} {output.rucio}
"""
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+3
to
+6
| # Uses snakemake-storage-plugin-rucio. Rucio connection details (host, auth) | ||
| # are read from the standard Rucio client configuration (~/.rucio/rucio.cfg or | ||
| # environment variables such as RUCIO_HOST, RUCIO_ACCOUNT, RUCIO_AUTH_TYPE). | ||
| # |
| """ | ||
| if _rucio_upload_enabled: | ||
| return storage(path) | ||
| local = re.sub(r"^rucio://[^/]*/", ".rucio_staging/", path) |
Comment on lines
+43
to
+44
| --outputFile {output.local} | ||
| exec cp {output.local} {output.rucio} |
Comment on lines
100
to
+102
| cd {wildcards.DETECTOR_CONFIG} | ||
| root -l -b -q '{input.script}+("{wildcards.PARTICLE}", true)' | ||
| cd - > /dev/null |
Comment on lines
+39
to
+41
| Benchmark outputs are uploaded to Rucio as part of each producing rule when | ||
| `config.rucio.host` is set (see `snakemake.yml`). Invoke with: | ||
|
|
Comment on lines
+14
to
+22
| # Uncomment the BNL block to upload to BNL instead of JLab. | ||
| rucio: | ||
| host: https://rucio-server.jlab.org:443 | ||
| rse: EIC-XRD | ||
| scope: epic | ||
| # BNL alternative: | ||
| # rucio: | ||
| # host: https://nprucio01.sdcc.bnl.gov:443 | ||
| # rse: BNL-XRD |
Comment on lines
+9
to
+16
| _rucio_upload_enabled = bool(config.get("rucio", {}).get("host")) | ||
| if not _rucio_upload_enabled: | ||
| logger.warning( | ||
| "Rucio host not configured (config.rucio.host); " | ||
| "benchmark outputs will not be uploaded to Rucio." | ||
| ) | ||
|
|
||
| _rucio_scope = config.get("rucio", {}).get("scope", "epic") |
Comment on lines
139
to
+141
| cd {wildcards.DETECTOR_CONFIG} | ||
| root -l -b -q '{input.script}+("../{input.sim}")' | ||
| cd - > /dev/null |
Comment on lines
+14
to
+16
| # Uncomment the BNL block to upload to BNL instead of JLab. | ||
| rucio: | ||
| host: https://rucio-server.jlab.org:443 |
veprbl
reviewed
Apr 24, 2026
| output: | ||
| "{DETECTOR_CONFIG}/sim_output/sim_emcal_barrel_{PARTICLE}_energies{E_MIN}_{E_MAX}.edm4hep.root" | ||
| local="{DETECTOR_CONFIG}/sim_output/sim_emcal_barrel_{PARTICLE}_energies{E_MIN}_{E_MAX}.edm4hep.root", | ||
| rucio=_rucio_output(f"{_RUCIO_BASE}/sim_output/sim_emcal_barrel_{{PARTICLE}}_energies{{E_MIN}}_{{E_MAX}}.edm4hep.root"), |
Member
There was a problem hiding this comment.
Have you considered using something like --default-storage-prefix rucio://user.wdconinck/j0001.?
Contributor
Author
There was a problem hiding this comment.
I didn't think we'd want users to write this; only CI VALIDATION.
Member
There was a problem hiding this comment.
And you can put this in the rucio profile Which is not what you should do because one can't remove/overwrite things from Rucio
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Briefly, what does this PR introduce? Please link to any relevant presentations or discussions.
rucio:andxrootd:config blocks tosnakemake.ymlwith JLab defaults and commented BNL alternativesSnakefile: add_rucio_upload_enabledflag,_rucio_output()helper, and shared_RUCIO_DID_PREFIX = rucio://{scope}/VALIDATION/benchmarks/detector/{job_id}Snakefile: add configurable_xrootd_base_url; fixget_remote_path()to support Rucio readsSnakefile: integrate Rucio storage outputs directly into the four producing rules (sim, particles_analysis, pi0_analysis, pion_rejection_analysis)profiles/rucio/config.yamlwithstorage-rucio-upload-rsesettingREADME: document Rucio upload configurationWhat is the urgency of this PR?
What kind of change does this PR introduce?
Please check if any of the following apply
Developed together with copilot. Wish I didn't have to steer it as mush as I did...