Refactor: shared job submission utility (eliminates duplication across 22 scripts)

[GernotMaier]

Summary

Refactors the duplicated job submission logic found in all analysis scripts into a single shared utility, eliminating ~320 lines of duplicated code and all remaining SC2086 shellcheck warnings.

Problem

Every one of the 22 analysis/IRF/SPANALYSIS scripts contained an identical (or nearly identical) 15–20 line if/elif block for job submission:

if [[ $SUBC == *qsub* ]]; then
    # shellcheck disable=SC2086
    JOBID=$($SUBC "$FSCRIPT".sh)
    ...
elif [[ $SUBC == *condor* ]]; then
    ...
elif [[ $SUBC == *sbatch* ]]; then
    # shellcheck disable=SC2086
    $SUBC "$FSCRIPT".sh
elif [[ $SUBC == *parallel* ]]; then
    ...
elif [[ "$SUBC" == *simple* ]]; then
    ...
fi

This caused:

• Massive code duplication (same block repeated 22 times)
• All remaining # shellcheck disable=SC2086 suppressions (for $SUBC)
• Inconsistent condor/condor_submit behaviour across scripts

Solution

New file: scripts/helper_scripts/UTILITY.submitJob.sh

Provides two functions sourced by all scripts:

• submit_job <fscript> [parallel_line [parallel_file]] — handles all scheduler types (qsub, condor, condor_submit, sbatch, parallel, simple, test)
• run_parallel_jobs [parallel_file] — runs collected jobs with GNU parallel after the main loop

Inside the function, $SUBC is split into an array with read -ra subc_arr <<< "$SUBC", which is equivalent to the previously unquoted usage but shellcheck-clean.

Each of the 22 scripts now does

source "$(dirname "$0")/helper_scripts/UTILITY.submitJob.sh"
# ...
submit_job "$FSCRIPT.sh" "$FSCRIPT.sh &> $FSCRIPT.log" "$LOGDIR/runscripts.dat"
echo "RUN $FILE JOBID $JOBID"   # per-script echo kept in place

Changes

	Before	After
Lines of code	+469	+151
SC2086 disables	52	0
Shellcheck violations	0	0

Scripts updated (22): ANALYSIS.anasum, ANALYSIS.anasum_combine, ANALYSIS.anasum_parallel_from_runlist, ANALYSIS.dispXGB, ANALYSIS.evndisp, ANALYSIS.evndisp_laser, ANALYSIS.evndisp_muon, ANALYSIS.mscw_energy, ANALYSIS.v2dl3, IRF.combine_effective_area_parts, IRF.combine_lookup_table_parts, IRF.compress_evndisp_MC, IRF.generate_effective_area_parts, IRF.generate_lookup_table_parts, IRF.generate_radial_acceptance, IRF.mscw_energy_MC, IRF.optimizeTMVAforGammaHadronSeparation, IRF.trainTMVAforAngularReconstruction, IRF.trainTMVAforGammaHadronSeparation, IRF.trainXGBforAngularReconstructionBinned, SPANALYSIS.lowgainped, SPANALYSIS.make_DST

Bug fix (bonus)

The condor / condor_submit distinction in submissionCommands.dat was not consistently respected. Some scripts always called condor_submit regardless of which condor variant the user had selected. The shared function now correctly checks *condor_submit* before *condor*, so automatic submission only happens when the user explicitly selects condor_submit mode.

Testing

• shellcheck --severity=info scripts/*.sh scripts/helper_scripts/*.sh → 0 violations ✅
• No # shellcheck disable=SC2086 lines remain anywhere ✅

[Copilot]

Pull request overview

Refactors repeated scheduler submission logic across analysis/IRF/SPANALYSIS shell scripts into a shared helper, reducing duplicated code and centralizing qsub/condor/sbatch/parallel/simple handling.

Changes:

• Adds scripts/helper_scripts/UTILITY.submitJob.sh with submit_job and run_parallel_jobs.
• Replaces inline job-submission conditionals in converted scripts with calls to the shared utility.
• Adds a maintenance changelog entry for the refactor.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/helper_scripts/UTILITY.submitJob.sh	Adds shared submission and parallel execution helpers.
scripts/ANALYSIS.anasum.sh	Replaces inline submission logic with utility calls.
scripts/ANALYSIS.anasum_combine.sh	Replaces inline submission logic with utility calls.
scripts/ANALYSIS.anasum_parallel_from_runlist.sh	Uses shared submission and parallel runner.
scripts/ANALYSIS.dispXGB.sh	Uses shared submission helper.
scripts/ANALYSIS.evndisp.sh	Uses shared submission helper and writes shared parallel runner into Run_me.sh.
scripts/ANALYSIS.evndisp_laser.sh	Uses shared submission and parallel runner.
scripts/ANALYSIS.evndisp_muon.sh	Uses shared submission and parallel runner.
scripts/ANALYSIS.mscw_energy.sh	Uses shared submission and parallel runner.
scripts/ANALYSIS.v2dl3.sh	Uses shared submission and parallel runner.
scripts/IRF.combine_effective_area_parts.sh	Uses shared submission helper.
scripts/IRF.combine_lookup_table_parts.sh	Uses shared submission helper.
scripts/IRF.compress_evndisp_MC.sh	Uses shared submission helper.
scripts/IRF.generate_effective_area_parts.sh	Uses shared submission helper.
scripts/IRF.generate_lookup_table_parts.sh	Uses shared submission helper.
scripts/IRF.generate_radial_acceptance.sh	Uses shared submission and parallel runner.
scripts/IRF.mscw_energy_MC.sh	Uses shared submission helper.
scripts/IRF.optimizeTMVAforGammaHadronSeparation.sh	Uses shared submission and parallel runner.
scripts/IRF.trainTMVAforAngularReconstruction.sh	Uses shared submission helper.
scripts/IRF.trainTMVAforGammaHadronSeparation.sh	Uses shared submission and parallel runner.
scripts/IRF.trainXGBforAngularReconstructionBinned.sh	Uses shared submission helper.
scripts/SPANALYSIS.lowgainped.sh	Uses shared submission and parallel runner.
scripts/SPANALYSIS.make_DST.sh	Uses shared submission and parallel runner.
docs/changes/148.maintenance.md	Adds changelog fragment for the submission refactor.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.