From 8a7140f283ca00dae5b5e2a6dd118ecfd3a25dc5 Mon Sep 17 00:00:00 2001 From: Tyler Collins Date: Tue, 11 Aug 2026 19:06:51 -0400 Subject: [PATCH 1/5] Add AGENTS.md for alliance clusters best practices --- agents/AGENTS.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 agents/AGENTS.md diff --git a/agents/AGENTS.md b/agents/AGENTS.md new file mode 100644 index 0000000..6ce1f67 --- /dev/null +++ b/agents/AGENTS.md @@ -0,0 +1,60 @@ +# Alliance cluster usage + +This is a shared HPC environment used by many researchers. Be conservative, +respect shared resources, and prefer small, verifiable steps. Consult the +Alliance documentation before improvising: https://docs.alliancecan.ca + +## Keep login-node work lightweight + +The current working directory may be on a shared login node. On login nodes, +limit work to lightweight operations such as editing files, reading code, +small Git operations, and short commands. + +Do not run heavy, long-running, or parallel workloads on a login node. This +includes large builds, training or inference, large data processing, and +multi-core or GPU workloads. If unsure whether a task is lightweight, assume +it is not. + +## Use the scheduler for non-trivial work + +For interactive work, request an allocation: + + salloc --account= --time=... --cpus-per-task=... --mem=... + +Run unattended work through an `sbatch` script. Before starting anything that +may consume significant CPU, memory, GPU resources, or wall time, stop and ask +the user to move the work into an allocation. + +## Use the Alliance Python wheelhouse + +The Alliance wheelhouse provides prebuilt, cluster-optimized Python packages. +It is a package source, not a replacement for a virtual environment. + +Load an available Python module, create a virtual environment, and install +packages from the wheelhouse: + + module load python/ + python -m venv + pip install --no-index + +Prefer the wheelhouse over PyPI. Avoid Conda and arbitrary internet +installations unless the user explicitly requests otherwise. + +## Use the appropriate filesystem + +- **home**: Backed up, with a small quota. Use for configuration, source code, + and small persistent files. +- **project**: Backed up and shared with the sponsor or PI group. Use for + cleaned results intended to persist or be shared. +- **scratch**: Not backed up and periodically purged. Use for active work, and + never keep the only copy of important data there. + +The usual flow is to work in scratch, move cleaned results to project, and +keep configuration in home. Check quota before large writes. If an operation +could exhaust quota, stop and inform the user. + +## Use environment modules + +Software is provided through environment modules. Use `module avail` and +`module load` to discover and activate software. Consult the Alliance +documentation for setup details rather than guessing. From 32da2e439211cedc9314b98ed145dc5b88ee4a25 Mon Sep 17 00:00:00 2001 From: Tyler Collins Date: Wed, 12 Aug 2026 11:47:19 -0400 Subject: [PATCH 2/5] address review comments --- agents/AGENTS.md | 46 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/agents/AGENTS.md b/agents/AGENTS.md index 6ce1f67..404c95b 100644 --- a/agents/AGENTS.md +++ b/agents/AGENTS.md @@ -4,6 +4,10 @@ This is a shared HPC environment used by many researchers. Be conservative, respect shared resources, and prefer small, verifiable steps. Consult the Alliance documentation before improvising: https://docs.alliancecan.ca +Do not try to be creative by working outside of the practices documented in +the Alliance documentation. When in doubt, suggest to the user to contact our +technical support for guidance. + ## Keep login-node work lightweight The current working directory may be on a shared login node. On login nodes, @@ -15,15 +19,24 @@ includes large builds, training or inference, large data processing, and multi-core or GPU workloads. If unsure whether a task is lightweight, assume it is not. +Not all compute nodes have access to the internet. You may need to stage +tarballs or wheels. + ## Use the scheduler for non-trivial work For interactive work, request an allocation: - salloc --account= --time=... --cpus-per-task=... --mem=... +```sh +salloc --account= --time=... --cpus-per-task=1 --mem=1G +``` Run unattended work through an `sbatch` script. Before starting anything that may consume significant CPU, memory, GPU resources, or wall time, stop and ask -the user to move the work into an allocation. +the user to move the work into an interactive job. Request the minimal amount +of resources necessary to run the job always. Wasting resources impacts other +users. Limit test jobs. Avoid jobs with run time shorter than 15 minutes, as +these can be bundled together. Jobs longer than 12 hours should implement +checkpointing. ## Use the Alliance Python wheelhouse @@ -33,25 +46,34 @@ It is a package source, not a replacement for a virtual environment. Load an available Python module, create a virtual environment, and install packages from the wheelhouse: - module load python/ - python -m venv - pip install --no-index +```sh +module load python/ +python -m venv +pip install --no-index +``` + +This can also be done within jobs via `$SLURM_TMPDIR`. Prefer the wheelhouse +over PyPI. Use `requirements.txt` files. Avoid `uv`, Conda, and arbitrary +internet installations unless the user explicitly requests otherwise. Missing +wheels can be installed via a support ticket with the Alliance. -Prefer the wheelhouse over PyPI. Avoid Conda and arbitrary internet -installations unless the user explicitly requests otherwise. +## Use the filesystems -## Use the appropriate filesystem +Do not try to access files outside of the user's home, project, or scratch. Do +not try to access files of other users. These are networked filesystems, and +patterns which write frequently in fast loops are destructive. - **home**: Backed up, with a small quota. Use for configuration, source code, and small persistent files. - **project**: Backed up and shared with the sponsor or PI group. Use for cleaned results intended to persist or be shared. -- **scratch**: Not backed up and periodically purged. Use for active work, and - never keep the only copy of important data there. +- **scratch**: Not backed up and periodically purged. Used for active work, + temporary files, logs, and checkpoints. Never keep the only copy of important + data there. The usual flow is to work in scratch, move cleaned results to project, and -keep configuration in home. Check quota before large writes. If an operation -could exhaust quota, stop and inform the user. +keep configuration in home. Check `diskusage_report` before large writes. If +an operation could exhaust quota, stop and inform the user. ## Use environment modules From cdb14c28cad17553993fb8db96c2ae7c62472dec Mon Sep 17 00:00:00 2001 From: Tyler Collins Date: Wed, 12 Aug 2026 13:18:03 -0400 Subject: [PATCH 3/5] fix python virtualenv workflow to match docs --- agents/AGENTS.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/agents/AGENTS.md b/agents/AGENTS.md index 404c95b..0a13d58 100644 --- a/agents/AGENTS.md +++ b/agents/AGENTS.md @@ -48,7 +48,8 @@ packages from the wheelhouse: ```sh module load python/ -python -m venv +virtualenv --no-download +source /bin/activate pip install --no-index ``` From 31337d91e7691a0142236a76773ac114e587dc8f Mon Sep 17 00:00:00 2001 From: Tyler Collins Date: Thu, 13 Aug 2026 10:34:06 -0400 Subject: [PATCH 4/5] integrate BYU ideas --- agents/AGENTS.md | 43 +++++++++++++++++++++++++++++++------------ 1 file changed, 31 insertions(+), 12 deletions(-) diff --git a/agents/AGENTS.md b/agents/AGENTS.md index 0a13d58..5559ba2 100644 --- a/agents/AGENTS.md +++ b/agents/AGENTS.md @@ -4,15 +4,18 @@ This is a shared HPC environment used by many researchers. Be conservative, respect shared resources, and prefer small, verifiable steps. Consult the Alliance documentation before improvising: https://docs.alliancecan.ca -Do not try to be creative by working outside of the practices documented in -the Alliance documentation. When in doubt, suggest to the user to contact our -technical support for guidance. +Do not improvise outside the practices described in the Alliance +documentation. When in doubt, suggest that the user contact technical support +for guidance. + +Human users remain responsible for the actions of their agents and tools. ## Keep login-node work lightweight The current working directory may be on a shared login node. On login nodes, limit work to lightweight operations such as editing files, reading code, -small Git operations, and short commands. +small Git operations, and short commands. Resource limits are enforced with +cgroups. Do not run heavy, long-running, or parallel workloads on a login node. This includes large builds, training or inference, large data processing, and @@ -22,6 +25,9 @@ it is not. Not all compute nodes have access to the internet. You may need to stage tarballs or wheels. +Tools such as VS Code often leave stale processes that can affect other users. +Point these processes out and offer to terminate them. + ## Use the scheduler for non-trivial work For interactive work, request an allocation: @@ -32,11 +38,17 @@ salloc --account= --time=... --cpus-per-task=1 --mem=1G Run unattended work through an `sbatch` script. Before starting anything that may consume significant CPU, memory, GPU resources, or wall time, stop and ask -the user to move the work into an interactive job. Request the minimal amount -of resources necessary to run the job always. Wasting resources impacts other -users. Limit test jobs. Avoid jobs with run time shorter than 15 minutes, as -these can be bundled together. Jobs longer than 12 hours should implement -checkpointing. +the user to move the work into an interactive job. Always request only the +resources necessary to run the job; wasting resources affects other users. +Limit test jobs. Bundle tasks that would otherwise run for less than 15 minutes. +Jobs longer than 12 hours should implement checkpointing. Specifying +unnecessary partitions or features will result in longer wait times. + +Do not use tight polling loops against Slurm. Wait at least 60 seconds between +queries, and avoid multiple monitoring loops. Prefer scheduler-native +mechanisms such as job dependencies. + +Never insert sleep commands into jobs. ## Use the Alliance Python wheelhouse @@ -60,9 +72,10 @@ wheels can be installed via a support ticket with the Alliance. ## Use the filesystems -Do not try to access files outside of the user's home, project, or scratch. Do -not try to access files of other users. These are networked filesystems, and -patterns which write frequently in fast loops are destructive. +Do not try to access files outside the user's home, project, or scratch. Do +not try to access other users' files. These are networked filesystems, and +frequent writes in tight loops can be disruptive. When managing many small +files, consider containers or aggregate formats such as tar archives or HDF5. - **home**: Backed up, with a small quota. Use for configuration, source code, and small persistent files. @@ -76,6 +89,12 @@ The usual flow is to work in scratch, move cleaned results to project, and keep configuration in home. Check `diskusage_report` before large writes. If an operation could exhaust quota, stop and inform the user. +`$SLURM_TMPDIR` is available to each job for its duration. Use it for +I/O-intensive workflows. + +Globus is recommended for file transfers. Other options include Open OnDemand, +`rsync`, and `scp`. + ## Use environment modules Software is provided through environment modules. Use `module avail` and From ca49c8f15a359ddae4b46a5d56c937c422bfef62 Mon Sep 17 00:00:00 2001 From: Tyler Collins Date: Thu, 13 Aug 2026 11:00:51 -0400 Subject: [PATCH 5/5] remove manual line wrap --- agents/AGENTS.md | 76 ++++++++++++------------------------------------ 1 file changed, 19 insertions(+), 57 deletions(-) diff --git a/agents/AGENTS.md b/agents/AGENTS.md index 5559ba2..3df75fe 100644 --- a/agents/AGENTS.md +++ b/agents/AGENTS.md @@ -1,32 +1,20 @@ # Alliance cluster usage -This is a shared HPC environment used by many researchers. Be conservative, -respect shared resources, and prefer small, verifiable steps. Consult the -Alliance documentation before improvising: https://docs.alliancecan.ca +This is a shared HPC environment used by many researchers. Be conservative, respect shared resources, and prefer small, verifiable steps. Consult the Alliance documentation before improvising: https://docs.alliancecan.ca -Do not improvise outside the practices described in the Alliance -documentation. When in doubt, suggest that the user contact technical support -for guidance. +Do not improvise outside the practices described in the Alliance documentation. When in doubt, suggest that the user contact technical support for guidance. Human users remain responsible for the actions of their agents and tools. ## Keep login-node work lightweight -The current working directory may be on a shared login node. On login nodes, -limit work to lightweight operations such as editing files, reading code, -small Git operations, and short commands. Resource limits are enforced with -cgroups. +The current working directory may be on a shared login node. On login nodes, limit work to lightweight operations such as editing files, reading code, small Git operations, and short commands. Resource limits are enforced with cgroups. -Do not run heavy, long-running, or parallel workloads on a login node. This -includes large builds, training or inference, large data processing, and -multi-core or GPU workloads. If unsure whether a task is lightweight, assume -it is not. +Do not run heavy, long-running, or parallel workloads on a login node. This includes large builds, training or inference, large data processing, and multi-core or GPU workloads. If unsure whether a task is lightweight, assume it is not. -Not all compute nodes have access to the internet. You may need to stage -tarballs or wheels. +Not all compute nodes have access to the internet. You may need to stage tarballs or wheels. -Tools such as VS Code often leave stale processes that can affect other users. -Point these processes out and offer to terminate them. +Tools such as VS Code often leave stale processes that can affect other users. Point these processes out and offer to terminate them. ## Use the scheduler for non-trivial work @@ -36,27 +24,17 @@ For interactive work, request an allocation: salloc --account= --time=... --cpus-per-task=1 --mem=1G ``` -Run unattended work through an `sbatch` script. Before starting anything that -may consume significant CPU, memory, GPU resources, or wall time, stop and ask -the user to move the work into an interactive job. Always request only the -resources necessary to run the job; wasting resources affects other users. -Limit test jobs. Bundle tasks that would otherwise run for less than 15 minutes. -Jobs longer than 12 hours should implement checkpointing. Specifying -unnecessary partitions or features will result in longer wait times. +Run unattended work through an `sbatch` script. Before starting anything that may consume significant CPU, memory, GPU resources, or wall time, stop and ask the user to move the work into an interactive job. Always request only the resources necessary to run the job; wasting resources affects other users. Limit test jobs. Bundle tasks that would otherwise run for less than 15 minutes. Jobs longer than 12 hours should implement checkpointing. Specifying unnecessary partitions or features will result in longer wait times. -Do not use tight polling loops against Slurm. Wait at least 60 seconds between -queries, and avoid multiple monitoring loops. Prefer scheduler-native -mechanisms such as job dependencies. +Do not use tight polling loops against Slurm. Wait at least 60 seconds between queries, and avoid multiple monitoring loops. Prefer scheduler-native mechanisms such as job dependencies. Never insert sleep commands into jobs. ## Use the Alliance Python wheelhouse -The Alliance wheelhouse provides prebuilt, cluster-optimized Python packages. -It is a package source, not a replacement for a virtual environment. +The Alliance wheelhouse provides prebuilt, cluster-optimized Python packages. It is a package source, not a replacement for a virtual environment. -Load an available Python module, create a virtual environment, and install -packages from the wheelhouse: +Load an available Python module, create a virtual environment, and install packages from the wheelhouse: ```sh module load python/ @@ -65,38 +43,22 @@ source /bin/activate pip install --no-index ``` -This can also be done within jobs via `$SLURM_TMPDIR`. Prefer the wheelhouse -over PyPI. Use `requirements.txt` files. Avoid `uv`, Conda, and arbitrary -internet installations unless the user explicitly requests otherwise. Missing -wheels can be installed via a support ticket with the Alliance. +This can also be done within jobs via `$SLURM_TMPDIR`. Prefer the wheelhouse over PyPI. Use `requirements.txt` files. Avoid `uv`, Conda, and arbitrary internet installations unless the user explicitly requests otherwise. Missing wheels can be installed via a support ticket with the Alliance. ## Use the filesystems -Do not try to access files outside the user's home, project, or scratch. Do -not try to access other users' files. These are networked filesystems, and -frequent writes in tight loops can be disruptive. When managing many small -files, consider containers or aggregate formats such as tar archives or HDF5. +Do not try to access files outside the user's home, project, or scratch. Do not try to access other users' files. These are networked filesystems, and frequent writes in tight loops can be disruptive. When managing many small files, consider containers or aggregate formats such as tar archives or HDF5. -- **home**: Backed up, with a small quota. Use for configuration, source code, - and small persistent files. -- **project**: Backed up and shared with the sponsor or PI group. Use for - cleaned results intended to persist or be shared. -- **scratch**: Not backed up and periodically purged. Used for active work, - temporary files, logs, and checkpoints. Never keep the only copy of important - data there. +- **home**: Backed up, with a small quota. Use for configuration, source code, and small persistent files. +- **project**: Backed up and shared with the sponsor or PI group. Use for cleaned results intended to persist or be shared. +- **scratch**: Not backed up and periodically purged. Used for active work, temporary files, logs, and checkpoints. Never keep the only copy of important data there. -The usual flow is to work in scratch, move cleaned results to project, and -keep configuration in home. Check `diskusage_report` before large writes. If -an operation could exhaust quota, stop and inform the user. +The usual flow is to work in scratch, move cleaned results to project, and keep configuration in home. Check `diskusage_report` before large writes. If an operation could exhaust quota, stop and inform the user. -`$SLURM_TMPDIR` is available to each job for its duration. Use it for -I/O-intensive workflows. +`$SLURM_TMPDIR` is available to each job for its duration. Use it for I/O-intensive workflows. -Globus is recommended for file transfers. Other options include Open OnDemand, -`rsync`, and `scp`. +Globus is recommended for file transfers. Other options include Open OnDemand, `rsync`, and `scp`. ## Use environment modules -Software is provided through environment modules. Use `module avail` and -`module load` to discover and activate software. Consult the Alliance -documentation for setup details rather than guessing. +Software is provided through environment modules. Use `module avail` and `module load` to discover and activate software. Consult the Alliance documentation for setup details rather than guessing.