uppmax_commands

UPPMAX MILOU COMMANDS

uquota
Will tell you how much space you are using compared to how much was allocated for this account

projinfo
Will tell you the amount of hours that you have used compared to how much hours you are allowed to use on MILOU
Default: 20000 hours per month
projinfo -M halvan
Will tell you the amount of hours that you have used compared to how much hours you are allowed to use on HALVAN

jobinfo | less -S
Will tell you about jobs that are currently running and in queue on MILOU
jobinfo -M halvan
Will tell you about jobs that are currently running and in queue on HALVAN

projmembers <projectid>
Will tell you accounts that are linked to a given project id

GET AN INTERACTIVE NODE
To get an interactive node, enter 
interactive -A <project_id>
You can always check your project_id with projinfo
If you want to use an interactive node just to test some things quickly you can apply for a short session with high priority
interactive -A <project_id> --qos=short

THE BATCH JOB
batch_job.sh:
===========================
#!/bin/bash -1

#SBATCH -A b2013066
#SBATCH -p core [core or node]
#SBATCH -n 1 [number of cores (32 if 2 nodes, 16 if single node, etc)]
#SBATCH -N 1 [number of nodes]
#SBATCH -t 12:00:00
#SBATCH -J some_job_name
--qos=short [if job is expected to be less then 15 minutes]
if specifying --qos=short, provide -t with less than 15 minutes

code...
===========================
Then
sbatch batch_job.sh

Alternatively, specify sbatch parameters on command line:
sbatch \
       --mail-user <email address> \
       -A <project_id> \
        -p node \
	-n <num_cores> \
	-t <days-hours:minutes:seconds> \
	-J <job_name> \
	-e <STDERR output> \
	-o <STDOUT output>
	<job_script> [job_script_argument1] [job_script_argument2] [etc]

LOADING MODULES
module avail				List available software
module spider <software>	     	Search for particular software
module load <software>/<version>	Load the software
module unload <software>		Unload the software
module purge  				Unload all loaded softwares
module help   				Find more information about module commands

MONITORING JOBS
jobinfo
jobinfo -u joranm
squeue
squeue -u joranm

SQUEUE SCREEN
The 'ST' field refers to the STate of the process
   'CG' = Job is in the process of completing
   'R'  = Job is running
   'PD' = Job is pending

MONITORING RUNNING JOBS
column -t /sw/share/slurm/milou/uppmax_jobstats/<node>/<job_id> | less -S

MONITORING STOPPED JOBS
finishedjobinfo

jobstats -p <jobid>
Gives you a graphical overview of how much memory and processing power the job is using over time. 
Requires graphical ssh connection. 
View the graph with
eog milou-<groupname>-<username>-<jobid>.png

jobstats -p -A <project id>
Shows you a history of jobs run in this project

CANCELING JOBS
scancel <jobid1> <jobid2> <etc>			Kills multiple jobs
scancel -i -u joranm				Kills all commands, but asks if you are really sure for each of them
scancel -u joranm --state=pending	      	Kills all jobs that are currently in queue
scancel -u joranm -n <jobname> -t running	Kills all running jobs with the name [jobname]

NODE INFORMATION
Standard	128GB RAM, 16 cores, -> 8GB RAM per core
Fat		512GB RAM, specify with '-C fat'. There are 17 of these nodes
256		256GB RAM, specify with '-C mem256GB'. There are 17 of these nodes
512		512GB RAM, specify with '-C mem512GB'. There are 17 of these nodes

WRITING JOB OUTPUT TO LOCAL DISK, NOT TO PICA
By default, a job will write output directly to the pica filesystem.
This requires a network connection, which may slow down the entire network, if used too much.
The admins therefore recommend you to write output to the disk of the node that your job is running on instead.
Thus, no network connection is required, and it will be faster as well.

You can use the special bash variable $SNIC_TMP in your job submission script to figure out on what node the job is done on.
For example, if you submit a job, and once it has started, it has be assigned to node m84, the $SNIC_TMP=m84.

Thus, in your job submission script:
PICADIR=$(pwd)
cp <input_files> $SNIC_TMP
<analysis_software_command> <input_files> > <output_files>
cp <output_files> $PICADIR

The output files on m84 or $SNIC_TMP will be wiped after a while

LOGGING IN TO WORKNODES
Sometimes you may want to check whether your local node analysis are producing any output before the job is finished.
To do that, simply check the node that your job is running on, and the jobid, with squeue, and then login to that node:

ssh joranm@m84
cd /scratch/<jobid>/

note: you can only login to nodes on which you have active jobs running

CPU TIME
1 corehour	-> Running 1 core for one 'wallclock' hour
16 corehours 	-> Running 1 node for one 'wallclock' hour

DISK INFORMATION
Also on Uppmax you have a home and your project folders.
You have about 32 GB available on your home.

RUNNING STUFF ON HALVAN
It is more or less the same but you need to specify that you run on HALVAN.
sbatch \
       -A <project_id> \
       -M halvan \
       -p halvan \
       -n <num_cores> \
       -t <days-hours:minutes:seconds> \
       -J <jobname> \
       <job_script> [arg1] [arg2] [etc]

The number of cores you set decides how much memory you demand from Halvan.
For example, 64 demands exclusive use of Halvan and thus 2048 GB RAM, whereas 32 demands only 1024 GB RAM

GENERAL TIPS
Use "rsync" to transfer files between Uppmax and your machine

POTENTIAL ISSUES
If a job runs, but generates output files of size 0, you might be over space quota. Clean your disks to create space