Submission on LSF fails (Sanger cluster)

[jan-engelmann]

Hello,
thank you for this tool!
Unfortunately, I am not able to run the LSF minimal example.
The documentation was not sufficient to solve the problem myself.

When running:

from psij import Job, JobSpec, JobExecutor

ex = JobExecutor.get_instance("lsf")

spec = JobSpec(executable="/bin/date")
job = Job(spec)

res = ex.submit(job)

I get:

SubmitException: 
Returning output by mail is not supported on this cluster.
Please use the -o option to write output to disk.
Request aborted by esub. Job not submitted.

I think this might be due to hard-coded lines in lsf.mustache

{{!since we redirect the output manually, below, tell LSF not to do its own thing, since it
only results in empty files that are not cleaned up}}
#BSUB -e /dev/null
#BSUB -o /dev/null

I am able to submit the generated script by specifying -o to bsub:

bsub -o output.log genreated_script.sh

Would really appreciate help on this!

[hategan]

Thank you for your report. I do not have access to an LSF machine, so I might need some help in figuring this out. What happens when you pass -o /dev/null directly to bsub? In other words, does the following work or fail:

bsub -o /dev/null generated_script.sh

[jan-engelmann]

Thank you for looking into this @hategan!

bsub -o /dev/null generated_script.sh

works.

How can I specify this using the psij package?

Also, I'm struggling to set the right job specs.
How would you translate the following command for Psij?

bsub -o log.txt -G GROUP_NAME -M1000 -R"select[mem>1000] rusage[mem=1000] span[hosts=1]" -n 8 -q QUEUE_NAME /bin/date

I tried as below but couldnt quite get it right

import logging

from psij import Job, JobSpec, JobExecutor, ResourceSpecV1

ex = JobExecutor.get_instance("lsf")

spec = JobSpec(
    executable="/bin/date",
    resources=ResourceSpecV1(
        node_count=1,
        processes_per_node=1,
        memory=1_024 * 1024,
    ),
)
spec.attributes.queue_name = "normal"
spec.attributes.custom_attributes = {}
spec.attributes.custom_attributes['lsf.R'] = "select[mem>1024] rusage[mem=1024] span[hosts=1]"

job = Job(spec)

self = ex

self._ensure_work_dir()

self._check_job(job)

context = self._create_script_context(job)

# assumes job ids are unique
submit_file_path = self.work_directory / (job.id + ".job")
with submit_file_path.open("w") as submit_file:
    self.generate_submit_script(job, context, submit_file)

this outputs:

#!/bin/bash
#BSUB -env all
#BSUB -nnodes 1
#BSUB -n 1
#BSUB -M 1024KB
#BSUB -W 0:10
#BSUB -q "normal"
#BSUB -R "'select[mem>1024] rusage[mem=1024] span[hosts=1]'"
#BSUB -e /dev/null
#BSUB -o /dev/null
PSIJ_NODEFILE="$LSB_HOSTS"
export PSIJ_NODEFILE
exec &>> "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.out"
echo "$?" > "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.ec"

[hategan]

Thank you for looking into this @hategan!

bsub -o /dev/null generated_script.sh

works.

That is odd. What is the contents of generated_script.sh?

How can I specify this using the psij package?

That is specified through the #BSUB -o /dev/null line(s). The fact that somehow it doesn't work is a bit puzzling.

Also, I'm struggling to set the right job specs. How would you translate the following command for Psij?

This might get a bit verbose, so it might be useful to submit a separate issue in the future if you have a question/issue different than the original one.

bsub -o log.txt -G GROUP_NAME -M1000 -R"select[mem>1000] rusage[mem=1000] span[hosts=1]" -n 8 -q QUEUE_NAME /bin/date

I tried as below but couldnt quite get it right

import logging

from psij import Job, JobSpec, JobExecutor, ResourceSpecV1

ex = JobExecutor.get_instance("lsf")

spec = JobSpec(
    executable="/bin/date",
    resources=ResourceSpecV1(
        node_count=1,
        processes_per_node=1,
        memory=1_024 * 1024,
    ),
)
spec.attributes.queue_name = "normal"
spec.attributes.custom_attributes = {}
spec.attributes.custom_attributes['lsf.R'] = "select[mem>1024] rusage[mem=1024] span[hosts=1]"

job = Job(spec)

self = ex

self._ensure_work_dir()

self._check_job(job)

context = self._create_script_context(job)

# assumes job ids are unique
submit_file_path = self.work_directory / (job.id + ".job")
with submit_file_path.open("w") as submit_file:
    self.generate_submit_script(job, context, submit_file)

this outputs:

#!/bin/bash
#BSUB -env all
#BSUB -nnodes 1
#BSUB -n 1
#BSUB -M 1024KB
#BSUB -W 0:10
#BSUB -q "normal"
#BSUB -R "'select[mem>1024] rusage[mem=1024] span[hosts=1]'"
#BSUB -e /dev/null
#BSUB -o /dev/null
PSIJ_NODEFILE="$LSB_HOSTS"
export PSIJ_NODEFILE
exec &>> "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.out"
echo "$?" > "/nfs/users/nfs_j/USER/.psij/work/lsf/$LSB_JOBID.ec"

I am not sure I fully follow. What is it that is different from what you were expecting?
Also, the launch line seems to be missing. Did you remove it by accident when reformatting the output?