Output of runSubmit.py

[nrsegovia]

I have recently started using the compas_run_submit utility, and noticed that my --output-container and --output-path settings are always ignored (a new directory is created wherever the script is run from, using default names). Looking at the source code, it seems that the priority was to develop the utility around docker implementations of COMPAS? And it was assumed that some environment variables would always be set. In my particular case, not having set COMPAS_LOGS_OUTPUT_DIR_PATH makes the script ignore the YAML options and use default values. Of course, I could "fix" the issue by setting those variables, but it would make the --output-container and --output-path options on the YAML file superfluous.

On a slightly related note: While looking at the source code I also noticed that the pythonProgramOptions class allows for extra arguments (random seed and output directory) at initialization, but these are not part of the CLI options. What should be done about them? If they are not meant to be accessible, then there is no need to keep them as arguments.

[ilyamandel]

I have a feeling that once we started using the YAML files, a lot of us stopped using runSubmit -- so we should decide whether to maintain it or do away with it. Maybe this is a conversation for after @jeffriley 's major re-write is complete?

[jeffriley]

@nrsegovia @ilyamandel

I don't use compas_run_submit - I always run COMPAS from the command line. Do we know if COMPAS at the command line honours the --output-container and --output-path options (it used to, but things change...)? I can check this, but it might be quicker if you do @nrsegovia. If the problem is a COMPAS code problem (rather than a compas_run_submit problem) I'll track it down. If it's a compas_run_submit it should be easily fixed - maybe it's just missing those options (I haven't looked).

[jeffriley]

Actually, doesn't runSubmit use the yaml file? I thought it did... The yaml template has those options in it. The command line check will tel us where to look to fix the problem.

[nrsegovia]

@ilyamandel

I have a feeling that once we started using the YAML files, a lot of us stopped using runSubmit -- so we should decide whether to maintain it or do away with it. Maybe this is a conversation for after @jeffriley 's major re-write is complete?

I'm slightly confused. Does this mean that it is possible to use the YAML files without having to go through python? I'm using the script because the YAML files are quite convenient and I thought there was no other way of using them. I used to run COMPAS using shell scripts specifying all options by hand.

@jeffriley

Actually, doesn't runSubmit use the yaml file? I thought it did... The yaml template has those options in it. The command line check will tel us where to look to fix the problem.

I think that the issue is described as follows: line 41 loads the YAML as a dictionary. Then, line 84 checks whether the environment variable COMPAS_LOGS_OUTPUT_DIR_PATH has been set. If it has not been set (which I think is always true, unless COMPAS is being run from Docker), --output-path gets set to cwd, while --output-container becomes None since this is the default of output_directory (and is never changed). It seems to me that output_directory was supposed to become a CLI option, but implementation was never completed.

The options work fine when running something simple like COMPAS --output-path /home/user/compas-data/output --output-container test.

[jeffriley]

@nrsegovia @ilyamandel

Not trying to dodge owning the problem, but I didn't write it and don't use it, so I don't know how the author or users expect it to work. I would have thought it should pass all options as declared in the yaml file through to COMPAS, but I just don't know. I think @avigna wrote the version that uses the yaml file. I think I did make some changes at some stage, but I think those were early after the original refactor, and preceded the adoption of yaml files.