payu runs concat_ice_daily.py after each run segment (via postscript.sh), but this occasionally fails with dask errors (in my experience 4-5 times out of 66 one-year run segments).
It would be nice if this would be automatically recovered from, but at present concat_ice_daily.py is only applied to the most recent run output, so previous failed concatentations never get fixed.
Should it do this instead?
- loop over all output directories
- if output directory contains files of the form
*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].nc then we have a concatenation failure (because these files are deleted on successful concatenation), so
- delete (or move to a DELETE subdirectory for safety?) any concatenated files of the form
*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-1][0-9].nc, as these will stop new attempts at concatenation and may be corrupt
- do
python3 $SCRIPTS_DIR/payu_config/postscript/concat_ice_daily.py --directory <output_directory_path> so that payu will clean up any previous failures in subsequent runs
IIRC this is the approach taken with collation and syncing.
This assumes all the required unconcatenated files*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].nc are still present.
payu runs concat_ice_daily.py after each run segment (via postscript.sh), but this occasionally fails with dask errors (in my experience 4-5 times out of 66 one-year run segments).
It would be nice if this would be automatically recovered from, but at present concat_ice_daily.py is only applied to the most recent run output, so previous failed concatentations never get fixed.
Should it do this instead?
*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].ncthen we have a concatenation failure (because these files are deleted on successful concatenation), so*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-1][0-9].nc, as these will stop new attempts at concatenation and may be corruptpython3 $SCRIPTS_DIR/payu_config/postscript/concat_ice_daily.py --directory <output_directory_path>so that payu will clean up any previous failures in subsequent runsIIRC this is the approach taken with collation and syncing.
This assumes all the required unconcatenated files
*cice.1day.*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].ncare still present.