Bart fails to recover from long slurm/cluster outage.

[kmjonsson]

Due to the fact that bart only looks for N days (default 7) into future from last check time an outage of more then N days will result in bart never recovers.

Bart should loop until "end_time" > "now" adding N days every loop to ensure that it will not stop reporting due to log outage.

[edelmann]

Yup.

The problem is this piece of code:

    # only update state if an entry is written
    if count > 0:
        self.state = tlp.end_str

I.e., it will update the the end_time of last run only if at least one job was found. Which obviously won't be the case if the cluster was down for longer than 'max_days' days.

Removing the 'if'-clause would be an simple solution, but it looks awfully deliberate, so I will have to give it some thought.

Possible work-arounds in the meantime:
Either set the parameter 'max_days' in the [slurm] section of the conf file large enough to bridge the gap, or hand edit /var/spool/bart/slurm.state (or wherever your state file is)

[kmjonsson]

Yes, there are several workarounds, but sometimes you just want it to work :-)

The reason for the if statement is to prevent sacct from return zero entries due to whatever reason.

I think the long term solution is to make it autoextend the time period with 'max_days' until "end_time" > "now". I might provide a PR for it later if time permits.