Add logic to restart from crashed jobs

[mattwthompson]

After #104, checkpoint files are used to capture state between apps. These ought to also be saved periodically during a simulation so that a crashed or cancelled job can be restarted partway through