Remove stats file production

[David Davies (DJDavies2)]

Fix #291. It does a bit more than fix that issue in that it removes stats file generation altogether. This doesn't affect the ctests as they continue to pass but if there is some use for stats file generation from opsinputs that isn't covered by this then this change isn't going to be appropriate.

To be clear, the problem in #291 is that because the stats file is written to fort.7, running opsinputs in parallel can therefore result in concurrent writes to the same file although this is a transient failure.

Test output:

https://cylchub/services/cylc-review/taskjobs/david.davies/?suite=OpsinputsStatsFile

[Adam Maycock (adammaycock)]

I believe the "stats" file contains only a line informing how many obs were output to varobs. I can't see any use for this, so happy with this change.

[Michael Cooke (mikecooke77)]

I can see why these need to be removed. Is this functionality that was used in OPS? Could it have utility here?

I have made one comment about the location of the changes files but apart from that I'm fine with this change.

[Michael Cooke (mikecooke77)]

I think now this is modified it should move to the stubs directory as it has diverged from the original file copied from OPS.

[David Davies (DJDavies2)]

Do you mean just this file or the whole directory?

[Michael Cooke (mikecooke77)]

I think now this is modified it should move to the stubs directory as it has diverged from the original file copied from OPS.

[David Davies (DJDavies2)]

Do you mean just this file or the whole directory?

[Michael Cooke (mikecooke77)]

I think now this is modified it should move to the stubs directory as it has diverged from the original file copied from OPS.

[David Davies (DJDavies2)]

Do you mean just this file or the whole directory?

[David Davies (DJDavies2)]

I can see why these need to be removed. Is this functionality that was used in OPS? Could it have utility here?

I have made one comment about the location of the changes files but apart from that I'm fine with this change.

Stats files were certainly used by people, they are produced operationally and were used to check on things like obs counts and obs stats. Adam Maycock (@adammaycock) probably knows more about these uses than me but they are/were certainly used.

However if we are working with ODB's I would assume that in a Jedi context there isn't anything in a stats file that cannot be obtained from an ODB. This is probably true of OPS as well, but stats files predate ODB in OPS by many years.

The CODB metdb to odb tasks that generate odb's via OpsScr_CreateODB still generate stats files, although all they do is write out ob counts. However that code is not affected as that is still using OPS code.

[Adam Maycock (adammaycock)]

I can see why these need to be removed. Is this functionality that was used in OPS? Could it have utility here?
I have made one comment about the location of the changes files but apart from that I'm fine with this change.

Stats files were certainly used by people, they are produced operationally and were used to check on things like obs counts and obs stats. Adam Maycock (Adam Maycock (@adammaycock)) probably knows more about these uses than me but they are/were certainly used.

However if we are working with ODB's I would assume that in a Jedi context there isn't anything in a stats file that cannot be obtained from an ODB. This is probably true of OPS as well, but stats files predate ODB in OPS by many years.

The CODB metdb to odb tasks that generate odb's via OpsScr_CreateODB still generate stats files, although all they do is write out ob counts. However that code is not affected as that is still using OPS code.

The stats files generated by OPS were certainly useful and used for a number of monitoring applications. They contained mean / RMS o-b stats, and rejection percentages etc. The stats files currently generated by JOPA don't contain any of that, they only contain a "Total number of Varobs" line which is of limited use. So I don't see a problem with getting rid, as proposed here.

For info, there is an open discussion on getting OPS like stats from JOPA: MetOffice/jedi-discussion#71.

[Michael Cooke (mikecooke77)]

I think when Wojciech created this the idea was for OPS files that are modified in opsinputs they should be in stubs. For those files that are still comparable with OPS they should be kept where they are. This was to help if and when we sync the OPS code with this repository.

[David Davies (DJDavies2)]

Has such syncing ever happened? I looked through the changelog for the past couple of years and didn't see anything, although it might not have been described as such in the summaries. There were changes related to OPS, such as addition of support for EPSMWS, but these look like they have just been manually added to opsinputs, not synced in any way.

I think a policy of syncing something like OPS code in to opsinputs is dangerous. Code in OPS runs in a particular context and environment and it is not tested outside of that environment and no consideration is given to it running outside of that environment by developers. The stats file is a good example of this: there are top level Fortran programs that ensure that the file is create and closed as appropriate and that information is passed down to subroutines that write to it such as unit numbers. On top of that there are scripts that control locations and names of stats file via environment variables. It looks like opsinputs doesn't have any of this so it isn't surprising that this code is having difficulty.