MetOffice · Michael Cooke (mikecooke77) · May 16, 2023 · May 3, 2023 · May 3, 2023 · May 3, 2023
@@ -1,6 +1,5 @@
 # (C) Crown Copyright 2020, the Met Office. All rights reserved.
 #
-# Refer to COPYRIGHT.txt of this distribution for details.
 
 ################################################################################
 # opsinputs

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2023, Met Office
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
@@ -1,15 +1,15 @@
 [![CI](https://github.com/MetOffice/opsinputs/actions/workflows/ci.yml/badge.svg)](https://github.com/MetOffice/opsinputs/actions/workflows/ci.yml)
 
-(C) Crown Copyright 2020, the Met Office. All rights reserved.
+&copy; Crown Copyright 2020, the Met Office. All rights reserved.
 
-Refer to COPYRIGHT.txt of this distribution for details.
+Refer to [COPYRIGHT](COPYRIGHT.txt) of this distribution for details.
 
 Overview
 ========
 
 This software makes it possible for NG-OPS to write observation data to files in the VarObs format and model data (geovals) to files in the Cx format.
 
-Files in both formats are read as inputs into the VAR data assimilation system at the Met Office. The VarObs file format is described in OTDP 16 (https://www-nwp/~opsrc/OPS/view/ops-latest/doc/OTDP16.html) and the Cx file format is described in OTDP 17 (https://www-nwp/~opsrc/OPS/view/ops-latest/doc/OTDP17.html).
+Files in both formats are read as inputs into the VAR data assimilation system at the Met Office. The VarObs file format is described in [OTDP 16](https://www-nwp/~opsrc/OPS/view/ops-latest/doc/OTDP16.html) and the Cx file format is described in [OTDP 17](https://www-nwp/~opsrc/OPS/view/ops-latest/doc/OTDP17.html).
 
 Dependencies
 ============
@@ -23,34 +23,25 @@ Note that both of the above need to be built with the same compiler and use the
 Building
 ========
 
-1. Clone the `ufo-bundle` into a folder of your choice. Consider using the `feature/wsmigaj` branch, which contains some Met Office-specific adjustments and includes the ropp-ufo package implementing operators for GNSSRO:
+1. Clone the `mo-bundle` into a folder of your choice.
 
-       git clone https://github.com/JCSDA/ufo-bundle.git -b feature/wsmigaj
-       cd ufo-bundle
-       cp -a ~jopa/jedi_repos/nightly/ropp-ufo .
+       git clone https://github.com/MetOffice/mo-bundle.git
+       cd mo-bundle
 
-2. Open the `CMakeLists.txt` file in the `ufo-bundle` directory and add the following line after all other lines starting with `ecbuild_bundle`:
+2. Open the `CMakeLists.txt` file in the `mo-bundle` directory and comment out the projects not required for opsinputs (i.e., lines beginning with ecbuild_bundle).
+    Note that `opsinputs` depends on `oops`, `ioda`, `ufo` and `ropp-ufo`.
 
-       ecbuild_bundle( PROJECT opsinputs GIT "https://github.com/MetOffice/opsinputs.git" BRANCH develop UPDATE)
 
-3. Run the following commands to set up your build environment:
+3. Run the following commands to set up your build environment (internal):
 
-       module use /home/h03/jopa/modulefiles
-       module load jedi/ufo-bundle-dev-stack
+       module use ~jopa/modulefiles
+       module load bb-env
 
-   (Alternatively, load jedi/ufo-bundle-dev-stack/intel if you want to use the Intel compiler or jedi/ufo-bundle-dev-stack/gnu/7.2.0 to use the GNU 7.2.0 compiler suite.)
+4. Configure and build:
 
-4. Create a build directory next to the `ufo-bundle` directory, enter it and run ecbuild to configure a build:
+       cmake --workflow --preset=vdi_gnu
 
-       ecbuild ../ufo-bundle
-
-5. Run make to build the the bundle:
-
-       make -j4
-
-   Note: if using the gfortran 7.2 compiler in the Debug configuration, you may need to remove the -finit-derived flag from the saber/cmake/compiler_flags_GNU_Fortran.cmake file to avoid an internal compiler error during compilation of some saber source files.
-
-8. Optionally, run tests to verify that components of the `opsinputs` package work correctly:
+5. Optionally, run tests to verify that components of the `opsinputs` package work correctly:
 
        ctest -R opsinputs
 
@@ -60,78 +51,81 @@ Usage
 VarObs and Cx files are written by the `VarObs Writer` and `Cx Writer` observation filters, respectively. See the Doxygen documentation of these filters in the `src/opsinputs/VarObsWriter.h` and `src/opsinputs/CxWriter.h` files (and the accompanying `...Parameters.h` files) for more information. 
 
 The following YAML snippet demonstrates the use of `VarObsWriter`. <ObsGroup> stands for the name of one of the observation groups known to OPS.
-
-    - ObsSpace:
-        name: <ObsGroup>
-        ObsDataIn:
-          obsfile: observations.nc4
-        simulate:
-          variables: [surface_pressure]        
-    - Filter: VarObs Writer
-        # The filter will output an <ObsGroup>.varobs file
-        # in the directory specified in the output_directory option.
-        output_directory: varobs
-        # By default, the filter will produce a VarObs file containing
-        # the varfields output by default by OPS for the observation
-        # group <ObsGroup>. The list of varfields can be changed by
-        # providing a namelist file <ObsGroup>.nl in the directory
-        # specified in the namelist_directory option. This file should
-        # contain a Fortran namelist in the format accepted by the
-        # Ops_ReadVarobsControlNL function from OPS.
-        namelist_directory: namelists
-        # Output only observations that passed the quality check in all
-        # variables.
-        reject_obs_with_any_variable_failing_qc: true
-        # Values of the following options are written to the UM fixed header
-        # embedded in the output VarObs file. In future (once the UM/LFRic JEDI
-        # interfaces are ready) they will probably be taken directly from
-        # the model. There are more options like this; the full list can be
-        # found in `src/opsinputs/VarObsWriterParameters.h`.
-        IC_XLen: 36
-        IC_YLen: 18
-        IC_PLevels: 8
-        IC_WetLevels: 9
-        RC_LongSpacing: 10
-        RC_LatSpacing: 10
+
+```yaml
+- ObsSpace:
+  name: <ObsGroup>
+  ObsDataIn:
+    obsfile: observations.nc4
+  simulate:
+    variables: [surface_pressure]
+- Filter: VarObs Writer
+  # The filter will output an <ObsGroup>.varobs file
+  # in the directory specified in the output_directory option.
+  output_directory: varobs
+  # By default, the filter will produce a VarObs file containing
+  # the varfields output by default by OPS for the observation
+  # group <ObsGroup>. The list of varfields can be changed by
+  # providing a namelist file <ObsGroup>.nl in the directory
+  # specified in the namelist_directory option. This file should
+  # contain a Fortran namelist in the format accepted by the
+  # Ops_ReadVarobsControlNL function from OPS.
+  namelist_directory: namelists
+  # Output only observations that passed the quality check in all variables.
+  reject_obs_with_any_variable_failing_qc: true
+  # Values of the following options are written to the UM fixed header
+  # embedded in the output VarObs file. In future (once the UM/LFRic JEDI
+  # interfaces are ready) they will probably be taken directly from
+  # the model. There are more options like this; the full list can be
+  # found in `src/opsinputs/VarObsWriterParameters.h`.
+  IC_XLen: 36
+  IC_YLen: 18
+  IC_PLevels: 8
+  IC_WetLevels: 9
+  RC_LongSpacing: 10
+  RC_LatSpacing: 10
+```
 
 And here is a YAML snippet demonstrating the use of `CxWriter`.
 
-    - ObsSpace:
-        name: <ObsGroup>
-        ObsDataIn:
-          obsfile: observations.nc4
-        simulate:
-          variables: [surface_pressure]        
-    - Filter: Cx Writer
-        # The filter will output an <ObsGroup>.cx file
-        # in the directory specified in the output_directory option.
-        output_directory: cx
-        # By default, the filter will produce a Cx file containing
-        # the cxfields output by default by OPS for the observation
-        # group <ObsGroup>. The list of cxfields can be changed by
-        # providing a namelist file <ObsGroup>.nl in the directory
-        # specified in the namelist_directory option. This file should
-        # contain a Fortran namelist in the format accepted by the
-        # Ops_ReadCXControlNL function from OPS.
-        namelist_directory: namelists
-        # Output only model columns corresponding to observations that 
-        # passed the quality check in all variables.
-        reject_obs_with_any_variable_failing_qc: true
-        # Values of the following options are written to the UM fixed header
-        # embedded in the output Cx file. In future (once the UM/LFRic JEDI
-        # interfaces are ready) they will probably be taken directly from
-        # the model. There are more options like this; the full list can be
-        # found in `src/opsinputs/VarObsWriterParameters.h`.
-        IC_XLen: 36
-        IC_YLen: 18
-        IC_PLevels: 3
-        IC_WetLevels: 4
-        RC_LongSpacing: 10
-        RC_LatSpacing: 10
-        # New dynamics vertical coordinate theta (length: IC_PLevels + 1).
-        eta_theta_levels: [300, 290, 280, 270]
-        # New dynamics vertical coordinate rho (length: IC_PLevels).
-        eta_rho_levels: [3, 2, 1]
+```yaml
+- ObsSpace:
+  name: <ObsGroup>
+  ObsDataIn:
+    obsfile: observations.nc4
+  simulate:
+    variables: [surface_pressure]
+- Filter: Cx Writer
+  # The filter will output an <ObsGroup>.cx file
+  # in the directory specified in the output_directory option.
+  output_directory: cx
+  # By default, the filter will produce a Cx file containing
+  # the cxfields output by default by OPS for the observation
+  # group <ObsGroup>. The list of cxfields can be changed by
+  # providing a namelist file <ObsGroup>.nl in the directory
+  # specified in the namelist_directory option. This file should
+  # contain a Fortran namelist in the format accepted by the
+  # Ops_ReadCXControlNL function from OPS.
+  namelist_directory: namelists
+  # Output only model columns corresponding to observations that
+  # passed the quality check in all variables.
+  reject_obs_with_any_variable_failing_qc: true
+  # Values of the following options are written to the UM fixed header
+  # embedded in the output Cx file. In future (once the UM/LFRic JEDI
+  # interfaces are ready) they will probably be taken directly from
+  # the model. There are more options like this; the full list can be
+  # found in `src/opsinputs/VarObsWriterParameters.h`.
+  IC_XLen: 36
+  IC_YLen: 18
+  IC_PLevels: 3
+  IC_WetLevels: 4
+  RC_LongSpacing: 10
+  RC_LatSpacing: 10
+  # New dynamics vertical coordinate theta (length: IC_PLevels + 1).
+  eta_theta_levels: [300, 290, 280, 270]
+  # New dynamics vertical coordinate rho (length: IC_PLevels).
+  eta_rho_levels: [3, 2, 1]
+```
 
 Further YAML files illustrating the use of these filters can be found in the `test/testinput` folder.
 
@@ -141,21 +135,21 @@ Development
 VarObs
 ------
 
-Only a subset of varfields recognised by OPS and VAR can currently be output. To add support for a new varfield:
+Only a subset of [varfields](Varfields.md) recognised by OPS and VAR can currently be output. To add support for a new varfield:
 
 1. Determine where the input data will come from (a variable stored in the `ObsSpace` object? a GeoVaL?).
 
 2. Edit the `case` corresponding to the varfield in question in the `select` statement in the `opsinputs_varobswriter_populateobservations` subroutine in the `src/opsinputs/opsinputs_varobswriter_mod.F90` file. In the vast majority of cases, you will simply need to replace a commented-out call to `Ops_Alloc` with a call to an appropriate `opsinputs_fill_fill...` subroutine. For example, suppose that the `VarField_logvis` varfield should be filled with the data (observed values, observation errors, gross error probabilities and QC flags) stored in the `logarithmic_visibility` observation variable in the `ObsSpace`. The comment
 
         ! call Ops_Alloc(Ob % Header % logvis, "logvis", Ob % Header % NumObsLocal, Ob % logvis)
 
-   should then be replaced by 
+   should then be replaced by
 
         call opsinputs_fill_fillelementtypefromsimulatedvariable( &
           Ob % Header % logvis, "logvis", Ob % Header % NumObsLocal, Ob % logvis, &
           ObsSpace, Flags, ObsErrors, "logarithmic_visibility")
 
-   If in doubt, look at similar varfields that have already been implemented or read the documentation of relevant subroutines from the `opsinputs_fill_mod.F90` module.
+   If in doubt, look at similar [varfields](Varfields.md) that have already been implemented or read the documentation of relevant subroutines from the `opsinputs_fill_mod.F90` module.
 
 3. Add a unit test for the new varfield:
 
@@ -165,7 +159,7 @@ Only a subset of varfields recognised by OPS and VAR can currently be output. To
 
    c. Add a call to the `ADD_WRITER_TEST` function in the `test/CMakeLists.txt` file, specifying the name of the test and its input YAML and data files.
 
-4. Update the list of implemented varfields in `Varfields.md`. 
+4. Update the list of implemented varfields in [Varfields.md](Varfields.md).
 
 Cx
 --

@@ -1,3 +1,8 @@
+<!---
+(C) Crown Copyright 2023, the Met Office. All rights reserved.
+
+--->
+
 | **Id** | **Varfield** | **Ob_type field** | **Description** | **How to get it from JEDI** | **TODO** |
 | --- | --- | --- | --- | --- | --- |
 | 1 | `VarField_pstar` | `pstar(:)` | surface pressure | `ObsValue/surfacePressure` | |
@@ -11,7 +16,7 @@
 | 9 | `VarField_lwp` | | | | Implement |
 | 10 | `VarField_britemp` | `CorBriTemp(:,:)` | bias-corrected brightness temperatures | `BiasCorrObsValue/brightnessTemperature` | |
 | 11 | `VarField_tskin` | `Tskin(:)` | skin temperature | `OneDVar/skinTemperature` | |
-| 12 | `VarField_gpstzdelay` | `gpstzdelay(:)`| Ground based GNSS Total Zenith Delay | BiasCorrObsValue/zenithTotalDelay | |
+| 12 | `VarField_gpstzdelay` | `gpstzdelay(:)`| Ground based GNSS Total Zenith Delay | `BiasCorrObsValue/zenithTotalDelay` | |
 | 15 | `VarField_cloud` | | | | Implement |
 | 16 | `VarField_rainrate` | | | | Implement |
 | 17 | `VarField_mwemiss` | `Emissivity(:,:)` | surface emissivity | `Emiss/emissivity` |  |
@@ -25,12 +30,12 @@
 | 25 | `VarField_chanword_ir` | | | | Implement |
 | 26 | `VarField_chanword_mw` | | | | Implement |
 | 27 | `VarField_stratt` | `t(:,-nlevels_strat_varobs:) `| temperature | Probably from a variable generated by the 1D-Var filter. | Implement |
-| 28 | `VarField_satid` | `SatId(:)` \[`INTEGER`\] | satellite ID | MetaData/satelliteIdentifier.` | |
+| 28 | `VarField_satid` | `SatId(:)` \[`INTEGER`\] | satellite ID | `MetaData/satelliteIdentifier` | |
 | 29 | `VarField_satazimth` | | | | Implement |
 | 30 | `VarField_localazimuth` | | | | Implement |
 | 31 | `VarField_solzenith` | `SolarZenith(:)` | solar zenith angle | `MetaData/solarZenithAngle` | |
 | 32 | `VarField_solazimth` | | | | Implement |
-| 34 | `VarField_iremiss` | `IREmiss(:)` | surface emissivity (satellite 1dvar) | 'Emiss/emissivityIR' | |
+| 34 | `VarField_iremiss` | `IREmiss(:)` | surface emissivity (satellite 1dvar) | `Emiss/emissivityIR` | |
 | 35 | `VarField_cloudtopp` | `CloudTopP(:)` | cloud top pressure (hPa) (satellite 1dvar) | `OutputToVAR/pressureAtTopOfCloud` | |
 | 36 | `VarField_cloudfrac` | `CloudFrac(:)` | cloud fraction (satellite 1dvar) | `OneDVar/cloudAmount` | |
 | 39 | `VarField_vnatovpp` | | | | Implement |
@@ -45,7 +50,7 @@
 | 53 | `VarField_pcorrect` | `AWPriorPCorrect(:,:)` | Probability for ambiguous wind | `ObsValue/probability` |  |
 | 54 | `VarField_NumChans` | `NumChans(:)` \[`INTEGER`\] | number of channels | From obsSpace.obsvariables().channels() and obs flags | |
 | 55 | `VarField_ChanNum` | `ChanNum(:,:)` \[`INTEGER`\] | channel numbers | From obsSpace.obsvariables().channels() and obs flags | |
-| 57 | `VarField_Emissivity` | `Emissivity(:,:)` | surface emissivity (satellite 1dvar) | OneDVar/emissivity | |
+| 57 | `VarField_Emissivity` | `Emissivity(:,:)` | surface emissivity (satellite 1dvar) | `OneDVar/emissivity` | |
 | 58 | `VarField_QCinfo` | `QCinfo(:)` | auxiliary QC variable for satellite observations | From a variable updated by multiple filters. | Specify variable name |
 | 59 | `VarField_refrac` | | | | Implement |
 | 60 | `VarField_z` | | | | Implement |
@@ -55,14 +60,14 @@
 | 64 | `VarField_RadarBeamElev` | | | | Implement |
 | 65 | `VarField_RadarObRange` | | | | Implement |
 | 66 | `VarField_RadarObAzim` | `RadarObAzim(:,:)` | Azimut of radar ob | `MetaData/radarAzimuth` | |
-| 67 | `VarField_GPS_Station_Height` | `GPS_Station_Height(:)`| Height of the Ground GNSS stations | MetaData/stationElevation | |
+| 67 | `VarField_GPS_Station_Height` | `GPS_Station_Height(:)`| Height of the Ground GNSS stations | `MetaData/stationElevation` | |
 | 68 | `VarField_clw` | | | | Implement |
 | 69 | `VarField_RadIdent` | | | | Implement Implement |
 | 70 | `VarField_Reflectivity` | | | | Implement |
 | 71 | `VarField_BendingAngle` | `BendingAngle(:,:)` or `BendingAngleAll(:,:)` | GPSRO bending angle (L1, L2, LC etc or just LC) | `ObsValue/bendingAngle` | Decide if a separate variable corresponding to BendingAngleAll is needed |
 | 72 | `VarField_ImpactParam` | `ImpactParam(:,:)` or `ImpactParamAll(:,:)` | GPSRO impact parameter (L1, L2, LC etc or just LC) | `ObsValue/impactParameterRO` | Decide if a separate variable corresponding to ImpactParamAll is needed |
 | 73 | `VarField_RO_Rad_Curv` | `RO_Rad_Curv(:)` | GPSRO Earth\'s local radius of curvature (m) | `MetaData/earthRadiusCurvature` | |
-| 74 | `VarField_RO_geoid_und` | `RO_geoid_und(:)` | GPSRO Geoid undulation (above WGS-84 ellipsoid, m) | MetaData/geoidUndulation | |
+| 74 | `VarField_RO_geoid_und` | `RO_geoid_und(:)` | GPSRO Geoid undulation (above WGS-84 ellipsoid, m) | `MetaData/geoidUndulation` | |
 | 75 | `VarField_RadAltAboveMSL` | | | | Implement |
 | 76 | `VarField_BriTempVarError` | `BriTempVarError(:,:)` | scene dependent obs errors for BTs (oblocation, channel) | `ObsErrorData/brightnessTemperature` | |
 | 77 | `VarField_AOD` | `AOD(:,:)` | aerosol optical depth | `ObsValue/aerosolOpticalDepth` | |

@@ -1,3 +1,6 @@
+# (C) Crown Copyright 2021, the Met Office. All rights reserved.
+#
+
 cmake_minimum_required( VERSION 3.12 FATAL_ERROR )
 
 project( opsinputs-ci VERSION 0.1 LANGUAGES C CXX Fortran )

@@ -1,4 +1,8 @@
 #!/bin/bash
+#
+# (C) Crown Copyright 2021, the Met Office. All rights reserved.
+#
+#
 set -euxo pipefail
 repo="$1"  # expect path to a repo clone