NetCDF output: fixed dims, batched writes, workers write one shared file (no merge)

[JoshCu]

Brings over the NetCDF performance lessons from rs_route's writer, then goes one step further so workers write straight into one shared file and the merge step goes away.

Changes

Fixed dimensions and batched writes (04e4a96)

• The id dimension is now fixed-size instead of UNLIMITED.
• Results are buffered 100 locations at a time. Each variable is written with a single call per batch, put_values(&data, (start..end, ..)), instead of one row per call.
• The writer thread now waits for results instead of flushing every 100 ms. The timer flush meant any model slower than 100 ms per location wrote batches of 1.

Shared file, no merge (9355080)
HDF5 can't have several processes writing one file through the library, because each open handle caches the file's metadata and writes it back. Since every size is known before the run, the library isn't needed for the data writes:

• Parent (create_netcdf_store, next to the existing create_zarr_store call):
  • Creates results.nc with fixed dims and writes all location IDs.
  • Stores each output variable as a contiguous, unfiltered f32 block with its disk space reserved up front.
  • NOFILL keeps that reservation sparse: 1.4 GB apparent size, 0 MB allocated in testing. _FillValue is re-added afterwards so readers still mask -9999.
  • Checks the storage layout once, so an unusable file fails before any worker starts.
• Workers (NetCdfWriter::new(path, global_start)):
  • Look up each variable's data offset with a read-only HDF5 open, then close it.
  • Write batches with pwrite at offset + row × n_times × 4. Each worker's rows form one contiguous byte range per variable, and no HDF5 handle is open while writing.
• Location IDs are written with one nc_put_var_string call through netcdf-sys, now a direct dependency with no extra features. The crate's per-string put_string took about 80 µs per ID: 6.2 s → 0.36 s for 80k IDs.
• The per-worker tmp_*.nc files and merge_netcdf_files are removed.

Behaviour changes

• Variable names come from the parent's pre-run discovery, the same list Zarr uses. Short rows are padded and missing variables written as -9999.
• If a run fails partway, rows that were never written read back as 0 rather than -9999. The file is still valid.
• NetCdfWriter::new now takes (path, start_idx).

Testing

• cargo test: 40 unit tests and 7 output-format tests pass. New tests cover several writers interleaving into one file, padding of short rows and missing variables, _FillValue surviving NOFILL, and a fixed id dimension.
• cargo build and cargo build --no-default-features are warning-free.
• Not yet benchmarked on a real model run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CHF3fXG3QAnmb8ZeJtsJ2F

---

Generated by Claude Code