Repository navigation
Enable fdb - #292
Draft
cosunae wants to merge 24 commits into
Draft
Enable fdb#292cosunae wants to merge 24 commits into
cosunae wants to merge 24 commits into
Conversation
Neither belongs in version control: core dumps are debug artifacts from crash investigation, and the eccodes-cosmo-mars checkout (needed locally by plot.smk/verification.smk for FDB reads outside the squashfs-mounted venv) is a clone of an external repo, not vendored content.
The previous implementation used pyfdb's list() to discover every archived
parameter and then fetched each one individually, regardless of which
params the caller actually asked for; bytes were also round-tripped through
a temp file. Rewrite to issue a single request per (levtype, timespan)
group for exactly the requested params, via earthkit-data's "fdb" source
(ekd.from_source) rather than raw pyfdb, with no intermediate file.
Also fixes FieldList.group_by("typeOfLevel") returning one group regardless
of distinct values (manual grouping instead), and adds the timespan=fs key
required for TOT_PREC (otherwise silently empty for any requested step).
cartopy, earthkit.plots and matplotlib.tri were imported eagerly at module load, forcing that whole stack onto any import of plotting.compat even for pure data-loading use (e.g. dashboard/meteogram reads). Move the viz-only content (DOMAINS, get_projection, StatePlotter) into plotting/viz.py and re-export it lazily via module-level __getattr__, so importing plotting only pulls in numpy until something viz-specific is actually touched.
squashfs-mount v10.1.0 replaced the old positional "path:mountpoint ..." syntax with a single -s "path:mnt,path2:mnt2" flag; update both call sites (inference_check_fdb and inference_execute) accordingly, and make the .venv-mount cleanup trap tolerate it already being gone. Also pin the eccodes-cosmo-mars clone to commit d04363540b (pre "add a grid Name") instead of a bare --depth 1 branch clone: a later commit on varda-ext adds a mars.gridName concept that the current realtime-varda FDB schema doesn't account for, breaking archival with a "Keywords not used" error. Un-pin once the schema is updated to match.
anemoi-inference, when using the copy-prognostic-from-forecaster post-processor (anemoi-plugins-meteoswiss), reliably crashes natively during process teardown strictly AFTER all forecast windows have been computed and written out -- glibc heap-corruption diagnostics (free(): invalid pointer, munmap_chunk(): invalid pointer, double free or corruption), with the exact message/signal (SIGABRT/134, SIGBUS/135) depending on memory layout at the time. Likely an eccodes/earthkit-data handle cleanup ordering bug tied to that filter's cached forecaster GRIB index being garbage-collected at interpreter shutdown. Verified empirically (including "Flushing FDB" x2 immediately preceding the crash, for FDB-backed runs) that all expected output is present and bit-correct despite the abort. Without this, the rule discarded valid output and forced a full re-run of expensive GPU inference for nothing. Detect the signature (no Python traceback, all forecast windows reached) and treat it as success instead.
plot_forecast_frame.py received showcase-facing param names like "TOT_PREC1"/"TOT_PREC6" (base param + accumulation window encoded in a numeric suffix) and passed them straight through to GRIB loading and de-accumulation, which only ever matched the literal base name "TOT_PREC" -- every precipitation frame failed with "None of the requested params ['TOT_PREC1'] found". Decompose via the existing parse_aggregated_param helper, using the base name for loading/de-accumulation/colormap lookup and the original suffixed name only for filenames/titles. Same root cause in resolve_leadtimes() (common.smk): the step-0 skip for accumulated params (PARAMS_WITHOUT_STEP_ZERO_VALUE) only recognized the literal "TOT_PREC", not its suffixed showcase variants, so frames for leadtime 0 failed with "accumulated fields ... undefined at step 0" where they should have simply been skipped.
Under heavy concurrent load on the shared postproc partition (many plot_forecast_frame jobs running at once), a single region's render has been observed taking ~16m vs ~1.5m uncontended. The 20m budget was killing otherwise-healthy jobs via TIMEOUT -- a resource/contention issue, not a correctness bug -- discarding completed region renders along with it.
# Conflicts: # resources/inference/templates/templates_index_icon.yaml # src/data_input/__init__.py # src/evalml/config.py # src/plotting/__init__.py # src/plotting/compat.py # workflow/Snakefile # workflow/rules/common.smk # workflow/rules/inference.smk # workflow/rules/plot.smk # workflow/rules/report.smk # workflow/rules/verification.smk # workflow/tools/config.schema.json
cosunae
marked this pull request as draft
November 12, 2026 07:34
# Conflicts: # resources/inference/templates/templates_index_icon.yaml # src/data_input/__init__.py # src/evalml/config.py # src/plotting/__init__.py # workflow/rules/common.smk # workflow/rules/verification.smk # workflow/scripts/plot_forecast_frame.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allows evalml to write inference directly to a FDB (local or global) and subsequent steps read from it