Skip to content

Update Zarr metadata refresh to include generated filename variable - #368

Draft
thommodin with Copilot wants to merge 1 commit into
mainfrom
copilot/update-zarr-attributes
Draft

Update Zarr metadata refresh to include generated filename variable#368
thommodin with Copilot wants to merge 1 commit into
mainfrom
copilot/update-zarr-attributes

Conversation

Copilot AI commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

update_metadata skipped filename for Zarr datasets when filename was auto-generated at write time but absent from config schema/add_variables. This left stale or missing attrs in .zattrs for that variable.

  • Root cause

    • Metadata refresh iterates only over config-derived schema (full_schema).
    • For datasets like satellite_austemp_dhd_14day, filename exists in store but is not represented in config schema, so it is never updated.
  • Behavior change

    • In update_store_varattrs_from_schema, when filename exists in the Zarr store but is missing from the schema input, inject a schema entry before attr reconciliation.
    • Schema source precedence:
      1. schema_transformation.add_variables.filename.schema
      2. schema.filename
      3. fallback: {type: "object", units: "1", long_name: "Filename of the source file"}
  • Targeted test coverage

    • Added a unit test that removes filename from config schema/add_variables, creates a store with filename, runs metadata update, and verifies filename attrs are updated from fallback metadata.
if "filename" in store and "filename" not in schema_to_apply:
    filename_schema = (
        self.dataset_config.get("schema_transformation", {})
        .get("add_variables", {})
        .get("filename", {})
        .get("schema")
    ) or self.dataset_config.get("schema", {}).get("filename") or {
        "type": "object",
        "units": "1",
        "long_name": "Filename of the source file",
    }
    schema_to_apply["filename"] = filename_schema

Co-authored-by: thommodin <98626996+thommodin@users.noreply.github.com>
Copilot AI changed the title Fix metadata updates for generated Zarr filename variable Update Zarr metadata refresh to include generated filename variable Sep 3, 2026
Copilot AI requested a review from thommodin September 3, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants