Skip to content

Preserve list attributes when reading with h5netcdf - #11504

Draft
hmaarrfk wants to merge 1 commit into
pydata:mainfrom
hmaarrfk:attr_list
Draft

Preserve list attributes when reading with h5netcdf#11504
hmaarrfk wants to merge 1 commit into
pydata:mainfrom
hmaarrfk:attr_list

Conversation

@hmaarrfk

@hmaarrfk hmaarrfk commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Warning

This PR was written by Claude Code (an AI assistant), acting on behalf of @hmaarrfk.
Please wait for Mark's human review before reading or reviewing this.
It is a draft proposal, not a reviewed change.

The bug

xr.Dataset(attrs={"a": [], "b": ["one"], "c": ["one", "two"]}).to_netcdf("f.nc", engine="h5netcdf")
xr.open_dataset("f.nc", engine="h5netcdf").attrs
# {'a': array([], dtype=float64), 'b': 'one', 'c': ['one', 'two']}

h5netcdf collapses every length-1 attribute to a scalar, mirroring netcdf4-python:

# h5netcdf/attrs.py
if not np.isscalar(output) and len(output) == 1:
    return output[0]

What this changes

Read side of the h5netcdf backend only. The collapse hides two things the file does record:

  • netCDF stores a single string as NC_CHAR, whose HDF5 dataspace is scalar, and a
    sequence of strings as NC_STRING, whose dataspace is not.
  • A zero-length attribute cannot be a scalar whatever its dataspace — whether it is the
    null dataspace netcdf-c writes or the (0,) h5netcdf writes.

_read_attributes now reads the dataspace off the underlying h5py attribute and restores
those two cases.

Nothing about how xarray writes files changed. I verified this by dumping
dtype/shape/dataspace/value for every attribute written by both engines: byte-for-byte
identical to main. No new attributes, no new conventions, existing files unaffected.

Scope, and why it stops here

case h5netcdf write + read why
["one"] ["one"] NC_STRING vs NC_CHAR is recorded in the dataspace
[] [] zero-length cannot be a scalar
["one", "two"] ✅ already worked
"plain" ✅ stays a scalar scalar dataspace
[1] 1 see below

Numeric attributes are deliberately left alone. netcdf-c stores a scalar number as a
length-1 vector, so 1 and [1] are the same bytes on disk. Guessing "sequence" would
turn the scale_factor of every existing file into a list and break CF decoding. There is
a test pinning this (test_numeric_attrs_are_not_turned_into_lists).

For the record: writing numbers as true HDF5 scalars would disambiguate them, and
netCDF-C reads that back fine (ncdump shows :true_scalar = 5LL ;, identical to a
length-1 vector). It still doesn't help — the reader rule needed to exploit it would
misread every file written to the existing convention.

The netcdf4 engine is untouched. netcdf4-python's _get_att collapses length-1
attributes before xarray sees them and exposes no way to ask for the length, so nothing
can be done from xarray's side. A file written by the netcdf4 engine also stores
["one"] as NC_CHAR, so the information is gone at write time — reading it with
engine="h5netcdf" still gives "one". Only the empty-list fix applies to those files.

An earlier revision of this PR changed the netCDF4 writer to emit NC_STRING for length-1
lists, which made both engines' files carry the distinction. It was dropped to keep the
on-disk format completely unchanged. Happy to bring it back if that tradeoff is preferred.

Worth noting the zarr backend round-trips all of these correctly, since it stores
attributes as JSON — so netCDF and zarr still disagree on the numeric cases.

Testing

Full local suite: 21471 passed, no regressions. New tests cover string sequences, empty
lists (including files written by the netcdf4 engine), and the numeric no-op.

<details><summary>Claude's draft</summary>

h5netcdf collapses every length-1 attribute to a scalar, mirroring
netcdf4-python, so `{"a": ["one"]}` read back as `{"a": "one"}` and `[]` as
`array([], dtype=float64)` (GH10275).

That collapse hides two things the file does record:

- netCDF stores a single string as NC_CHAR, whose HDF5 dataspace is scalar,
  and a sequence of strings as NC_STRING, whose dataspace is not;
- a zero length attribute cannot be a scalar whatever its dataspace, whether
  it is the null dataspace netcdf-c writes or the (0,) h5netcdf writes.

`_read_attributes` now reads the dataspace off the underlying h5py attribute
and restores those two cases. Numeric attributes are deliberately left
alone: netcdf-c stores a scalar number as a length-1 vector, so 1 and [1]
are the same bytes on disk, and guessing "sequence" would turn the
scale_factor of every existing file into a list.

Nothing about how xarray writes files changed -- verified by dumping
dtype/shape/dataspace/value for every attribute written by both engines,
which is byte-for-byte identical to main. The netCDF4 engine is untouched,
since netcdf4-python collapses length-1 attributes before xarray sees them
and exposes no way to ask for the length.

Resume this Claude session:
```
cd /Users/mark/git/xarray/xarray
claude --resume 8659c07e-0bfb-4785-8161-4b65723eeb66
```
</details>

Co-authored-by: Claude <noreply@anthropic.com>
@hmaarrfk hmaarrfk changed the title Preserve length-1 string lists in netCDF attributes Preserve list attributes when reading with h5netcdf Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant