Skip to content

Opening multifile dataset with separate_times=True immediately loads all data #14

@ZedThree

Description

@ZedThree

separate_times=False:

<xarray.DataArray 'Electric Field/Ey' (time: 51, X_Grid_mid: 250,
                                       Y_Grid_mid: 250)> Size: 26MB
dask.array<where, shape=(51, 250, 250), dtype=float64, chunksize=(1, 250, 250), chunktype=numpy.ndarray>
Coordinates:
  * X_Grid_mid  (X_Grid_mid) float64 2kB -9.96e-06 -9.88e-06 ... 9.96e-06
  * Y_Grid_mid  (Y_Grid_mid) float64 2kB -9.96e-06 -9.88e-06 ... 9.96e-06
  * time        (time) float64 408B 8.963e-17 1.076e-15 ... 4.912e-14 5.001e-14
Attributes:
    units:       V/m
    point_data:  False

separate_times=True:

<xarray.DataArray 'Electric Field/Ey' (time0: 51, X_Grid_mid: 250,
                                       Y_Grid_mid: 250)> Size: 26MB
array([[[ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00, ...,
          0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
        [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00, ...,
          0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
        [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00, ...,
          0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
        ...,
        [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00, ...,
          0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
        [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00, ...,
          0.00000000e+00,  0.00000000e+00,  0.00000000e+00],
        [ 0.00000000e+00,  0.00000000e+00,  0.00000000e+00, ...,
          0.00000000e+00,  0.00000000e+00,  0.00000000e+00]],

       [[ 9.69136707e+05,  1.45450376e+06,  1.94499080e+06, ...,
          1.94499080e+06,  1.45450376e+06,  9.69136707e+05],
        [ 1.44010639e+06,  1.74127173e+06,  2.20128447e+06, ...,
          2.20128447e+06,  1.74127173e+06,  1.44010639e+06],
        [ 1.32602883e+06,  1.54845058e+06,  1.94702953e+06, ...,
          1.94702953e+06,  1.54845058e+06,  1.32602883e+06],
...
        [ 1.17550855e-43,  1.25565768e-43,  1.42221304e-43, ...,
          1.43461604e-43,  1.26341511e-43,  1.17815023e-43],
        [ 5.60076522e-45,  5.98834772e-45,  6.79031193e-45, ...,
          6.83507099e-45,  6.01653508e-45,  5.61039643e-45],
        [ 2.92406830e-46,  3.12923275e-46,  3.55228981e-46, ...,
          3.56974126e-46,  3.14030567e-46,  2.92786815e-46]],

       [[-5.27739575e+10, -5.28289740e+10, -5.29271139e+10, ...,
         -5.28762368e+10, -5.27975081e+10, -5.27641672e+10],
        [-5.24617548e+10, -5.25239749e+10, -5.26336159e+10, ...,
         -5.25485324e+10, -5.24737035e+10, -5.24440332e+10],
        [-5.19791781e+10, -5.20505032e+10, -5.21703297e+10, ...,
         -5.20500749e+10, -5.19788063e+10, -5.19543593e+10],
        ...,
        [ 8.29600913e-39,  8.81580924e-39,  9.92955236e-39, ...,
          1.01625444e-38,  8.95868772e-39,  8.34416144e-39],
        [ 4.89437945e-40,  5.20716306e-40,  5.87157946e-40, ...,
          5.98527753e-40,  5.27708363e-40,  4.91797844e-40],
        [ 3.15606367e-41,  3.36131486e-41,  3.79414634e-41, ...,
          3.85427555e-41,  3.39841345e-41,  3.16860621e-41]]])
Coordinates:
  * X_Grid_mid  (X_Grid_mid) float64 2kB -9.96e-06 -9.88e-06 ... 9.96e-06
  * Y_Grid_mid  (Y_Grid_mid) float64 2kB -9.96e-06 -9.88e-06 ... 9.96e-06
  * time0       (time0) float64 408B 8.963e-17 1.076e-15 ... 4.912e-14 5.001e-14
Attributes:
    units:       V/m
    point_data:  False

Need to study xr.open_mfdataset more closely to see why this would be different. Looks like expand_dims is causing the data to be loaded

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions