ERA5 Monthly Single-Level (CDS) → GRIB → Parquet (grid-split)

This project downloads ERA5 monthly averaged data on single levels from the Copernicus Climate Data Store (CDS) and converts each downloaded GRIB file into one-or-more Parquet files, grouped by exact lat/lon grid.

The main goal is to:

• keep all variables that share the exact same grid in the same Parquet file, and
• make Parquet outputs easy to bulk-load into a SQL database with a consistent schema: time, lat, lon, var_1, var_2, ... var_n

---

Dataset

• Dataset name (CDS): reanalysis-era5-single-levels-monthly-means
• DOI: 10.24381/cds.f17050d7
• Temporal resolution: monthly
• Spatial grid: regular lat/lon
• Typical resolution for reanalysis fields: 0.25° x 0.25°
• Note on mixed resolutions: some categories are on coarser grids (commonly 0.5°) and will naturally produce separate Parquet outputs.

---

Variables downloaded (current defaults)

The downloader requests monthly_averaged_reanalysis and currently includes these variables:

Atmosphere / surface:

• 10m_u_component_of_wind
• 10m_v_component_of_wind
• 2m_dewpoint_temperature
• 2m_temperature
• mean_sea_level_pressure
• surface_pressure
• total_cloud_cover
• cloud_base_height

Radiation:

• downward_uv_radiation_at_the_surface
• top_net_solar_radiation
• top_net_thermal_radiation
• uv_visible_albedo_for_direct_radiation

Ocean/surface:

• sea_surface_temperature

Hydrology:

• total_precipitation
• mean_total_precipitation_rate

Short vs long names:
GRIB “shortName”, “longName”, units, paramId, stepType, etc. are saved per-download in metadata_vars.csv (generated automatically).

Optional wave variables (commented out)

The script includes commented-out wave variables. We can uncomment them; they may land on a different grid and therefore will be written to a different Parquet file automatically.

---

Metadata CSVs

Alongside GRIB download, two metadata tables are written:

• metadata_dataset.csv – one row per cfgrib “dataset group”, includes:

  • grid size (grid_nlat, grid_nlon)
  • spatial spacing (dlat_deg, dlon_deg)
  • bounding box and time-like coordinates

• metadata_vars.csv – one row per variable, includes:

  • xarray_var, shortName, longName, units, paramId
  • typeOfLevel, level, stepType, dataType, gridType
  • (optionally) quick stats like nan fraction, min, max

These metadata files are extremely useful for:

• Simons CMAP ingestion process
• validating expected grid resolution,
• detecting mixed grids in a single GRIB,
• tracking GRIB shortName ↔ longName ↔ units for SQL schema generation.

---

Parquet schema

Each Parquet file is written in “wide” form:

• Required columns:
  • time (canonicalized; see below)
  • lat
  • lon
• One column per variable on that grid:
  • e.g., t2m, d2m, u10, v10, sp, msl, sst, tp, etc.

Time handling (important)

GRIB messages often contain:

• time (analysis/initial time)
• step (forecast/accumulation lead time)
• valid_time (computed time)

Some variables—especially accumulations—use non-zero step and a valid_time that can look “shifted” relative to other fields.

To keep the monthly outputs consistent across variables, the code:

1. builds a “best available” timestamp (prefers valid_time, else time + step, else time), then
2. canonicalizes it (default: month_start) so everything ends up with the same month timestamp.

Supported canonical_time modes:

• month_start (default): all rows for a month become YYYY-MM-01 00:00:00
• day_start: floor to day
• valid_time: keep true valid time

This is what lets tp and “instantaneous-style” variables live together cleanly in SQL.

Longitude convention

For Parquet outputs only, longitudes are shifted to the common GIS convention:

• values > 180 are converted by lon = lon - 360
• final range is approximately [-180, 180]

(The GRIB file remains unchanged.)

---

Installation

1) CDS account + API key

Required:

• a CDS account
• a configured CDS API key in ~/.cdsapirc

See the CDS “User guide” for CDS API setup: https://cds.climate.copernicus.eu/how-to-api

2) Python environment

conda env create -f cresst-etl.yml
conda activate cresst-etl

---

Usage

Download + convert a single month

python era5_monthly_cli.py --ym 2025-11 --skip-existing

or

from era5_monthly_single_level import get_era5_and_process

paths = get_era5_and_process("2025", "11")
print(paths)  # dict: grid_id -> parquet_path

Bulk download a range of months

python era5_monthly_cli.py --range 2024-01 2024-12 --skip-existing

or

from era5_monthly_single_level import get_era5_and_process

def iter_ym(start_ym, end_ym):
    sy, sm = map(int, start_ym.split("-"))
    ey, em = map(int, end_ym.split("-"))
    y, m = sy, sm
    while (y < ey) or (y == ey and m <= em):
        yield f"{y:04d}", f"{m:02d}"
        m += 1
        if m == 13:
            m = 1
            y += 1

for year, month in iter_ym("2020-01", "2020-12"):
    paths = get_era5_and_process(year, month)
    print(year, month, paths)