Skip to content

hourly: multi-variable requests, levels compatibility, concurrent downloads#7

Merged
glwagner merged 3 commits into
mainfrom
glw/hourly-multivariable
Jul 14, 2026
Merged

hourly: multi-variable requests, levels compatibility, concurrent downloads#7
glwagner merged 3 commits into
mainfrom
glw/hourly-multivariable

Conversation

@glwagner

@glwagner glwagner commented Jul 12, 2026

Copy link
Copy Markdown
Member

NumericalEarth's ERA5 download extension now batches downloads: one hourly invocation per calendar-month batch covering all pending variables (NumericalEarth/NumericalEarth.jl, NumericalEarthCopernicusClimateDataStoreExt). This PR closes three gaps in v0.2's hourly() relative to the registered v0.1 (era5cli) interface that the extension targets, plus two request-lifecycle fixes, and bumps the version to 0.2.1.

Changes

  • variables accepts a Vector{String} as well as a String. Each variable is its own CDS request (matching era5cli semantics); requests are submitted concurrently via asyncmap, up to threads at a time, so a bundle of variables waits in the Copernicus queue together instead of one at a time. Multi-variable calls append the variable name to outputprefix so each request gets its own file; single-variable naming is unchanged (era5.nc / era5_2020_1_1.nc).

  • The v0.1 levels keyword is accepted again as a spelling of pressure_levels. Previously it fell silently into additional_kw..., so a v0.1-style pressure-level call downloaded the single-levels product without warning (e.g. u10 instead of u on pressure levels). (levels = :surface and nothing mean single-levels, as before.)

  • Fast-start poll backoff: poll_request_status polled at a fixed 10 s; small extractions complete in ~15–25 s, so up to 10 s per request was dead time. Polling now starts at 1 s and backs off exponentially to poll_interval.

  • Transient-error retries: submit, status poll, results fetch, and file download retry with exponential backoff (request_with_retries) instead of dying on the first 502 — which previously took down every request of a concurrent bundle, possibly minutes into a queue wait.

  • overwrite=false skips per variable; unused dates_str construction removed.

Benchmark

5 single-level variables, one datetime, 5°×5° box, distinct datetimes per run (so the CDS result cache can't cross-contaminate):

mode run 1 run 2 mean
main (5 sequential calls) 185.0 s 229.6 s ~207 s
PR (1 bundled call, threads=6) 152.1 s 97.9 s ~125 s

≈1.7× faster. A timestamped timeline shows all 5 requests accepted simultaneously at t=6 s, but completions spaced ~20 s apart — CDS executes roughly one request at a time per user, so bundling eliminates the client-side dead time (sequential submits, poll quantization) while server-side processing stays serialized. The best bundled run (97.9 s) sits at that server-side floor (~5 × 20 s). Consequence: reducing request count (many datetimes per request, as NumericalEarth's month-batching does) remains the dominant lever; variable bundling is a solid secondary win.

Testing

  • Offline testsets pin the file-naming/skip-existing contract and request_with_retries (transient recovery + rethrow-after-exhaustion).
  • Full suite passes locally, 19/19, including the live CDS integration download.

🤖 Generated with Claude Code

glwagner and others added 3 commits July 12, 2026 04:19
…nloads

Three gaps in hourly() relative to the registered v0.1 (era5cli) interface
that NumericalEarth's batched download extension relies on:

- variables now accepts a Vector{String} as well as a String. Each variable
  is its own CDS request (matching era5cli semantics); requests are submitted
  concurrently via asyncmap, up to `threads` at a time, so a bundle of
  variables waits in the Copernicus queue together instead of sequentially.
  Multi-variable calls append the variable name to outputprefix so each
  request gets its own file; single-variable naming is unchanged.

- The v0.1 `levels` keyword is accepted again as a spelling of
  pressure_levels. Previously it fell silently into additional_kw, so a
  v0.1-style pressure-level request downloaded the single-levels product
  without warning (e.g. u10 instead of u on levels).

- Existing files are skipped per variable when overwrite=false, and the
  unused dates_str construction is removed.

Adds offline tests pinning the file-naming and skip-existing contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two request-lifecycle fixes that matter for batched multi-request use:

- poll_request_status polled at a fixed 10 s; small extractions complete
  in ~15-25 s, so up to 10 s per request was dead time. Polling now starts
  at 1 s and backs off exponentially to poll_interval, so short requests
  return promptly while long queue waits still poll gently.

- The CDS gateway intermittently answers valid requests with transient
  errors (e.g. 502). Submit, status poll, results fetch, and file download
  now retry with exponential backoff (request_with_retries) instead of
  dying on the first blip - which previously took down every request of a
  concurrent bundle, possibly minutes into a queue wait.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documenter's missing_docs check fails on docstrings absent from the
manual; list the new helper alongside the other request-lifecycle
helpers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@glwagner
glwagner merged commit 7432f78 into main Jul 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant