hourly: multi-variable requests, levels compatibility, concurrent downloads#7
Merged
Conversation
…nloads
Three gaps in hourly() relative to the registered v0.1 (era5cli) interface
that NumericalEarth's batched download extension relies on:
- variables now accepts a Vector{String} as well as a String. Each variable
is its own CDS request (matching era5cli semantics); requests are submitted
concurrently via asyncmap, up to `threads` at a time, so a bundle of
variables waits in the Copernicus queue together instead of sequentially.
Multi-variable calls append the variable name to outputprefix so each
request gets its own file; single-variable naming is unchanged.
- The v0.1 `levels` keyword is accepted again as a spelling of
pressure_levels. Previously it fell silently into additional_kw, so a
v0.1-style pressure-level request downloaded the single-levels product
without warning (e.g. u10 instead of u on levels).
- Existing files are skipped per variable when overwrite=false, and the
unused dates_str construction is removed.
Adds offline tests pinning the file-naming and skip-existing contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two request-lifecycle fixes that matter for batched multi-request use: - poll_request_status polled at a fixed 10 s; small extractions complete in ~15-25 s, so up to 10 s per request was dead time. Polling now starts at 1 s and backs off exponentially to poll_interval, so short requests return promptly while long queue waits still poll gently. - The CDS gateway intermittently answers valid requests with transient errors (e.g. 502). Submit, status poll, results fetch, and file download now retry with exponential backoff (request_with_retries) instead of dying on the first blip - which previously took down every request of a concurrent bundle, possibly minutes into a queue wait. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documenter's missing_docs check fails on docstrings absent from the manual; list the new helper alongside the other request-lifecycle helpers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NumericalEarth's ERA5 download extension now batches downloads: one
hourlyinvocation per calendar-month batch covering all pending variables (NumericalEarth/NumericalEarth.jl,NumericalEarthCopernicusClimateDataStoreExt). This PR closes three gaps in v0.2'shourly()relative to the registered v0.1 (era5cli) interface that the extension targets, plus two request-lifecycle fixes, and bumps the version to 0.2.1.Changes
variablesaccepts aVector{String}as well as aString. Each variable is its own CDS request (matching era5cli semantics); requests are submitted concurrently viaasyncmap, up tothreadsat a time, so a bundle of variables waits in the Copernicus queue together instead of one at a time. Multi-variable calls append the variable name tooutputprefixso each request gets its own file; single-variable naming is unchanged (era5.nc/era5_2020_1_1.nc).The v0.1
levelskeyword is accepted again as a spelling ofpressure_levels. Previously it fell silently intoadditional_kw..., so a v0.1-style pressure-level call downloaded the single-levels product without warning (e.g.u10instead ofuon pressure levels). (levels = :surfaceandnothingmean single-levels, as before.)Fast-start poll backoff:
poll_request_statuspolled at a fixed 10 s; small extractions complete in ~15–25 s, so up to 10 s per request was dead time. Polling now starts at 1 s and backs off exponentially topoll_interval.Transient-error retries: submit, status poll, results fetch, and file download retry with exponential backoff (
request_with_retries) instead of dying on the first 502 — which previously took down every request of a concurrent bundle, possibly minutes into a queue wait.overwrite=falseskips per variable; unuseddates_strconstruction removed.Benchmark
5 single-level variables, one datetime, 5°×5° box, distinct datetimes per run (so the CDS result cache can't cross-contaminate):
main(5 sequential calls)threads=6)≈1.7× faster. A timestamped timeline shows all 5 requests accepted simultaneously at t=6 s, but completions spaced ~20 s apart — CDS executes roughly one request at a time per user, so bundling eliminates the client-side dead time (sequential submits, poll quantization) while server-side processing stays serialized. The best bundled run (97.9 s) sits at that server-side floor (~5 × 20 s). Consequence: reducing request count (many datetimes per request, as NumericalEarth's month-batching does) remains the dominant lever; variable bundling is a solid secondary win.
Testing
request_with_retries(transient recovery + rethrow-after-exhaustion).🤖 Generated with Claude Code