Repository navigation
Infrastructure: release successful environmental refreshes despite source outages - #161
Conversation
- Release notes list the files that changed and, after a partial refresh, link the run, instead of always claiming weather, ocean, tide and ENSO. - A full live fetch took 48 minutes on 4 Nov 2026 (NOAA ERDDAP 502/503s and six 120 s stalls), too close to the 60-minute limit. - AGENTS.md: the #159 note sits with the refresh-external.yml bullet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reviewed with a full live fetch: all seven sources, written into a scratch copy of Live run (4 Nov 2026, 48 minutes)
What I checked in the code
Changes pushed (a61fd2f)
One thing to watch (no change here). Final OISST is currently 16 days behind, so the 17-day allowance has about 3 days of slack. If NOAA publishes late, the ocean job fails and Coral Reef Watch and waves wait a week, but that no longer blocks the other sources. |
A failed environmental source currently prevents every successful snapshot from being committed and released. The refresh now stages each source's files until that source completes, publishes successful sources, and reports failures after the release/image jobs. For example, a weather outage retains the previous weather files while refreshed tide and climate data can still ship; the workflow remains failed and lists source status in its summary.
HTTP 429 responses now wait at least one minute with increasing backoff and honor both seconds and HTTP-date
Retry-Aftervalues. Weather chunks pause 10 seconds. Historical ERDDAP 404s retry within the bounded request attempts, while final unpublished chunks still return immediately. Wave request errors and final chlorophyll outages now fail their source instead of publishing truncated data. Staging stays outsideexternal/so an interrupted fetch cannot accidentally commit partial files. The workflow budget increases to 60 minutes to accommodate backoff.Validation: lint, Svelte checks (zero errors/warnings), 22 fetch/refresh regression tests, application unit suite, production build, 37 Playwright/axe tests, actionlint, Docker build and synthetic-fixture container smoke test. Tests cover 429 recovery, both
Retry-Afterforms, retry exhaustion, transient 404s, response-body failures, preservation of multiple old files, and the CLI publishing tide data while reporting weather failure. No external snapshots or private data changed. The production weekly workflow was not dispatched; publication sequencing was checked with actionlint and review, without making a release.Closes #159.