Skip to content

Infrastructure: release successful environmental refreshes despite source outages - #161

Merged
Luen merged 2 commits into
mainfrom
codex/159-resilient-refresh
Nov 4, 2026
Merged

Luen merged 2 commits into
mainfrom
codex/159-resilient-refresh

Conversation

@Luen

@Luen Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

A failed environmental source currently prevents every successful snapshot from being committed and released. The refresh now stages each source's files until that source completes, publishes successful sources, and reports failures after the release/image jobs. For example, a weather outage retains the previous weather files while refreshed tide and climate data can still ship; the workflow remains failed and lists source status in its summary.

HTTP 429 responses now wait at least one minute with increasing backoff and honor both seconds and HTTP-date Retry-After values. Weather chunks pause 10 seconds. Historical ERDDAP 404s retry within the bounded request attempts, while final unpublished chunks still return immediately. Wave request errors and final chlorophyll outages now fail their source instead of publishing truncated data. Staging stays outside external/ so an interrupted fetch cannot accidentally commit partial files. The workflow budget increases to 60 minutes to accommodate backoff.

Validation: lint, Svelte checks (zero errors/warnings), 22 fetch/refresh regression tests, application unit suite, production build, 37 Playwright/axe tests, actionlint, Docker build and synthetic-fixture container smoke test. Tests cover 429 recovery, both Retry-After forms, retry exhaustion, transient 404s, response-body failures, preservation of multiple old files, and the CLI publishing tide data while reporting weather failure. No external snapshots or private data changed. The production weekly workflow was not dispatched; publication sequencing was checked with actionlint and review, without making a release.

Closes #159.

Luen and others added 2 commits November 4, 2026 21:12
- Release notes list the files that changed and, after a partial refresh,
  link the run, instead of always claiming weather, ocean, tide and ENSO.
- A full live fetch took 48 minutes on 4 Nov 2026 (NOAA ERDDAP 502/503s
  and six 120 s stalls), too close to the 60-minute limit.
- AGENTS.md: the #159 note sits with the refresh-external.yml bullet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Luen

Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Reviewed with a full live fetch: all seven sources, written into a scratch copy of external/, nothing committed. It works. I pushed two small changes in a61fd2f.

Live run (4 Nov 2026, 48 minutes)

  • Result: all seven sources refreshed, the exit code was 0, and the step summary was written.

  • The new snapshot only adds data:

    • ocean and weather: 2 new days each (48 hours of hourly weather);
    • ENSO: 1 new month;
    • waves: ERA5 replaced the forecast model's values for 27–28 Oct.

    No historical value changed. OISST still ends on 19 Oct, 14 days before the 2 Nov end date and inside the 17-day allowance.

  • The retry paths ran for real:

    • OISST: two 503s and six 120 s timeouts, each recovered on retry;
    • chlorophyll: eight 502s, each recovered on retry;
    • weather: no 429 this time, with chunks 10 s apart.

What I checked in the code

  • Staging: files go to a temporary folder next to external/ and are moved in only when the source finishes. So a failed weather job keeps both of its old files, and a test covers that.
  • Workflow:
    • Because of continue-on-error, steps.fetch.outcome is failure while the step's conclusion is success, so the report job sees the failure.
    • always() makes the report job run even when Docker is skipped.
    • actionlint, with shellcheck, passes.
  • 429: waits 60 s × the attempt number or Retry-After, whichever is longer. A failed body read is now retried too; before, a rejected res.text() escaped the retry loop.
  • 404: a historical ERDDAP chunk is retried. A final chunk's 404 still returns at once and means "not published yet". The OISST recovery requests count as historical.
  • Waves and the final chlorophyll chunk now fail their source instead of publishing gaps. With staging, that costs one week of that source and nothing else.

Changes pushed (a61fd2f)

  • Workflow timeout, 60 → 120 minutes. The live run took 48 minutes, mostly NOAA stalls, before any 429 backoff. A timeout cancels the whole job, so nothing would be published, which is exactly what this PR is meant to prevent.
  • Release notes: they always said "Weekly refresh of weather, ocean, tide and ENSO snapshots", which is wrong after a partial refresh. They now list the files that changed, and link the run when a source failed. I tested the snippet in a scratch repo.
  • AGENTS.md: moved the Infrastructure: weekly environmental refresh failed on its first run (Open-Meteo 429, ERDDAP outages) #159 note next to the refresh-external.yml bullet and shortened it.

One thing to watch (no change here). Final OISST is currently 16 days behind, so the 17-day allowance has about 3 days of slack. If NOAA publishes late, the ocean job fails and Coral Reef Watch and waves wait a week, but that no longer blocks the other sources.

@Luen
Luen merged commit a1b317d into main Nov 4, 2026
5 checks passed
@Luen
Luen deleted the codex/159-resilient-refresh branch November 4, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Infrastructure: weekly environmental refresh failed on its first run (Open-Meteo 429, ERDDAP outages)

1 participant