You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Infrastructure: weekly environmental refresh failed on its first run (Open-Meteo 429, ERDDAP outages) #159
What happened. The first scheduled run of refresh-external.yml failed on 2 Nov 2026 (run 41543545970) and committed nothing:
weather: Open-Meteo's archive API answered 429 (too many requests) for the 2026 chunk. Two more tries then failed outright, so the job threw. Open-Meteo counts a long request with many variables as several calls. Each yearly chunk asks for a year of 7 hourly and about 12 daily variables, and the job sends 14 of them 1.5 s apart, so it probably goes over the free per-minute limit. Runners share IP addresses, which doesn't help.
ocean: the OISST _LonPM180 requests returned 503 and hung for about 16 minutes (there was no request timeout). PacIOOS also returned 404 for the Coral Reef Watch chunks from 2023 to 2026. The old code wrote ocean-daily.csv with all of those blank.
Effect: a job that throws sets the exit code, so the "Fetch snapshots" step fails and nothing is committed. That includes the tide and climate files, which had refreshed fine.
The failed weather job is the only reason the blanked CRW columns weren't committed and released.
Since then.#158 makes ERDDAP failures fatal: a historical chunk that fails or comes back blank stops the ocean job before it writes. It also adds a 120 s request timeout. That fixes the blanking, but an OISST, CRW or Open-Meteo outage now fails the whole weekly run. The CRW request that returned 404 answered normally on 4 Nov.
What to change
Weather: survive a 429. Wait longer between chunks, and back off for at least a minute on 429 (honour Retry-After if it's sent).
Isolate jobs: a job that fails should keep its old file without blocking the others. Commit the snapshots that did refresh, then fail the workflow (or report which job failed) so the failure stays visible.
Maybe: retry a 404 on a historical ERDDAP chunk a few times before failing, since PacIOOS's 404s were transient. A 404 on the final chunk is expected, because that data isn't published yet.
Acceptance
A run with one source down still commits and releases the other snapshots, and still shows which job failed.
#161 fixes this. Each source is now staged and replaces its files only when it finishes, so one failed source keeps its old snapshot while the others still commit and release. The run still turns red afterwards and the step summary lists each source. Open-Meteo 429s back off for at least a minute (or Retry-After), weather chunks are 10 s apart, and historical ERDDAP 404s are retried. A full live fetch on 4 Nov refreshed all seven sources in 48 minutes, recovering from NOAA 502/503s and timeouts. It only appended new days. Review notes are on #161.
Activity
Refresh: name updated files in release notes; allow 120 minutes
Luen commentedon Nov 4, 2026
#161 fixes this. Each source is now staged and replaces its files only when it finishes, so one failed source keeps its old snapshot while the others still commit and release. The run still turns red afterwards and the step summary lists each source. Open-Meteo 429s back off for at least a minute (or
Retry-After), weather chunks are 10 s apart, and historical ERDDAP 404s are retried. A full live fetch on 4 Nov refreshed all seven sources in 48 minutes, recovering from NOAA 502/503s and timeouts. It only appended new days. Review notes are on #161.Infrastructure: release successful environmental refreshes despite so…