Skip to content

Environment: restore OISST and disclose SST fallback sources - #158

Merged
Luen merged 3 commits into
mainfrom
codex/154-oisst-source
Nov 4, 2026
Merged

Luen merged 3 commits into
mainfrom
codex/154-oisst-source

Conversation

@Luen

@Luen Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

The site labelled its SST as OISST, but the committed OISST column was entirely empty and analyses used Coral Reef Watch. This restores 4,828 daily OISST values through 2026-10-19, fixes source credits, and shows live OISST/fallback/missing coverage on /ocean. All other snapshot columns and dates are unchanged.

The fetch now uses NOAA upwell ncdcOisst21Agg with 0–360 longitude and the surface selector. That aggregation silently omits 19 historical dates; the fetch recovers them from NOAA's oisst_rclone representation of the same final product. It validates daily continuity and bounded publication lag before writing. Historical request failures, final-chunk outages and all-null variables are fatal. Only an absent final chunk may be tolerated, subject to the whole-series checks. Requests have a timeout. The --only=oisst repair mode preserves independent ocean variables.

Source copy is shared and explicitly describes OISST plus CRW fallback. The current snapshot has ten CRW fallback days and two days without either source. No import logic changed; the external fingerprint already rebuilds caches.

Real-data recomputation preserves the calendar interpretation of the raw SST association: fully adjusted RR per °C is 0.974 (95% CI 0.835–1.136), versus 0.939 before. Nursery-minus-satellite mean changes from +0.43 to +0.42 °C. Seasonal sampling-night means change by −0.078 to +0.247 °C. Aggregate validation and source details are in docs/dev/OISST-154.md; private records and caches are excluded.

Validation: nine fetch regression tests, 107 unit tests, lint, svelte-check (zero errors/warnings), read-only real-data import and before/after analysis comparison, production build, Playwright/axe (37 tests). Fetch tests are included in pnpm test.

Closes #154.

@Luen Luen added bug Something isn't working environment Weather, ocean, tide, climate and astronomy data data-quality Issues in the source spreadsheets for the lab to review labels Nov 4, 2026
@Luen

Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review finding — [P2] OISST refresh can fail around New Year within the allowed publication lag. Calendar-year chunks make December 31 a fixed non-final endpoint even when final OISST is only published through mid/late December. The new fatal historical-error handling then aborts the weekly refresh. ERDDAP requires coordinate values to be inside the published axis range (documentation).

Fix prepared in f7e59f8: keep the full 17-day publication-lag window in the final request ending at last, retaining continuity and staleness checks. Three regression cases cover January 1, 17 and 18. The January 1 case fails with the original chunking and passes with the fix.

Validation: lint passed; Svelte checks returned zero errors/warnings; 12 fetch tests and 107 application tests passed (two opt-in real-data tests skipped). Verified all 4,840 snapshot dates and all non-OISST columns are unchanged, with 4,828 restored OISST values. No other actionable findings from the diff review. No live full NOAA refresh or real-data recomputation was performed during this review.

- /ocean: the coverage line moves from the page header into the SST chart's
  note, with formatted numbers and singular/plural wording.
- AGENTS.md and docs/dev/OISST-154.md: upwell redirects ncdcOisst21Agg to
  coastwatch.pfeg.noaa.gov (only oisst_rclone is served by upwell itself);
  note that ERDDAP failures now stop the ocean job for Coral Reef Watch too.
- erddap.test.mjs: drop a redundant re-import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Luen

Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Reviewed against NOAA's live data and the real survey data (aggregates only). It's right, and I've pushed small fixes in aa459c5.

What I checked

  • Snapshot: only the oisst column changed. None of the other 43,560 cells or dates moved, and the column now has 4,828 values.
  • Against NOAA today: all 4,809 values that ncdcOisst21Agg returns match the snapshot. The 19 days the aggregation skips include 19 and 21–31 December 2023; they come from oisst_rclone, and the three I checked match. Both datasets stamp days at 12:00 UTC, so the one-day recovery requests can't pick up the neighbouring day.
  • OISST against Coral Reef Watch, on days with both: OISST reads 0.02 °C higher on average, and 90% of days are within ±0.4 °C.
  • Real data: /drivers now says 22% fewer sharks per °C on its own, and a rate ratio of 1.04 (0.92–1.19) once day of the season is in. Both match the PR, and the calendar explanation still holds. Nursery water vs satellite uses 2,716 days. All of that copy is computed live, and the −22% in AGENTS.md agrees.
  • Timeout: 120 s is enough. A yearly OISST chunk takes about 22 s; the whole series in one request took 4 min 47 s, so the yearly chunks matter. A year of chlorophyll took 20 s and IBTrACS 16 s.
  • Checks: pnpm lint, pnpm check, 107 unit tests and 12 fetch tests, pnpm build, and the pages served on the real data.

Why the stricter fetch matters. The first weekly refresh on 2 Nov failed:

  • OISST returned 503, and the retries hung for about 16 minutes with no timeout.
  • PacIOOS returned 404 for the Coral Reef Watch chunks from 2023 to 2026, and the old code wrote ocean-daily.csv with those columns blank.
  • It wasn't committed only because the weather job also failed (Open-Meteo 429).

With this PR, either ERDDAP failure stops the ocean job before it writes. The cost is that one outage leaves the whole ocean snapshot (waves too) unchanged that week and fails the workflow, as chlorophyll already does. I filed #159 so one failed job stops blocking the others and the weather job gets through a 429.

Fixes pushed (aa459c5)

  • /ocean: the coverage sentence moves from the page header into the SST chart's note, where the mixed sources are. Numbers are formatted ("4,828", not "4828"), and singular days read correctly.
  • AGENTS.md and the validation doc: upwell redirects ncdcOisst21Agg to coastwatch.pfeg.noaa.gov, the same server whose _LonPM180 copy returns 503. Only oisst_rclone is served by upwell itself. I tightened the section and noted that Coral Reef Watch failures are fatal now too.
  • Test: removed a redundant re-import of validateDaily.

@Luen
Luen merged commit 9a890a4 into main Nov 4, 2026
5 checks passed
@Luen
Luen deleted the codex/154-oisst-source branch November 4, 2026 11:06
@Luen Luen mentioned this pull request Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working data-quality Issues in the source spreadsheets for the lab to review environment Weather, ocean, tide, climate and astronomy data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Environment: OISST has never been fetched; SST on the site is Coral Reef Watch

1 participant