Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions .github/workflows/refresh-external.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@ permissions:
jobs:
refresh:
runs-on: ubuntu-latest
timeout-minutes: 30
# A full run took 48 minutes on 4 Nov 2026 with NOAA ERDDAP stalls and 429 backoff possible on top.
timeout-minutes: 120
outputs:
version: ${{ steps.version.outputs.version }}
changed: ${{ steps.diff.outputs.changed }}
fetch-outcome: ${{ steps.fetch.outcome }}
steps:
- uses: actions/checkout@v7
with:
Expand All @@ -36,6 +38,8 @@ jobs:
node-version: 26

- name: Fetch snapshots
id: fetch
continue-on-error: true
run: node scripts/fetch-external.mjs

- name: Detect changes
Expand All @@ -61,16 +65,24 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ steps.version.outputs.version }}
FETCH_OUTCOME: ${{ steps.fetch.outcome }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add external package.json
# Name what changed: after a partial refresh, not every source was updated (#159).
FILES="$(git diff --cached --name-only -- external | sed 's#^external/##' | paste -sd, - | sed 's/,/, /g')"
NOTES="Weekly refresh of the environmental snapshots. Updated: ${FILES}."
if [ "$FETCH_OUTCOME" = "failure" ]; then
NOTES="${NOTES} Some sources failed and kept their previous snapshot; see ${RUN_URL}."
fi
git commit -m "chore(data): weekly environmental data refresh (v${VERSION})"
git pull --rebase origin main
git tag "v${VERSION}"
git push origin HEAD:main "v${VERSION}"
gh release create "v${VERSION}" --target main --title "v${VERSION}" --notes "Weekly refresh of weather, ocean, tide and ENSO snapshots."
gh release create "v${VERSION}" --target main --title "v${VERSION}" --notes "$NOTES"

docker:
needs: refresh
Expand All @@ -81,3 +93,15 @@ jobs:
packages: write
with:
ref: v${{ needs.refresh.outputs.version }}

# Wait for the release/image before surfacing partial failure. Otherwise a failed
# refresh dependency would skip Docker and leave successful snapshots undeployed.
report:
needs: [refresh, docker]
if: always() && needs.refresh.outputs.fetch-outcome == 'failure'
runs-on: ubuntu-latest
steps:
- name: Report source failures
run: |
echo "::error::Some environmental sources failed. See the refresh job summary and fetch log. Successful snapshots were eligible for release; failed sources retained their previous snapshots."
exit 1
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,6 @@ vite.config.ts.timestamp-*

# Local preview config
/.claude/

# Incomplete environmental refresh staging (normally removed automatically)
.external-refresh-*/
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ Drop the workbook in `data/` named like `Physioshark_Shark Log 2026-2027.xlsx` (
- `docker-release.yml`: builds linux/amd64 + arm64 and pushes on `v*` tag pushes, manual dispatch or `workflow_call`. Image name is lower-cased from the repo (`rummerlab/data-stats`).
- `nightly-minor-deps-release.yml`: `npm-check-updates --target semver --cooldown 3`, full suite, then a patch release + image **only when a package that reaches the image moved** (tooling list in `scripts/nightlyRelease.mjs`; add new lint/test devDependencies there). It pushes to `main`, so branch protection needs a bypass for Actions.
- `refresh-external.yml`: weekly `fetch-external.mjs`; when `external/` changed it commits, tags a patch release and publishes the image.
- Partial refreshes (#159): each source writes into a staging folder (`external-refresh.mjs`) and replaces its files only when it finishes, so a failed source keeps its old snapshot and the others still ship. The fetch step may fail (`continue-on-error`); the workflow still commits and releases what changed, and the `report` job turns the run red afterwards (also when Docker was skipped). Don't fail the refresh job before publishing. Release notes list the updated files. The job allows 120 minutes: a full fetch took 48 on 4 Nov 2026, mostly NOAA ERDDAP stalls (each request times out after 120 s and retries). HTTP 429 waits at least a minute (or `Retry-After`), weather chunks are 10 s apart (Open-Meteo counts a long request with many variables as several calls; 1.5 s apart hit its per-minute limit on 2 Nov 2026), and historical ERDDAP 404s are retried; a final chunk's 404 means not yet published.
- Dependabot covers GitHub Actions only: its pnpm runs on Node < 26 and `engineStrict` fails every npm job. npm majors are upgraded by hand.
- `ci.yml`: lint + check + unit tests, build + Playwright/axe, and a Docker build + smoke test (synthetic fixture) on both architectures.
- Runner is distroless `cc-debian13:nonroot` (uid 65532) with the build stage's Node binary: no shell. Debug with `node -e`.
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"preview": "vite preview",
"prepare": "svelte-kit sync || echo ''",
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
"test": "node --test scripts/erddap.test.mjs && vitest run",
"test": "node --test scripts/erddap.test.mjs scripts/external-refresh.test.mjs && vitest run",
"test:related": "vitest related --run",
"test:watch": "vitest",
"test:e2e": "playwright test",
Expand Down
4 changes: 3 additions & 1 deletion scripts/erddap.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@ export async function readSeries({
`${v}%5B(${start}T12:00:00Z):1:(${final && latest ? 'last' : end + 'T12:00:00Z'})%5D${zlev ? '%5B(0.0)%5D' : ''}%5B(${lat})%5D%5B(${lon})%5D`;
let text;
try {
text = await get(`${base}/griddap/${dataset}.csv?${vars.map(select).join(',')}`);
text = await get(`${base}/griddap/${dataset}.csv?${vars.map(select).join(',')}`, {
retry404: !final || !latest
});
} catch (err) {
// Only a final chunk with no published observations may be absent. Outages are fatal.
if (!final || !String(err).includes('404')) throw err;
Expand Down
74 changes: 74 additions & 0 deletions scripts/external-refresh.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
import { mkdtempSync, renameSync, rmSync, writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';

export const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
export const WEATHER_CHUNK_DELAY_MS = 10_000;

/** HTTP retries include body reads; injected I/O keeps outage tests fast and offline. */
export function createGet({
fetch: request = globalThis.fetch,
wait = sleep,
now = Date.now
} = {}) {
return async function get(url, { json = false, tries = 5, retry404 = false } = {}) {
let error;
for (let i = 0; i < tries; i++) {
let delay = 2000 * (i + 1) ** 2;
try {
const res = await request(url, {
signal: AbortSignal.timeout(120_000),
headers: {
'user-agent': 'physioshark-data-stats (+https://github.com/RummerLab/data-stats)'
}
});
if (res.ok) return await (json ? res.json() : res.text());
error = new Error(`${res.status} ${url}`);
error.status = res.status;
if (res.status === 429) {
const value = res.headers.get('retry-after');
const retryAfter =
value == null
? 0
: /^\d+(\.\d+)?$/.test(value.trim())
? Number(value) * 1000
: Date.parse(value) - now();
delay = Math.max(60_000 * (i + 1), Number.isFinite(retryAfter) ? retryAfter : 0);
}
await res.body?.cancel();
if (res.status === 404 && !retry404) throw error;
} catch (err) {
error = err;
if (err.status === 404 && !retry404) throw err;
}
if (i + 1 < tries) {
console.warn(`[external] ${error.message}; retrying in ${delay / 1000}s`);
await wait(delay);
}
}
throw error;
};
}

/** Fetch into staging so a failed source never publishes its partially written files. */
export async function refreshJobs(out, jobs, { report = console.log } = {}) {
const failed = [];
for (const [name, job] of Object.entries(jobs)) {
// Keep staging outside external/: interrupted runs must never git-add partial files.
const stage = mkdtempSync(join(dirname(out), '.external-refresh-'));
const files = new Set();
try {
await job((file, body) => {
writeFileSync(join(stage, file), body);
files.add(file);
});
for (const file of files) renameSync(join(stage, file), join(out, file));
report(`[external] ${name}: refreshed (${files.size} files)`);
} catch (err) {
failed.push(name);
report(`[external] ${name}: FAILED — ${err.message ?? err}`);
} finally {
rmSync(stage, { recursive: true, force: true });
}
}
return failed;
}
151 changes: 151 additions & 0 deletions scripts/external-refresh.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
import assert from 'node:assert/strict';
import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { test } from 'node:test';
import { createGet, refreshJobs } from './external-refresh.mjs';

for (const [header, expected] of [
[null, 60_000],
['5', 60_000],
['120', 120_000],
['Wed, 04 Nov 2026 00:02:00 GMT', 120_000],
['invalid', 60_000]
]) {
test(`429 recovers after Retry-After ${header}`, async () => {
const waits = [];
let calls = 0;
const get = createGet({
now: () => Date.parse('2026-11-04T00:00:00Z'),
wait: async (ms) => waits.push(ms),
fetch: async () =>
calls++ === 0
? new Response('', { status: 429, headers: header ? { 'Retry-After': header } : {} })
: Response.json({ daily: { time: ['2026-11-01'] } })
});
assert.deepEqual(await get('https://example.test/weather', { json: true }), {
daily: { time: ['2026-11-01'] }
});
assert.deepEqual(waits, [expected]);
assert.equal(calls, 2);
});
}

test('persistent 429 has bounded attempts and increasing backoff', async () => {
const waits = [];
let calls = 0;
const get = createGet({
wait: async (ms) => waits.push(ms),
fetch: async () => {
calls++;
return new Response('', { status: 429 });
}
});
await assert.rejects(get('https://example.test/weather', { tries: 3 }), { status: 429 });
assert.equal(calls, 3);
assert.deepEqual(waits, [60_000, 120_000]);
});

test('historical 404 retries, final 404 remains immediate', async () => {
let calls = 0;
const get = createGet({
wait: async () => {},
fetch: async () => (++calls % 2 ? new Response('', { status: 404 }) : new Response('recovered'))
});
assert.equal(await get('https://example.test/history', { retry404: true }), 'recovered');
assert.equal(calls, 2);
await assert.rejects(get('https://example.test/final'), { status: 404 });
assert.equal(calls, 3);
});

test('failed response body reads retry within the attempt limit', async () => {
let calls = 0;
const get = createGet({
wait: async () => {},
fetch: async () =>
++calls === 1
? {
ok: true,
text: async () => {
throw new Error('body timeout');
}
}
: new Response('complete')
});
assert.equal(await get('https://example.test/body'), 'complete');
assert.equal(calls, 2);
});

test('failed source retains all old files while other sources publish and report', async () => {
const out = mkdtempSync(join(tmpdir(), 'external-refresh-'));
const reports = [];
try {
writeFileSync(join(out, 'weather-daily.csv'), 'old daily');
writeFileSync(join(out, 'weather-hourly.csv.gz'), 'old hourly');
writeFileSync(join(out, 'climate.csv'), 'old climate');
const failed = await refreshJobs(
out,
{
weather: async (write) => {
write('weather-daily.csv', 'new daily');
write('weather-hourly.csv.gz', 'new hourly');
throw new Error('source down');
},
climate: async (write) => write('climate.csv', 'new climate'),
tide: async (write) => write('tide.csv', 'new tide')
},
{ report: (line) => reports.push(line) }
);
assert.deepEqual(failed, ['weather']);
assert.equal(readFileSync(join(out, 'weather-daily.csv'), 'utf8'), 'old daily');
assert.equal(readFileSync(join(out, 'weather-hourly.csv.gz'), 'utf8'), 'old hourly');
assert.equal(readFileSync(join(out, 'climate.csv'), 'utf8'), 'new climate');
assert.equal(readFileSync(join(out, 'tide.csv'), 'utf8'), 'new tide');
assert.ok(reports.some((line) => line.includes('weather: FAILED')));
assert.deepEqual(readdirSync(out).sort(), [
'climate.csv',
'tide.csv',
'weather-daily.csv',
'weather-hourly.csv.gz'
]);
} finally {
rmSync(out, { recursive: true, force: true });
}
});

test('CLI reports a failed weather source and publishes tide data with a failing exit status', async () => {
const { spawnSync } = await import('node:child_process');
const { gunzipSync } = await import('node:zlib');
const out = mkdtempSync(join(tmpdir(), 'external-cli-'));
try {
writeFileSync(join(out, 'weather-daily.csv'), 'old weather');
const mock = `globalThis.fetch = async (url) => {
if (url.includes('archive-api.open-meteo.com')) return new Response('', { status: 404 });
if (url.includes('/h015.csv')) return new Response('2026,11,1,12,1000');
throw new Error('Unexpected request: ' + url);
};`;
const summary = join(out, 'summary.md');
const result = spawnSync(
process.execPath,
[
'--import',
`data:text/javascript,${encodeURIComponent(mock)}`,
'scripts/fetch-external.mjs',
'--only=weather,tide'
],
{ encoding: 'utf8', env: { ...process.env, EXTERNAL_DIR: out, GITHUB_STEP_SUMMARY: summary } }
);
assert.equal(result.status, 1, result.stderr);
assert.match(result.stdout, /weather: FAILED/);
assert.match(result.stdout, /tide: refreshed/);
assert.equal(readFileSync(join(out, 'weather-daily.csv'), 'utf8'), 'old weather');
assert.match(
gunzipSync(readFileSync(join(out, 'tide-papeete-hourly.csv.gz'))).toString(),
/2026-11-01T02,1000/
);
assert.match(readFileSync(summary, 'utf8'), /weather: FAILED/);
assert.match(readFileSync(summary, 'utf8'), /tide: refreshed/);
} finally {
rmSync(out, { recursive: true, force: true });
}
});
Loading
Loading