Only fetch the github data on one os in ci

[dougdonohoe]

Follow-up to #497, which was closed after the other fixes were cherry-picked. This is the one that was not taken, rebased onto current master and reduced to a single commit.

The problem

Enriching the listing costs one github api request per repository. Doing that on all three operating systems triples the total spend, against an hourly budget shared by every job and every run in the repository:

Error: API rate limit exceeded for installation.
Request to https://api.github.com/repos/abakan-zz/ablog failed with status 403: Forbidden

The pool: new PromisePool(100) added in 95753b7 caps how many requests are in flight at once, which is the concurrency ceiling. This is the hourly quota, a separate limit — the pool does not reduce how many requests are made in total.

The change

The enriched result is identical on every platform, so within ci only linux fetches it. Run locally it always fetches, whatever the platform, so regenerating the listing by hand is unaffected.

The suite still runs everywhere, because it writes raw.json and hydrated.json, which source/index.ts imports and our:verify resolves. Skipping the whole suite off linux instead fails with:

error  Can't resolve '../hydrated.json'
error  Can't resolve '../raw.json'

So this adds a github option to hydrate that skips only the api calls; the entries are still assembled and written, without the github fields.

Verified against the real listing with a stubbed fetch:

	api requests	entries
hydrate(list, { github: false })	0	478, ids intact
hydrate(list, {})	430	478
ci on darwin	0	writes both json files

Why this may unblock the site

https://staticsitegenerators.bevry.me/ currently returns:

SyntaxError: Unexpected token 'N', "Not found:"... is not valid JSON

The worker fetches the listing from the npm cdn and gets a plain text 404 back, then parses it as json — the 'N' is the first character of Not found: /staticsitegenerators@4.0.1/hydrated.json. Unpacking the published tarballs shows why:

version	raw.json + hydrated.json
4.0.0	present, cdn serves them
4.0.1	absent

Which is #498, and 7d855f3 is the right fix for it. But publish declares needs: test, so it only runs when all three test jobs pass — and the run for 7d855f3 failed on windows with the rate limit above, so publish was skipped and the packaging fix has not shipped.

This PR does not fix the site directly. It removes the thing that has been intermittently failing a job and therefore skipping publish, so 7d855f3 gets a chance to run. Cutting the per-run spend by two thirds should make that considerably more reliable.

If the site needs restoring before that lands, pinning the worker to 4.0.0 would do it immediately, since that version still has the files on the cdn.

[balupton]

Happy for this is PR to persist as a reference resource in case I change my mind in the future.

Why this may unblock the site

https://staticsitegenerators.bevry.me/ currently returns:

SyntaxError: Unexpected token 'N', "Not found:"... is not valid JSON

The worker fetches the listing from the npm cdn and gets a plain text 404 back, then parses it as json — the 'N' is the first character of Not found: /staticsitegenerators@4.0.1/hydrated.json. Unpacking the published tarballs shows why:

version raw.json + hydrated.json
4.0.0 present, cdn serves them
4.0.1 absent
Which is #498, and 7d855f3 is the right fix for it. But publish declares needs: test, so it only runs when all three test jobs pass — and the run for 7d855f3 failed on windows with the rate limit above, so publish was skipped and the packaging fix has not shipped.

This PR does not fix the site directly. It removes the thing that has been intermittently failing a job and therefore skipping publish, so 7d855f3 gets a chance to run. Cutting the per-run spend by two thirds should make that considerably more reliable.

If the site needs restoring before that lands, pinning the worker to 4.0.0 would do it immediately, since that version still has the files on the cdn.

Site break was due to #498 now fixed by 7d855f3 and the v4.0.2 publish.

---

For now, I'm going to leave it there. Ping me back if there is anything to follow-up on. Thanks so much for the help.

[dougdonohoe]

I think it makes sense to limit checks to one OS - not quite clear why this site worries about Mac and Windows to begin with, given the purpose is to produce a website.

[balupton]

I think it makes sense to limit checks to one OS - not quite clear why this site worries about Mac and Windows to begin with, given the purpose is to produce a website.

The package can be consumed beyond the website. The package is intended for other listings/websites to fetch the data themselves. At least that was the original idea.

I'd be more inclined to just drop windows and mac ci runners from the matrix. But for now, it's good enough. Rate limited hits can be restarted, and so far all is green.

[dougdonohoe]

Closing per comments above.