[anth-volk]

Fixes #278

Stacked on #241 (base microcosm-releases-json-readers), which reads releases.json with the release writer's rules (releasesRecordFromResponse in release-record.ts). This PR's own changes are the diff against that base, which GitHub shows.

• This PR's commits: f8614fe, ce68777, bc72806, a13909c, c4c8ae0, 4555362, 847b8fb, 4bbc59a, 20c4cca and 4a93ac3.
• Read Microcosm releases from releases.json wherever a repository has it, and from tags otherwise #241's commits, brought in by the merge commits bdfca47 and 4e6b89f, are reviewed in Read Microcosm releases from releases.json wherever a repository has it, and from tags otherwise #241.

Once #241 merges, change the base to main.

Why

Builds that Microcosm publishes into build folders get no Hugging Face tag. The webhook's Slack release alert has only ever fired for new tags, so these publications would go unannounced. This PR adds alerts from releases.json. It has to be live before the UK repository switches to releases.json; until then, UK alerts keep coming from new tags.

What changes

When an update to the main of a release repository whose country has a Slack channel arrives (the US and the UK; Belgium has none), the webhook first reads releases.json at the update's oldSha and newSha (only newSha for a new main). It then dispatches the tree publication and posts the tag alerts, as before, and posts one Slack message to the country's channel for each change in the record:

Change between the two records	Message
a newly listed build that became current	New Microcosm UK build `<release>/<build>`: now current
a newly listed build that did not	New Microcosm UK build `<release>/<build>`: published, not current (current stays `<build>`) (or (`<release>` has no current build))
a release's current build changed, other than to a new build	Microcosm UK `<release>`: current build changed from `<old>` to `<new>`
a build newly marked withdrawn	Microcosm UK build `<release>/<build>` withdrawn
releases.json did not exist before the update (Microcosm's switch commit)	one summary: Microcosm UK now records its releases in `releases.json`: N builds in M releases, then one entry per release with its current build

• Each message keeps the existing context text: the repository and the dashboard link. Only new-build messages also carry a time, the build's published_at. UK is the country code, so US and Belgian messages read the same way.
• Escaping: values the dashboard does not check (published_at, a tag's name in tag alerts) are escaped for Slack's formatted text (&, <, >), so none can mention a channel or form a link.
• The summary lists each release's current build in sections of at most 3,000 characters, within Slack's 50 blocks. An empty release list adds no section.
• The order is: new builds, then current-build changes, then withdrawals. Releases and builds are in name order.
• No record: a repository without releases.json on both sides gets none of these messages. Only Hugging Face's 404 for the file counts as no record; a 404 for a missing repository or revision does not.
• Transient failure, retried: Hugging Face answers 5xx or 429, or the request or its body fails or times out (15 seconds for both). The webhook logs it and answers 503 before dispatching or posting anything, and Hugging Face retries the delivery. Since nothing was done, the retry dispatches, posts the tag alerts and posts the record alerts once each. Only a repository that can get record alerts reads the record, so only it risks a 503.
  • Why this and not a retry after posting: the handler has no store to remember which alerts a delivery already posted, so answering 5xx after the dispatch and the tag alerts would repeat them. Reading first leaves nothing to repeat.
  • The cost: while the Hub fails, a US or UK release repository's main updates are neither dispatched nor alerted until a retry succeeds.
    • A long outage reaches every dataset: one Hugging Face webhook serves all five watched datasets (the US, UK and Belgian release repositories and the US and UK staging repositories). A long run of 503 answers can make Hugging Face suspend it, and then deliveries stop for all five.
    • Recovery: the guide's failure list covers it, and applies whenever deliveries were answered 503: Hugging Face documents no retry limit, and doesn't promise that the Activity page shows an event's last answer.
      1. Use the function logs to find each delivery answered 503 and never later answered 200.
      2. Replay those, oldest first.
      3. For updates made during a suspension, send a hand-made release payload. Its main entry runs from the newSha of the repository's latest delivery answered 200 to its head, and it has one refs/tags/<tag> entry, with oldSha null, for each tag created meanwhile. Send one staging payload as well.
      4. A delivery answered with release_record_problem is replayed once its cause is corrected. That replay repeats any tag alert it carried.
    • Verification: after a deployment, the guide's replay check also requires an answer without release_record_problem, which shows the token can read the UK repository.
    • Late retries: a retry that comes late can post after newer messages, so its "current stays <build>" may be out of date.
    • Still with Anthony: whether he accepts this design, which holds back the dispatch during Hub failures.
• Lasting failure, logged and named: a token that cannot read the repository (401 or 403), a missing repository or revision, or a record the writer's rules refuse, on either side.
  • It is logged (console.error), and the answer names it as release_record_problem, so Hugging Face's activity page tells it apart from "nothing changed" (release_record_alerted: [] alone).
  • The dispatch and the tag alerts go ahead, and no record alert is posted. Retrying would not help.
  • Both reads are settled first. A lasting failure on one side wins over a transient one on the other.
• Changes with no alert: a deleted releases.json, a change of default, and a build that is no longer withdrawn are logged as warnings (console.warn), and nothing is posted.
• Tags: alerts for new tags are unchanged.
• Staging: staging repositories read no releases.json.
• Slack failures are logged, as for tag alerts.
• Response: when the webhook reads releases.json (a main update of a release repository whose country has a Slack channel, so not Belgium's), its JSON answer adds release_record_alerted, describing each alert it posted, and release_record_problem after a lasting failure. Any other delivery, including a Belgian main update, answers exactly as before.

Code:

• lib/microcosm/release-record-alerts.ts: the comparison.
• lib/slack.ts: releaseRecordAlertPayload and postReleaseRecordAlert. The tag alert shares their posting and context code, and its payload is unchanged.
• lib/microcosm/hf-webhook-handler.ts: the two reads and the posting.
• hf-webhook-config.ts: huggingFaceToken and releaseAlertCountries, both required and both read when the server starts. A change to HF_TOKEN or to a SLACK_WEBHOOK_POPULACE_* variable takes effect only with the next Deploy production run.
• huggingFaceToken is resolved by huggingFaceReadToken, new in latest-artifact.ts: HF_TOKEN, else HUGGINGFACE_TOKEN, as set, and an empty value sends no token.
  • That is the rule every dashboard Hugging Face read uses, but only latest-artifact.ts calls the function. Three modules keep their own copies of the same rule, all already on main: publish-calibration-tree.ts, calibration-release-locator.ts and staging-artifact.ts.
  • 4555362's commit message says the other readers "now share" it; that overstates it.
  • The one difference from the other readers is when it is read: once, when the server module starts, like the rest of the webhook's configuration (route.ts resolves it there by design). So a new token takes a redeploy, as a new HF_WEBHOOK_SECRET does.
• Docs: docs/calibration-tree-artifacts.md ("Slack release alerts", and the replay advice) and frontend/HOSTED_RUNTIME.md (Hugging Face reads).

Decided by Anthony on 2026-11-04

• the three message texts above, each with the existing context text;
• one summary message when an update first creates releases.json;
• an invalid or unreadable record is logged, and nothing is posted.

Still with Anthony, unchanged here: the summary's wording, the "published, not current (<release> has no current build)" variant and "current build changed from none to <build>" are mine and await his agreement. The Vercel checks on HF_TOKEN are also his.

Known limit, left for later (review round 1, quality 2): the webhook compares only the two ends of an update. A delivery that carries several commits reports their net change. For example, a build published and then withdrawn within one push is announced as "published, not current", with no withdrawal. A build published as not current while another became current says "current stays <build>" before the separate change message. Microcosm's release writer pushes one commit per change through the Hub API, so this needs a push of several commits.

Environment variables

This PR adds one new read of an existing variable: the webhook now reads HF_TOKEN (or HUGGINGFACE_TOKEN). It adds no new variable and changes no workflow or deployment file. None of the Vercel project settings could be inspected from here, so each of the four is unverified:

Setting to check	Read by	Expected provider	Wiring	Status
SLACK_WEBHOOK_POPULACE_UK	lib/slack.ts, at runtime, for UK alerts (tag and releases.json); also, new here, hf-webhook-config.ts at server start, to decide whether the UK release repository's releases.json is read	Vercel project environment variable, Production (runtime)	Read from process.env when an alert is posted, and once at server start (hasReleaseAlertChannel). HOSTED_RUNTIME.md lists it among the Vercel Production settings (names listed on 2026-10-26). If it were unset, the UK would read no record and get no record alerts.	unverified
SLACK_WEBHOOK_POPULACE_US	the same, for US alerts	Vercel project environment variable, Production (runtime)	the same	unverified
HF_WEBHOOK_SECRET	hf-webhook-config.ts, at server start, to authenticate deliveries	Vercel project environment variable, Production (runtime), sensitive	Unchanged	unverified
HF_TOKEN (or HUGGINGFACE_TOKEN)	hf-webhook-config.ts, at server start (new here), to read releases.json of the private UK repository	Vercel runtime, expected	deploy.yml passes the repository Actions secret HF_RELEASE_READ_TOKEN (it exists) to deploy-dashboard.sh, which sets it with vercel deploy --env HF_TOKEN=… on production deployments. The Vercel project also stores its own Production HF_TOKEN; which value a deployment uses, and whether the token can read the UK repository, are not verified. Previews use the Vercel Preview HF_TOKEN.	unverified

Belgium has no Slack webhook variable, so it gets no alerts, as before. If HF_TOKEN cannot read the UK repository (401), every UK update logs a failed releases.json read and posts no releases.json alert. That failure is lasting, so it is not retried, and tag alerts and the dispatch are unaffected.

Tests

Only the tests this PR adds or changes were run locally, each selected by exact name: bun test <file> -t "^(<names>)$". That covers the webhook route tests whose shared helper this PR changes. CI runs the rest. All repositories, records and the Slack incoming webhook address are synthetic; nothing is sent to Slack or read from Hugging Face.

Commit	Red	Green
f8614fe Test Slack release alerts from changes to releases.json	app/api/hf-webhook/route.test.ts, 20 selected: the 10 new alert tests failed on their assertions (no releases.json read and no message posted). The 9 existing dispatch tests and "a staging repository's main update reads no releases.json" passed. lib/microcosm/hf-webhook-config.test.ts "reads the Hugging Face token for releases.json alerts from HF_TOKEN, or else HUGGINGFACE_TOKEN" failed on toBe. The new tests didn't type-check at red: they pass the new huggingFaceToken setting.	—
ce68777 Post Slack release alerts from changes to releases.json	—	the same 21 tests passed
bc72806 / a13909c Answer a delivery without a main update as before	CI at ce68777 failed two existing tests in lib/microcosm/reviewed-repository-import.test.ts: a tag delivery's answer gained an empty release_record_alerted. The new route test "a delivery without a main update answers as before, naming no releases.json alerts" failed on toEqual. "a main update's answer lists the releases.json alerts it posted" passed: it guards the main-update answer.	both passed, and so did the 21 above (23 in all). CI re-runs the two existing tests.
bdfca47 Merge #241's review round 1 fixes (no conflicts)	—	the 23 above passed
c4c8ae0 Test review round 1's fixes to the releases.json alerts	15 tests (details below): 7 failed on assertions, 8 passed	—
4555362 Retry transient releases.json failures, escape Slack text, log silent changes	—	the 15 passed, and so did the earlier 21 still selected by name (two of the 23 were renamed or replaced in c4c8ae0)
4e6b89f Merge #241's review round 2 fixes (no conflicts)	—	bun run lint clean
847b8fb Test review round 2's points	7 tests: 5 failed on assertions, 2 passed (details below)	—
4bbc59a Retry body failures, let lasting failures decide, and read only where alerted	—	the 7 passed, and so did all the earlier ones (8 direct, 6 + 21 route and 2 configuration tests by name)
20c4cca The guide: the 503's scope and the recovery steps	— (docs)	—
4a93ac3 Review round 3: the recovery steps' open cases, what is read at server start, and the verification check	— (docs and comments; the configuration test drops ?? [] on the now-required field)	"reads releases.json only for the countries whose Slack channel is configured" passed; bun run lint clean

The existing dispatch tests' helper answers releases.json with Hugging Face's "entry not found" 404 before their own fetch, so their GitHub call counts are unchanged. The configuration's helper in the route test gains huggingFaceToken: undefined, because the field is now required (a type change only, in 4555362).

The 7 failures at c4c8ae0:

• lib/microcosm/release-record-alerts.test.ts:
  • "the summary of a releases.json created with no releases has no empty section" (toBeGreaterThan);
  • "the summary of a long releases.json stays within Slack's limits and names every release" (toBeLessThanOrEqual);
  • "a publication time is escaped for Slack's formatted text" (toContain);
  • "a tag name is escaped in its release alert" (toContain).
• app/api/hf-webhook/route.test.ts:
  • "a transient failure reading releases.json answers 503 before dispatching or posting anything, so Hugging Face's retry posts each alert once" (toBe: 200);
  • "a deleted releases.json, a change of default release and an un-withdrawal are logged and post nothing" (toContain).
• lib/microcosm/hf-webhook-config.test.ts: "reads the Hugging Face token for releases.json alerts as the dashboard's other readers do: HF_TOKEN, else HUGGINGFACE_TOKEN" (toBe). It replaces the token test from f8614fe, which expected trimming.

The 5 failures at 847b8fb, all in app/api/hf-webhook/route.test.ts except the last:

• "a connection that fails while the body of releases.json is read is transient, so the delivery is retried" (toBe: 200 instead of 503, the review's bug);
• "when one side fails for a lasting reason and the other transiently, the lasting failure decides: no retry" (toBe: 503);
• "a lasting failure is named in the answer, and a record that did not change is not" (toContain);
• "a release repository whose country has no Slack channel reads no releases.json and is never answered 503" (toBe: 503);
• lib/microcosm/hf-webhook-config.test.ts "reads releases.json only for the countries whose Slack channel is configured" (toEqual).

The 2 that passed: the summary's "…and N more" branch (2,000 releases, in release-record-alerts.test.ts), and "a delivery without a main update answers as before, naming no releases.json alerts", rerun to check the answer's content still holds.

The 8 that passed guard behaviour that already held:

• in release-record-alerts.test.ts: several alerts in one update and their order; a new release; "from none"; an unchanged or absent record;
• in the route test: a lasting failure (RevisionNotFound, 401) with the dispatch and tag alert going ahead. This replaces the earlier "cannot be read" test, whose 503 case is now the transient test;
• a null oldSha;
• a Slack failure for one alert;
• no token, no Authorization header.

bun run lint (tsc --noEmit for the app and the package) is clean.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Nov 4, 2026 4:49pm UTC