From 241fc5be5ae2c2bfedc3973f099def76b26d9b72 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Mon, 16 Nov 2026 20:10:20 +0200 Subject: [PATCH 1/5] feat(release-mirror): publish complete Stable and Beta desktop update feeds The release mirror worker now copies each channel's whole electron-updater set (channel manifests, macOS ZIPs and DMGs, NSIS EXE, AppImage, blockmaps) from the desktop GitHub releases into the immutable scient-desktop// directories and then promotes generic feeds at scient-desktop/updates/{stable,beta}/. - Every manifest must match the release handoff's size and SHA-256; every file it names must be a bare name the handoff lists with the same size, and must hash to the manifest's SHA-512 while streaming (the last chunk is held back until it does). - Channel manifests are written last, each as a compare-and-swap on the version seen before asking GitHub, so a feed never names missing bytes and a slow run cannot undo a newer promotion. Old objects are kept; nothing is redirected. - The feed copy only re-points file URLs to ../..// (resolved by electron-updater against the feed) and adds releaseName/releaseNotes from the release's Atom entry, as GitHub's provider showed them. Digests are untouched. - Beta is discovered from the beta repository's releases feed, by version. - C1: versioned objects stay one-year immutable; feed manifests are no-store. - The mirror worker remains the only publisher; README has the deploy/verify runbook. Co-Authored-By: Claude Opus 5.5 --- README.md | 47 ++ docs/architecture/desktop-downloads.md | 41 ++ src/lib/release-handoff.ts | 76 ++- src/lib/release-source.ts | 19 + workers/release-mirror/src/index.ts | 37 +- workers/release-mirror/src/mirror.ts | 89 +++- workers/release-mirror/src/releaseFeed.ts | 77 +++ workers/release-mirror/src/updateFeed.test.ts | 466 ++++++++++++++++++ workers/release-mirror/src/updateFeed.ts | 293 +++++++++++ .../release-mirror/src/updateManifest.test.ts | 76 +++ workers/release-mirror/src/updateManifest.ts | 177 +++++++ 11 files changed, 1367 insertions(+), 31 deletions(-) create mode 100644 workers/release-mirror/src/releaseFeed.ts create mode 100644 workers/release-mirror/src/updateFeed.test.ts create mode 100644 workers/release-mirror/src/updateFeed.ts create mode 100644 workers/release-mirror/src/updateManifest.test.ts create mode 100644 workers/release-mirror/src/updateManifest.ts diff --git a/README.md b/README.md index 11d8ed5..3abca0b 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,53 @@ Do not deploy production from a feature branch or store Cloudflare credentials i See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution workflow. The cross-repository operating model is maintained in [`ScientFactory/Scient`](https://github.com/ScientFactory/Scient). +## Desktop update feeds: deploy and verify + +The `scientfactory-release-mirror` worker is the only publisher of +`downloads.scientfactory.com/scient-desktop/`. It copies the Stable installers and the complete +Stable and Beta electron-updater sets; see +[docs/architecture/desktop-downloads.md](docs/architecture/desktop-downloads.md). Pages +deployments do not deploy it. + +1. After merge, from `main`: `bun run mirror:types:check && bun run mirror:deploy`. +2. Watch the next runs (every ten minutes) with + `bunx wrangler tail scientfactory-release-mirror --config workers/release-mirror/wrangler.jsonc`. + Expect `Stable update feed synchronized` and `Beta update feed synchronized` with `promoted` + naming three manifests each. The first run per release copies up to about 1.8 GB per channel and + hashes it; on the Workers Free plan (50 external subrequests, 10 ms CPU) it cannot finish, + so confirm the account is on Workers Paid. A run cut short resumes from what is already in the + bucket. +3. Cache rules on the `downloads.scientfactory.com` zone (C1): + - `starts_with(http.request.uri.path, "/scient-desktop/updates/")`: **Bypass cache**. + - `/scient-desktop/v*` paths: **Eligible for cache**, edge TTL **use cache-control header**, + and a status-code TTL of **no-store** for 404 (and 4xx/5xx). Without such a rule, + Cloudflare caches 404s of default-cacheable types (`.exe`, `.zip`, `.dmg`) for a few + minutes, and does not cache `.AppImage`, `.blockmap` or `.yml` at all. Files above + 512 MB are not cached on non-Enterprise plans. +4. Verify each channel (`stable`/`latest`, then `beta`/`beta`): + + ```sh + feed=https://downloads.scientfactory.com/scient-desktop/updates/stable + for manifest in latest.yml latest-mac.yml latest-linux.yml; do + curl -fsS -D- "$feed/$manifest?noCache=$RANDOM" -o "/tmp/$manifest" | grep -i '^cache-control' # no-store + grep -E '^(version| - url| size| sha512|releaseName)' "/tmp/$manifest" + done + # For each url: it must resolve under scient-desktop//, answer 200 with the listed size, + # and hash to the listed sha512. + url=https://downloads.scientfactory.com/scient-desktop/v0.6.30/Scient-0.6.30-x64.exe + curl -fsSI "$url" | grep -iE '^(content-length|cache-control|cf-cache-status)' + curl -fsS "$url" | openssl dgst -sha512 -binary | base64 + curl -fsSI "$url.blockmap" | head -1 # 200 + curl -sS -r 0-99 -o /dev/null -w '%{http_code} %{size_download}\n' "$url" # 206 100 + curl -sS -o /dev/null -w '%{http_code}\n' https://downloads.scientfactory.com/scient-desktop/v0.0.0/missing.exe # 404, twice: never HIT + ``` + +5. Only then merge the desktop change that makes these feeds the updater's default, and qualify + real updates on each platform from a build that uses them. + +To stop publishing, remove the cron trigger and redeploy; the bucket's objects stay valid. Never +delete or rewrite a versioned object: installed clients and caches may still use it. + ## Scient Docs publishing The public `/docs` experience is generated from reviewed Markdown in diff --git a/docs/architecture/desktop-downloads.md b/docs/architecture/desktop-downloads.md index 8722fa3..9c23b80 100644 --- a/docs/architecture/desktop-downloads.md +++ b/docs/architecture/desktop-downloads.md @@ -47,6 +47,47 @@ Each run: A failed run leaves the previous pointer untouched, so the pointer only names installers that are in the bucket. Older versions stay in the bucket. +## The update feeds + +The same worker publishes the desktop app's generic electron-updater feeds: + +| Channel | Feed URL (`generic` provider) | Source | +| ------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| Stable | `https://downloads.scientfactory.com/scient-desktop/updates/stable` | `ScientFactory/scient-desktop`, GitHub's latest release | +| Beta | `https://downloads.scientfactory.com/scient-desktop/updates/beta` | `ScientFactory/scient-desktop-beta`, the newest `-beta.` tag in its releases Atom feed | + +Each run, per channel (`workers/release-mirror/src/updateFeed.ts`): + +1. Lists the channel's feed objects, recording each version, before asking GitHub anything. +2. Reads and validates the channel's handoff (`parseChannelHandoff`): the channel's version + shape, the official source repository, and that channel's distribution. If every channel + manifest already offers this release (`tag` and the manifest's SHA-256 in its metadata), + the run ends here. +3. Downloads the release's channel manifests (`latest.yml`, `latest-mac.yml`, + `latest-linux.yml`, or the `beta*` set) and accepts them only if size and SHA-256 match the + handoff. Every file a manifest names must be a bare file name the handoff lists, with the + same size. +4. Copies every payload (macOS ZIPs and DMGs, the NSIS EXE, the AppImage), every blockmap, and + the manifests themselves into `scient-desktop//`, with the same immutable writes as the + installers. Payloads must also hash to the manifest's SHA-512 while streaming; the last chunk + is held back until it does, so a mismatched copy never completes. +5. Writes `scient-desktop/updates//` last, each as a compare-and-swap on the + version seen in step 1. The feed copy differs from the release's manifest only in two ways: + every file URL becomes `../..//`, which electron-updater resolves against the feed + URL to the version directory, and `releaseName`/`releaseNotes` are added from the release's + Atom entry, which is what GitHub's provider showed before. Sizes and SHA-512s are untouched. + +Nothing is redirected: an old version's objects stay at their own paths, and the updater finds +the previous version's blockmap by substituting its version into the new file's path, which +the version directories satisfy for every release mirrored since the feed began. Two runs that +race can leave platforms on different releases for one interval; each platform's manifest is +self-consistent, and the next run moves the rest. + +Caching (C1): versioned objects carry `public, max-age=31536000, immutable`; feed manifests and +`stable.json` carry `no-store`, and clients add a random `noCache` query to every check. A +response for a missing object is not one the worker writes, so the zone's cache rules decide +how long a 404 is kept (see the runbook in the README). + Deploy the worker with `bun run mirror:deploy`. The Pages project reads the bucket through its `DESKTOP_DOWNLOADS` binding in `wrangler.jsonc`. diff --git a/src/lib/release-handoff.ts b/src/lib/release-handoff.ts index 2a0a0c1..fdecfc6 100644 --- a/src/lib/release-handoff.ts +++ b/src/lib/release-handoff.ts @@ -4,6 +4,7 @@ import type { Release, ReleaseAsset } from "./release-schema"; import { + DESKTOP_BETA_RELEASE_REPOSITORY, DESKTOP_RELEASE_REPOSITORY, desktopMirrorUrl, isOfficialDesktopProvenanceRepository, @@ -59,6 +60,7 @@ export function releaseAssetContentType(name: string): string { if (name.endsWith(".txt")) return "text/plain"; if (name.endsWith(".json")) return "application/json"; if (name.endsWith(".tgz")) return "application/gzip"; + if (name.endsWith(".zip")) return "application/zip"; return "application/octet-stream"; } @@ -72,8 +74,68 @@ export function requiredInstallerNames(version: string): readonly string[] { ]; } -export function githubReleaseDownloadUrl(tag: string, name: string): string { - return `https://github.com/${DESKTOP_RELEASE_REPOSITORY}/releases/download/${tag}/${encodeURIComponent(name)}`; +export function githubReleaseDownloadUrl( + tag: string, + name: string, + repository: string = DESKTOP_RELEASE_REPOSITORY, +): string { + return `https://github.com/${repository}/releases/download/${tag}/${encodeURIComponent(name)}`; +} + +function parseHandoffAssets(value: unknown): readonly ReleaseHandoffAsset[] { + if (!Array.isArray(value) || value.length === 0 || value.length > 128) { + throw new Error("Release handoff is invalid."); + } + const assets = value.map(parseAsset); + if (new Set(assets.map((asset) => asset.name)).size !== assets.length) { + throw new Error("Release handoff contains duplicate assets."); + } + return assets; +} + +const BETA_VERSION = /^\d+\.\d+\.\d+-beta\.\d{8}\.[1-9]\d*$/u; + +/** + * A handoff for one update channel: Stable's x.y.z or Beta's x.y.z-beta.YYYYMMDD.N, from the + * official source repository, and, when it names its distribution, distributed on that channel + * from that channel's repository. Unlike {@link parseReleaseHandoff} it requires no website + * installers; the update feed checks the electron-updater manifests instead. + */ +export function parseChannelHandoff(value: unknown, channel: "stable" | "beta"): ReleaseHandoff { + const versionShape = channel === "stable" ? /^\d+\.\d+\.\d+$/u : BETA_VERSION; + const repository = + channel === "stable" ? DESKTOP_RELEASE_REPOSITORY : DESKTOP_BETA_RELEASE_REPOSITORY; + if ( + !isRecord(value) || + value.schemaVersion !== 1 || + value.product !== "Scient" || + typeof value.version !== "string" || + !versionShape.test(value.version) || + value.tag !== `v${value.version}` || + !isRecord(value.source) || + typeof value.source.repository !== "string" || + !isOfficialDesktopProvenanceRepository(value.source.repository) || + typeof value.source.commit !== "string" || + !/^[a-f0-9]{40}$/u.test(value.source.commit) + ) { + throw new Error(`The ${channel} release handoff is invalid.`); + } + // Older Stable handoffs predate the distribution record; every Beta handoff carries it. + if (value.distribution !== undefined || channel === "beta") { + const distribution = value.distribution; + if ( + !isRecord(distribution) || + distribution.repository !== repository || + distribution.channel !== channel + ) { + throw new Error(`The ${channel} release handoff names another distribution.`); + } + } + return { + version: value.version, + tag: `v${value.version}`, + assets: parseHandoffAssets(value.assets), + }; } export function parseReleaseHandoff(value: unknown): ReleaseHandoff { @@ -90,18 +152,12 @@ export function parseReleaseHandoff(value: unknown): ReleaseHandoff { typeof value.source.commit !== "string" || !/^[a-f0-9]{40}$/u.test(value.source.commit) || typeof value.source.tree !== "string" || - !/^[a-f0-9]{40}$/u.test(value.source.tree) || - !Array.isArray(value.assets) || - value.assets.length === 0 || - value.assets.length > 128 + !/^[a-f0-9]{40}$/u.test(value.source.tree) ) { throw new Error("Release handoff is invalid."); } - const assets = value.assets.map(parseAsset); - if (new Set(assets.map((asset) => asset.name)).size !== assets.length) { - throw new Error("Release handoff contains duplicate assets."); - } + const assets = parseHandoffAssets(value.assets); const names = new Set(assets.map((asset) => asset.name)); if (!requiredInstallerNames(value.version).every((name) => names.has(name))) { diff --git a/src/lib/release-source.ts b/src/lib/release-source.ts index 333eaac..491c6d1 100644 --- a/src/lib/release-source.ts +++ b/src/lib/release-source.ts @@ -3,6 +3,8 @@ // Layer: Shared marketing and Cloudflare utility export const DESKTOP_RELEASE_REPOSITORY = "ScientFactory/scient-desktop"; +/** Beta builds are published only here, never in the Stable release index. */ +export const DESKTOP_BETA_RELEASE_REPOSITORY = "ScientFactory/scient-desktop-beta"; const TRANSITIONAL_DESKTOP_RELEASE_REPOSITORY = "ScientFactory/scient-desktop-next"; const TRUSTED_PROVENANCE_REPOSITORIES = new Set([ @@ -27,6 +29,23 @@ export function desktopMirrorKey(tag: string, name: string): string { return `${DESKTOP_MIRROR_PREFIX}${tag}/${name}`; } +/** + * The desktop app's generic update feeds. Each channel's electron-updater metadata + * (`latest*.yml` for Stable, `beta*.yml` for Beta) is promoted here only after every file it + * names is in that release's immutable `scient-desktop//` directory. + */ +export type DesktopUpdateFeedChannel = "stable" | "beta"; +export const DESKTOP_UPDATE_FEED_PREFIX = `${DESKTOP_MIRROR_PREFIX}updates/`; + +export function desktopUpdateFeedKey(channel: DesktopUpdateFeedChannel, file: string): string { + return `${DESKTOP_UPDATE_FEED_PREFIX}${channel}/${file}`; +} + +/** The URL a desktop client configures as its `generic` provider feed. */ +export function desktopUpdateFeedUrl(channel: DesktopUpdateFeedChannel): string { + return `${DESKTOP_MIRROR_ORIGIN}/${DESKTOP_UPDATE_FEED_PREFIX}${channel}`; +} + export function desktopMirrorUrl(tag: string, name: string): string { return `${DESKTOP_MIRROR_ORIGIN}/${DESKTOP_MIRROR_PREFIX}${tag}/${encodeURIComponent(name)}`; } diff --git a/workers/release-mirror/src/index.ts b/workers/release-mirror/src/index.ts index bfecaac..35c6d1d 100644 --- a/workers/release-mirror/src/index.ts +++ b/workers/release-mirror/src/index.ts @@ -1,15 +1,40 @@ // FILE: index.ts -// Purpose: Keeps downloads.scientfactory.com's copy of the Stable desktop installers current. +// Purpose: Keeps downloads.scientfactory.com's copy of the Stable desktop installers and the +// desktop app's Stable and Beta update feeds current. // Layer: Cloudflare Worker (cron only; no public routes) import { syncDesktopMirror } from "./mirror"; +import { syncUpdateFeed } from "./updateFeed"; export default { async scheduled(controller, env) { - const result = await syncDesktopMirror( - env.DESKTOP_DOWNLOADS, - new Date(controller.scheduledTime), - ); - console.log(JSON.stringify({ message: "Desktop mirror synchronized", ...result })); + const bucket = env.DESKTOP_DOWNLOADS; + // Each step is independent: one failing (a release still uploading, GitHub refusing) must not + // hold back the others. Steps run in turn so a release's shared installers are copied once. + const steps = [ + [ + "Desktop mirror synchronized", + () => syncDesktopMirror(bucket, new Date(controller.scheduledTime)), + ], + ["Stable update feed synchronized", () => syncUpdateFeed(bucket, "stable")], + ["Beta update feed synchronized", () => syncUpdateFeed(bucket, "beta")], + ] as const; + const failures: unknown[] = []; + for (const [message, run] of steps) { + try { + console.log(JSON.stringify({ message, ...(await run()) })); + } catch (error) { + failures.push(error); + console.error( + JSON.stringify({ + message: message.replace("synchronized", "failed"), + error: error instanceof Error ? error.message : String(error), + }), + ); + } + } + if (failures.length > 0) { + throw new AggregateError(failures, `${failures.length} release mirror step(s) failed.`); + } }, } satisfies ExportedHandler; diff --git a/workers/release-mirror/src/mirror.ts b/workers/release-mirror/src/mirror.ts index 5af0ce1..f5c3143 100644 --- a/workers/release-mirror/src/mirror.ts +++ b/workers/release-mirror/src/mirror.ts @@ -1,5 +1,6 @@ // FILE: mirror.ts -// Purpose: Copies the current Stable desktop installers from GitHub into ScientFactory's R2 bucket. +// Purpose: Copies the current Stable desktop installers from GitHub into ScientFactory's R2 bucket, +// and provides the verified, immutable asset copy the update feeds share. // Layer: Cloudflare Worker (scheduled) import { readHandoff } from "../../../functions/_lib/latest-release"; @@ -13,13 +14,16 @@ import { } from "../../../src/lib/release-handoff"; import { DESKTOP_MIRROR_STABLE_KEY, + DESKTOP_RELEASE_REPOSITORY, desktopMirrorKey, GITHUB_RELEASE_HANDOFF_URL, } from "../../../src/lib/release-source"; -const USER_AGENT = "ScientFactory-release-mirror"; +export const USER_AGENT = "ScientFactory-release-mirror"; const HANDOFF_TIMEOUT_MS = 30_000; const INSTALLER_TIMEOUT_MS = 10 * 60_000; +/** Versioned objects never change, so browsers and the edge may keep them for a year. */ +export const IMMUTABLE_CACHE_CONTROL = "public, max-age=31536000, immutable"; export interface MirrorSyncResult { readonly tag: string; @@ -48,15 +52,61 @@ async function fetchLatestHandoff(): Promise<{ raw: unknown; publishedAt: string }; } +/** What the mirror needs to copy one published release asset. */ +export interface ReleaseAssetSource { + /** The GitHub repository whose release holds the asset. */ + readonly repository: string; + readonly tag: string; + readonly asset: ReleaseHandoffAsset; + /** + * The base64 SHA-512 the release's electron-updater manifest records for these bytes. When + * set, the copy is refused unless the streamed bytes hash to it as well as to the handoff's + * SHA-256. + */ + readonly sha512?: string; +} + +function base64(bytes: ArrayBuffer): string { + let binary = ""; + for (const byte of new Uint8Array(bytes)) binary += String.fromCharCode(byte); + return btoa(binary); +} + +/** + * Passes bytes through while hashing them, holding back the last chunk until the SHA-512 is + * known. A mismatch errors the stream before the final bytes reach R2, so a fixed-length + * upload can never complete with them. + */ +function verifySha512(expected: string, name: string): TransformStream { + const digest = new crypto.DigestStream("SHA-512"); + const hashing = digest.getWriter(); + let held: Uint8Array | null = null; + return new TransformStream({ + async transform(chunk, controller) { + await hashing.write(chunk); + if (held !== null) controller.enqueue(held); + held = chunk; + }, + async flush(controller) { + await hashing.close(); + const actual = base64(await digest.digest); + if (actual !== expected) { + throw new Error(`${name} does not match its update manifest's SHA-512.`); + } + if (held !== null) controller.enqueue(held); + }, + }); +} + /** - * Copies one installer unless the bucket already holds it with the handoff's digest. - * Installer keys are immutable: bytes already published under a tag are never replaced, since - * the current pointer and year-long caches may still serve them. A changed digest needs a new tag. + * Copies one release asset to `scient-desktop//` unless the bucket already holds it + * with the handoff's digest. Keys are immutable: bytes already published under a tag are never + * replaced, since the current pointer, update feeds and year-long caches may still serve them. + * A changed digest needs a new tag. Returns true when this call wrote the object. */ -async function mirrorInstaller( +export async function mirrorReleaseAsset( bucket: R2Bucket, - tag: string, - asset: ReleaseHandoffAsset, + { repository, tag, asset, sha512 }: ReleaseAssetSource, ): Promise { const key = desktopMirrorKey(tag, asset.name); const verifyExisting = (existing: R2Object | null): false => { @@ -65,13 +115,13 @@ async function mirrorInstaller( } throw new Error( `${key} already holds other bytes (sha256 ${existing?.customMetadata?.sha256 ?? "unrecorded"}); ` + - `the handoff now says ${asset.sha256}. Published installers are never replaced.`, + `the handoff now says ${asset.sha256}. Published assets are never replaced.`, ); }; const existing = await bucket.head(key); if (existing !== null) return verifyExisting(existing); - const response = await fetch(githubReleaseDownloadUrl(tag, asset.name), { + const response = await fetch(githubReleaseDownloadUrl(tag, asset.name, repository), { headers: { "User-Agent": USER_AGENT }, signal: AbortSignal.timeout(INSTALLER_TIMEOUT_MS), }); @@ -89,7 +139,11 @@ async function mirrorInstaller( // hash to `sha256`. `If-None-Match: *` keeps a concurrent run from replacing the object. const sized = new FixedLengthStream(asset.size); const stopPiping = new AbortController(); - const piping = response.body.pipeTo(sized.writable, { signal: stopPiping.signal }); + const source = + sha512 === undefined + ? response.body + : response.body.pipeThrough(verifySha512(sha512, asset.name)); + const piping = source.pipeTo(sized.writable, { signal: stopPiping.signal }); let written: R2Object | null; try { written = await bucket.put(key, sized.readable, { @@ -98,14 +152,18 @@ async function mirrorInstaller( httpMetadata: { contentType: releaseAssetContentType(asset.name), contentDisposition: `attachment; filename="${asset.name}"`, - cacheControl: "public, max-age=31536000, immutable", + cacheControl: IMMUTABLE_CACHE_CONTROL, }, customMetadata: { sha256: asset.sha256 }, }); } catch (error) { stopPiping.abort(); - await piping.catch(() => undefined); - throw error; + // The stream's own failure (a SHA-512 mismatch, a short body) explains R2's refusal best. + const piped = await piping.then( + () => null, + (cause: unknown) => cause, + ); + throw piped instanceof Error && !(piped.name === "AbortError") ? piped : error; } if (written === null) { stopPiping.abort(); @@ -133,7 +191,8 @@ export async function syncDesktopMirror(bucket: R2Bucket, now: Date): Promise candidate.name === name); if (!asset) throw new Error(`Handoff ${handoff.tag} has no ${name}.`); - if (await mirrorInstaller(bucket, handoff.tag, asset)) copied.push(name); + const source = { repository: DESKTOP_RELEASE_REPOSITORY, tag: handoff.tag, asset }; + if (await mirrorReleaseAsset(bucket, source)) copied.push(name); } const sameRelease = diff --git a/workers/release-mirror/src/releaseFeed.ts b/workers/release-mirror/src/releaseFeed.ts new file mode 100644 index 0000000..b80d067 --- /dev/null +++ b/workers/release-mirror/src/releaseFeed.ts @@ -0,0 +1,77 @@ +// FILE: releaseFeed.ts +// Purpose: Reads a desktop repository's GitHub releases Atom feed: the same feed electron-updater's +// GitHub provider reads for prerelease discovery, release names and release notes. +// Layer: Cloudflare Worker (scheduled) + +const FEED_TIMEOUT_MS = 30_000; +const MAX_FEED_BYTES = 4 * 1024 * 1024; +/** Notes larger than this are left out of the feed copy rather than truncated mid-markup. */ +export const MAX_RELEASE_NOTES_BYTES = 256 * 1024; + +export interface ReleaseFeedEntry { + readonly tag: string; + /** The entry title, which electron-updater's GitHub provider reports as `releaseName`. */ + readonly title: string; + /** The rendered release body (HTML), which that provider reports as `releaseNotes`. */ + readonly content: string; +} + +const XML_ENTITIES: Readonly> = { + amp: "&", + apos: "'", + gt: ">", + lt: "<", + quot: '"', +}; + +function decodeXml(value: string): string { + return value.replace(/&(#x[0-9a-fA-F]+|#\d+|[a-z]+);/gu, (entity, name: string) => { + if (name.startsWith("#x")) return String.fromCodePoint(Number.parseInt(name.slice(2), 16)); + if (name.startsWith("#")) return String.fromCodePoint(Number.parseInt(name.slice(1), 10)); + return XML_ENTITIES[name] ?? entity; + }); +} + +function escapeRegExp(value: string): string { + return value.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"); +} + +/** Entries whose release link points into `repository`; anything else in the feed is ignored. */ +export function parseReleaseFeed(xml: string, repository: string): ReleaseFeedEntry[] { + const link = new RegExp( + `]*\\bhref="https://github\\.com/${escapeRegExp(repository)}/releases/tag/([^"/?#]+)"`, + "u", + ); + const entries: ReleaseFeedEntry[] = []; + for (const [, entry = ""] of xml.matchAll(/]*>([\s\S]*?)<\/entry>/gu)) { + const tag = link.exec(entry)?.[1]; + if (tag === undefined) continue; + const title = /]*>([\s\S]*?)<\/title>/u.exec(entry)?.[1] ?? ""; + const content = /]*>([\s\S]*?)<\/content>/u.exec(entry)?.[1] ?? ""; + entries.push({ + tag: decodeURIComponent(tag), + title: decodeXml(title).trim(), + content: decodeXml(content), + }); + } + return entries; +} + +export async function fetchReleaseFeed( + repository: string, + userAgent: string, +): Promise { + const response = await fetch(`https://github.com/${repository}/releases.atom`, { + headers: { Accept: "application/atom+xml, application/xml", "User-Agent": userAgent }, + signal: AbortSignal.timeout(FEED_TIMEOUT_MS), + }); + if (!response.ok) { + await response.body?.cancel(); + throw new Error(`GitHub returned ${response.status} for the ${repository} release feed.`); + } + const body = await response.arrayBuffer(); + if (body.byteLength > MAX_FEED_BYTES) { + throw new Error(`The ${repository} release feed is too large.`); + } + return parseReleaseFeed(new TextDecoder().decode(body), repository); +} diff --git a/workers/release-mirror/src/updateFeed.test.ts b/workers/release-mirror/src/updateFeed.test.ts new file mode 100644 index 0000000..ce19a47 --- /dev/null +++ b/workers/release-mirror/src/updateFeed.test.ts @@ -0,0 +1,466 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { desktopMirrorUrl, desktopUpdateFeedUrl } from "../../../src/lib/release-source"; +import { syncUpdateFeed } from "./updateFeed"; + +// The platform's WebCrypto, which both Node and Workers provide, captured before tests stub it. +const subtle = crypto.subtle; +const digest = async (algorithm: string, bytes: BufferSource) => + new Uint8Array(await subtle.digest(algorithm, bytes)); +const sha256 = async (body: string) => + [...(await digest("SHA-256", new TextEncoder().encode(body)))] + .map((byte) => byte.toString(16).padStart(2, "0")) + .join(""); +const sha512 = async (body: string) => + btoa(String.fromCharCode(...(await digest("SHA-512", new TextEncoder().encode(body))))); + +interface Release { + readonly repository: string; + readonly version: string; + readonly channel: "stable" | "beta"; + /** Every published file's bytes, by name, including the manifests. */ + readonly files: Map; + readonly handoff: Record; +} + +/** A release shaped like the desktop workflow's: three manifests, their payloads and blockmaps. */ +async function release( + version: string, + channel: "stable" | "beta" = "stable", + tamper: { readonly sha512Of?: string } = {}, +): Promise { + const prefix = channel === "stable" ? "latest" : "beta"; + const repository = + channel === "stable" ? "ScientFactory/scient-desktop" : "ScientFactory/scient-desktop-beta"; + const payload = (name: string) => `${name} bytes`; + const names = { + mac: [ + `Scient-${version}-arm64.zip`, + `Scient-${version}-arm64.dmg`, + `Scient-${version}-x64.zip`, + `Scient-${version}-x64.dmg`, + ], + win: [`Scient-${version}-x64.exe`], + linux: [`Scient-${version}-x86_64.AppImage`], + }; + const files = new Map(); + const manifest = async (list: readonly string[]) => { + const entries: string[] = []; + for (const name of list) { + files.set(name, payload(name)); + const recorded = await sha512(tamper.sha512Of === name ? "other bytes" : payload(name)); + entries.push( + ` - url: ${name}\n sha512: ${recorded}\n size: ${payload(name).length}\n`, + ); + } + return `version: ${version}\nfiles:\n${entries.join("")}path: ${list[0]}\nsha512: ${await sha512(payload(list[0]!))}\nreleaseDate: '2026-11-16T10:00:00.000Z'\n`; + }; + files.set(`${prefix}-mac.yml`, await manifest(names.mac)); + files.set(`${prefix}.yml`, await manifest(names.win)); + files.set(`${prefix}-linux.yml`, await manifest(names.linux)); + for (const name of [...names.mac.filter((n) => n.endsWith(".zip")), ...names.win]) { + files.set(`${name}.blockmap`, `${name} blockmap`); + } + files.set(`scient-server-${version}.tgz`, "server"); + const handoff = { + schemaVersion: 1, + product: "Scient", + version, + tag: `v${version}`, + distribution: { repository, channel }, + source: { + repository: "ScientFactory/scient-desktop", + commit: "a".repeat(40), + tree: "b".repeat(40), + }, + assets: await Promise.all( + [...files].map(async ([name, body]) => ({ + name, + size: body.length, + sha256: await sha256(body), + })), + ), + }; + return { repository, version, channel, files, handoff }; +} + +function atom(repository: string, releases: readonly Release[]): string { + const entries = releases.map( + (r) => ` + tag:github.com,2008:Repository/1/v${r.version} + + Scient v${r.version} + <p>Notes for ${r.version} &amp; more</p> +`, + ); + return `${entries.join("")}`; +} + +interface StoredObject { + readonly body: string; + readonly etag: string; + readonly httpMetadata?: R2HTTPMetadata; + readonly customMetadata?: Record; +} + +/** An in-memory bucket with the R2 write rules the feed relies on. */ +function memoryBucket() { + let version = 0; + const objects = new Map(); + const writes: string[] = []; + const describe = (key: string, object: StoredObject | undefined) => + object + ? { + key, + size: object.body.length, + etag: object.etag, + customMetadata: object.customMetadata ?? {}, + text: async () => object.body, + } + : null; + const bucket = { + head: vi.fn(async (key: string) => describe(key, objects.get(key))), + list: vi.fn(async ({ prefix }: R2ListOptions) => ({ + objects: [...objects.keys()] + .filter((key) => key.startsWith(prefix ?? "")) + .map((key) => describe(key, objects.get(key))), + truncated: false, + })), + put: vi.fn(async (key: string, value: ReadableStream | string, options?: R2PutOptions) => { + const onlyIf = options?.onlyIf; + const existing = objects.get(key); + if (onlyIf instanceof Headers && onlyIf.get("If-None-Match") === "*" && existing) { + return null; + } + if (onlyIf && !(onlyIf instanceof Headers) && onlyIf.etagMatches !== existing?.etag) { + return null; + } + const body = typeof value === "string" ? value : await new Response(value).text(); + if (options?.sha256 !== undefined && options.sha256 !== (await sha256(body))) { + throw new Error("put: The SHA-256 checksum you specified did not match what we received."); + } + const stored = { + body, + etag: `e${(version += 1)}`, + httpMetadata: options?.httpMetadata as R2HTTPMetadata | undefined, + customMetadata: options?.customMetadata as Record | undefined, + }; + objects.set(key, stored); + writes.push(key); + return describe(key, stored); + }), + }; + return { bucket: bucket as unknown as R2Bucket, objects, writes }; +} + +/** GitHub as the feed sees it: each repository's releases, newest Stable as `latest`. */ +function github( + releases: () => readonly Release[], + overrides: Record Response | Promise> = {}, +) { + return vi.fn(async (input) => { + const url = String(input); + if (overrides[url]) return overrides[url]!(); + for (const repository of [ + "ScientFactory/scient-desktop", + "ScientFactory/scient-desktop-beta", + ]) { + const own = releases().filter((r) => r.repository === repository); + if (url === `https://github.com/${repository}/releases.atom`) { + return new Response(atom(repository, own)); + } + const latest = own.at(-1); + if ( + latest && + url === + `https://github.com/${repository}/releases/latest/download/scient-release-handoff.json` + ) { + return Response.json(latest.handoff); + } + const prefix = `https://github.com/${repository}/releases/download/`; + if (!url.startsWith(prefix)) continue; + const [tag = "", encoded = ""] = url.slice(prefix.length).split("/"); + const match = own.find((r) => `v${r.version}` === tag); + const name = decodeURIComponent(encoded); + if (match && name === "scient-release-handoff.json") return Response.json(match.handoff); + const body = match?.files.get(name); + if (body !== undefined) { + return new Response(body, { headers: { "Content-Length": String(body.length) } }); + } + } + return new Response(null, { status: 404 }); + }); +} + +const FEED = "scient-desktop/updates/stable/"; + +function feedFile(objects: Map, key: string): string { + return objects.get(key)!.body; +} + +beforeEach(() => { + vi.stubGlobal( + "FixedLengthStream", + class extends TransformStream { + constructor(_length: number) { + super(); + } + }, + ); + // Workers' streaming digest, over the platform's WebCrypto. + vi.stubGlobal("crypto", { + subtle, + DigestStream: class extends WritableStream { + readonly digest: Promise; + constructor(algorithm: string) { + const chunks: Uint8Array[] = []; + let resolve!: (value: ArrayBuffer) => void; + const result = new Promise((done) => { + resolve = done; + }); + super({ + write: (chunk) => { + chunks.push(chunk); + }, + close: async () => { + const all = new Uint8Array(chunks.reduce((size, chunk) => size + chunk.length, 0)); + let offset = 0; + for (const chunk of chunks) { + all.set(chunk, offset); + offset += chunk.length; + } + resolve(await subtle.digest(algorithm, all)); + }, + }); + this.digest = result; + } + }, + }); +}); + +afterEach(() => { + vi.unstubAllGlobals(); + vi.restoreAllMocks(); +}); + +describe("desktop update feed", () => { + it("copies the whole updater set into the version directory, then promotes the manifests", async () => { + const stable = await release("0.6.30"); + const { bucket, objects, writes } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + const result = await syncUpdateFeed(bucket, "stable"); + + expect(result.tag).toBe("v0.6.30"); + expect(result.promoted).toEqual(["latest-linux.yml", "latest-mac.yml", "latest.yml"]); + expect(result.superseded).toEqual([]); + // Every manifest, payload and blockmap, never the server package. + expect(result.copied.toSorted()).toEqual( + [...stable.files.keys()].filter((name) => !name.endsWith(".tgz")).toSorted(), + ); + // Channel metadata is written only after every object it names. + const firstFeedWrite = writes.findIndex((key) => key.startsWith(FEED)); + expect(writes.slice(firstFeedWrite).every((key) => key.startsWith(FEED))).toBe(true); + + // The release's manifests are kept byte for byte; the feed copy resolves into them. + expect(feedFile(objects, "scient-desktop/v0.6.30/latest.yml")).toBe( + stable.files.get("latest.yml"), + ); + const windows = feedFile(objects, `${FEED}latest.yml`); + const urls = [...windows.matchAll(/^ {2}- url: (.+)$/gmu)].map(([, url]) => url!); + expect(urls.map((url) => new URL(url, `${desktopUpdateFeedUrl("stable")}/`).href)).toEqual([ + desktopMirrorUrl("v0.6.30", "Scient-0.6.30-x64.exe"), + ]); + expect(windows).toContain('releaseName: "Scient v0.6.30"'); + expect(windows).toContain('releaseNotes: "

Notes for 0.6.30 & more

"'); + + // C1: versioned objects are immutable for a year; channel metadata is never stored. + expect(objects.get("scient-desktop/v0.6.30/Scient-0.6.30-x64.exe")?.httpMetadata).toMatchObject( + { cacheControl: "public, max-age=31536000, immutable" }, + ); + expect(objects.get(`${FEED}latest-mac.yml`)?.httpMetadata).toMatchObject({ + cacheControl: "no-store", + contentType: "text/yaml; charset=utf-8", + }); + expect(objects.get(`${FEED}latest-mac.yml`)?.customMetadata).toEqual({ + tag: "v0.6.30", + version: "0.6.30", + sourceSha256: (stable.handoff.assets as { name: string; sha256: string }[]).find( + (asset) => asset.name === "latest-mac.yml", + )!.sha256, + }); + }); + + it("only reads the handoff while the feed already offers GitHub's release", async () => { + const stable = await release("0.6.30"); + const { bucket, writes } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + await syncUpdateFeed(bucket, "stable"); + const writesBefore = writes.length; + const fetchMock = github(() => [stable]); + vi.stubGlobal("fetch", fetchMock); + + await expect(syncUpdateFeed(bucket, "stable")).resolves.toEqual({ + channel: "stable", + tag: "v0.6.30", + copied: [], + promoted: [], + superseded: [], + }); + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(writes).toHaveLength(writesBefore); + }); + + it("moves to a new release and keeps every object of the previous one", async () => { + const older = await release("0.6.30"); + const newer = await release("0.6.31"); + let releases = [older]; + const { bucket, objects } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => releases), + ); + await syncUpdateFeed(bucket, "stable"); + releases = [older, newer]; + + await expect(syncUpdateFeed(bucket, "stable")).resolves.toMatchObject({ + tag: "v0.6.31", + promoted: ["latest-linux.yml", "latest-mac.yml", "latest.yml"], + }); + expect(feedFile(objects, `${FEED}latest.yml`)).toContain("../../v0.6.31/Scient-0.6.31-x64.exe"); + // The updater finds the old blockmap by substituting the old version into the new path. + expect(objects.has("scient-desktop/v0.6.30/Scient-0.6.30-x64.exe.blockmap")).toBe(true); + expect(objects.has("scient-desktop/v0.6.30/Scient-0.6.30-arm64.zip")).toBe(true); + }); + + it("refuses bytes the handoff accepts but the release's manifest does not", async () => { + const stable = await release("0.6.30", "stable", { sha512Of: "Scient-0.6.30-x64.exe" }); + const { bucket, objects } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + await expect(syncUpdateFeed(bucket, "stable")).rejects.toThrow(/SHA-512/); + expect(objects.has("scient-desktop/v0.6.30/Scient-0.6.30-x64.exe")).toBe(false); + expect([...objects.keys()].some((key) => key.startsWith(FEED))).toBe(false); + }); + + it("refuses a manifest that differs from the handoff", async () => { + const stable = await release("0.6.30"); + const { bucket, objects } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable], { + "https://github.com/ScientFactory/scient-desktop/releases/download/v0.6.30/latest.yml": + () => new Response(stable.files.get("latest.yml")!.replace("0.6.30", "0.6.29")), + }), + ); + + await expect(syncUpdateFeed(bucket, "stable")).rejects.toThrow( + /does not match the release handoff/, + ); + expect(objects.size).toBe(0); + }); + + it("keeps a newer promotion when an older, slower run finishes later", async () => { + const older = await release("0.6.30"); + const newer = await release("0.6.31"); + let releases = [older]; + let releaseSlowCopy!: () => void; + const slowCopy = new Promise((resolve) => { + releaseSlowCopy = resolve; + }); + const { bucket, objects } = memoryBucket(); + const exe = older.files.get("Scient-0.6.30-x64.exe")!; + vi.stubGlobal( + "fetch", + github(() => releases, { + "https://github.com/ScientFactory/scient-desktop/releases/download/v0.6.30/Scient-0.6.30-x64.exe": + async () => { + await slowCopy; + return new Response(exe, { headers: { "Content-Length": String(exe.length) } }); + }, + }), + ); + + const olderRun = syncUpdateFeed(bucket, "stable"); + await vi.waitFor(() => + expect(objects.has("scient-desktop/v0.6.30/Scient-0.6.30-x64.dmg")).toBe(true), + ); + releases = [older, newer]; + await expect(syncUpdateFeed(bucket, "stable")).resolves.toMatchObject({ tag: "v0.6.31" }); + releaseSlowCopy(); + + await expect(olderRun).resolves.toMatchObject({ + tag: "v0.6.30", + promoted: [], + superseded: ["latest-linux.yml", "latest-mac.yml", "latest.yml"], + }); + expect(feedFile(objects, `${FEED}latest.yml`)).toContain("v0.6.31"); + }); + + it("publishes the newest Beta, by version, to its own feed", async () => { + const first = await release("0.6.31-beta.20261116.1", "beta"); + const second = await release("0.6.31-beta.20261116.2", "beta"); + const stable = await release("0.6.30"); + const { bucket, objects } = memoryBucket(); + // The feed lists the newer build first, as GitHub usually does. + vi.stubGlobal( + "fetch", + github(() => [stable, second, first]), + ); + + await expect(syncUpdateFeed(bucket, "beta")).resolves.toMatchObject({ + tag: "v0.6.31-beta.20261116.2", + promoted: ["beta-linux.yml", "beta-mac.yml", "beta.yml"], + }); + expect(feedFile(objects, "scient-desktop/updates/beta/beta.yml")).toContain( + " - url: ../../v0.6.31-beta.20261116.2/Scient-0.6.31-beta.20261116.2-x64.exe", + ); + expect([...objects.keys()].some((key) => key.startsWith(FEED))).toBe(false); + }); + + it("leaves the Beta feed alone while no Beta is published", async () => { + const stable = await release("0.6.30"); + const { bucket, writes } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + await expect(syncUpdateFeed(bucket, "beta")).resolves.toEqual({ + channel: "beta", + tag: null, + copied: [], + promoted: [], + superseded: [], + }); + expect(writes).toEqual([]); + }); + + it("refuses a Stable handoff on the Beta feed", async () => { + const beta = await release("0.6.31-beta.20261116.1", "beta"); + const { bucket, writes } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [beta], { + "https://github.com/ScientFactory/scient-desktop-beta/releases/download/v0.6.31-beta.20261116.1/scient-release-handoff.json": + () => + Response.json({ + ...beta.handoff, + distribution: { repository: "ScientFactory/scient-desktop", channel: "stable" }, + }), + }), + ); + + await expect(syncUpdateFeed(bucket, "beta")).rejects.toThrow(/another distribution/); + expect(writes).toEqual([]); + }); +}); diff --git a/workers/release-mirror/src/updateFeed.ts b/workers/release-mirror/src/updateFeed.ts new file mode 100644 index 0000000..4648f80 --- /dev/null +++ b/workers/release-mirror/src/updateFeed.ts @@ -0,0 +1,293 @@ +// FILE: updateFeed.ts +// Purpose: Publishes the desktop app's Stable and Beta electron-updater feeds from verified copies +// of the GitHub releases. +// Layer: Cloudflare Worker (scheduled) + +import { readHandoff } from "../../../functions/_lib/latest-release"; +import { + githubReleaseDownloadUrl, + parseChannelHandoff, + type ReleaseHandoff, + type ReleaseHandoffAsset, +} from "../../../src/lib/release-handoff"; +import { + DESKTOP_BETA_RELEASE_REPOSITORY, + DESKTOP_RELEASE_REPOSITORY, + desktopUpdateFeedKey, + DESKTOP_UPDATE_FEED_PREFIX, + type DesktopUpdateFeedChannel, +} from "../../../src/lib/release-source"; +import { mirrorReleaseAsset, USER_AGENT } from "./mirror"; +import { fetchReleaseFeed, MAX_RELEASE_NOTES_BYTES, type ReleaseFeedEntry } from "./releaseFeed"; +import { feedUpdateManifest, parseUpdateManifest } from "./updateManifest"; + +const HANDOFF_TIMEOUT_MS = 30_000; +const MANIFEST_TIMEOUT_MS = 30_000; +const MAX_MANIFEST_BYTES = 1024 * 1024; +/** + * Channel metadata moves on every release, so nothing may keep a copy; clients also add a + * random `noCache` query to each check. Only the versioned objects it names are cacheable. + */ +export const FEED_CACHE_CONTROL = "no-store"; + +interface ChannelSource { + readonly repository: string; + /** electron-updater's channel file prefix: `latest` for Stable, `beta` for Beta. */ + readonly manifestPrefix: "latest" | "beta"; +} + +const CHANNELS: Readonly> = { + stable: { repository: DESKTOP_RELEASE_REPOSITORY, manifestPrefix: "latest" }, + beta: { repository: DESKTOP_BETA_RELEASE_REPOSITORY, manifestPrefix: "beta" }, +}; + +/** The manifests every release carries: macOS, Windows and Linux x64. */ +function requiredManifestNames(prefix: string): readonly string[] { + return [`${prefix}-mac.yml`, `${prefix}.yml`, `${prefix}-linux.yml`]; +} + +function isChannelManifest(name: string, prefix: string): boolean { + return new RegExp(`^${prefix}(?:-mac|-linux(?:-[a-z0-9]+)?)?\\.yml$`, "u").test(name); +} + +export interface UpdateFeedSyncResult { + readonly channel: DesktopUpdateFeedChannel; + /** Null only when the channel has no published release yet. */ + readonly tag: string | null; + /** Versioned objects copied this run; empty when the bucket already held every one. */ + readonly copied: readonly string[]; + /** Channel manifests this run moved to `tag`. */ + readonly promoted: readonly string[]; + /** Channel manifests another run wrote first, so this run left them alone. */ + readonly superseded: readonly string[]; +} + +const BETA_TAG = /^v(\d+)\.(\d+)\.(\d+)-beta\.(\d{8})\.([1-9]\d*)$/u; + +function betaOrder(tag: string): readonly number[] | null { + const match = BETA_TAG.exec(tag); + return match ? match.slice(1).map(Number) : null; +} + +function compareOrder(a: readonly number[], b: readonly number[]): number { + for (let index = 0; index < a.length; index += 1) { + if (a[index] !== b[index]) return a[index]! - b[index]!; + } + return 0; +} + +/** The newest Beta in the feed, by version rather than by the feed's order. */ +export function newestBetaEntry(entries: readonly ReleaseFeedEntry[]): ReleaseFeedEntry | null { + let newest: { entry: ReleaseFeedEntry; order: readonly number[] } | null = null; + for (const entry of entries) { + const order = betaOrder(entry.tag); + if (order !== null && (newest === null || compareOrder(order, newest.order) > 0)) { + newest = { entry, order }; + } + } + return newest?.entry ?? null; +} + +async function fetchHandoff(url: string): Promise { + const response = await fetch(url, { + headers: { Accept: "application/octet-stream, application/json", "User-Agent": USER_AGENT }, + signal: AbortSignal.timeout(HANDOFF_TIMEOUT_MS), + }); + if (!response.ok) { + await response.body?.cancel(); + throw new Error(`GitHub returned ${response.status} for the release handoff.`); + } + return readHandoff(response); +} + +function hex(bytes: ArrayBuffer): string { + return [...new Uint8Array(bytes)].map((byte) => byte.toString(16).padStart(2, "0")).join(""); +} + +/** A release manifest's text, only if it is exactly the bytes the handoff attests. */ +async function fetchManifest( + repository: string, + tag: string, + asset: ReleaseHandoffAsset, +): Promise { + if (asset.size > MAX_MANIFEST_BYTES) throw new Error(`${asset.name} is too large.`); + const response = await fetch(githubReleaseDownloadUrl(tag, asset.name, repository), { + headers: { "User-Agent": USER_AGENT }, + signal: AbortSignal.timeout(MANIFEST_TIMEOUT_MS), + }); + if (!response.ok) { + await response.body?.cancel(); + throw new Error(`GitHub returned ${response.status} for ${asset.name}.`); + } + const body = await response.arrayBuffer(); + const digest = hex(await crypto.subtle.digest("SHA-256", body)); + if (body.byteLength !== asset.size || digest !== asset.sha256) { + throw new Error(`${asset.name} does not match the release handoff.`); + } + return new TextDecoder("utf-8", { fatal: true, ignoreBOM: false }).decode(body); +} + +interface VerifiedManifest { + readonly asset: ReleaseHandoffAsset; + readonly text: string; +} + +/** + * Everything one release's feed needs, checked against both of the release's own records: + * each manifest against the handoff's SHA-256, and each file it names against the handoff's + * size (here) and the manifest's SHA-512 (while copying). + */ +function planFeedObjects(handoff: ReleaseHandoff, manifests: readonly VerifiedManifest[]) { + const byName = new Map(handoff.assets.map((asset) => [asset.name, asset])); + const payloads = new Map(); + for (const { asset, text } of manifests) { + const manifest = parseUpdateManifest(text, asset.name); + if (manifest.version !== handoff.version) { + throw new Error( + `${asset.name} is for ${manifest.version}; the handoff is ${handoff.version}.`, + ); + } + for (const file of manifest.files) { + const payload = byName.get(file.url); + if (payload === undefined) { + throw new Error(`${asset.name} names ${file.url}, which the handoff does not list.`); + } + if (payload.size !== file.size) { + throw new Error( + `${asset.name} says ${file.url} is ${file.size} bytes; the handoff says ${payload.size}.`, + ); + } + const known = payloads.get(file.url); + if (known !== undefined && known.sha512 !== file.sha512) { + throw new Error(`The release's manifests disagree about ${file.url}.`); + } + payloads.set(file.url, { asset: payload, sha512: file.sha512 }); + } + } + const blockmaps = [...payloads.keys()] + .map((name) => byName.get(`${name}.blockmap`)) + .filter((asset): asset is ReleaseHandoffAsset => asset !== undefined); + return { payloads: [...payloads.values()], blockmaps }; +} + +function releaseOf(entry: ReleaseFeedEntry, tag: string) { + return { + name: entry.title || tag, + notes: + new TextEncoder().encode(entry.content).byteLength > MAX_RELEASE_NOTES_BYTES + ? "" + : entry.content, + }; +} + +/** + * Brings one channel's generic update feed up to GitHub's newest release on that channel. + * Every versioned object the release's manifests need (manifests, payloads, blockmaps) is copied + * into `scient-desktop//` first; the channel manifests under + * `scient-desktop/updates//` move last, each as a compare-and-swap on the version this + * run saw at its start, so a feed never names bytes that are not there and a slow run cannot + * undo a newer promotion. Older versions stay in the bucket. + */ +export async function syncUpdateFeed( + bucket: R2Bucket, + channel: DesktopUpdateFeedChannel, +): Promise { + const { repository, manifestPrefix } = CHANNELS[channel]; + // Captured before anything this run learns from GitHub, like the Stable pointer. + const listed = await bucket.list({ + prefix: `${DESKTOP_UPDATE_FEED_PREFIX}${channel}/`, + include: ["customMetadata"], + }); + const observed = new Map(listed.objects.map((object) => [object.key, object])); + + let feed: ReleaseFeedEntry[] | null = null; + let handoff: ReleaseHandoff; + if (channel === "beta") { + // Beta releases are prereleases, which GitHub's `latest` never names. electron-updater's + // GitHub provider finds them the same way, in the releases feed. + feed = await fetchReleaseFeed(repository, USER_AGENT); + const newest = newestBetaEntry(feed); + if (newest === null) { + return { channel, tag: null, copied: [], promoted: [], superseded: [] }; + } + handoff = parseChannelHandoff( + await fetchHandoff( + githubReleaseDownloadUrl(newest.tag, "scient-release-handoff.json", repository), + ), + channel, + ); + if (handoff.tag !== newest.tag) { + throw new Error(`The ${newest.tag} release carries the ${handoff.tag} handoff.`); + } + } else { + handoff = parseChannelHandoff( + await fetchHandoff( + `https://github.com/${repository}/releases/latest/download/scient-release-handoff.json`, + ), + channel, + ); + } + const { tag } = handoff; + + const manifestAssets = handoff.assets.filter((asset) => + isChannelManifest(asset.name, manifestPrefix), + ); + for (const required of requiredManifestNames(manifestPrefix)) { + if (!manifestAssets.some((asset) => asset.name === required)) { + throw new Error(`Handoff ${tag} has no ${required}.`); + } + } + + const current = (asset: ReleaseHandoffAsset) => { + const metadata = observed.get(desktopUpdateFeedKey(channel, asset.name))?.customMetadata; + return metadata?.tag === tag && metadata.sourceSha256 === asset.sha256; + }; + if (manifestAssets.every(current)) { + return { channel, tag, copied: [], promoted: [], superseded: [] }; + } + + // The release's name and notes, as GitHub's provider showed them to clients. + feed ??= await fetchReleaseFeed(repository, USER_AGENT); + const entry = feed.find((candidate) => candidate.tag === tag); + if (entry === undefined) { + throw new Error(`The ${repository} release feed does not list ${tag} yet.`); + } + + const manifests: VerifiedManifest[] = []; + for (const asset of manifestAssets) { + manifests.push({ asset, text: await fetchManifest(repository, tag, asset) }); + } + const { payloads, blockmaps } = planFeedObjects(handoff, manifests); + + const copied: string[] = []; + const copy = async (asset: ReleaseHandoffAsset, sha512?: string) => { + const source = { repository, tag, asset, ...(sha512 === undefined ? {} : { sha512 }) }; + if (await mirrorReleaseAsset(bucket, source)) copied.push(asset.name); + }; + for (const payload of payloads) await copy(payload.asset, payload.sha512); + for (const blockmap of blockmaps) await copy(blockmap); + // The release's own manifests, byte for byte, as the record of what each version offered. + for (const { asset } of manifests) await copy(asset); + + const release = releaseOf(entry, tag); + const promoted: string[] = []; + const superseded: string[] = []; + for (const { asset, text } of manifests.toSorted((a, b) => + a.asset.name.localeCompare(b.asset.name), + )) { + if (current(asset)) continue; + const key = desktopUpdateFeedKey(channel, asset.name); + const seen = observed.get(key); + const written = await bucket.put(key, feedUpdateManifest(text, asset.name, tag, release), { + onlyIf: + seen === undefined ? new Headers({ "If-None-Match": "*" }) : { etagMatches: seen.etag }, + httpMetadata: { + contentType: "text/yaml; charset=utf-8", + cacheControl: FEED_CACHE_CONTROL, + }, + customMetadata: { tag, version: handoff.version, sourceSha256: asset.sha256 }, + }); + (written === null ? superseded : promoted).push(asset.name); + } + return { channel, tag, copied, promoted, superseded }; +} diff --git a/workers/release-mirror/src/updateManifest.test.ts b/workers/release-mirror/src/updateManifest.test.ts new file mode 100644 index 0000000..ebf852c --- /dev/null +++ b/workers/release-mirror/src/updateManifest.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from "vitest"; + +import { desktopMirrorUrl, desktopUpdateFeedUrl } from "../../../src/lib/release-source"; +import { feedUpdateManifest, parseUpdateManifest } from "./updateManifest"; + +const WINDOWS = `version: 0.6.30 +files: + - url: Scient-0.6.30-x64.exe + sha512: c2hhNTEyLWV4ZQ== + size: 123 +path: Scient-0.6.30-x64.exe +sha512: c2hhNTEyLWV4ZQ== +releaseDate: '2026-11-16T10:00:00.000Z' +`; + +/** How electron-updater 6.8.9's Provider resolves a manifest file URL against a generic feed. */ +function resolveAgainstFeed(url: string, feed: string): string { + return new URL(url, feed.endsWith("/") ? feed : `${feed}/`).href; +} + +describe("update manifest feed copy", () => { + it("points every file at the release's immutable directory without touching its digests", () => { + const copy = feedUpdateManifest(WINDOWS, "latest.yml", "v0.6.30", null); + + expect(copy).toBe( + WINDOWS.replace(" - url: Scient", " - url: ../../v0.6.30/Scient").replace( + "path: Scient", + "path: ../../v0.6.30/Scient", + ), + ); + const [file] = parseUpdateManifest(copy.replaceAll("../../v0.6.30/", ""), "copy").files; + expect(file).toEqual({ url: "Scient-0.6.30-x64.exe", sha512: "c2hhNTEyLWV4ZQ==", size: 123 }); + expect( + resolveAgainstFeed("../../v0.6.30/Scient-0.6.30-x64.exe", desktopUpdateFeedUrl("stable")), + ).toBe(desktopMirrorUrl("v0.6.30", "Scient-0.6.30-x64.exe")); + expect( + resolveAgainstFeed( + "../../v0.6.31-beta.20261116.2/Scient-0.6.31-beta.20261116.2-x64.exe", + desktopUpdateFeedUrl("beta"), + ), + ).toBe(desktopMirrorUrl("v0.6.31-beta.20261116.2", "Scient-0.6.31-beta.20261116.2-x64.exe")); + }); + + it("adds the GitHub release's name and notes as YAML double-quoted scalars", () => { + const notes = '

What\'s new

\n

"Quoted" \\ and a line\u2028separator

'; + const copy = feedUpdateManifest(WINDOWS, "latest.yml", "v0.6.30", { + name: "Scient v0.6.30", + notes, + }); + + expect(copy).toContain('releaseName: "Scient v0.6.30"\n'); + const line = copy.split("\n").find((candidate) => candidate.startsWith("releaseNotes: "))!; + expect(line).toContain("\\u2028"); + expect(JSON.parse(line.slice("releaseNotes: ".length))).toBe(notes); + }); + + it("keeps a manifest's own release notes and omits empty ones", () => { + const own = `${WINDOWS}releaseNotes: 'From the build'\n`; + expect( + feedUpdateManifest(own, "latest.yml", "v0.6.30", { name: "Scient v0.6.30", notes: "x" }), + ).toContain("releaseNotes: 'From the build'\n"); + expect( + feedUpdateManifest(WINDOWS, "latest.yml", "v0.6.30", { name: "Scient v0.6.30", notes: "" }), + ).not.toContain("releaseNotes"); + }); + + it.each([ + ["an absolute URL", WINDOWS.replace("url: Scient", "url: https://github.com/x/Scient")], + ["a nested URL", WINDOWS.replace("url: Scient", "url: ../Scient")], + ["a block scalar", `${WINDOWS}releaseNotes: |\n - url: evil.exe\n`], + ["a file without its digest", WINDOWS.replace(" sha512: c2hhNTEyLWV4ZQ==\n", "")], + ["a path outside its files", WINDOWS.replace("path: Scient-0.6.30", "path: Other-0.6.30")], + ])("refuses %s", (_label, text) => { + expect(() => feedUpdateManifest(text, "latest.yml", "v0.6.30", null)).toThrow(); + }); +}); diff --git a/workers/release-mirror/src/updateManifest.ts b/workers/release-mirror/src/updateManifest.ts new file mode 100644 index 0000000..3e4b5bf --- /dev/null +++ b/workers/release-mirror/src/updateManifest.ts @@ -0,0 +1,177 @@ +// FILE: updateManifest.ts +// Purpose: Reads and re-points the electron-updater channel manifests a desktop release publishes. +// Layer: Cloudflare Worker (scheduled) + +/** One downloadable file a manifest names, as electron-builder writes it. */ +export interface UpdateManifestFile { + readonly url: string; + readonly sha512: string; + readonly size: number; +} + +export interface UpdateManifest { + readonly version: string; + readonly files: readonly UpdateManifestFile[]; + /** The legacy single-file field; electron-updater uses it only when `files` is empty. */ + readonly path: string | null; + readonly hasReleaseName: boolean; + readonly hasReleaseNotes: boolean; +} + +/** What GitHub's release feed says about the release, as electron-updater's GitHub provider shows it. */ +export interface UpdateManifestRelease { + readonly name: string; + readonly notes: string; +} + +const FILE_URL = /^ {2}- url: (.+)$/u; +const FILE_FIELD = /^ {4}([A-Za-z][A-Za-z0-9]*): (.+)$/u; +const TOP_LEVEL = /^([A-Za-z][A-Za-z0-9]*):(?: (.*))?$/u; +const ASSET_NAME = /^[A-Za-z0-9][A-Za-z0-9._+-]*$/u; + +function scalar(raw: string): string { + const value = raw.trim(); + if (value.length >= 2 && value.startsWith("'") && value.endsWith("'")) { + return value.slice(1, -1).replace(/''/gu, "'"); + } + if (value.length >= 2 && value.startsWith('"') && value.endsWith('"')) { + return JSON.parse(value) as string; + } + return value; +} + +/** + * Parses the flat subset of YAML electron-builder and the release's manifest merge write: + * top-level scalars, and a `files` list of scalar maps. Anything else (block scalars, nested + * maps, anchors) is refused rather than guessed at, because the feed copy is rewritten line by + * line and must not misread a line inside a multi-line value. + */ +export function parseUpdateManifest(text: string, name: string): UpdateManifest { + let version: string | null = null; + let path: string | null = null; + let hasReleaseName = false; + let hasReleaseNotes = false; + let inFiles = false; + const files: UpdateManifestFile[] = []; + let current: { url: string; sha512?: string; size?: number } | null = null; + const finish = () => { + if (current === null) return; + if (current.sha512 === undefined || current.size === undefined) { + throw new Error(`${name} has a file entry without sha512 and size.`); + } + files.push({ url: current.url, sha512: current.sha512, size: current.size }); + current = null; + }; + + for (const rawLine of text.split(/\r?\n/u)) { + const line = rawLine.trimEnd(); + if (line === "") continue; + const fileUrl = inFiles ? FILE_URL.exec(line) : null; + if (fileUrl) { + finish(); + current = { url: scalar(fileUrl[1]!) }; + continue; + } + const field = current !== null ? FILE_FIELD.exec(line) : null; + if (field) { + const value = scalar(field[2]!); + if (/^[|>]/u.test(value)) throw new Error(`${name} has an unsupported block value.`); + if (field[1] === "sha512") current!.sha512 = value; + if (field[1] === "size") { + if (!/^\d+$/u.test(value)) throw new Error(`${name} has an invalid file size.`); + current!.size = Number(value); + } + continue; + } + finish(); + inFiles = false; + const top = TOP_LEVEL.exec(line); + if (!top) throw new Error(`${name} has an unsupported line.`); + const [, key, rawValue = ""] = top; + if (key === "files" && rawValue.trim() === "") { + inFiles = true; + continue; + } + const value = scalar(rawValue); + if (value === "" || /^[|>&*!]/u.test(value)) { + throw new Error(`${name} has an unsupported value for ${key}.`); + } + if (key === "version") version = value; + if (key === "path") path = value; + if (key === "releaseName") hasReleaseName = true; + if (key === "releaseNotes") hasReleaseNotes = true; + } + finish(); + + if (version === null) throw new Error(`${name} has no version.`); + if (files.length === 0) throw new Error(`${name} names no files.`); + const urls = new Set(); + for (const file of files) { + // A bare file name is what makes the release's relative URLs resolvable at all; an absolute + // or nested URL would keep clients tied to another host. + if (!ASSET_NAME.test(file.url) || urls.has(file.url)) { + throw new Error(`${name} names an unsafe or duplicate file: ${file.url}.`); + } + urls.add(file.url); + } + if (path !== null && !urls.has(path)) { + throw new Error(`${name} has a path that is not one of its files.`); + } + return { version, files, path, hasReleaseName, hasReleaseNotes }; +} + +/** A YAML double-quoted scalar: JSON's escapes, plus the characters YAML refuses unescaped. */ +function quoted(value: string): string { + return JSON.stringify(value).replace( + /[\u007f-\u009f\u2028\u2029\ufeff]/gu, + (character) => `\\u${character.charCodeAt(0).toString(16).padStart(4, "0")}`, + ); +} + +/** + * The feed copy of a release's manifest. electron-updater resolves each file URL against the + * feed (`new URL(url, feed + "/")`), so every bare name becomes `../..//`: from + * `scient-desktop/updates//` that is the release's immutable directory. The bytes' + * sizes and SHA-512s are left exactly as the release recorded them. Blockmaps need no entry: + * the updater derives `.blockmap` beside the resolved file, and the previous version's + * by substituting the version in that path, which the version directories also satisfy. + * + * `release` adds the name and notes GitHub's provider would have shown, unless the manifest + * already carries its own. + */ +export function feedUpdateManifest( + text: string, + name: string, + tag: string, + release: UpdateManifestRelease | null, +): string { + const manifest = parseUpdateManifest(text, name); + const relative = (file: string) => `../../${tag}/${file}`; + const lines = text + .split(/\r?\n/u) + .map((rawLine) => { + const line = rawLine.trimEnd(); + const fileUrl = FILE_URL.exec(line); + if (fileUrl) return ` - url: ${relative(scalar(fileUrl[1]!))}`; + const top = TOP_LEVEL.exec(line); + if (top?.[1] === "path") return `path: ${relative(scalar(top[2]!))}`; + return line; + }) + .filter((line, index, all) => line !== "" || index < all.length - 1); + if (release !== null) { + if (!manifest.hasReleaseName) lines.push(`releaseName: ${quoted(release.name)}`); + if (!manifest.hasReleaseNotes && release.notes !== "") { + lines.push(`releaseNotes: ${quoted(release.notes)}`); + } + } + const rewritten = `${lines.join("\n")}\n`; + // The copy must still describe exactly the same bytes. + const check = parseUpdateManifest( + rewritten.replaceAll(`../../${tag}/`, ""), + `${name} (feed copy)`, + ); + if (JSON.stringify(check.files) !== JSON.stringify(manifest.files)) { + throw new Error(`${name} changed while being re-pointed.`); + } + return rewritten; +} From efeae9578e262b8f2260efc5afc23c3148b0bc1b Mon Sep 17 00:00:00 2001 From: Yaacov Date: Mon, 16 Nov 2026 22:40:41 +0200 Subject: [PATCH 2/5] fix(release-mirror): verify reused bytes and guard feed promotion --- README.md | 14 +- docs/architecture/desktop-downloads.md | 25 ++-- workers/release-mirror/src/mirror.ts | 25 ++-- workers/release-mirror/src/updateFeed.test.ts | 130 +++++++++++++++++- workers/release-mirror/src/updateFeed.ts | 47 ++++++- 5 files changed, 209 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 3abca0b..ce418db 100644 --- a/README.md +++ b/README.md @@ -66,13 +66,13 @@ deployments do not deploy it. hashes it; on the Workers Free plan (50 external subrequests, 10 ms CPU) it cannot finish, so confirm the account is on Workers Paid. A run cut short resumes from what is already in the bucket. -3. Cache rules on the `downloads.scientfactory.com` zone (C1): - - `starts_with(http.request.uri.path, "/scient-desktop/updates/")`: **Bypass cache**. - - `/scient-desktop/v*` paths: **Eligible for cache**, edge TTL **use cache-control header**, - and a status-code TTL of **no-store** for 404 (and 4xx/5xx). Without such a rule, - Cloudflare caches 404s of default-cacheable types (`.exe`, `.zip`, `.dmg`) for a few - minutes, and does not cache `.AppImage`, `.blockmap` or `.yml` at all. Files above - 512 MB are not cached on non-Enterprise plans. +3. In the `scientfactory.com` zone, the `Scient download cache policy` ruleset (C1) + applies only to the `downloads.scientfactory.com` hostname: + - `/scient-desktop/updates/*` and `/scient-desktop/stable.json`: **Bypass cache**. + - `/scient-desktop/v*`: **Eligible for cache**, edge and browser TTLs **respect origin**, + with **no-store** for HTTP 300–599. Keep errors and redirects out of immutable caches. + Without such a rule, Cloudflare caches some 404s and does not cache `.AppImage`, + `.blockmap` or `.yml` by default. Files above 512 MB are not cached on non-Enterprise plans. 4. Verify each channel (`stable`/`latest`, then `beta`/`beta`): ```sh diff --git a/docs/architecture/desktop-downloads.md b/docs/architecture/desktop-downloads.md index 9c23b80..20176de 100644 --- a/docs/architecture/desktop-downloads.md +++ b/docs/architecture/desktop-downloads.md @@ -65,17 +65,20 @@ Each run, per channel (`workers/release-mirror/src/updateFeed.ts`): the run ends here. 3. Downloads the release's channel manifests (`latest.yml`, `latest-mac.yml`, `latest-linux.yml`, or the `beta*` set) and accepts them only if size and SHA-256 match the - handoff. Every file a manifest names must be a bare file name the handoff lists, with the - same size. -4. Copies every payload (macOS ZIPs and DMGs, the NSIS EXE, the AppImage), every blockmap, and - the manifests themselves into `scient-desktop//`, with the same immutable writes as the - installers. Payloads must also hash to the manifest's SHA-512 while streaming; the last chunk - is held back until it does, so a mismatched copy never completes. -5. Writes `scient-desktop/updates//` last, each as a compare-and-swap on the - version seen in step 1. The feed copy differs from the release's manifest only in two ways: - every file URL becomes `../..//`, which electron-updater resolves against the feed - URL to the version directory, and `releaseName`/`releaseNotes` are added from the release's - Atom entry, which is what GitHub's provider showed before. Sizes and SHA-512s are untouched. + handoff. The macOS manifest must offer ZIPs and DMGs for both x64 and arm64; + Windows must offer the x64 NSIS EXE and Linux the x64 AppImage. Extra Linux + architecture manifests and their listed packages, including debs, are copied too. + Every file must be a bare file name with the handoff's size, and ZIP/EXE payloads + must have differential blockmaps. +4. Copies every payload, blockmap and manifest into `scient-desktop//` with immutable + writes. Payloads also match the manifest's SHA-512 while streaming. An existing object + needs a matching SHA-512 verification receipt or is streamed from R2 and hashed before + reuse; a matching handoff SHA-256 alone is insufficient. +5. Rejects a candidate older than any observed platform manifest, then writes + `scient-desktop/updates//` last with a compare-and-swap on the + observed version. A stale source cannot regress a later promotion. The feed copy changes + file URLs to `../..//` and adds release name/notes from the Atom entry; + sizes and SHA-512s remain unchanged. Nothing is redirected: an old version's objects stay at their own paths, and the updater finds the previous version's blockmap by substituting its version into the new file's path, which diff --git a/workers/release-mirror/src/mirror.ts b/workers/release-mirror/src/mirror.ts index f5c3143..751dded 100644 --- a/workers/release-mirror/src/mirror.ts +++ b/workers/release-mirror/src/mirror.ts @@ -109,14 +109,23 @@ export async function mirrorReleaseAsset( { repository, tag, asset, sha512 }: ReleaseAssetSource, ): Promise { const key = desktopMirrorKey(tag, asset.name); - const verifyExisting = (existing: R2Object | null): false => { - if (existing?.size === asset.size && existing.customMetadata?.sha256 === asset.sha256) { - return false; + const verifyExisting = async (existing: R2Object | null): Promise => { + if (existing?.size !== asset.size || existing.customMetadata?.sha256 !== asset.sha256) { + throw new Error( + `${key} already holds other bytes (sha256 ${existing?.customMetadata?.sha256 ?? "unrecorded"}); ` + + `the handoff now says ${asset.sha256}. Published assets are never replaced.`, + ); } - throw new Error( - `${key} already holds other bytes (sha256 ${existing?.customMetadata?.sha256 ?? "unrecorded"}); ` + - `the handoff now says ${asset.sha256}. Published assets are never replaced.`, - ); + if (sha512 !== undefined && existing.customMetadata?.sha512 !== sha512) { + const stored = await bucket.get(key); + if (stored === null) throw new Error(`${key} disappeared before verification.`); + const digest = new crypto.DigestStream("SHA-512"); + await stored.body.pipeTo(digest); + if (base64(await digest.digest) !== sha512) { + throw new Error(`${key} does not match its update manifest's SHA-512.`); + } + } + return false; }; const existing = await bucket.head(key); if (existing !== null) return verifyExisting(existing); @@ -154,7 +163,7 @@ export async function mirrorReleaseAsset( contentDisposition: `attachment; filename="${asset.name}"`, cacheControl: IMMUTABLE_CACHE_CONTROL, }, - customMetadata: { sha256: asset.sha256 }, + customMetadata: { sha256: asset.sha256, ...(sha512 === undefined ? {} : { sha512 }) }, }); } catch (error) { stopPiping.abort(); diff --git a/workers/release-mirror/src/updateFeed.test.ts b/workers/release-mirror/src/updateFeed.test.ts index ce19a47..c5d5963 100644 --- a/workers/release-mirror/src/updateFeed.test.ts +++ b/workers/release-mirror/src/updateFeed.test.ts @@ -27,7 +27,12 @@ interface Release { async function release( version: string, channel: "stable" | "beta" = "stable", - tamper: { readonly sha512Of?: string } = {}, + tamper: { + readonly sha512Of?: string; + readonly omitPayload?: string; + readonly omitBlockmap?: string; + readonly extraLinux?: boolean; + } = {}, ): Promise { const prefix = channel === "stable" ? "latest" : "beta"; const repository = @@ -45,20 +50,28 @@ async function release( }; const files = new Map(); const manifest = async (list: readonly string[]) => { + const included = list.filter((name) => name !== tamper.omitPayload); const entries: string[] = []; - for (const name of list) { + for (const name of included) { files.set(name, payload(name)); const recorded = await sha512(tamper.sha512Of === name ? "other bytes" : payload(name)); entries.push( ` - url: ${name}\n sha512: ${recorded}\n size: ${payload(name).length}\n`, ); } - return `version: ${version}\nfiles:\n${entries.join("")}path: ${list[0]}\nsha512: ${await sha512(payload(list[0]!))}\nreleaseDate: '2026-11-16T10:00:00.000Z'\n`; + return `version: ${version}\nfiles:\n${entries.join("")}path: ${included[0]}\nsha512: ${await sha512(payload(included[0]!))}\nreleaseDate: '2026-11-16T10:00:00.000Z'\n`; }; files.set(`${prefix}-mac.yml`, await manifest(names.mac)); files.set(`${prefix}.yml`, await manifest(names.win)); files.set(`${prefix}-linux.yml`, await manifest(names.linux)); + if (tamper.extraLinux) { + files.set( + `${prefix}-linux-arm64.yml`, + await manifest([`Scient-${version}-arm64.AppImage`, `scient_${version}_arm64.deb`]), + ); + } for (const name of [...names.mac.filter((n) => n.endsWith(".zip")), ...names.win]) { + if (name === tamper.omitBlockmap) continue; files.set(`${name}.blockmap`, `${name} blockmap`); } files.set(`scient-server-${version}.tgz`, "server"); @@ -120,6 +133,12 @@ function memoryBucket() { : null; const bucket = { head: vi.fn(async (key: string) => describe(key, objects.get(key))), + get: vi.fn(async (key: string) => { + const object = objects.get(key); + return object === undefined + ? null + : { ...describe(key, object), body: new Response(object.body).body }; + }), list: vi.fn(async ({ prefix }: R2ListOptions) => ({ objects: [...objects.keys()] .filter((key) => key.startsWith(prefix ?? "")) @@ -339,6 +358,24 @@ describe("desktop update feed", () => { expect(objects.has("scient-desktop/v0.6.30/Scient-0.6.30-arm64.zip")).toBe(true); }); + it("publishes an offered extra-architecture Linux feed including its deb package", async () => { + const stable = await release("0.6.30", "stable", { extraLinux: true }); + const { bucket, objects } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + const result = await syncUpdateFeed(bucket, "stable"); + expect(result.promoted).toContain("latest-linux-arm64.yml"); + expect(result.copied).toContain("scient_0.6.30_arm64.deb"); + expect(objects.get("scient-desktop/v0.6.30/scient_0.6.30_arm64.deb")?.body).toBe( + stable.files.get("scient_0.6.30_arm64.deb"), + ); + const manifest = feedFile(objects, `${FEED}latest-linux-arm64.yml`); + expect(manifest).toContain("../../v0.6.30/scient_0.6.30_arm64.deb"); + expect(manifest).toContain("../../v0.6.30/Scient-0.6.30-arm64.AppImage"); + }); it("refuses bytes the handoff accepts but the release's manifest does not", async () => { const stable = await release("0.6.30", "stable", { sha512Of: "Scient-0.6.30-x64.exe" }); const { bucket, objects } = memoryBucket(); @@ -352,6 +389,93 @@ describe("desktop update feed", () => { expect([...objects.keys()].some((key) => key.startsWith(FEED))).toBe(false); }); + it.each([false, true])( + "checks an existing installer's SHA-512 before promotion (mismatch: %s)", + async (mismatch) => { + const name = "Scient-0.6.30-x64.exe"; + const stable = await release("0.6.30", "stable", mismatch ? { sha512Of: name } : {}); + const { bucket, objects } = memoryBucket(); + const body = stable.files.get(name)!; + objects.set(`scient-desktop/v0.6.30/${name}`, { + body, + etag: "installer-already-mirrored", + customMetadata: { sha256: await sha256(body) }, + }); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + if (mismatch) { + await expect(syncUpdateFeed(bucket, "stable")).rejects.toThrow(/SHA-512/); + expect([...objects.keys()].some((key) => key.startsWith(FEED))).toBe(false); + } else { + await expect(syncUpdateFeed(bucket, "stable")).resolves.toMatchObject({ + promoted: ["latest-linux.yml", "latest-mac.yml", "latest.yml"], + }); + } + }, + ); + + it("refuses a macOS manifest without both architectures' updater ZIPs", async () => { + const stable = await release("0.6.30", "stable", { + omitPayload: "Scient-0.6.30-arm64.zip", + }); + const { bucket, objects } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + await expect(syncUpdateFeed(bucket, "stable")).rejects.toThrow( + /does not reference required Scient-0.6.30-arm64\.zip/, + ); + expect(objects.size).toBe(0); + }); + + it("refuses a ZIP without its required differential blockmap", async () => { + const stable = await release("0.6.30", "stable", { + omitBlockmap: "Scient-0.6.30-arm64.zip", + }); + const { bucket, objects } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => [stable]), + ); + + await expect(syncUpdateFeed(bucket, "stable")).rejects.toThrow(/no required .*\.zip\.blockmap/); + expect(objects.size).toBe(0); + }); + + it.each([ + { channel: "stable" as const, older: "0.6.30", newer: "0.6.31" }, + { + channel: "beta" as const, + older: "0.6.31-beta.20261116.2", + newer: "0.6.31-beta.20261116.10", + }, + ])( + "refuses a stale source after a newer $channel promotion", + async ({ channel, older, newer }) => { + const oldRelease = await release(older, channel); + const newRelease = await release(newer, channel); + let releases = [newRelease]; + const { bucket, objects, writes } = memoryBucket(); + vi.stubGlobal( + "fetch", + github(() => releases), + ); + await syncUpdateFeed(bucket, channel); + const before = new Map(objects); + const written = writes.length; + releases = [oldRelease]; + + await expect(syncUpdateFeed(bucket, channel)).rejects.toThrow(/Refusing to regress/); + expect(objects).toEqual(before); + expect(writes).toHaveLength(written); + }, + ); + it("refuses a manifest that differs from the handoff", async () => { const stable = await release("0.6.30"); const { bucket, objects } = memoryBucket(); diff --git a/workers/release-mirror/src/updateFeed.ts b/workers/release-mirror/src/updateFeed.ts index 4648f80..46e53db 100644 --- a/workers/release-mirror/src/updateFeed.ts +++ b/workers/release-mirror/src/updateFeed.ts @@ -147,6 +147,23 @@ function planFeedObjects(handoff: ReleaseHandoff, manifests: readonly VerifiedMa `${asset.name} is for ${manifest.version}; the handoff is ${handoff.version}.`, ); } + const required = asset.name.endsWith("-mac.yml") + ? [ + `Scient-${handoff.version}-arm64.dmg`, + `Scient-${handoff.version}-arm64.zip`, + `Scient-${handoff.version}-x64.dmg`, + `Scient-${handoff.version}-x64.zip`, + ] + : asset.name === "latest.yml" || asset.name === "beta.yml" + ? [`Scient-${handoff.version}-x64.exe`] + : asset.name.endsWith("-linux.yml") + ? [`Scient-${handoff.version}-x86_64.AppImage`] + : []; + for (const name of required) { + if (!manifest.files.some((file) => file.url === name)) { + throw new Error(`${asset.name} does not reference required ${name}.`); + } + } for (const file of manifest.files) { const payload = byName.get(file.url); if (payload === undefined) { @@ -164,9 +181,14 @@ function planFeedObjects(handoff: ReleaseHandoff, manifests: readonly VerifiedMa payloads.set(file.url, { asset: payload, sha512: file.sha512 }); } } - const blockmaps = [...payloads.keys()] - .map((name) => byName.get(`${name}.blockmap`)) - .filter((asset): asset is ReleaseHandoffAsset => asset !== undefined); + const blockmaps: ReleaseHandoffAsset[] = []; + for (const name of payloads.keys()) { + const blockmap = byName.get(`${name}.blockmap`); + if (blockmap !== undefined) blockmaps.push(blockmap); + else if (/\.(?:zip|exe)$/u.test(name)) { + throw new Error(`Handoff ${handoff.tag} has no required ${name}.blockmap.`); + } + } return { payloads: [...payloads.values()], blockmaps }; } @@ -238,6 +260,25 @@ export async function syncUpdateFeed( } } + const candidateOrder = + channel === "beta" ? betaOrder(handoff.tag)! : handoff.version.split(".").map(Number); + for (const asset of manifestAssets) { + const metadata = observed.get(desktopUpdateFeedKey(channel, asset.name))?.customMetadata; + if (metadata === undefined) continue; + const version = metadata.version ?? ""; + const previousOrder = + channel === "beta" + ? betaOrder(`v${version}`) + : /^\d+\.\d+\.\d+$/u.test(version) + ? version.split(".").map(Number) + : null; + if (previousOrder === null || compareOrder(candidateOrder, previousOrder) < 0) { + throw new Error( + `Refusing to regress ${asset.name} from ${metadata.version ?? "unknown"} to ${handoff.version}.`, + ); + } + } + const current = (asset: ReleaseHandoffAsset) => { const metadata = observed.get(desktopUpdateFeedKey(channel, asset.name))?.customMetadata; return metadata?.tag === tag && metadata.sourceSha256 === asset.sha256; From 4711f4f0b34da02220e346e194b000e926ffcc33 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Mon, 16 Nov 2026 23:05:07 +0200 Subject: [PATCH 3/5] test(events): sync updater and recovery conformance cases --- workers/events/fixtures/contract-v11.json | 253 +++++++++++++++++++++- 1 file changed, 242 insertions(+), 11 deletions(-) diff --git a/workers/events/fixtures/contract-v11.json b/workers/events/fixtures/contract-v11.json index d676a78..edf9d2c 100644 --- a/workers/events/fixtures/contract-v11.json +++ b/workers/events/fixtures/contract-v11.json @@ -189,7 +189,10 @@ "privacyLevel": "product", "consentLevel": "product", "properties": { - "workKinds": ["scientific", "code"], + "workKinds": [ + "scientific", + "code" + ], "field": "biology", "role": "researcher", "otherWork": "Teaching", @@ -869,7 +872,6 @@ "action": "unknown", "runtimeSource": "unknown", "stage": "unknown", - "failureClass": "unknown", "durationBucket": "unknown", "trigger": "user", "appVersion": "0.6.8", @@ -887,9 +889,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -909,7 +911,6 @@ "action": "unknown", "runtimeSource": "unknown", "stage": "unknown", - "failureClass": "unknown", "durationBucket": "unknown", "trigger": "user", "appVersion": "0.6.8", @@ -927,9 +928,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -970,6 +971,7 @@ "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -989,7 +991,6 @@ "action": "unknown", "runtimeSource": "unknown", "stage": "unknown", - "failureClass": "unknown", "durationBucket": "unknown", "trigger": "user", "appVersion": "0.6.8", @@ -1007,9 +1008,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -1908,6 +1909,78 @@ "contractRevision": "11" } }, + { + "case": "app.update.outcome:fallbacks", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "unknown", + "outcome": "unknown", + "trigger": "unknown", + "channel": "unknown", + "sourceKind": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.outcome:representative", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "download", + "outcome": "completed", + "trigger": "background", + "channel": "stable", + "sourceKind": "github", + "targetVersion": "1.3.0", + "packageKind": "zip", + "attemptRef": "9f2c47a1-b0d3-4e6f-8c1a-7e4d92b03f6a", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:fallbacks", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "status": "unknown", + "behind": "unknown", + "channel": "unknown", + "sourceKind": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:representative", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "status": "downloading", + "behind": "yes", + "channel": "stable", + "sourceKind": "github", + "targetVersion": "1.3.0", + "stageAgeBucket": "1-10m", + "progressBucket": "25-49", + "sinceProgressBucket": "under-1m", + "readyAgeBucket": "1-6h", + "lastSuccessfulCheckAgeBucket": "10-60m", + "pendingFactsDroppedBucket": "2-3", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, { "case": "scient.operation.started:capture-v2-interactive-started", "name": "scient.operation.started", @@ -2500,9 +2573,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -2609,9 +2682,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -2718,9 +2791,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -2827,9 +2900,9 @@ "action": "repair", "runtimeSource": "scient_managed", "stage": "downloading", - "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", + "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", "compressed": true, @@ -2839,6 +2912,164 @@ "contractRevision": "11" } }, + { + "case": "app.update.outcome:update-outcome-signature-failure", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "download", + "outcome": "failed", + "trigger": "background", + "channel": "beta", + "sourceKind": "github", + "targetVersion": "1.3.0-beta.20261116.1", + "packageKind": "nsis", + "attemptRef": "9f2c47a1-b0d3-4e6f-8c1a-7e4d92b03f6a", + "failureClass": "signature", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.outcome:update-outcome-hostile", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "check", + "outcome": "started", + "trigger": "unknown", + "channel": "stable", + "sourceKind": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:update-summary-hostile", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "status": "unknown", + "behind": "unknown", + "channel": "stable", + "sourceKind": "unknown", + "stageAgeBucket": "unknown", + "progressBucket": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "provider.lifecycle.started:download-diagnostics", + "name": "provider.lifecycle.started", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "scient", + "action": "install", + "runtimeSource": "unknown", + "stage": "unknown", + "durationBucket": "unknown", + "trigger": "user", + "sourceKind": "scient_mirror", + "variant": "raw", + "networkKind": "reset", + "progressBucket": "50_74", + "throughputBucket": "64k_1m", + "lastFailureClass": "http_5xx", + "lastSourceKind": "github", + "compressedFailureClass": "checksum", + "downloadAttempts": 7, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "provider.lifecycle.completed:download-diagnostics", + "name": "provider.lifecycle.completed", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "scient", + "action": "install", + "runtimeSource": "unknown", + "stage": "unknown", + "durationBucket": "unknown", + "trigger": "user", + "sourceKind": "scient_mirror", + "variant": "raw", + "networkKind": "reset", + "progressBucket": "50_74", + "throughputBucket": "64k_1m", + "lastFailureClass": "http_5xx", + "lastSourceKind": "github", + "compressedFailureClass": "checksum", + "downloadAttempts": 7, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "provider.lifecycle.cancelled:download-diagnostics", + "name": "provider.lifecycle.cancelled", + "privacyLevel": "product", + "consentLevel": "product", + "properties": { + "provider": "scient", + "action": "install", + "runtimeSource": "unknown", + "stage": "unknown", + "durationBucket": "unknown", + "trigger": "user", + "sourceKind": "scient_mirror", + "variant": "raw", + "networkKind": "reset", + "progressBucket": "50_74", + "throughputBucket": "64k_1m", + "lastFailureClass": "http_5xx", + "lastSourceKind": "github", + "compressedFailureClass": "checksum", + "downloadAttempts": 7, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "provider.lifecycle.failed:download-diagnostics", + "name": "provider.lifecycle.failed", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "provider": "scient", + "action": "install", + "runtimeSource": "unknown", + "stage": "unknown", + "failureClass": "stalled", + "durationBucket": "unknown", + "trigger": "user", + "sourceKind": "scient_mirror", + "variant": "raw", + "networkKind": "reset", + "progressBucket": "50_74", + "throughputBucket": "64k_1m", + "lastFailureClass": "http_5xx", + "lastSourceKind": "github", + "compressedFailureClass": "checksum", + "downloadAttempts": 7, + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, { "case": "client.turn.requested:passthrough", "name": "client.turn.requested", From 3d40abd0ff1211a6e608bde3b354c60e34aed9a6 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Mon, 16 Nov 2026 23:31:34 +0200 Subject: [PATCH 4/5] test(events): align corpus with reconciled updater outcomes --- workers/events/fixtures/contract-v11.json | 474 +++++++++++----------- 1 file changed, 228 insertions(+), 246 deletions(-) diff --git a/workers/events/fixtures/contract-v11.json b/workers/events/fixtures/contract-v11.json index edf9d2c..ac5071c 100644 --- a/workers/events/fixtures/contract-v11.json +++ b/workers/events/fixtures/contract-v11.json @@ -118,6 +118,70 @@ "contractRevision": "11" } }, + { + "case": "app.update.outcome:fallbacks", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "unknown", + "outcome": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.outcome:representative", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "unknown", + "outcome": "unknown", + "sourceKind": "github", + "durationBucket": "under-1m", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:fallbacks", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "reason": "unknown", + "status": "unknown", + "channel": "unknown", + "behind": "unknown", + "stageAgeBucket": "unknown", + "lastCheckAgeBucket": "never", + "droppedFactsBucket": "0", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:representative", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "reason": "unknown", + "status": "unknown", + "channel": "unknown", + "behind": "unknown", + "stageAgeBucket": "unknown", + "lastCheckAgeBucket": "never", + "droppedFactsBucket": "0", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, { "case": "server.boot.heartbeat:fallbacks", "name": "server.boot.heartbeat", @@ -891,11 +955,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -930,11 +999,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -971,11 +1045,16 @@ "failureClass": "permission", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -1010,11 +1089,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -1909,78 +1993,6 @@ "contractRevision": "11" } }, - { - "case": "app.update.outcome:fallbacks", - "name": "app.update.outcome", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "step": "unknown", - "outcome": "unknown", - "trigger": "unknown", - "channel": "unknown", - "sourceKind": "unknown", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "app.update.outcome:representative", - "name": "app.update.outcome", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "step": "download", - "outcome": "completed", - "trigger": "background", - "channel": "stable", - "sourceKind": "github", - "targetVersion": "1.3.0", - "packageKind": "zip", - "attemptRef": "9f2c47a1-b0d3-4e6f-8c1a-7e4d92b03f6a", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "app.update.summary:fallbacks", - "name": "app.update.summary", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "status": "unknown", - "behind": "unknown", - "channel": "unknown", - "sourceKind": "unknown", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "app.update.summary:representative", - "name": "app.update.summary", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "status": "downloading", - "behind": "yes", - "channel": "stable", - "sourceKind": "github", - "targetVersion": "1.3.0", - "stageAgeBucket": "1-10m", - "progressBucket": "25-49", - "sinceProgressBucket": "under-1m", - "readyAgeBucket": "1-6h", - "lastSuccessfulCheckAgeBucket": "10-60m", - "pendingFactsDroppedBucket": "2-3", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, { "case": "scient.operation.started:capture-v2-interactive-started", "name": "scient.operation.started", @@ -2134,6 +2146,114 @@ "contractRevision": "11" } }, + { + "case": "app.update.outcome:download-failed", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "download", + "outcome": "failed", + "failureClass": "signature", + "trigger": "background", + "targetVersion": "0.6.30", + "sourceKind": "github", + "packageKind": "nsis", + "attemptRef": "3f9a0c1d2e4b5a67", + "durationBucket": "1-10m", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.outcome:apply-not-applied", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "apply", + "outcome": "not-applied", + "targetVersion": "0.6.30", + "packageKind": "zip", + "attemptRef": "3f9a0c1d2e4b5a67", + "durationBucket": "1-3d", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.outcome:outcome-hostile", + "name": "app.update.outcome", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "step": "install", + "outcome": "unknown", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:summary-downloading", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "reason": "change", + "status": "downloading", + "channel": "latest", + "behind": "yes", + "stageAgeBucket": "10-60m", + "lastCheckAgeBucket": "10-60m", + "droppedFactsBucket": "0", + "progressBucket": "25-49", + "sinceProgressBucket": "10-60m", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:summary-ready", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "reason": "daily", + "status": "downloaded", + "channel": "beta", + "behind": "yes", + "stageAgeBucket": "3-7d", + "lastCheckAgeBucket": "1-6h", + "droppedFactsBucket": "11-50", + "readyAgeBucket": "3-7d", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, + { + "case": "app.update.summary:summary-disabled", + "name": "app.update.summary", + "privacyLevel": "essential", + "consentLevel": "essential", + "properties": { + "reason": "launch", + "status": "disabled", + "channel": "latest", + "behind": "unknown", + "stageAgeBucket": "under-1m", + "lastCheckAgeBucket": "never", + "droppedFactsBucket": "0", + "disabledClass": "policy", + "appVersion": "0.6.8", + "buildChannel": "stable", + "contractRevision": "11" + } + }, { "case": "scient.operation.completed:capture-v2-retained", "name": "scient.operation.completed", @@ -2575,11 +2695,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -2684,11 +2809,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -2793,11 +2923,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, + "sourceKind": "github", + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" @@ -2902,169 +3037,16 @@ "stage": "downloading", "durationBucket": "5-15s", "trigger": "agent", - "sourceKind": "github", "attempt": 2, "downloadSource": "fallback", - "compressed": true, - "resumedBucket": "10-50mb", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "app.update.outcome:update-outcome-signature-failure", - "name": "app.update.outcome", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "step": "download", - "outcome": "failed", - "trigger": "background", - "channel": "beta", "sourceKind": "github", - "targetVersion": "1.3.0-beta.20261116.1", - "packageKind": "nsis", - "attemptRef": "9f2c47a1-b0d3-4e6f-8c1a-7e4d92b03f6a", - "failureClass": "signature", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "app.update.outcome:update-outcome-hostile", - "name": "app.update.outcome", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "step": "check", - "outcome": "started", - "trigger": "unknown", - "channel": "stable", - "sourceKind": "unknown", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "app.update.summary:update-summary-hostile", - "name": "app.update.summary", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "status": "unknown", - "behind": "unknown", - "channel": "stable", - "sourceKind": "unknown", - "stageAgeBucket": "unknown", - "progressBucket": "unknown", - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "provider.lifecycle.started:download-diagnostics", - "name": "provider.lifecycle.started", - "privacyLevel": "product", - "consentLevel": "product", - "properties": { - "provider": "scient", - "action": "install", - "runtimeSource": "unknown", - "stage": "unknown", - "durationBucket": "unknown", - "trigger": "user", - "sourceKind": "scient_mirror", - "variant": "raw", - "networkKind": "reset", - "progressBucket": "50_74", - "throughputBucket": "64k_1m", - "lastFailureClass": "http_5xx", - "lastSourceKind": "github", - "compressedFailureClass": "checksum", - "downloadAttempts": 7, - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "provider.lifecycle.completed:download-diagnostics", - "name": "provider.lifecycle.completed", - "privacyLevel": "product", - "consentLevel": "product", - "properties": { - "provider": "scient", - "action": "install", - "runtimeSource": "unknown", - "stage": "unknown", - "durationBucket": "unknown", - "trigger": "user", - "sourceKind": "scient_mirror", - "variant": "raw", - "networkKind": "reset", - "progressBucket": "50_74", - "throughputBucket": "64k_1m", - "lastFailureClass": "http_5xx", - "lastSourceKind": "github", - "compressedFailureClass": "checksum", - "downloadAttempts": 7, - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "provider.lifecycle.cancelled:download-diagnostics", - "name": "provider.lifecycle.cancelled", - "privacyLevel": "product", - "consentLevel": "product", - "properties": { - "provider": "scient", - "action": "install", - "runtimeSource": "unknown", - "stage": "unknown", - "durationBucket": "unknown", - "trigger": "user", - "sourceKind": "scient_mirror", - "variant": "raw", - "networkKind": "reset", - "progressBucket": "50_74", - "throughputBucket": "64k_1m", - "lastFailureClass": "http_5xx", - "lastSourceKind": "github", - "compressedFailureClass": "checksum", - "downloadAttempts": 7, - "appVersion": "0.6.8", - "buildChannel": "stable", - "contractRevision": "11" - } - }, - { - "case": "provider.lifecycle.failed:download-diagnostics", - "name": "provider.lifecycle.failed", - "privacyLevel": "essential", - "consentLevel": "essential", - "properties": { - "provider": "scient", - "action": "install", - "runtimeSource": "unknown", - "stage": "unknown", - "failureClass": "stalled", - "durationBucket": "unknown", - "trigger": "user", - "sourceKind": "scient_mirror", - "variant": "raw", - "networkKind": "reset", - "progressBucket": "50_74", - "throughputBucket": "64k_1m", - "lastFailureClass": "http_5xx", - "lastSourceKind": "github", - "compressedFailureClass": "checksum", - "downloadAttempts": 7, + "downloadVariant": "raw", + "downloadAttempts": 5, + "networkKind": "stalled", + "compressedFailure": "checksum", + "resumedBucket": "10-50mb", + "progressBucket": "25-49", + "throughputBucket": "1-5mbps", "appVersion": "0.6.8", "buildChannel": "stable", "contractRevision": "11" From 84664fea286d6cd091eb57c4f0396ba8438bb911 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Tue, 17 Nov 2026 01:02:28 +0200 Subject: [PATCH 5/5] fix(ci): format generated analytics fixture with website conventions --- README.md | 6 +++++- workers/events/fixtures/contract-v11.json | 5 +---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index ce418db..a0f00e1 100644 --- a/README.md +++ b/README.md @@ -143,9 +143,13 @@ and sufficient consent for the event's declared privacy level. It rejects nested properties such as `prompt` or `error`, and any value that looks like a path, link or email address. `workers/events/src/eventContract.ts` is generated from `scient-desktop/packages/scient-analytics/src/wireContract.ts`; do not edit it here. The shared -conformance fixtures (revisions 2 to 10) must keep passing. Since revision 10, new desktop events +conformance fixtures (revisions 2 to 11) must keep passing. Since revision 10, new desktop events and fields need no gateway change, so the desktop and gateway no longer have to ship in order. +After regenerating a shared fixture, apply this repository's formatter before validation: +`bunx --no-install oxfmt workers/events/fixtures/contract-v11.json`, then `bun run check`. +The desktop generator's JSON layout is not the website's formatting contract. + The public desktop endpoint is disabled unless the Cloudflare runtime variable `DESKTOP_INGESTION_ENABLED` is exactly `true`. It also requires a random installation-owned deletion token and applies the configured Cloudflare rate diff --git a/workers/events/fixtures/contract-v11.json b/workers/events/fixtures/contract-v11.json index ac5071c..68a19a2 100644 --- a/workers/events/fixtures/contract-v11.json +++ b/workers/events/fixtures/contract-v11.json @@ -253,10 +253,7 @@ "privacyLevel": "product", "consentLevel": "product", "properties": { - "workKinds": [ - "scientific", - "code" - ], + "workKinds": ["scientific", "code"], "field": "biology", "role": "researcher", "otherWork": "Teaching",