Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 23 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,21 +58,35 @@ Stable and Beta electron-updater sets; see
deployments do not deploy it.

1. After merge, from `main`: `bun run mirror:types:check && bun run mirror:deploy`.
2. Watch the next runs (every ten minutes) with
2. Watch the next runs (at three minutes past every ten, and whenever a desktop release is
published) with
`bunx wrangler tail scientfactory-release-mirror --config workers/release-mirror/wrangler.jsonc`.
Expect `Stable update feed synchronized` and `Beta update feed synchronized` with `promoted`
naming three manifests each. The first run per release copies up to about 1.8 GB per channel and
hashes it; on the Workers Free plan (50 external subrequests, 10 ms CPU) it cannot finish,
so confirm the account is on Workers Paid. A run cut short resumes from what is already in the
bucket.
3. In the `scientfactory.com` zone, the `Scient download cache policy` ruleset (C1)
hashes it in about 40 seconds of CPU. It needs Workers Paid and the configured
`limits.cpu_ms` of 300000; on the Workers Free plan (50 external subrequests, 10 ms CPU) it
cannot finish. A run cut short resumes from what is already in the bucket.
3. The release workflow's trigger needs one shared random token, set in both places (never
printed or committed):

```sh
token=$(openssl rand -hex 32)
printf %s "$token" | bunx wrangler secret put MIRROR_SYNC_TOKEN --config workers/release-mirror/wrangler.jsonc
printf %s "$token" | gh secret set RELEASE_MIRROR_SYNC_TOKEN --repo ScientFactory/scient-desktop
gh secret set RELEASE_MIRROR_SYNC_URL --repo ScientFactory/scient-desktop \
--body "https://scientfactory-release-mirror.<account subdomain>.workers.dev/sync"
```

Without the secret, `POST /sync` answers 404 and only the schedule runs.

4. In the `scientfactory.com` zone, the `Scient download cache policy` ruleset (C1)
applies only to the `downloads.scientfactory.com` hostname:
- `/scient-desktop/updates/*` and `/scient-desktop/stable.json`: **Bypass cache**.
- `/scient-desktop/v*`: **Eligible for cache**, edge and browser TTLs **respect origin**,
with **no-store** for HTTP 300–599. Keep errors and redirects out of immutable caches.
Without such a rule, Cloudflare caches some 404s and does not cache `.AppImage`,
`.blockmap` or `.yml` by default. Files above 512 MB are not cached on non-Enterprise plans.
4. Verify each channel (`stable`/`latest`, then `beta`/`beta`):
5. Verify each channel (`stable`/`latest`, then `beta`/`beta`):

```sh
feed=https://downloads.scientfactory.com/scient-desktop/updates/stable
Expand All @@ -90,10 +104,11 @@ deployments do not deploy it.
curl -sS -o /dev/null -w '%{http_code}\n' https://downloads.scientfactory.com/scient-desktop/v0.0.0/missing.exe # 404, twice: never HIT
```

5. Only then merge the desktop change that makes these feeds the updater's default, and qualify
6. Only then merge the desktop change that makes these feeds the updater's default, and qualify
real updates on each platform from a build that uses them.

To stop publishing, remove the cron trigger and redeploy; the bucket's objects stay valid. Never
To stop publishing, remove the cron trigger, delete the `MIRROR_SYNC_TOKEN` secret and redeploy;
the bucket's objects stay valid. Never
delete or rewrite a versioned object: installed clients and caches may still use it.

## Scient Docs publishing
Expand Down
35 changes: 32 additions & 3 deletions docs/architecture/desktop-downloads.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,19 @@ download path, or `downloads.scientfactory.com/scient-desktop/v<semver>/<install

## The mirror

The `scientfactory-release-mirror` worker (`workers/release-mirror/`) runs every ten minutes
against the `scient-downloads` R2 bucket, the same bucket that serves Scient Agent's copies.
Each run:
The `scientfactory-release-mirror` worker (`workers/release-mirror/`) runs against the
`scient-downloads` R2 bucket, the same bucket that serves Scient Agent's copies. It runs at three
minutes past every ten (off the round minutes, when many Workers reach GitHub at once), and as
soon as a release is published: the desktop release workflow calls its authenticated
`POST /sync?channel=stable|beta` (see [Run on publish](#run-on-publish)). A run publishes the
Stable update feed first, then the Stable pointer below, then the Beta feed, so the pointer step
normally finds every Stable installer already verified against both digests instead of hashing
it again. When the feed step fails and the pointer step copies the installers alone, later
feed runs read them back to check their SHA-512 until one publishes the release (the copies keep
no SHA-512 receipt). The pointer step also refuses a GitHub handoff older than the pointer it
read, and stops if it cannot read the pointer, so a stale answer cannot move the pointer back.
GitHub answers of 429 or a gateway error are retried up to three attempts, honoring `Retry-After`
up to 30 seconds. Each run:

1. Reads GitHub's latest handoff and validates it.
2. For each required installer, keeps an existing `scient-desktop/<tag>/<name>` whose size
Expand Down Expand Up @@ -94,6 +104,25 @@ how long a 404 is kept (see the runbook in the README).
Deploy the worker with `bun run mirror:deploy`. The Pages project reads the bucket through
its `DESKTOP_DOWNLOADS` binding in `wrangler.jsonc`.

## Run on publish

The desktop release workflow (`.github/workflows/release.yml` in `scient-desktop`) calls
`POST https://<worker>.workers.dev/sync?channel=<stable|beta>` right after it publishes a Stable
or Beta release, with `Authorization: Bearer` and the token held as the worker secret
`MIRROR_SYNC_TOKEN` and the desktop repository secret `RELEASE_MIRROR_SYNC_TOKEN` (the address is
`RELEASE_MIRROR_SYNC_URL`). The worker answers 404 while the secret is unset, 401 for any other
token, and otherwise streams one JSON object per line: each step's result, a heartbeat every 15
seconds while it copies, and finally `{"message":"Release mirror finished","failures":n}`. The
run lasts only while the caller stays connected, since an HTTP invocation has no wall-time limit
but stops when its client leaves; whatever it copied stays, and the next scheduled run continues
from the bucket. The scheduled runs remain the fallback whenever the call fails.

The worker allows 5 minutes of CPU per invocation (`limits.cpu_ms`). Hashing one channel's
release (about 1.8 GB, SHA-256 by R2 and SHA-512 while streaming) takes about 40 seconds; with
the default 30 seconds, and the Stable pointer step's installers hashed a second time by the
feed step, the first run per release was cut short, and a release reached the feeds two or three
runs (20–30 minutes) after it was published.

## Why

Before this, every cold edge cache asked GitHub for the handoff while the visitor waited,
Expand Down
24 changes: 23 additions & 1 deletion functions/_lib/release-mirror.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,33 @@ export function parseMirrorPointer(value: unknown): MirroredRelease {
/** The current Stable pointer, null when nothing has been mirrored yet. Throws when unreadable. */
export async function readMirroredRelease(bucket: R2Bucket): Promise<MirroredRelease | null> {
const object = await bucket.get(DESKTOP_MIRROR_STABLE_KEY);
if (object === null) return null;
return object === null ? null : parseMirrorPointerObject(object);
}

async function parseMirrorPointerObject(
object: Pick<R2ObjectBody, "size" | "text">,
): Promise<MirroredRelease> {
if (object.size > MAX_POINTER_BYTES) throw new Error("Mirror pointer is too large.");
return parseMirrorPointer(JSON.parse(await object.text()) as unknown);
}

/**
* The pointer in an object a run has just read from the bucket, or null when its bytes are
* malformed (so the run may replace them). A failed read rejects instead: the run must not
* replace a pointer it could not see.
*/
export async function readMirrorPointerSnapshot(
object: Pick<R2ObjectBody, "size" | "text">,
): Promise<MirroredRelease | null> {
if (object.size > MAX_POINTER_BYTES) return null;
const text = await object.text();
try {
return parseMirrorPointer(JSON.parse(text) as unknown);
} catch {
return null;
}
}

/**
* Writes the pointer only if it is still the version this run observed (`observedEtag`, or
* absent when null). Returns false when another run wrote it first, so a slow run that saw an
Expand Down
103 changes: 103 additions & 0 deletions workers/release-mirror/src/github.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
import { afterEach, describe, expect, it, vi } from "vitest";

import { fetchFromGitHub, retryDelay } from "./github";

afterEach(() => {
vi.unstubAllGlobals();
});

function answers(...statuses: Array<number | [number, Record<string, string>]>) {
const fetchMock = vi.fn<typeof fetch>();
for (const status of statuses) {
const [code, headers] = Array.isArray(status) ? status : [status, {}];
fetchMock.mockResolvedValueOnce(
new Response(code === 200 ? "ok" : null, { status: code, headers }),
);
}
vi.stubGlobal("fetch", fetchMock);
return fetchMock;
}

describe("fetchFromGitHub", () => {
it("waits as long as GitHub asks, then tries again", async () => {
const fetchMock = answers([429, { "Retry-After": "2" }], 200);
const sleep = vi.fn(async () => undefined);

const response = await fetchFromGitHub("https://github.com/x", {}, 1_000, sleep);

expect(response.status).toBe(200);
expect(fetchMock).toHaveBeenCalledTimes(2);
expect(sleep).toHaveBeenCalledWith(2_000);
});

it("caps a long Retry-After and backs off when GitHub gives none", async () => {
answers([503, { "Retry-After": "600" }], 502, 200);
const sleep = vi.fn(async () => undefined);

await fetchFromGitHub("https://github.com/x", {}, 1_000, sleep);

expect(sleep.mock.calls).toEqual([[30_000], [10_000]]);
});

it("returns the third refusal for the caller to report", async () => {
const fetchMock = answers(429, 429, 429);

const response = await fetchFromGitHub(
"https://github.com/x",
{},
1_000,
async () => undefined,
);

expect(response.status).toBe(429);
expect(fetchMock).toHaveBeenCalledTimes(3);
});

it("does not retry an answer that will not change", async () => {
const fetchMock = answers(404);

const response = await fetchFromGitHub(
"https://github.com/x",
{},
1_000,
async () => undefined,
);

expect(response.status).toBe(404);
expect(fetchMock).toHaveBeenCalledTimes(1);
});

it("reads Retry-After as an HTTP date too", () => {
const now = Date.parse("Sat, 21 Nov 2026 20:00:00 GMT");
const at = (value: string) =>
retryDelay(new Response(null, { status: 429, headers: { "Retry-After": value } }), 1, now);

expect(at("Sat, 21 Nov 2026 20:00:25 GMT")).toBe(25_000);
expect(at("Sat, 21 Nov 2026 20:05:00 GMT")).toBe(30_000);
expect(at("Sat, 21 Nov 2026 19:59:00 GMT")).toBe(0);
expect(at("soon")).toBe(5_000);
});

it("retries even when the refused response's body has already failed", async () => {
const broken = new ReadableStream({
start(controller) {
controller.error(new Error("connection reset"));
},
});
const fetchMock = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(new Response(broken, { status: 503 }))
.mockResolvedValueOnce(new Response("ok"));
vi.stubGlobal("fetch", fetchMock);

const response = await fetchFromGitHub(
"https://github.com/x",
{},
1_000,
async () => undefined,
);

expect(response.status).toBe(200);
expect(fetchMock).toHaveBeenCalledTimes(2);
});
});
41 changes: 41 additions & 0 deletions workers/release-mirror/src/github.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// FILE: github.ts
// Purpose: Fetches from github.com for the release mirror, retrying GitHub's brief refusals.
// Layer: Cloudflare Worker

/** Statuses GitHub uses for a refusal or outage that usually clears within seconds. */
const RETRYABLE = new Set([429, 502, 503, 504]);
const ATTEMPTS = 3;
const MAX_RETRY_AFTER_MS = 30_000;

/**
* How long GitHub asked us to wait (`Retry-After` in seconds or as an HTTP date), at most 30
* seconds, or a short growing pause when it did not say.
*/
export function retryDelay(response: Response, attempt: number, now = Date.now()): number {
const header = response.headers.get("Retry-After")?.trim() ?? "";
let delay = Number.NaN;
if (/^\d+$/u.test(header)) delay = Number(header) * 1000;
else if (header !== "") delay = Date.parse(header) - now;
if (Number.isFinite(delay)) return Math.min(Math.max(delay, 0), MAX_RETRY_AFTER_MS);
return 5_000 * attempt;
}

/**
* `fetch` that tries again, up to three attempts in all, when GitHub answers 429 or a gateway
* error. Each attempt gets its own timeout. Any other answer, including the last refusal, is
* returned for the caller to judge.
*/
export async function fetchFromGitHub(
url: string,
headers: Record<string, string>,
timeoutMs: number,
sleep: (ms: number) => Promise<void> = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
): Promise<Response> {
for (let attempt = 1; ; attempt += 1) {
const response = await fetch(url, { headers, signal: AbortSignal.timeout(timeoutMs) });
if (!RETRYABLE.has(response.status) || attempt >= ATTEMPTS) return response;
// Only frees the connection; a body that already failed must not stop the retry.
await response.body?.cancel().catch(() => undefined);
await sleep(retryDelay(response, attempt));
}
}
Loading