diff --git a/.agents/upstream-review.md b/.agents/upstream-review.md index 238201627..ef1d2f3f7 100644 --- a/.agents/upstream-review.md +++ b/.agents/upstream-review.md @@ -15,6 +15,29 @@ Two standing sections outlive any single batch and must be read on every review: ## Review batches +## 2026-09-05 — targeted Astra model classification + +The maintainer directed completing the Astra integration after review identified +upstream `bc03c3640d6d3bb44e5fb477bfd78d7484cd0e00` / `#9762`. Adopted on +`upstream/2026-09-05-astra`: add `gpt-6-astra` to the current Codex models. +Upstream's `updatedAt` field is omitted because Pylon's schema does not accept it. +The separate default-selection proposal `#9957` closed unmerged and is not part +of this classification change. + +The end-to-end check found that Pylon's publisher had failed since September 2: +it still rejected the `providers` field adopted with `#9084`. Publication now +uses the server decoder and emits a complete `model-catalog.json` plus the +original classification-only `model-manifest.json` for older strict readers. +Current servers fetch the catalog; both feeds receive the Astra classification. + +This is a filtered review at upstream head +`f8b4c464b4760d73e0ece7e68011c738803d8b69`; `reviewed-through` remains unchanged. +DEF-8 is due (Expo 57 is merged and its September 5 floor is reached); DEF-7 +has not reached its November 1 floor. WATCH-1's upstream issue `#5760` is closed, +Pylon `#178` remains open, and upstream `#8097` remains open. WATCH-2 and WATCH-3 +remain closed-unmerged upstream efforts with November 1 revisit floors. No +register or owner-issue decisions are implied by this targeted Astra change. + ## 2026-09-05 — first catch-up batch and watch recheck (partial) The maintainer approved A1–A4 after reviewing the bounded upstream head diff --git a/.github/workflows/publish-model-manifest.yml b/.github/workflows/publish-model-manifest.yml index edaa130a3..485cc4f90 100644 --- a/.github/workflows/publish-model-manifest.yml +++ b/.github/workflows/publish-model-manifest.yml @@ -7,6 +7,8 @@ on: paths: - apps/server/src/provider/model-manifest.json - .github/workflows/publish-model-manifest.yml + - apps/server/scripts/prepare-model-manifests.ts + - apps/server/src/provider/modelManifestPublication.ts permissions: contents: read @@ -19,8 +21,28 @@ jobs: publish: name: Publish public manifest runs-on: ubuntu-latest - timeout-minutes: 5 + timeout-minutes: 15 steps: + - name: Checkout approved source revision + uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + with: + ref: ${{ github.sha }} + persist-credentials: false + sparse-checkout: | + /* + !/.repos/ + sparse-checkout-cone-mode: false + + - name: Setup Vite+ + uses: voidzero-dev/setup-vp@250f29ce396baf5e8f24498e17c0dfdebabc26eb # v1 + with: + node-version-file: package.json + cache: true + run-install: true + + - name: Validate and prepare public manifests + run: node apps/server/scripts/prepare-model-manifests.ts /tmp/pylon-model-manifests + - name: Check source revision is current id: source_revision shell: bash @@ -76,79 +98,6 @@ jobs: exit 1 fi - - name: Fetch exact approved Pylon manifest - if: steps.source_revision.outputs.should_publish == 'true' - shell: bash - env: - GH_TOKEN: ${{ github.token }} - SOURCE_REPOSITORY: ${{ github.repository }} - SOURCE_SHA: ${{ github.sha }} - run: | - set -euo pipefail - curl \ - --proto '=https' \ - --tlsv1.2 \ - --fail \ - --silent \ - --show-error \ - --location \ - --connect-timeout 10 \ - --max-time 30 \ - --max-filesize 32768 \ - --retry 2 \ - --retry-all-errors \ - --retry-delay 1 \ - --header "Accept: application/vnd.github.raw+json" \ - --header "Authorization: Bearer ${GH_TOKEN}" \ - --header "X-GitHub-Api-Version: 2022-11-28" \ - --output /tmp/pylon-source-model-manifest.json \ - "${GITHUB_API_URL}/repos/${SOURCE_REPOSITORY}/contents/apps/server/src/provider/model-manifest.json?ref=${SOURCE_SHA}" - - - name: Validate source manifest - if: steps.source_revision.outputs.should_publish == 'true' - shell: bash - run: | - set -euo pipefail - node <<'NODE' - const fs = require("node:fs"); - const sourcePath = "/tmp/pylon-source-model-manifest.json"; - const publicPath = "/tmp/pylon-model-manifest.json"; - const maxBytes = 32 * 1024; - const maxModelsPerProvider = 256; - const maxSlugLength = 256; - const supportedDrivers = new Set(["codex", "claudeAgent"]); - const source = fs.readFileSync(sourcePath); - if (source.byteLength > maxBytes) { - throw new Error("Model manifest exceeds its public byte limit"); - } - const manifest = JSON.parse(source.toString("utf8")); - const topLevelFields = Object.keys(manifest).sort(); - if (JSON.stringify(topLevelFields) !== JSON.stringify(["currentModels", "version"])) { - throw new Error("Model manifest has unsupported top-level fields"); - } - if (manifest.version !== 1 || typeof manifest.currentModels !== "object" || manifest.currentModels === null || Array.isArray(manifest.currentModels)) { - throw new Error("Unsupported model manifest schema"); - } - const currentModels = Object.entries(manifest.currentModels); - if (currentModels.length > supportedDrivers.size || currentModels.some(([driver]) => !supportedDrivers.has(driver))) { - throw new Error("Model manifest has an unsupported driver kind"); - } - for (const [driver, models] of currentModels) { - if (!Array.isArray(models) || models.length > maxModelsPerProvider || new Set(models).size !== models.length || models.some((model) => typeof model !== "string" || !model || model.length > maxSlugLength)) { - throw new Error(`Invalid model list for ${driver}`); - } - } - const publicManifest = { - version: manifest.version, - currentModels: Object.fromEntries(currentModels), - }; - const serialized = `${JSON.stringify(publicManifest, null, 2)}\n`; - if (Buffer.byteLength(serialized) > maxBytes) { - throw new Error("Serialized public model manifest exceeds its byte limit"); - } - fs.writeFileSync(publicPath, serialized); - NODE - - name: Checkout public releases repository if: steps.source_revision.outputs.should_publish == 'true' uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 @@ -164,14 +113,15 @@ jobs: shell: bash run: | set -euo pipefail - cp /tmp/pylon-model-manifest.json public/model-manifest.json + cp /tmp/pylon-model-manifests/model-manifest.json public/model-manifest.json + cp /tmp/pylon-model-manifests/model-catalog.json public/model-catalog.json cd public - if [[ -z "$(git status --porcelain -- model-manifest.json)" ]]; then + if [[ -z "$(git status --porcelain -- model-manifest.json model-catalog.json)" ]]; then echo "The public manifest is already current." exit 0 fi git config user.name "pylon-release-bot" git config user.email "release-bot@users.noreply.github.com" - git add model-manifest.json + git add model-manifest.json model-catalog.json git commit -m "chore: update model manifest" git push origin HEAD:main diff --git a/apps/server/scripts/prepare-model-manifests.ts b/apps/server/scripts/prepare-model-manifests.ts new file mode 100644 index 000000000..79cc2f284 --- /dev/null +++ b/apps/server/scripts/prepare-model-manifests.ts @@ -0,0 +1,30 @@ +import * as NodeRuntime from "@effect/platform-node/NodeRuntime"; +import * as NodeServices from "@effect/platform-node/NodeServices"; +import * as Effect from "effect/Effect"; +import * as FileSystem from "effect/FileSystem"; +import * as Path from "effect/Path"; + +import { decodeManifestJson } from "../src/provider/ModelManifest.ts"; +import { serializeModelManifestPublication } from "../src/provider/modelManifestPublication.ts"; + +const destination = process.argv[2]; +if (!destination) { + throw new Error("Usage: prepare-model-manifests.ts "); +} + +Effect.gen(function* () { + const fileSystem = yield* FileSystem.FileSystem; + const path = yield* Path.Path; + const sourcePath = yield* path.fromFileUrl( + new URL("../src/provider/model-manifest.json", import.meta.url), + ); + const source = yield* fileSystem.readFileString(sourcePath); + // Reuse runtime validation, including driver bounds and provider adapter metadata, + // before either public file is written. + const manifest = yield* decodeManifestJson(source); + const files = serializeModelManifestPublication(manifest); + yield* fileSystem.makeDirectory(destination, { recursive: true }); + for (const { name, contents } of files) { + yield* fileSystem.writeFileString(path.join(destination, name), contents); + } +}).pipe(Effect.provide(NodeServices.layer), NodeRuntime.runMain); diff --git a/apps/server/src/provider/ModelManifest.test.ts b/apps/server/src/provider/ModelManifest.test.ts index 86e723848..41906fdf4 100644 --- a/apps/server/src/provider/ModelManifest.test.ts +++ b/apps/server/src/provider/ModelManifest.test.ts @@ -287,7 +287,7 @@ describe("ModelManifest service", () => { yield* service.refresh; assert.strictEqual( fetchedUrl, - "https://raw.githubusercontent.com/pylon-code/pylon-releases/main/model-manifest.json", + "https://raw.githubusercontent.com/pylon-code/pylon-releases/main/model-catalog.json", ); }).pipe( Effect.scoped, diff --git a/apps/server/src/provider/ModelManifest.ts b/apps/server/src/provider/ModelManifest.ts index 16c8ce683..6b1cdd837 100644 --- a/apps/server/src/provider/ModelManifest.ts +++ b/apps/server/src/provider/ModelManifest.ts @@ -38,7 +38,7 @@ import bundledManifestJson from "./model-manifest.json" with { type: "json" }; import type { ServerProviderDraft } from "./providerSnapshot.ts"; const MODEL_MANIFEST_URL = - "https://raw.githubusercontent.com/pylon-code/pylon-releases/main/model-manifest.json"; + "https://raw.githubusercontent.com/pylon-code/pylon-releases/main/model-catalog.json"; /** How long a fetched manifest stays fresh before the next probe re-fetches. */ const MANIFEST_TTL_MS = 60 * 60 * 1000; @@ -48,7 +48,7 @@ const MANIFEST_TTL_MS = 60 * 60 * 1000; const MANIFEST_RETRY_MS = 5 * 60 * 1000; const FETCH_TIMEOUT_MS = 10_000; -const MODEL_MANIFEST_MAX_BYTES = 32 * 1024; +export const MODEL_MANIFEST_MAX_BYTES = 32 * 1024; const MODEL_MANIFEST_MAX_PROVIDERS = 8; const MODEL_MANIFEST_MAX_MODELS_PER_PROVIDER = 256; const MODEL_MANIFEST_MAX_SLUG_LENGTH = 256; @@ -148,7 +148,7 @@ const decodeManifestJsonSchema = Schema.decodeUnknownEffect( ModelManifestSchema as unknown as Schema.Codec, ), ); -const decodeManifestJson = (input: string) => +export const decodeManifestJson = (input: string) => decodeManifestJsonSchema(input).pipe(Effect.flatMap(validateManifestDrivers)); export const BUNDLED_MODEL_MANIFEST: ModelManifestData = diff --git a/apps/server/src/provider/model-manifest.json b/apps/server/src/provider/model-manifest.json index 713015965..82475aa62 100644 --- a/apps/server/src/provider/model-manifest.json +++ b/apps/server/src/provider/model-manifest.json @@ -2,6 +2,7 @@ "version": 1, "currentModels": { "codex": [ + "gpt-6-astra", "gpt-5.6-luna", "gpt-5.6-terra", "gpt-5.6-sol", diff --git a/apps/server/src/provider/modelManifestPublication.test.ts b/apps/server/src/provider/modelManifestPublication.test.ts new file mode 100644 index 000000000..5855d4005 --- /dev/null +++ b/apps/server/src/provider/modelManifestPublication.test.ts @@ -0,0 +1,57 @@ +import { assert, describe, it } from "@effect/vitest"; +import * as Schema from "effect/Schema"; + +import { type ModelManifestData, MODEL_MANIFEST_MAX_BYTES } from "./ModelManifest.ts"; +import { serializeModelManifestPublication } from "./modelManifestPublication.ts"; + +const manifest: ModelManifestData = { + version: 1, + currentModels: { codex: ["current-model"] }, + providers: { + claudeAgent: { + profiles: {}, + models: [{ slug: "catalog-model", name: "Catalog Model", status: "current" }], + }, + }, +}; + +describe("model manifest publication", () => { + it("keeps the classification feed compatible with strict pre-catalog readers", () => { + const files = serializeModelManifestPublication(manifest); + const classification = JSON.parse( + files.find((file) => file.name === "model-manifest.json")!.contents, + ); + const legacySchema = Schema.Struct({ + version: Schema.Literal(1), + currentModels: Schema.Record(Schema.String, Schema.Array(Schema.String)), + }).annotate({ parseOptions: { onExcessProperty: "error" } }); + + assert.deepStrictEqual(Schema.decodeUnknownSync(legacySchema)(classification), { + version: 1, + currentModels: manifest.currentModels, + }); + assert.deepStrictEqual( + JSON.parse(files.find((file) => file.name === "model-catalog.json")!.contents), + manifest, + ); + }); + + it("rejects oversized catalog metadata before producing any files", () => { + const oversized = { + ...manifest, + providers: { + claudeAgent: { + profiles: {}, + models: [ + { + slug: "large-model", + name: "x".repeat(MODEL_MANIFEST_MAX_BYTES), + status: "current" as const, + }, + ], + }, + }, + }; + assert.throws(() => serializeModelManifestPublication(oversized), /model-catalog.json exceeds/); + }); +}); diff --git a/apps/server/src/provider/modelManifestPublication.ts b/apps/server/src/provider/modelManifestPublication.ts new file mode 100644 index 000000000..0317e22af --- /dev/null +++ b/apps/server/src/provider/modelManifestPublication.ts @@ -0,0 +1,24 @@ +import { MODEL_MANIFEST_MAX_BYTES, type ModelManifestData } from "./ModelManifest.ts"; + +/** Publish the validated catalog alongside the classification feed older servers accept. */ +export function serializeModelManifestPublication(manifest: ModelManifestData) { + const classification = { + version: manifest.version, + currentModels: manifest.currentModels, + }; + const files = { + "model-manifest.json": classification, + "model-catalog.json": { + ...classification, + ...(manifest.providers ? { providers: manifest.providers } : {}), + }, + }; + + return Object.entries(files).map(([name, value]) => { + const contents = `${JSON.stringify(value, null, 2)}\n`; + if (Buffer.byteLength(contents) > MODEL_MANIFEST_MAX_BYTES) { + throw new Error(`${name} exceeds the model manifest byte limit`); + } + return { name, contents }; + }); +} diff --git a/docs/internals/model-manifest.md b/docs/internals/model-manifest.md index bebf5c2ee..932cf3ad3 100644 --- a/docs/internals/model-manifest.md +++ b/docs/internals/model-manifest.md @@ -1,7 +1,8 @@ # Model manifest `apps/server/src/provider/model-manifest.json` is bundled for offline startup and fetched from -`main` at runtime. A remote fetch replaces the in-memory and on-disk cache only after generic +`model-catalog.json` on Pylon's public releases repository at runtime. A remote fetch replaces +the in-memory and on-disk cache only after generic catalog references and provider-owned adapter data validate. A failed or invalid fetch keeps the last successful remote manifest. The bundle is used only when no valid remote cache exists. @@ -18,6 +19,10 @@ a capability combination that does not already exist. catalog discovery. New Claude models do not need to be added there. Codex still discovers models from its app server and uses `currentModels.codex` only as a legacy-classification overlay. +Publication also produces `model-manifest.json` with only `version` and `currentModels` for +older releases whose strict schemas reject provider catalogs. Both files are generated from +the same validated source and published in one commit. See the [publishing runbook](../operations/model-manifest.md). + Claude model entries support: - `aliases`, `status`, `badge`, and `profile` for client presentation and selection. diff --git a/docs/operations/model-manifest.md b/docs/operations/model-manifest.md index 404c7ced0..5fae8926a 100644 --- a/docs/operations/model-manifest.md +++ b/docs/operations/model-manifest.md @@ -5,30 +5,34 @@ Pylon uses a small public manifest to decide which Codex and Claude models appea ## Locations - Source: `apps/server/src/provider/model-manifest.json` in `pylon-code/pylon`. -- Public mirror: `model-manifest.json` on the `main` branch of `pylon-code/pylon-releases`. -- Runtime URL: `https://raw.githubusercontent.com/pylon-code/pylon-releases/main/model-manifest.json`. +- Public catalog: `model-catalog.json` on the `main` branch of `pylon-code/pylon-releases`. +- Compatibility feed: `model-manifest.json` on the same branch, containing only `version` and `currentModels`. +- Runtime URL: `https://raw.githubusercontent.com/pylon-code/pylon-releases/main/model-catalog.json`. Older releases continue fetching the compatibility feed. - Publisher: `.github/workflows/publish-model-manifest.yml`. -The manifest contains schema version `1` and a map from provider driver kind to current model slugs. A built-in model is legacy when its driver has a manifest entry and its slug is absent from that list. Providers with no entry are not classified. Custom models are never classified. +The catalog contains schema version `1`, current model slugs, and optional provider catalogs. The compatibility feed omits provider catalogs because older releases reject unknown fields. A built-in model is legacy when its driver has a manifest entry and its slug is absent from that list. Providers with no entry are not classified. Custom models are never classified. ## Publish an update 1. Edit the source manifest on a task branch in `pylon-code/pylon`. -2. Run `vp test run apps/server/src/provider/ModelManifest.test.ts` and the server typecheck. +2. Run `vp test run apps/server/src/provider/ModelManifest.test.ts apps/server/src/provider/modelManifestPublication.test.ts` and the server typecheck. 3. Open and merge a PR against `pylon`. 4. Confirm the **Publish model manifest** workflow succeeds. -5. Verify the public repository at its exact `main` commit, then compare the JSON value with the source: +5. Generate both expected files and compare their JSON values with the public repository at its exact `main` commit: ```bash public_sha="$(gh api repos/pylon-code/pylon-releases/commits/main --jq .sha)" - gh api "repos/pylon-code/pylon-releases/contents/model-manifest.json?ref=$public_sha" --jq .content | base64 --decode | jq -S . > /tmp/public-model-manifest.json - jq -S . apps/server/src/provider/model-manifest.json > /tmp/source-model-manifest.json - diff -u /tmp/source-model-manifest.json /tmp/public-model-manifest.json + node apps/server/scripts/prepare-model-manifests.ts /tmp/pylon-expected-model-manifests + for manifest_file in model-manifest.json model-catalog.json; do + gh api "repos/pylon-code/pylon-releases/contents/$manifest_file?ref=$public_sha" --jq .content | base64 --decode | jq -S . > "/tmp/public-$manifest_file" + jq -S . "/tmp/pylon-expected-model-manifests/$manifest_file" > "/tmp/source-$manifest_file" + diff -u "/tmp/source-$manifest_file" "/tmp/public-$manifest_file" + done ``` The branch-based raw URL can remain cached for up to five minutes after publication. The commit-pinned check above is the authoritative immediate verification. -The workflow fetches only the source manifest blob at the exact merged `pylon` commit that triggered it. Before any release credential is exposed, it confirms that commit is still the current `pylon` head; delayed or rerun older revisions exit without publishing. Runs are serialized so an older revision cannot overwrite a newer mirror. The protected `pylon` branch requires a PR and rejects administrator bypass, force pushes, and deletion. The workflow validates a strict size-bounded schema and constructs a new allowlisted public object instead of copying arbitrary source fields. Its public commit message is the generic `chore: update model manifest`; it does not publish a private source commit, branch name, or commit history. +The workflow checks out the exact merged `pylon` revision that triggered it without persisting checkout credentials. Before accessing the release credential, it installs the pinned dependencies, validates the source with the server's manifest decoder, prepares both size-bounded public files, and confirms that the source commit is still the current `pylon` head. Delayed or rerun older revisions exit without publishing. Runs are serialized, and both files publish in one public commit. The protected `pylon` branch requires a PR and rejects administrator bypass, force pushes, and deletion. Public output is constructed from the allowlisted `version`, `currentModels`, and `providers` fields. Its generic commit message does not include source commit or branch metadata. The public `main` branch rejects force pushes and deletion, including from administrators. Normal fast-forward publication remains enabled for the release bot. A dedicated manifest-only credential would reduce the impact of a future publisher-token compromise; until one exists, keep `RELEASES_REPO_TOKEN` fine-grained and limited to `pylon-code/pylon-releases` contents. @@ -43,7 +47,7 @@ gh run rerun "$run_id" --repo pylon-code/pylon Rerunning the latest trusted `pylon` revision also covers manual recovery when the previous run succeeded but the public repository needs reconciliation. An older revision exits as superseded instead of rolling the public mirror back. Do not add an unrestricted manual-dispatch path that could expose the cross-repository credential to a branch workflow. -Do not change the runtime URL to the private product repository. Unauthenticated clients cannot depend on private source access. +Keep runtime feeds in the public releases repository so installed clients have a stable, unauthenticated endpoint independent of product repository access. ## Roll back a bad classification