From 9d8c4427a63847c6a2b97fd005d9d67a20f09fb5 Mon Sep 17 00:00:00 2001 From: Bruno Fernandez-Ruiz Date: Fri, 21 Aug 2026 08:44:50 +0200 Subject: [PATCH] ci: create the component release as a draft, upload, then publish `gh release create` produces a published release. GitHub refuses asset uploads into one, so every release since 2026-08-12 was created and left empty. From run 31833185321, verbatim: HTTP 422: Cannot upload assets to an immutable release. (.../releases/370780254/assets?label=&name=SHA256SUMS.txt) Immutability is not the cause, despite what that message says. aegis-agent-runtime carries 46 immutable releases and all 46 hold assets. What differs is the publishing shape: WORKS create a draft, upload assets, PATCH draft=false BREAKS create a published release, then upload -> 422 A draft is mutable. Publishing it afterwards produces an immutable release that already holds its assets, which is a required outcome rather than a hazard to work around: the publisher asserts it at line 376. The skip guard is why the empty releases persisted. Every re-run found the release already present and left its assets untouched, so the state was self-sustaining. It is deleted. scripts/publish_component_release.py is vendored byte-identically from aegis-agent-runtime, sha256 0ccf4b252a85a5547cfb642aea1b0e1c7871df88deab02d13672a559c1165340, matching the aegis-engine precedent. It carries upload-URL validation and draft resumption, which are the parts a fresh implementation gets wrong. Its tests are vendored too, which aegis-engine did not do. Shipping 444 lines that nothing local exercises is the failure mode this change exists to end, and map-cli ran no Python tests at all before this. ci.yml gains one step that runs them, and ci.yml runs on pull_request, so they execute before a merge rather than after. A concurrency block is added. The publisher is not internally serialised; its own comment says GitHub exposes no publication CAS. This workflow triggers on both push to main and workflow_dispatch, so without a group two runs on one revision compute the same tag, both create a draft (a draft makes no git ref, so GitHub permits two), and both then fail on multiple releases matching the intended tag. That leaves two orphan drafts and the revision permanently unpublishable, including by re-dispatch. Keyed on github.sha rather than github.ref, because the contended resource is the tag and the tag is map-cli-macos/${GITHUB_SHA}. cancel-in-progress is false so a publisher is never cancelled mid-publish. The signing block is unchanged. It moved out of the deleted else branch and was re-indented; de-indented and compared, both sides are 33 non-blank lines and identical, so it survived in content rather than only in appearance. The asset list is equivalent to the glob it replaces, in all three cases: signing succeeds *.tar.gz + SHA256SUMS.txt + SHA256SUMS.txt.sig signing skipped *.tar.gz + SHA256SUMS.txt nothing built the publisher raises, naming the missing path The third case improves on the old behaviour. intended_assets refuses a path that is not a file and refuses an empty set outright, so a glob that expanded to nothing fails loudly instead of uploading whatever matched. __pycache__/ is added to .gitignore because this change puts a Python test in CI, and .worktrees/ because this repository lacked it. Proof: vendored tests rc=0 Ran 23 tests OK mutation: publish call removed rc=1 FAILED (failures=3, errors=5) restore cmp rc=0, tests OK both workflows parse rc=0 publisher copy vs source cmp rc=0, sha256 identical gh release create / upload / view 0 / 0 / 0 publish_component_release.py 1 concurrency: 1 Independent review returned no blocking findings. Two majors it raised as explicitly unconfirmed were settled by measurement rather than argument, and both are refuted: target_commitish round-trip two existing published immutable releases report the full 40-hex SHA, not "main", so the post-publication check passes snapshot() requires an ETag gh api --include returns Etag: W/"1f450956a20ac7..." It also could not corroborate the precedent, because both reference clones on this machine are stale working trees: aegis-agent-runtime is 374 commits behind origin/main and aegis-engine is 29. The publisher exists on origin/main in both, and aegis-engine's bca355d, "fix(ci): publish component releases through a draft", landed 2026-08-20. That was a briefing error rather than a review error. The workflow comment now says why the existence guard was deleted and that a red re-dispatch is the publisher refusing to agree a release is what it claims to be. Without it the obvious repair for that red is to re-add the guard, which restores this defect. The wording deliberately does not quote the old command, because a comment containing it makes `gh release view` grep to 1 and turns a regression check ambiguous. Not proven: the workflow has never been executed. Nothing here shows the publish step works against GitHub, or that the concurrency group behaves under a real double-trigger. Those need the push run this change enables. Fixes #55 Agent-Session: 715d3479.coder-mapcli-55 --- .github/workflows/ci.yml | 1 + .github/workflows/component-artifacts.yml | 106 ++-- .gitignore | 2 + scripts/publish_component_release.py | 444 ++++++++++++++++ scripts/test_publish_component_release.py | 584 ++++++++++++++++++++++ 5 files changed, 1095 insertions(+), 42 deletions(-) create mode 100755 scripts/publish_component_release.py create mode 100755 scripts/test_publish_component_release.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 010daa2..de3094e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,3 +14,4 @@ jobs: - run: cargo fmt --check - run: cargo test - run: python3 scripts/package_component.py target/map-cli-component.tar.gz + - run: python3 scripts/test_publish_component_release.py diff --git a/.github/workflows/component-artifacts.yml b/.github/workflows/component-artifacts.yml index b958bc2..07e26bb 100644 --- a/.github/workflows/component-artifacts.yml +++ b/.github/workflows/component-artifacts.yml @@ -12,6 +12,14 @@ on: branches: [main] workflow_dispatch: +concurrency: + # The publisher is not internally serialised. Queue same-revision writers so + # they cannot create duplicate drafts for the same immutable release tag. + # Key this on github.sha, not github.ref: the contended resource is the tag. + # Keep cancel-in-progress false so a publisher is never cancelled mid-publish. + group: ${{ github.workflow }}-${{ github.sha }} + cancel-in-progress: false + permissions: contents: read @@ -57,7 +65,8 @@ jobs: # ADR 0022: durable, immutable Release assets so the pin never expires. # Per-commit, per-family tag; SHA256SUMS.txt recorded for the consume/re-pin - # side. Immutable = first-publish-wins. macOS runner -> shasum(1). + # side. Immutable, and published from a draft that already holds its assets + # (#55): a published release refuses uploads. macOS runner -> shasum(1). - name: Publish durable Release assets if: github.ref == 'refs/heads/main' env: @@ -67,48 +76,61 @@ jobs: # signing is skipped and the release still publishes SHA256SUMS.txt. AEGIS_COMPONENT_SIGNING_KEY: ${{ secrets.AEGIS_COMPONENT_SIGNING_KEY }} run: | + # The vendored publisher subsumes the old existence guard and is stronger. + # On an existing published tag it validates the release against the intended + # assets -- refusing extras, refusing missing assets, and comparing asset + # bytes against a rebuild. The old guard skipped on mere existence, which is + # how the empty releases accumulated unnoticed. Their disposition is recorded + # in #55: superseded, not deleted and not recreated. + # + # Do not re-add an existence check that skips when the tag is present. A + # red re-dispatch is the publisher refusing to agree that a release is what + # it claims to be, and that guard restores exactly the defect #55 fixed. tag="map-cli-macos/${GITHUB_SHA}" - if gh release view "$tag" >/dev/null 2>&1; then - echo "Release $tag already exists (immutable); leaving assets untouched." - else - ( cd dist/map-cli || exit 1 - shasum -a 256 -- *.tar.gz > SHA256SUMS.txt - # Self-signed component provenance (aegis#1110): detached raw Ed25519 - # signature over the exact SHA256SUMS.txt bytes. Inert until the org secret - # AEGIS_COMPONENT_SIGNING_KEY is provisioned; a release without it still - # publishes SHA256SUMS.txt and does not fail. Verified aegis-side with - # `openssl pkeyutl -verify -pubin -rawin -sigfile SHA256SUMS.txt.sig`. - if [ -n "${AEGIS_COMPONENT_SIGNING_KEY:-}" ]; then - # Resolve an OpenSSL 3.x `openssl` (only 3.x has `pkeyutl -rawin`). macOS system - # openssl is LibreSSL and lacks it; prefer Homebrew openssl@3 when present. - openssl_bin="" - if command -v brew >/dev/null 2>&1; then - brew_openssl="$(brew --prefix openssl@3 2>/dev/null)/bin/openssl" - if [ -x "$brew_openssl" ] && "$brew_openssl" version 2>/dev/null | grep -qE '^OpenSSL [3-9]'; then - openssl_bin="$brew_openssl" - fi - fi - if [ -z "$openssl_bin" ] && command -v openssl >/dev/null 2>&1 && openssl version 2>/dev/null | grep -qE '^OpenSSL [3-9]'; then - openssl_bin=openssl - fi - if [ -n "$openssl_bin" ]; then - # A real signing failure here is a hard error (fails the release, by design). - printf '%s\n' "$AEGIS_COMPONENT_SIGNING_KEY" > "$RUNNER_TEMP/component-signing.key" - "$openssl_bin" pkeyutl -sign -inkey "$RUNNER_TEMP/component-signing.key" -rawin -in SHA256SUMS.txt -out SHA256SUMS.txt.sig - rm -f "$RUNNER_TEMP/component-signing.key" - else - # Capability gap (e.g. a self-hosted runner without openssl@3): degrade - # gracefully. Producer signs best-effort; the aegis consumer's required mode - # blocks anything unsigned at re-pin, so a skipped signature is caught - # downstream, never a silent gap. Still publish SHA256SUMS.txt (unsigned). - echo "::warning::component-signing skipped: no OpenSSL 3.x on this runner; SHA256SUMS.txt published unsigned - consumer required-mode will flag it" + ( cd dist/map-cli || exit 1 + shasum -a 256 -- *.tar.gz > SHA256SUMS.txt + # Self-signed component provenance (aegis#1110): detached raw Ed25519 + # signature over the exact SHA256SUMS.txt bytes. Inert until the org secret + # AEGIS_COMPONENT_SIGNING_KEY is provisioned; a release without it still + # publishes SHA256SUMS.txt and does not fail. Verified aegis-side with + # `openssl pkeyutl -verify -pubin -rawin -sigfile SHA256SUMS.txt.sig`. + if [ -n "${AEGIS_COMPONENT_SIGNING_KEY:-}" ]; then + # Resolve an OpenSSL 3.x `openssl` (only 3.x has `pkeyutl -rawin`). macOS system + # openssl is LibreSSL and lacks it; prefer Homebrew openssl@3 when present. + openssl_bin="" + if command -v brew >/dev/null 2>&1; then + brew_openssl="$(brew --prefix openssl@3 2>/dev/null)/bin/openssl" + if [ -x "$brew_openssl" ] && "$brew_openssl" version 2>/dev/null | grep -qE '^OpenSSL [3-9]'; then + openssl_bin="$brew_openssl" fi fi - ) - gh release create "$tag" \ - --prerelease \ - --target "$GITHUB_SHA" \ - --title "map-cli macOS component ${GITHUB_SHA:0:12}" \ - --notes "Durable macOS MAP CLI (\`map\`) component assets (ADR 0022 / aegis#980). Immutable per commit; consumed via aegis component-pins by tag + sha256." - gh release upload "$tag" dist/map-cli/*.tar.gz dist/map-cli/SHA256SUMS.txt* + if [ -z "$openssl_bin" ] && command -v openssl >/dev/null 2>&1 && openssl version 2>/dev/null | grep -qE '^OpenSSL [3-9]'; then + openssl_bin=openssl + fi + if [ -n "$openssl_bin" ]; then + # A real signing failure here is a hard error (fails the release, by design). + printf '%s\n' "$AEGIS_COMPONENT_SIGNING_KEY" > "$RUNNER_TEMP/component-signing.key" + "$openssl_bin" pkeyutl -sign -inkey "$RUNNER_TEMP/component-signing.key" -rawin -in SHA256SUMS.txt -out SHA256SUMS.txt.sig + rm -f "$RUNNER_TEMP/component-signing.key" + else + # Capability gap (e.g. a self-hosted runner without openssl@3): degrade + # gracefully. Producer signs best-effort; the aegis consumer's required mode + # blocks anything unsigned at re-pin, so a skipped signature is caught + # downstream, never a silent gap. Still publish SHA256SUMS.txt (unsigned). + echo "::warning::component-signing skipped: no OpenSSL 3.x on this runner; SHA256SUMS.txt published unsigned - consumer required-mode will flag it" + fi + fi + ) + # Build the exact intended asset list. SHA256SUMS.txt.sig exists only when + # AEGIS_COMPONENT_SIGNING_KEY is provisioned and an OpenSSL 3.x was found. + release_assets=(dist/map-cli/*.tar.gz dist/map-cli/SHA256SUMS.txt) + if [ -f dist/map-cli/SHA256SUMS.txt.sig ]; then + release_assets+=(dist/map-cli/SHA256SUMS.txt.sig) fi + + python3 "$GITHUB_WORKSPACE/scripts/publish_component_release.py" \ + --tag "$tag" \ + --target-commit "$GITHUB_SHA" \ + --name "map-cli macOS component ${GITHUB_SHA:0:12}" \ + --body "Durable macOS MAP CLI (\`map\`) component assets (ADR 0022 / aegis#980). Immutable per commit; consumed via aegis component-pins by tag + sha256." \ + "${release_assets[@]}" diff --git a/.gitignore b/.gitignore index 4095b6e..006aa7c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,4 @@ /target/ /dist +__pycache__/ +.worktrees/ diff --git a/scripts/publish_component_release.py b/scripts/publish_component_release.py new file mode 100755 index 0000000..d116ef4 --- /dev/null +++ b/scripts/publish_component_release.py @@ -0,0 +1,444 @@ +#!/usr/bin/env python3 +"""Publish an exact immutable component release, resuming equivalent drafts.""" + +from __future__ import annotations + +import argparse +from dataclasses import dataclass +import json +import os +from pathlib import Path +import subprocess +from typing import Protocol +from urllib.error import URLError +from urllib.parse import urlencode, urlsplit +from urllib.request import HTTPRedirectHandler, Request, build_opener + + +class PublicationError(RuntimeError): + """The remote release cannot safely converge to the intended release.""" + + +@dataclass(frozen=True) +class Snapshot: + etag: str + release: dict[str, object] + + +class UploadOpener(Protocol): + def open(self, request: Request): ... + + +class RefuseUploadRedirects(HTTPRedirectHandler): + def redirect_request(self, request, response, code, message, headers, new_url): + raise PublicationError("release asset upload refused an HTTP redirect") + + +class ReleaseClient(Protocol): + def create(self, metadata: dict[str, object]) -> int | None: ... + + def find_release_ids(self, tag: str) -> list[int]: ... + + def snapshot(self, release_id: int) -> Snapshot: ... + + def download_asset(self, asset_id: int) -> bytes: ... + + def upload_asset(self, release_id: int, upload_url: str, path: Path) -> None: ... + + def publish(self, release_id: int) -> None: ... + + +class GhReleaseClient: + def __init__( + self, + endpoint: str = "repos/{owner}/{repo}/releases", + upload_opener: UploadOpener | None = None, + ) -> None: + self.endpoint = endpoint + self.upload_opener = upload_opener or build_opener(RefuseUploadRedirects()) + + @staticmethod + def _run(arguments: list[str], *, check: bool = True) -> subprocess.CompletedProcess[bytes]: + return subprocess.run( + ["gh", *arguments], + check=check, + stdout=subprocess.PIPE, + stderr=None, + ) + + def create(self, metadata: dict[str, object]) -> int | None: + result = self._run( + [ + "api", + "--method", + "POST", + self.endpoint, + "-f", + f"tag_name={metadata['tag_name']}", + "-F", + f"target_commitish={metadata['target_commitish']}", + "-f", + f"name={metadata['name']}", + "-f", + f"body={metadata['body']}", + "-F", + "draft=true", + "-F", + "prerelease=true", + ], + check=False, + ) + if result.returncode != 0: + return None + try: + release_id = json.loads(result.stdout)["id"] + except (json.JSONDecodeError, KeyError, TypeError) as error: + raise PublicationError("release creation returned no numeric id") from error + if not isinstance(release_id, int): + raise PublicationError("release creation returned no numeric id") + return release_id + + def find_release_ids(self, tag: str) -> list[int]: + result = self._run( + [ + "api", + "--method", + "GET", + "--paginate", + "--slurp", + "-F", + "per_page=100", + self.endpoint, + ] + ) + try: + pages = json.loads(result.stdout) + except json.JSONDecodeError as error: + raise PublicationError("release discovery returned invalid JSON") from error + if not isinstance(pages, list): + raise PublicationError("release discovery returned a non-list") + + release_ids: list[int] = [] + for page in pages: + if not isinstance(page, list): + raise PublicationError("release discovery returned a malformed page") + for release in page: + if not isinstance(release, dict): + raise PublicationError("release discovery returned a malformed release") + release_tag = release.get("tag_name") + release_id = release.get("id") + if ( + not isinstance(release_tag, str) + or not isinstance(release_id, int) + or isinstance(release_id, bool) + ): + raise PublicationError("release discovery omitted a tag or numeric id") + if release_tag == tag: + release_ids.append(release_id) + return release_ids + + def snapshot(self, release_id: int) -> Snapshot: + result = self._run(["api", "--include", f"{self.endpoint}/{release_id}"]) + normalized = result.stdout.replace(b"\r\n", b"\n") + try: + header_bytes, body = normalized.split(b"\n\n", 1) + except ValueError as error: + raise PublicationError("release snapshot omitted HTTP headers") from error + headers: dict[str, str] = {} + for line in header_bytes.decode("utf-8").splitlines()[1:]: + if ":" in line: + key, value = line.split(":", 1) + headers[key.lower()] = value.strip() + etag = headers.get("etag", "") + if not etag: + raise PublicationError("release snapshot omitted ETag") + try: + release = json.loads(body) + except json.JSONDecodeError as error: + raise PublicationError("release snapshot returned invalid JSON") from error + if not isinstance(release, dict): + raise PublicationError("release snapshot returned a non-object") + return Snapshot(etag=etag, release=release) + + def download_asset(self, asset_id: int) -> bytes: + result = self._run( + [ + "api", + "-H", + "Accept: application/octet-stream", + f"{self.endpoint}/assets/{asset_id}", + ] + ) + return result.stdout + + def upload_asset(self, release_id: int, upload_url: str, path: Path) -> None: + token = os.environ.get("GH_TOKEN") + if not token: + raise PublicationError("release asset upload requires GH_TOKEN") + endpoint = validated_upload_endpoint(upload_url, release_id) + destination = f"{endpoint}?{urlencode({'name': path.name})}" + request = Request( + destination, + data=path.read_bytes(), + headers={ + "Accept": "application/vnd.github+json", + "Authorization": f"Bearer {token}", + "Content-Type": "application/octet-stream", + "X-GitHub-Api-Version": "2022-11-28", + }, + method="POST", + ) + try: + with self.upload_opener.open(request) as response: + response.read() + except URLError as error: + raise PublicationError( + f"release asset upload failed for release {release_id}" + ) from error + + def publish(self, release_id: int) -> None: + self._run( + [ + "api", + "--silent", + "--method", + "PATCH", + "-F", + "draft=false", + f"{self.endpoint}/{release_id}", + ] + ) + + +def intended_assets(paths: list[Path]) -> dict[str, Path]: + assets: dict[str, Path] = {} + for path in paths: + if not path.is_file(): + raise PublicationError(f"intended release asset is not a file: {path}") + if path.name in assets: + raise PublicationError(f"duplicate intended release asset name: {path.name}") + assets[path.name] = path + if not assets: + raise PublicationError("component release has no intended assets") + return assets + + +def validated_upload_endpoint(upload_url: object, release_id: int) -> str: + if not isinstance(release_id, int) or isinstance(release_id, bool): + raise PublicationError("release upload requires a numeric release id") + if not isinstance(upload_url, str): + raise PublicationError("release snapshot omitted its upload URL") + + template = "{?name,label}" + if not upload_url.endswith(template): + raise PublicationError("release upload URL omitted its name template") + endpoint = upload_url[: -len(template)] + parsed = urlsplit(endpoint) + try: + port = parsed.port + except ValueError as error: + raise PublicationError("release upload URL has an invalid port") from error + if parsed.scheme != "https": + raise PublicationError("release upload URL must use HTTPS") + if parsed.hostname != "uploads.github.com": + raise PublicationError("release upload URL host is not uploads.github.com") + if parsed.username is not None or parsed.password is not None or port is not None: + raise PublicationError("release upload URL has unexpected authority fields") + if parsed.query or parsed.fragment: + raise PublicationError("release upload URL has an unexpected query or fragment") + + path_parts = parsed.path.split("/") + if ( + len(path_parts) != 7 + or path_parts[0] != "" + or path_parts[1] != "repos" + or not path_parts[2] + or not path_parts[3] + or path_parts[4:] != ["releases", str(release_id), "assets"] + ): + raise PublicationError("release upload URL is not bound to the selected release id") + return endpoint + + +def validate_snapshot( + client: ReleaseClient, + snapshot: Snapshot, + expected: dict[str, object], + assets: dict[str, Path], + *, + allow_missing: bool, +) -> list[str]: + release = snapshot.release + for field, expected_value in expected.items(): + if release.get(field) != expected_value: + raise PublicationError(f"release {field} does not match intended metadata") + + remote_assets = release.get("assets") + if not isinstance(remote_assets, list): + raise PublicationError("release snapshot omitted its asset inventory") + by_name: dict[str, dict[str, object]] = {} + for item in remote_assets: + if not isinstance(item, dict) or not isinstance(item.get("name"), str): + raise PublicationError("release asset inventory is malformed") + name = item["name"] + if name in by_name: + raise PublicationError(f"release contains duplicate asset name: {name}") + by_name[name] = item + + extras = sorted(set(by_name) - set(assets)) + if extras: + raise PublicationError(f"release contains unexpected assets: {', '.join(extras)}") + missing = sorted(set(assets) - set(by_name)) + if missing and not allow_missing: + raise PublicationError(f"published release is missing assets: {', '.join(missing)}") + + for name, item in by_name.items(): + asset_id = item.get("id") + size = item.get("size") + local_bytes = assets[name].read_bytes() + if item.get("state") not in (None, "uploaded"): + raise PublicationError(f"release asset is not uploaded: {name}") + if not isinstance(asset_id, int) or size != len(local_bytes): + raise PublicationError(f"release asset metadata differs from intended bytes: {name}") + if client.download_asset(asset_id) != local_bytes: + raise PublicationError(f"release asset bytes differ from intended bytes: {name}") + return missing + + +def validated_release_ids(release_ids: list[int]) -> set[int]: + validated: set[int] = set() + for release_id in release_ids: + if not isinstance(release_id, int) or isinstance(release_id, bool): + raise PublicationError("release discovery returned a non-numeric id") + validated.add(release_id) + return validated + + +def unique_matching_release_id(client: ReleaseClient, tag: str) -> int | None: + release_ids = validated_release_ids(client.find_release_ids(tag)) + if len(release_ids) > 1: + raise PublicationError(f"multiple releases match intended tag: {tag}") + if not release_ids: + return None + return next(iter(release_ids)) + + +def select_created_release_id( + created_release_id: int | None, discovered_release_ids: list[int], tag: str +) -> int: + release_ids = validated_release_ids(discovered_release_ids) + if created_release_id is not None: + if not isinstance(created_release_id, int) or isinstance( + created_release_id, bool + ): + raise PublicationError("release creation returned no numeric id") + release_ids.add(created_release_id) + if len(release_ids) > 1: + raise PublicationError(f"multiple releases match intended tag: {tag}") + if not release_ids: + raise PublicationError("release creation failed and no matching release exists") + return next(iter(release_ids)) + + +def publish_component_release( + client: ReleaseClient, + *, + tag: str, + target_commit: str, + name: str, + body: str, + asset_paths: list[Path], +) -> str: + assets = intended_assets(asset_paths) + metadata: dict[str, object] = { + "tag_name": tag, + "target_commitish": target_commit, + "name": name, + "body": body, + "prerelease": True, + } + release_id = unique_matching_release_id(client, tag) + if release_id is not None: + disposition = "resumed" + else: + created_release_id = client.create(metadata) + release_id = select_created_release_id( + created_release_id, client.find_release_ids(tag), tag + ) + disposition = "created" if created_release_id is not None else "resumed" + + snapshot = client.snapshot(release_id) + draft = snapshot.release.get("draft") + upload_url = snapshot.release.get("upload_url") + validated_upload_endpoint(upload_url, release_id) + expected = {**metadata, "id": release_id, "upload_url": upload_url} + if draft is False: + if snapshot.release.get("immutable") is not True: + raise PublicationError("published release is not immutable") + validate_snapshot(client, snapshot, expected, assets, allow_missing=False) + return "published-equivalent" + if draft is not True: + raise PublicationError("release draft state is invalid") + + missing = validate_snapshot(client, snapshot, expected, assets, allow_missing=True) + for asset_name in missing: + client.upload_asset(release_id, upload_url, assets[asset_name]) + + final_snapshot = client.snapshot(release_id) + if final_snapshot.release.get("draft") is not True: + raise PublicationError("release draft state changed before publication") + validate_snapshot(client, final_snapshot, expected, assets, allow_missing=False) + + # GitHub exposes no publication CAS. Repository Contents writers are trusted + # release authorities; workflow concurrency serializes repository-owned writers. + # These adjacent validations detect an authority breach but cannot prevent one. + client.publish(release_id) + published_snapshot = client.snapshot(release_id) + if published_snapshot.release.get("draft") is not False: + raise PublicationError( + "post-publication authority breach: release remained a draft" + ) + if published_snapshot.release.get("immutable") is not True: + raise PublicationError( + "post-publication authority breach: published release is not immutable" + ) + try: + validate_snapshot( + client, published_snapshot, expected, assets, allow_missing=False + ) + except PublicationError as error: + raise PublicationError( + f"post-publication authority breach: {error}" + ) from error + return disposition + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tag", required=True) + parser.add_argument("--target-commit", required=True) + parser.add_argument("--name", required=True) + parser.add_argument("--body", required=True) + parser.add_argument("assets", nargs="+", type=Path) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + try: + result = publish_component_release( + GhReleaseClient(), + tag=args.tag, + target_commit=args.target_commit, + name=args.name, + body=args.body, + asset_paths=args.assets, + ) + except (PublicationError, subprocess.CalledProcessError) as error: + raise SystemExit(f"component release publication refused: {error}") from error + print(f"component_release={result} target={args.target_commit} tag={args.tag}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/test_publish_component_release.py b/scripts/test_publish_component_release.py new file mode 100755 index 0000000..cfaefdb --- /dev/null +++ b/scripts/test_publish_component_release.py @@ -0,0 +1,584 @@ +#!/usr/bin/env python3 +"""Deterministic tests for resumable immutable component publication.""" + +from __future__ import annotations + +import copy +from email.message import Message +import importlib.util +import json +from pathlib import Path +import subprocess +import sys +import tempfile +from typing import Callable +import unittest +from unittest import mock + + +SCRIPT = Path(__file__).with_name("publish_component_release.py") +SPEC = importlib.util.spec_from_file_location("publish_component_release", SCRIPT) +assert SPEC and SPEC.loader +MODULE = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = MODULE +SPEC.loader.exec_module(MODULE) + + +class ConditionalUnsafePatchRejectingGhClient(MODULE.GhReleaseClient): + """Fake command boundary reproducing GitHub's captured HTTP 400.""" + + CAPTURED_HTTP_400 = ( + "HTTP 400: Conditional request headers are not allowed in unsafe requests " + "unless supported by the endpoint" + ) + + def __init__(self) -> None: + super().__init__(endpoint="repos/example/runtime/releases") + self.commands: list[list[str]] = [] + + def _run( + self, arguments: list[str], *, check: bool = True + ) -> subprocess.CompletedProcess[bytes]: + self.commands.append(arguments) + is_unsafe_patch = "--method" in arguments and "PATCH" in arguments + has_if_match = any(argument.startswith("If-Match: ") for argument in arguments) + if is_unsafe_patch and has_if_match: + raise MODULE.PublicationError(self.CAPTURED_HTTP_400) + return subprocess.CompletedProcess(["gh", *arguments], 0, stdout=b"") + + def publish_as_old_client(self, release_id: int, etag: str) -> None: + self._run( + [ + "api", + "--silent", + "--method", + "PATCH", + "-H", + f"If-Match: {etag}", + "-F", + "draft=false", + f"{self.endpoint}/{release_id}", + ] + ) + + +class ReleaseDiscoveryGhClient(MODULE.GhReleaseClient): + """Fake command boundary for paginated same-tag release discovery.""" + + def __init__(self, pages: object) -> None: + super().__init__(endpoint="repos/example/runtime/releases") + self.pages = pages + self.commands: list[list[str]] = [] + + def _run( + self, arguments: list[str], *, check: bool = True + ) -> subprocess.CompletedProcess[bytes]: + self.commands.append(arguments) + return subprocess.CompletedProcess( + ["gh", *arguments], 0, stdout=json.dumps(self.pages).encode() + ) + + +class RedirectResponseOpener: + """Return one redirect, then record any unsafe follow-up request.""" + + def __init__(self, handler: MODULE.HTTPRedirectHandler) -> None: + self.handler = handler + self.handler.parent = self + self.requests: list[MODULE.Request] = [] + + def open(self, request: MODULE.Request, **kwargs): + self.requests.append(request) + response = mock.MagicMock() + if len(self.requests) == 1: + request.timeout = None + headers = Message() + headers["Location"] = "https://attacker.example/collect" + return self.handler.http_error_302( + request, response, 302, "Found", headers + ) + response.__enter__.return_value.read.return_value = b"{}" + return response + + +class FakeClient: + """GitHub-like client that permits duplicate draft tags on create.""" + + def __init__( + self, + *, + draft: bool = True, + snapshot_etag: str = 'W/"weak-etag-current"', + ) -> None: + self.release_id = 42 + self.release: dict[str, object] | None = None + self.draft = draft + self.snapshot_etag = snapshot_etag + self.snapshot_count = 0 + self.create_calls = 0 + self.find_calls = 0 + self.create_result: int | None = self.release_id + self.find_results: dict[int, list[int]] = {} + self.matching_release_ids: list[int] = [] + self.operations: list[str] = [] + self.before_snapshot: dict[int, Callable[[FakeClient], None]] = {} + self.after_publish: Callable[[FakeClient], None] | None = None + self.asset_bytes: dict[int, bytes] = {} + self.uploaded: list[str] = [] + self.uploaded_release_ids: list[int] = [] + self.uploaded_urls: list[str] = [] + self.published: list[int] = [] + self.next_asset_id = 100 + + def seed(self, metadata: dict[str, object], assets: dict[str, bytes]) -> None: + if self.release_id not in self.matching_release_ids: + self.matching_release_ids.append(self.release_id) + self.release = { + **metadata, + "id": self.release_id, + "draft": self.draft, + "immutable": not self.draft, + "upload_url": ( + "https://uploads.github.com/repos/example/runtime/releases/" + f"{self.release_id}/assets{{?name,label}}" + ), + "assets": [], + } + for name, content in assets.items(): + self._add_asset(name, content) + + def _add_asset(self, name: str, content: bytes) -> None: + assert self.release is not None + asset_id = self.next_asset_id + self.next_asset_id += 1 + self.asset_bytes[asset_id] = content + self.release["assets"].append( + {"id": asset_id, "name": name, "size": len(content)} + ) + + def create(self, metadata: dict[str, object]) -> int | None: + self.create_calls += 1 + self.operations.append("create") + if self.matching_release_ids: + self.release_id = max(self.matching_release_ids) + 1 + if self.create_result is not None: + self.create_result = self.release_id + self.seed(metadata, {}) + return self.create_result + + def find_release_ids(self, tag: str) -> list[int]: + self.find_calls += 1 + self.operations.append("find") + if self.release is not None: + assert self.release["tag_name"] == tag + return self.find_results.get( + self.find_calls, self.matching_release_ids + ).copy() + + def snapshot(self, release_id: int): + assert release_id == self.release_id + assert self.release is not None + self.operations.append("snapshot") + self.snapshot_count += 1 + mutation = self.before_snapshot.get(self.snapshot_count) + if mutation is not None: + mutation(self) + return MODULE.Snapshot(self.snapshot_etag, copy.deepcopy(self.release)) + + def download_asset(self, asset_id: int) -> bytes: + return self.asset_bytes[asset_id] + + def upload_asset(self, release_id: int, upload_url: str, path: Path) -> None: + assert self.release is not None + assert release_id == self.release_id + assert upload_url == self.release["upload_url"] + self.operations.append("upload") + self.uploaded.append(path.name) + self.uploaded_release_ids.append(release_id) + self.uploaded_urls.append(upload_url) + self._add_asset(path.name, path.read_bytes()) + + def publish(self, release_id: int) -> None: + assert self.release is not None + self.operations.append("publish") + self.published.append(release_id) + self.release["draft"] = False + self.release["immutable"] = True + if self.after_publish is not None: + self.after_publish(self) + + +class PublicationTests(unittest.TestCase): + def setUp(self) -> None: + self.temp = tempfile.TemporaryDirectory() + root = Path(self.temp.name) + self.assets = [root / "runtime.tar.gz", root / "runtime.tar.gz.sig"] + self.assets[0].write_bytes(b"archive-exact-bytes") + self.assets[1].write_bytes(b"signature-exact-bytes") + self.kwargs = { + "tag": "components/" + "7" * 40, + "target_commit": "7" * 40, + "name": "components " + "7" * 12, + "body": "owned immutable component release", + "asset_paths": self.assets, + } + self.metadata = { + "tag_name": self.kwargs["tag"], + "target_commitish": self.kwargs["target_commit"], + "name": self.kwargs["name"], + "body": self.kwargs["body"], + "prerelease": True, + } + + def tearDown(self) -> None: + self.temp.cleanup() + + def test_old_conditional_client_gets_http_400_and_repaired_client_omits_header( + self, + ) -> None: + client = ConditionalUnsafePatchRejectingGhClient() + with self.assertRaisesRegex( + MODULE.PublicationError, + "Conditional request headers are not allowed in unsafe requests", + ) as caught: + client.publish_as_old_client(42, 'W/"weak-etag-current"') + self.assertEqual(str(caught.exception), client.CAPTURED_HTTP_400) + self.assertIn("If-Match: W/\"weak-etag-current\"", client.commands[0]) + + client.publish(42) + self.assertEqual(len(client.commands), 2) + self.assertNotIn("-H", client.commands[1]) + self.assertNotIn("If-Match:", " ".join(client.commands[1])) + + def test_release_discovery_lists_every_exact_tag_across_pages(self) -> None: + tag = self.kwargs["tag"] + client = ReleaseDiscoveryGhClient( + [ + [{"id": 40, "tag_name": "other"}, {"id": 41, "tag_name": tag}], + [{"id": 42, "tag_name": tag}], + ] + ) + self.assertEqual(client.find_release_ids(tag), [41, 42]) + self.assertEqual( + client.commands, + [[ + "api", + "--method", + "GET", + "--paginate", + "--slurp", + "-F", + "per_page=100", + "repos/example/runtime/releases", + ]], + ) + + def test_fresh_release_accepts_weak_etag_and_validates_after_publish(self) -> None: + client = FakeClient() + result = MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(result, "created") + self.assertEqual(client.uploaded, [path.name for path in self.assets]) + self.assertEqual(client.published, [42]) + self.assertEqual(client.snapshot_count, 3) + self.assertEqual(client.create_calls, 1) + self.assertEqual(client.find_calls, 2) + self.assertEqual(client.uploaded_release_ids, [42, 42]) + self.assertEqual( + client.uploaded_urls, + [client.release["upload_url"], client.release["upload_url"]], + ) + + def test_create_response_id_survives_empty_post_create_listing(self) -> None: + client = FakeClient() + client.find_results[2] = [] + result = MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(result, "created") + self.assertEqual(client.find_calls, 2) + self.assertEqual(client.operations[:4], ["find", "create", "find", "snapshot"]) + self.assertEqual(client.uploaded_release_ids, [42, 42]) + self.assertEqual(client.published, [42]) + + def test_failed_create_resumes_one_release_from_post_create_listing(self) -> None: + client = FakeClient() + client.create_result = None + result = MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(result, "resumed") + self.assertEqual(client.find_calls, 2) + self.assertEqual(client.uploaded_release_ids, [42, 42]) + self.assertEqual(client.published, [42]) + + def test_failed_create_without_discovered_release_is_rejected(self) -> None: + client = FakeClient() + client.create_result = None + client.find_results[2] = [] + with self.assertRaisesRegex( + MODULE.PublicationError, + "release creation failed and no matching release exists", + ): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.snapshot_count, 0) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_created_and_discovered_distinct_ids_are_rejected(self) -> None: + client = FakeClient() + client.find_results[2] = [43] + with self.assertRaisesRegex( + MODULE.PublicationError, "multiple releases match intended tag" + ): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.snapshot_count, 0) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_release_asset_upload_uses_validated_id_url_and_in_process_token( + self, + ) -> None: + opener = mock.MagicMock() + client = MODULE.GhReleaseClient( + endpoint="repos/example/runtime/releases", upload_opener=opener + ) + upload_url = ( + "https://uploads.github.com/repos/example/runtime/releases/" + "42/assets{?name,label}" + ) + response = mock.MagicMock() + response.__enter__.return_value.read.return_value = b"{}" + opener.open.return_value = response + with mock.patch.dict("os.environ", {"GH_TOKEN": "secret-token"}, clear=True): + client.upload_asset(42, upload_url, self.assets[0]) + + request = opener.open.call_args.args[0] + self.assertEqual( + request.full_url, + "https://uploads.github.com/repos/example/runtime/releases/42/" + "assets?name=runtime.tar.gz", + ) + self.assertEqual(request.method, "POST") + self.assertEqual(request.data, self.assets[0].read_bytes()) + self.assertEqual(request.get_header("Authorization"), "Bearer secret-token") + self.assertNotIn("secret-token", request.full_url) + + def test_release_asset_upload_refuses_cross_origin_redirect_without_token_egress( + self, + ) -> None: + opener = RedirectResponseOpener(MODULE.RefuseUploadRedirects()) + client = MODULE.GhReleaseClient( + endpoint="repos/example/runtime/releases", upload_opener=opener + ) + upload_url = ( + "https://uploads.github.com/repos/example/runtime/releases/" + "42/assets{?name,label}" + ) + with mock.patch.dict("os.environ", {"GH_TOKEN": "secret-token"}, clear=True): + with self.assertRaisesRegex( + MODULE.PublicationError, "refused an HTTP redirect" + ): + client.upload_asset(42, upload_url, self.assets[0]) + + self.assertEqual(len(opener.requests), 1) + self.assertEqual(opener.requests[0].host, "uploads.github.com") + self.assertEqual( + opener.requests[0].get_header("Authorization"), "Bearer secret-token" + ) + self.assertNotIn("attacker.example", opener.requests[0].full_url) + + def test_release_asset_upload_rejects_missing_token(self) -> None: + client = MODULE.GhReleaseClient(endpoint="repos/example/runtime/releases") + upload_url = ( + "https://uploads.github.com/repos/example/runtime/releases/" + "42/assets{?name,label}" + ) + with mock.patch.dict("os.environ", {}, clear=True): + with self.assertRaisesRegex(MODULE.PublicationError, "requires GH_TOKEN"): + client.upload_asset(42, upload_url, self.assets[0]) + + def test_release_asset_upload_rejects_wrong_scheme_host_and_id(self) -> None: + client = MODULE.GhReleaseClient(endpoint="repos/example/runtime/releases") + invalid_urls = { + "scheme": ( + "http://uploads.github.com/repos/example/runtime/releases/" + "42/assets{?name,label}" + ), + "host": ( + "https://github.com/repos/example/runtime/releases/" + "42/assets{?name,label}" + ), + "id": ( + "https://uploads.github.com/repos/example/runtime/releases/" + "43/assets{?name,label}" + ), + } + with mock.patch.dict("os.environ", {"GH_TOKEN": "secret-token"}, clear=True): + for label, upload_url in invalid_urls.items(): + with self.subTest(label=label): + with self.assertRaises(MODULE.PublicationError): + client.upload_asset(42, upload_url, self.assets[0]) + + def test_complete_equivalent_draft_is_discovered_before_create(self) -> None: + client = FakeClient() + client.seed( + self.metadata, + {path.name: path.read_bytes() for path in self.assets}, + ) + result = MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(result, "resumed") + self.assertEqual(client.create_calls, 0) + self.assertEqual(client.operations[:2], ["find", "snapshot"]) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, [42]) + + def test_multiple_same_tag_drafts_are_rejected_before_mutation(self) -> None: + client = FakeClient() + client.seed(self.metadata, {}) + client.matching_release_ids.append(43) + with self.assertRaisesRegex( + MODULE.PublicationError, "multiple releases match intended tag" + ): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.create_calls, 0) + self.assertEqual(client.snapshot_count, 0) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_partial_equivalent_draft_uploads_only_missing_then_publishes(self) -> None: + client = FakeClient() + client.seed(self.metadata, {self.assets[0].name: self.assets[0].read_bytes()}) + result = MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(result, "resumed") + self.assertEqual(client.uploaded, [self.assets[1].name]) + self.assertEqual(client.published, [42]) + + def test_changed_draft_immediately_before_publish_is_rejected(self) -> None: + client = FakeClient() + client.seed( + self.metadata, + {path.name: path.read_bytes() for path in self.assets}, + ) + + def change_body(fake: FakeClient) -> None: + assert fake.release is not None + fake.release["body"] = "concurrent foreign body" + + client.before_snapshot[2] = change_body + with self.assertRaisesRegex(MODULE.PublicationError, "body does not match"): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_post_publication_authority_breach_byte_mismatch_is_detected(self) -> None: + client = FakeClient() + client.seed( + self.metadata, + {path.name: path.read_bytes() for path in self.assets}, + ) + + def change_published_bytes(fake: FakeClient) -> None: + assert fake.release is not None + first_asset = fake.release["assets"][0] + fake.asset_bytes[first_asset["id"]] = b"X" * first_asset["size"] + + client.after_publish = change_published_bytes + with self.assertRaisesRegex( + MODULE.PublicationError, + "post-publication authority breach:.*bytes differ", + ): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.published, [42]) + + def test_post_publication_authority_breach_non_immutable_snapshot_is_detected( + self, + ) -> None: + client = FakeClient() + client.seed( + self.metadata, + {path.name: path.read_bytes() for path in self.assets}, + ) + + def keep_mutable(fake: FakeClient) -> None: + assert fake.release is not None + fake.release["immutable"] = False + + client.after_publish = keep_mutable + with self.assertRaisesRegex( + MODULE.PublicationError, + "post-publication authority breach:.*is not immutable", + ): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.published, [42]) + + def test_post_publication_authority_breach_metadata_mismatch_is_detected( + self, + ) -> None: + client = FakeClient() + client.seed( + self.metadata, + {path.name: path.read_bytes() for path in self.assets}, + ) + + def change_published_body(fake: FakeClient) -> None: + assert fake.release is not None + fake.release["body"] = "foreign published body" + + client.after_publish = change_published_body + with self.assertRaisesRegex( + MODULE.PublicationError, + "post-publication authority breach:.*body does not match", + ): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.published, [42]) + + def test_existing_asset_byte_mismatch_is_rejected_without_mutation(self) -> None: + client = FakeClient() + client.seed( + self.metadata, + {self.assets[0].name: b"X" * len(self.assets[0].read_bytes())}, + ) + with self.assertRaisesRegex(MODULE.PublicationError, "bytes differ"): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_extra_asset_is_rejected_without_mutation(self) -> None: + client = FakeClient() + client.seed(self.metadata, {"unexpected.txt": b"extra"}) + with self.assertRaisesRegex(MODULE.PublicationError, "unexpected assets"): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_draft_with_foreign_body_is_rejected_without_mutation(self) -> None: + client = FakeClient() + foreign = {**self.metadata, "body": "foreign release ownership"} + client.seed(foreign, {}) + with self.assertRaisesRegex(MODULE.PublicationError, "body does not match"): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.create_calls, 0) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_existing_published_immutable_equivalent_resumes_without_mutation( + self, + ) -> None: + client = FakeClient(draft=False) + client.seed( + self.metadata, + {path.name: path.read_bytes() for path in self.assets}, + ) + result = MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(result, "published-equivalent") + self.assertEqual(client.snapshot_count, 1) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + def test_incomplete_published_release_is_rejected_as_immutable(self) -> None: + client = FakeClient(draft=False) + client.seed(self.metadata, {self.assets[0].name: self.assets[0].read_bytes()}) + with self.assertRaisesRegex(MODULE.PublicationError, "is missing assets"): + MODULE.publish_component_release(client, **self.kwargs) + self.assertEqual(client.uploaded, []) + self.assertEqual(client.published, []) + + +if __name__ == "__main__": + unittest.main()