Repository navigation
ci: publish workspace contract checks and runbooks #126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| # Operating the public demo | ||
|
|
||
| The demo at <https://selvage-demo.dontblameme.dev> runs the tracked | ||
| [`deploy/`](../deploy) shape. For general self-hosting and TLS requirements, see | ||
| [Deploying](deploying.md); for page-serving privacy requirements, see | ||
| [Serving the page](serving-the-page.md). This guide describes the operational | ||
| contract, not a live host inventory. Read image versions, digests, credentials, | ||
| firewall rules, and installed files from their authoritative systems. | ||
|
|
||
| ## Installation and privilege boundary | ||
|
|
||
| The Compose project is `selvage-prod`, rooted at `/srv/selvage`. It contains the | ||
| TLS front, `selvaged`, and the separate page image. Only the front publishes a | ||
| port. The proxy configuration is baked into its locally built image; a proxy | ||
| change requires a rebuild, not just a restart. | ||
|
|
||
| Initial provisioning is an administrator's operation, not a CI deploy: | ||
|
|
||
| 1. Provision Docker Compose, the host's firewall, TLS origin certificate, and | ||
| Tailscale SSH access. Restrict origin ingress to the intended edge. Refresh | ||
| the firewall's Cloudflare ranges together with | ||
| [`cloudflare-ips.conf`](../deploy/proxy/conf.d/cloudflare-ips.conf). | ||
| 2. Install `deploy/compose.yaml` and `deploy/proxy/` from a reviewed commit into | ||
| `/srv/selvage`. Copy `deploy/.env.example` to `.env` once, edit it for this | ||
| instance, and make `.env` mode `0600`; never overwrite it during an upgrade. | ||
| 3. Install the origin certificate in `tls/origin.pem` (`root:root`, `0644`) and | ||
| key in `tls/origin.key` (`root:101`, `0640`), inside `tls/` (`root:root`, | ||
| `0750`). Certificates and keys are instance material, not repository files. | ||
| 4. Install `deploy/selvage-update.service` and `deploy/selvage-update.timer` in | ||
| `/etc/systemd/system/`, then run `systemctl daemon-reload`. Bring the shape | ||
| up from `/srv/selvage` with `docker compose up -d --build proxy`; enable the | ||
| timer only after checking the configured images and origin responses. | ||
| 5. Create the unprivileged `deployci` account. Install `deploy/deploy.py` as | ||
| `/usr/local/sbin/selvage-deploy` and `deploy/install_shape.py` as | ||
| `/usr/local/sbin/selvage-install-shape`, both `root:root`, `0755`. Validate | ||
| the corresponding `deployci.sudoers` and `installci.sudoers` with `visudo -c` | ||
| before installing them in `/etc/sudoers.d/` as `root:root`, `0440`. | ||
|
|
||
| The two sudoers commands must retain the trailing `""`: it permits **no | ||
| arguments**. Omitting that argument constraint permits arbitrary arguments. | ||
| Read the result with `sudo -l -U deployci`. Do not grant CI Docker-group | ||
| membership, blanket sudo, or permission to replace these scripts or sudoers | ||
| files. Changing either privileged program, the systemd units, `.env` settings | ||
| other than image references, or TLS material remains administrator work. | ||
|
|
||
| The image deployer accepts only tags of the two allowed GHCR images on stdin. | ||
| The shape installer accepts only `compose.yaml` and `proxy/` from a tar stream; | ||
| it rejects traversal, absolute paths, links, and special files. Shape access is | ||
| still powerful: a reviewed Compose file controls mounts, ports, and capabilities. | ||
| The installer does not delete obsolete proxy files; removed files need an | ||
| administrator's cleanup before rebuilding the front. | ||
|
|
||
| Instance-specific `compose.override.yaml` and notice files are not installed by | ||
| CI. A notice mounted into the front must survive shape upgrades. In particular, | ||
| nginx `sub_filter` directives at a more specific location replace the inherited | ||
| list rather than appending to it; `nginx -t` alone cannot prove a notice renders. | ||
|
|
||
| ## CI access | ||
|
|
||
| [deploy-prod.yml](../.github/workflows/deploy-prod.yml) is dispatched at `main`. | ||
| It has **no environment approval gate**. Tailscale workload identity federation | ||
| matches the GitHub OIDC subject exactly: | ||
|
|
||
| ```text | ||
| repo:selvage-protocol@328844633/reference_server@1368869493:ref:refs/heads/main | ||
| ``` | ||
|
|
||
| The numeric IDs are part of the organization's immutable subject form. Adding | ||
| an `environment:` to this job changes the subject and breaks the exchange. | ||
| Repository secrets `TS_OAUTH_CLIENT_ID` and `TS_AUDIENCE` name the credential | ||
| pair; their values belong in the credential console and GitHub, not this guide. | ||
| The ephemeral runner receives `tag:ci`. Tailnet policy must restrict it to SSH on | ||
| `tag:selvaged-prod`, with the SSH rule permitting only `deployci`. Tailscale SSH | ||
| cannot constrain the command: the host's two argument-free sudo rules are the | ||
| command boundary. Tailnet Lock requires a separate authentication design if | ||
| enabled; do not silently replace federation with a broad reusable key. | ||
|
|
||
| To rotate the pair, create a replacement identity with the same constrained | ||
| subject, update both repository secrets, verify a deploy, then remove the old | ||
| identity. An exchange refusal needs the actual claims and credential inspected: | ||
| a missing identity and a rejected subject are different failures. Never loosen | ||
| the subject or host privileges to make a retry pass. | ||
|
|
||
| ## Deploy and verify | ||
|
|
||
| From any checkout, explicitly name this repository: | ||
|
|
||
| ```sh | ||
| gh workflow run deploy-prod.yml -R selvage-protocol/reference_server --ref main \ | ||
| -f server_version=<version> | ||
| gh workflow run deploy-prod.yml -R selvage-protocol/reference_server --ref main \ | ||
| -f server_version=<version> -f install_shape=true | ||
| ``` | ||
|
|
||
| An empty image input leaves that line unchanged; `latest` unpins it. The server | ||
| release changes its own `SELVAGED_IMAGE` line. The page repository normally | ||
| changes `SELVAGE_WEB_IMAGE` independently. The deploy accepts tags, not digests: | ||
| a version tag is a deliberate pin but is still mutable by a registry writer. | ||
|
|
||
| With `install_shape=true`, CI streams the tracked pair from the run's `main` | ||
| ref, installs it under the update lock, and reads installed file hashes back. | ||
| Matching files are a no-op; a Compose-only change runs `up -d`; a proxy change | ||
| rebuilds the front. To install the same reviewed shape by hand from this repo: | ||
|
|
||
| ```sh | ||
| git archive --format=tar <sha>:deploy compose.yaml proxy \ | ||
| | ssh deployci@selvage-protocol-prod \ | ||
| '/usr/bin/sudo /usr/local/sbin/selvage-install-shape' | ||
| ``` | ||
|
|
||
| The image deploy holds `/srv/selvage/.update.lock`, stages `.env.deploy`, and | ||
| writes `.env` only after convergence, keeping `.env.prev`. A failed deploy can | ||
| already have changed running containers even though `.env` is unchanged; inspect | ||
| both, then reconcile with the intended references. A healthy origin is asserted | ||
| through the front over SSH: `/meta` must report the requested server version | ||
| when one was specified, and `/` must answer successfully. Read current state: | ||
|
|
||
| ```sh | ||
| ssh selvage@selvage-protocol-prod 'cd /srv/selvage && docker compose ps && docker compose images' | ||
| ssh selvage@selvage-protocol-prod \ | ||
| "curl -sk -H 'Host: selvage-demo.dontblameme.dev' https://127.0.0.1/meta" | ||
| ``` | ||
|
|
||
| The public edge may challenge a datacenter client. Verification reports a | ||
| challenge or unreadable public response; it does not bypass bot controls or | ||
| claim that the public path passed. A readable public version different from the | ||
| requested version fails the deploy. `latest` has no expected version to compare. | ||
| A page answering 200 is **not proof of its build identity**. Neither these reads | ||
| nor local proxy tests verify the separately hosted site's deployed response | ||
| headers or their limits. Read those from the deployed response before claiming | ||
| them. Automated checks do not establish human acceptance. | ||
|
|
||
| ## Timer, recovery, and downtime | ||
|
|
||
| The update timer pulls configured tags every five minutes under the same lock. | ||
| A line left at `latest` follows registry updates without another approval. | ||
| Inspect with `systemctl list-timers selvage-update.timer` and | ||
| `journalctl -u selvage-update.service -n 50`. | ||
|
|
||
| For manual operations, run Compose as the instance operator in `/srv/selvage`, | ||
| without a `-f` override, so it reads the same `.env` and optional override as the | ||
| timer. **Stop the timer before stopping services** or its next tick undoes the | ||
| stop. To keep it stopped across boots, disable it as well. | ||
|
|
||
| ```sh | ||
| sudo systemctl stop selvage-update.timer | ||
| cd /srv/selvage | ||
| docker compose down | ||
| # Restore the reviewed shape and intended image references before restarting. | ||
| docker compose pull --ignore-buildable | ||
| docker compose up -d --build proxy | ||
| sudo systemctl start selvage-update.timer | ||
| ``` | ||
|
|
||
| Rollback is an explicit pin to a known previous image tag followed by pull and | ||
| up. After a successful deploy, `.env.prev` is the previous environment; inspect | ||
| it before copying it back. Restore older proxy files from a reviewed commit and | ||
| rebuild for a front rollback. These are recovery procedures, not a claim that a | ||
| live rollback has been exercised. | ||
|
|
||
| Replacing `selvaged` ends every room: all state is in memory, with no drain or | ||
| restart reclaim. A front replacement drops WebSockets and requires clients to | ||
| reconnect; it does not itself restore lost server state. Coordinate disruptive | ||
| changes with users rather than treating a green deploy as a seamless upgrade. | ||
|
|
||
| The repository's [publishing guide](publishing.md) covers release/image recovery. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| # Publishing the server | ||
|
|
||
| This repository versions the server independently of the clients. `Cargo.toml`'s | ||
| workspace version owns the release number; `scripts/bump-version.sh` updates it, | ||
| and `scripts/release-tags.sh` asserts image/tag agreement. A server release does | ||
| not change the protocol's wire version. | ||
|
|
||
| ## Dispatch and gates | ||
|
|
||
| [release.yml](../.github/workflows/release.yml) is dispatch-only at `main`: | ||
|
|
||
| ```sh | ||
| gh workflow run release.yml -R selvage-protocol/reference_server --ref main \ | ||
| -f bump=patch -f dry_run=true | ||
| gh workflow run release.yml -R selvage-protocol/reference_server --ref main \ | ||
| -f bump=patch | ||
| ``` | ||
|
|
||
| Choose `patch`, `minor`, or `major` deliberately. The read-only plan requires an | ||
| exact remote tag for the current manifest version and no tag for the target. | ||
| The write job rechecks these preconditions before bumping; a retried write job | ||
| cannot rely on the plan's earlier remote read. Dry run prints the plan and stops | ||
| before modifying the tree, running the build gates, or publishing anything. | ||
|
|
||
| Merge with green [CI](../.github/workflows/ci.yml), including the public | ||
| workspace-contract job comparing vendored vectors and bounds with the | ||
| specification. The release runs its repository-local gate on the bumped tree | ||
| before an atomic push of its commit and lightweight tag. It never force-pushes | ||
| or moves a release tag. The workflow commits as the owner; it does not open a | ||
| bump PR whose checks would never start under its own `GITHUB_TOKEN`. | ||
|
|
||
| [image.yml](../.github/workflows/image.yml) builds and publishes the server-only | ||
| image for `linux/amd64` and `linux/arm64`. Its tag-gated publish uses | ||
| `GITHUB_TOKEN` with `packages: write`; no separate owner token or approval | ||
| environment is configured in that workflow. Tags are `<version>-<short-sha>`, | ||
| `<version>`, and, for non-prereleases, `latest`. The release explicitly | ||
| dispatches the image workflow because a tag push using `GITHUB_TOKEN` does not | ||
| start it automatically. | ||
|
|
||
| ## Deployment and recovery | ||
|
|
||
| After the image becomes available, release dispatches `deploy-prod.yml` at | ||
| `main` for its own server version. It sets `install_shape=true` when `deploy/` | ||
| changed, waits for the resulting deploy run, and fails unless that run succeeds. | ||
| A published tag or image can therefore remain after a red release. A green run | ||
| means the deploy's assertions passed, not that human acceptance or a live | ||
| rollback was performed. The host privilege boundary, credentials by name, | ||
| verification limits, and rollback procedure are in | ||
| [Operating the demo](operating-the-demo.md). | ||
|
|
||
| Recover the failed stage rather than dispatching a new bump blindly: | ||
|
|
||
| - **Tag exists, image missing:** inspect the original image run and rerun the | ||
| workflow at the existing tag: | ||
|
|
||
| ```sh | ||
| gh workflow run image.yml -R selvage-protocol/reference_server --ref 'v<version>' | ||
| ``` | ||
|
|
||
| The image workflow reasserts tag/manifest agreement. Do not move the tag to | ||
| include a packaging fix; fix forward with a new version instead. | ||
| - **Image exists, demo deploy failed:** inspect the deploy's origin/read-back | ||
| failure, then retry the bounded deploy at `main` with the intended version | ||
| and shape input. Another release dispatch would compute a new version. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| - **Image exists, GitHub Release missing:** the image workflow creates the | ||
| GitHub Release in a separate job; the image and even the deployed demo can | ||
| exist while that job is red. Inspect its failure, then rerun `image.yml` at | ||
| the existing `v<version>` tag as above to create the missing release. Do not | ||
| dispatch another version bump merely to recover the release record. | ||
| - **Atomic push refused:** reconcile with current `main` and tags before | ||
| planning again. Do not force the old candidate over concurrent work. | ||
|
|
||
| Verify the artifact and deployed version, not just a tag's existence: | ||
|
|
||
| ```sh | ||
| gh release view 'v<version>' -R selvage-protocol/reference_server --json tagName,assets | ||
| docker run --rm ghcr.io/selvage-protocol/selvaged:<version> --version | ||
| ``` | ||
|
|
||
| Inspect the published multi-platform manifest and its per-platform layers when | ||
| checking architecture claims. A two-platform manifest alone does not prove the | ||
| binaries inside differ. The image workflow's smoke and layer checks cover that | ||
| artifact contract; origin `/meta` checks cover the running server version. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.