Skip to content

Promote each staging deployment's tested entrypoint revision - #854

Closed
anth-volk wants to merge 2 commits into
mainfrom
apiv2/staging-promotion
Closed

anth-volk wants to merge 2 commits into
mainfrom
apiv2/staging-promotion

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #853

Summary

  • simulation-deploy.yml (beta) and stage12-deploy.yml (deploy-stage12-staging) now pass promote_entrypoint: true, as their production jobs already do.
  • No change to simulation-deploy.reusable.yml. Staging now runs the same promote_entrypoint job as production:
    1. Check that stable traffic still points at the revision recorded before deployment, and that the tested revision belongs to the staging service and is Ready.
    2. Move 100% of stable traffic to that exact revision.
    3. Check the stable URL's /health revision header, /ready, /versions, the release-identity fields and /ping.
    4. Reproduce the release sentinel numbers (full-stack deploys only).
    5. Run the authenticated tests against the stable URL.
    6. Restore the previous revision if any check fails.
  • cleanup_entrypoint_revisions now also runs for staging. It keeps the promoted revision and the previous one, and deletes the rest. The first promoted run deletes about 67 old tagged staging revisions.
  • prod still starts only after the whole beta job, now including promotion, succeeds.
  • A test pins both staging callers to promote_entrypoint: true; the old assertion that one caller passed false is replaced. Docs and changelog_entry.yaml updated.
  • The test is registered as rule sim.deploy.staging-promotes-tested-revision in rules/registry.toml, with docs/generated/rules.md and the AGENTS.md rules block re-rendered.

Why

See #853. The staging stable URL has served Cloud Run's placeholder hello revision since the service was created. Every staging request in the last 5,000 log entries went to a tagged candidate URL. Callers that use the stable URL cannot reach a tested build. Those callers are the Public API's staging service, once it is pointed at staging sim, and the job store's Cloud Tasks and Cloud Scheduler calls.

Before merging: one operator step

The stable-URL smoke step parses /health as JSON and tolerates only the previous revision's header while traffic settles. The placeholder revision returns HTML. So if the stable URL serves it for even one request after the traffic change, the first promoted run fails. It then restores the placeholder and fails beta, which blocks prod. Move staging traffic once, before merging, to the newest staging revision that passed its deployment tests. That is policyengine-simulation-entry-staging-00069-saj (tag s-68), the #712 build that passed run 39309691903's beta deploy and authenticated tests:

SIMULATION_ENTRYPOINT_GCP_PROJECT_ID=policyengine-simulation-entry \
SIMULATION_ENTRYPOINT_GCP_REGION=us-central1 \
SIMULATION_ENTRYPOINT_SERVICE=policyengine-simulation-entry-staging \
SIMULATION_ENTRYPOINT_TARGET_REVISION=policyengine-simulation-entry-staging-00069-saj \
SIMULATION_ENTRYPOINT_EXPECTED_CURRENT_REVISION=policyengine-simulation-entry-staging-00001-cuh \
  .github/scripts/set-cloud-run-simulation-entry-revision.sh
.github/scripts/cloud-run-simulation-entry-smoke.sh \
  https://policyengine-simulation-entry-staging-4xy55an4yq-uc.a.run.app \
  policyengine-simulation-entry-staging-00069-saj

If a newer staging deployment runs first, use its tested revision instead.

Environment variables and secrets used by the staging promotion

No new variable or secret. The staging promotion and cleanup jobs read only values the beta environment already supplies to its candidate tests:

Name Provider Wiring Status
SIMULATION_ENTRYPOINT_GCP_PROJECT_ID, SIMULATION_ENTRYPOINT_GCP_REGION beta environment variables job env: in promote_entrypoint and cleanup_entrypoint_revisions verified (values set)
SIMULATION_ENTRYPOINT_GCP_WIF_PROVIDER, SIMULATION_ENTRYPOINT_GCP_DEPLOY_SERVICE_ACCOUNT beta environment secrets google-github-actions/auth inputs, via secrets: inherit secrets exist. The project's only deploy service account with a Cloud Run role is sim-entry-gh-deployer (roles/run.admin, enough for traffic updates and revision deletes). The secret's value can't be read, so the mapping is inferred.
SIMULATION_ENTRYPOINT_AUTH_ISSUER, SIMULATION_ENTRYPOINT_AUTH_AUDIENCE, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET beta environment secrets step env: for the stable-URL sentinel and authentication checks secrets exist; the same values already serve beta's authenticated_test job
SIMULATION_ENTRYPOINT_ENFORCE_PERMISSIONS beta environment variable (unset, so it deploys 0) step env: unchanged
SIMULATION_ENTRYPOINT_CUSTOM_URL beta environment variable job env: unset in beta, so the custom-hostname steps are skipped

The promoted revision carries its own runtime environment and its runtime service account (sim-entry-beta-runtime), both set when the candidate was deployed. The staging service's ingress is all, and its roles/run.invoker binding is allUsers, the same as production. Callers authenticate in the application with Auth0 tokens.

Tests

  • New test_staging_deployments_promote_the_tested_entrypoint_revision (both staging callers). It failed before the workflow change (assert False is True, two cases) and passes after.
  • projects/policyengine-simulation-entry: uv run pytest tests/ passed (536 tests).
  • python -m unittest discover -s scripts/rules/tests -t . and python scripts/run_quality_guards.py (shellcheck 0.11.0.1) passed. Formatted with ruff format; ruff check clean on the changed test.

Staging tests that follow from this

  • Once merged, every full-stack staging deploy runs the stable-URL smoke, release-sentinel and authentication checks against the staging stable URL.
  • Pointing the Public API's staging service at that URL (a separate Public API change, plus an environment-secret update) makes its staging live tests run against staging sim and staging Modal instead of production.
  • An end-to-end test of background result collection (submit a job, let Cloud Tasks collect it, read the stored result through the stable URL) cannot be written yet. No job kind is registered and the job store has no submit route until the first API v2 migration PR that registers a job kind. That PR carries the test as an acceptance requirement.

🤖 Generated with Claude Code

Staging deployments tested a tagged candidate and left the staging
service's stable traffic on its first revision, Cloud Run's placeholder
image. Callers of the staging stable URL, such as the Public API's
staging service and the job store's Cloud Tasks and Cloud Scheduler
calls, could therefore never reach a tested build.

Both staging deploy callers now promote, as production does: the
promotion job moves stable traffic to the exact tested revision, runs
the stable-URL smoke, release-sentinel and authentication checks, and
restores the previous revision if they fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rules registry requires every deployment guard test to belong to a
rule. Register sim.deploy.staging-promotes-tested-revision for the new
staging promotion test and render the rule views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk anth-volk closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Staging deployments leave the staging stable URL on Cloud Run's placeholder revision

1 participant