Skip to content

Check that each release calls its own environment's simulation service - #3982

Closed
anth-volk wants to merge 1 commit into
masterfrom
apiv2/staging-simulation-entrypoint
Closed

anth-volk wants to merge 1 commit into
masterfrom
apiv2/staging-simulation-entrypoint

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #3981

Summary

  • New .github/scripts/check_simulation_entrypoint_environment.sh:
    • it requests /health from the URL that SIM_ENTRYPOINT selects and reads the x-policyengine-simulation-revision header;
    • it removes Cloud Run's -NNNNN-xxx suffix to get the service name;
    • it fails unless that service is policyengine-simulation-entry-staging (staging) or policyengine-simulation-entry (production).
    • A missing header, which is what Cloud Run's placeholder image returns, also fails. old_gateway_direct is not checked, because the old gateway is one shared Modal app that reports no environment.
  • push.yml: both ensure-*-model-version-aligns-with-sim-api jobs set DEPLOYMENT_ENVIRONMENT and run the check first. These jobs already run before every release's database migrations and deploys.
  • docs/migration/cloud-run-operations.md documents which URL each environment's secret holds, and the check. Changelog fragment added.

Migration scope: no route groups or user-visible API contracts change. Only the release workflow and its documentation do.

Merge order (do not merge first)

Today the staging secret points at production, so the new check would fail every release at its staging step. Merge only after these steps:

  1. Sim, before Promote each staging deployment's tested entrypoint revision policyengine-sim-api#854 merges: move staging traffic off the placeholder revision. The exact commands are in that PR.
  2. Merge Promote each staging deployment's tested entrypoint revision policyengine-sim-api#854. Its staging deployment then promotes a tested revision and verifies the staging stable URL.
  3. Confirm the staging stable URL serves a staging revision:
    curl -sSi https://policyengine-simulation-entry-staging-4xy55an4yq-uc.a.run.app/health | grep -i x-policyengine-simulation-revision
    # expect: policyengine-simulation-entry-staging-<number>-<suffix>
  4. Update the staging environment secret (a GitHub setting):
    gh secret set SIMULATION_ENTRYPOINT_URL \
      --repo PolicyEngine/policyengine-api \
      --env staging \
      --body "https://policyengine-simulation-entry-staging-4xy55an4yq-uc.a.run.app"
    Leave the production secret unchanged.
  5. Merge this PR. Merging starts a release. Its staging step now runs the check against staging sim. The staging candidate is deployed with the new URL, and the staging live tests run against staging sim and the staging Modal environment.

Rollback: set the staging secret back to https://simulation.api.policyengine.org and revert this PR.

Environment variables

Name Provider Wiring Status
DEPLOYMENT_ENVIRONMENT literal in push.yml (staging / production) job env: of each ensure-* job verified (in this diff)
SIM_ENTRYPOINT staging / production environment variable, both cloud_run_simulation_entrypoint job env: (${{ vars.SIM_ENTRYPOINT }}) in jobs bound to that environment verified
SIMULATION_ENTRYPOINT_URL staging / production environment secret job env: (${{ secrets.SIMULATION_ENTRYPOINT_URL }}); the deploy jobs pass it to Cloud Run through deploy_cloud_run_candidate.sh the secrets exist and both runtime services resolve them to the production URL. The staging value changes in step 4.
GATEWAY_AUTH_ISSUER, GATEWAY_AUTH_AUDIENCE, GATEWAY_AUTH_CLIENT_ID, GATEWAY_AUTH_CLIENT_SECRET_RESOURCE repository secrets (the last names a Secret Manager version) deploy-job step env:, then Cloud Run env through deploy_cloud_run_candidate.sh; not read by the new script unchanged. Staging and production use the same Auth0 issuer, audience and client. The staging simulation service accepts the same issuer and audience, so no authentication change is needed.

The staging simulation service's Cloud Run invoker role is allUsers, and it authenticates callers in the application, as production does. No IAM change is needed.

Testing

  • tests/unit/test_simulation_entrypoint_environment_script.py (new, 11 tests). The script runs under subprocess with a fake curl first on PATH. The tests cover a matching staging and production service, staging calling production, production calling staging, the placeholder response, a malformed revision name, a failed request, an unknown environment, old_gateway_direct, and the wiring in both push.yml jobs. Before the script existed, 10 failed; all 11 pass now.
  • The script was also run read-only against the live /health endpoints:
    • staging with the production URL fails: "The staging deployment calls https://simulation.api.policyengine.org, which revision policyengine-simulation-entry-00122-low of policyengine-simulation-entry serves";
    • production with the production URL passes;
    • staging with the staging stable URL fails today, because the placeholder returns no revision header.
  • tests/unit/test_cloud_run_deploy_scripts.py, test_ci_complete_gate.py and the new file: 143 passed.
  • python scripts/run_quality_guards.py: passed, including ci-scripts with shellcheck 0.11.0.1.
  • ruff format and ruff check are clean on the new test file.

🤖 Generated with Claude Code

The staging and production releases both read SIMULATION_ENTRYPOINT_URL
from their GitHub environment, and both secrets point at the production
Simulation Entrypoint, so staging's live simulation tests ran on
production simulation workers.

Before each release deploys, a new script reads the revision header
from the selected URL's /health response and fails unless the revision
belongs to that environment's service: policyengine-simulation-entry-staging
for staging, policyengine-simulation-entry for production. The shared
old gateway reports no environment and is not checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.08%. Comparing base (14707a9) to head (dd1bc61).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #3982       +/-   ##
===========================================
+ Coverage   46.17%   84.08%   +37.91%     
===========================================
  Files         239      255       +16     
  Lines       13987    15368     +1381     
  Branches     2272     2555      +283     
===========================================
+ Hits         6458    12922     +6464     
+ Misses       6958     1686     -5272     
- Partials      571      760      +189     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk anth-volk closed this Oct 20, 2026

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The staging release calls the production simulation service

1 participant