[anth-volk]

Fixes #4001

Summary

On 2026-10-20 it was disclosed that the private keys of eight eMudhra (emSign) intermediate CA certificates can be recovered from the public certificates (https://emudhra.we-have-a-problem.com). Anyone can issue a certificate for any domain that chains to "emSign ECC Root CA - G3" or, through Sectigo's cross-signatures, to "AAA Certificate Services", so a client that carries its own CA list (or reads an outdated operating-system store) and does not check revocation can have its HTTPS traffic intercepted. No exploitation has been reported. Browsers and current Windows and macOS already block these certificates; the exposure is in server-side clients like the ones in the Public API.

What in the Public API trusted the affected roots

Established from uv.lock, the Dockerfiles, the workflows, the application code and the Actions run history on master at 392e653:

• certifi 2026.4.22 (uv.lock) includes emSign ECC Root CA - G3. httpx uses it for the Auth0 token request (libs/gateway_auth.py), the Auth0 Management API client (libs/auth0_management.py), the simulation service client (libs/simulation_entrypoint.py) and Hugging Face downloads (services/economy_comparison.py). Checked: httpx 0.28.1 with certifi 2026.4.22 completed a request to https://dv-g3b-emudhra.we-have-a-problem.com/ (HTTP 200).
• Operating-system CA store in the Cloud Run image. Production was last built on 2026-10-20 (run 39467627051) from python:3.12-slim@sha256:a6e34c59…, which has Debian ca-certificates 20250419 with the affected roots. PyJWT's Auth0 signing-key fetch (libs/user_auth.py) reads this store. gcp/cloud_run/Dockerfile never installs ca-certificates, so the image only gets a fixed store when the base image happens to carry one.
• grpcio 1.80.0 bundles its own CA list (grpc/_cython/_credentials/roots.pem), which contains both emSign ECC Root CA - G3 and AAA Certificate Services. The Google Cloud Logging, Secret Manager and OTLP export clients use it. No grpcio release removes them yet (fixed in source in [security] Update gRPC repo roots. grpc/grpc#43699).
• uv 0.12.1 (Cloud Run image build, the versioning job in push.yml, the collector deploy workflow) uses a compiled-in CA list. Checked: uv 0.12.1 completed the TLS handshake with the emSign test endpoint; uv 0.13.4 rejects it with invalid peer certificate: UnknownIssuer.
• otelcol-google 0.160.0 (observability collector image) bundles both affected roots; 0.163.1 does not.

Not affected: the Redis cache client trusts only the Memorystore CA from RUNTIME_CACHE_CA_CERT, and Cloud SQL connections use the instance's own CA. docker/Dockerfile (ghcr.io/policyengine/policyengine) needs no change: today's python:3.12 base has ca-certificates 20250419+deb13u1 and pip install resolves certifi 2026.10.19, so the next build of that image is fixed.

Changes

• uv.lock: certifi 2026.4.22 → 2026.10.19 (uv lock --upgrade-package certifi with uv 0.13.4; no other package changed). uv 0.13.4 also raises the lockfile revision from 3 to 5; uv 0.12.1 still accepts the file with uv lock --locked.
• gcp/cloud_run/Dockerfile:
  • install ca-certificates in the existing apt-get install, so every build takes Debian's current CA store even when the base image is older;
  • set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt, so gRPC loads that bundle instead of the list bundled in grpcio;
  • copy uv from ghcr.io/astral-sh/uv:0.13.4 instead of 0.12.1.
• gcp/observability/collector/Dockerfile and the config validation step in .github/workflows/deploy-observability-collector.yml: otelcol-google 0.160.0 → 0.163.1.
• .github/workflows/push.yml (versioning job) and .github/workflows/deploy-observability-collector.yml: setup-uv version 0.12.1 → 0.13.4. tests/unit/test_alembic_workflows.py asserts the versioning job's uv version and is updated to match.
• changelog.d/emudhra-ca-trust-update.fixed.md.

Testing

• uv lock --locked passes with uv 0.13.4 and with uv 0.12.1.
• pytest tests/unit/test_alembic_workflows.py tests/unit/test_cloud_run_deploy_scripts.py: 142 passed.
• make format (no files changed), ruff check tests/unit/test_alembic_workflows.py: passed.
• python scripts/run_quality_guards.py: passed.
• docker build -f gcp/cloud_run/Dockerfile . (local, linux/arm64): succeeded. The image has ca-certificates 20250419+deb13u1, certifi 2026.10.19, grpcio 1.80.0, uv 0.13.4 and GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt.
• Inside that image:
  • httpx (certifi) and the standard library's default SSL context (operating-system store) both fail with CERTIFICATE_VERIFY_FAILED against dv-g3b-emudhra.we-have-a-problem.com and dv-g3b-aaa-emudhra.we-have-a-problem.com, and both still reach www.google.com.
  • Parsing each CA file: the Debian bundle and certifi 2026.10.19 contain neither emSign ECC Root CA - G3 nor AAA Certificate Services; grpcio's bundled list contains both.
  • gRPC reads the file named by GRPC_DEFAULT_SSL_ROOTS_FILE_PATH: pointed at a file holding only ISRG Root X1, a call to www.google.com:443 fails with CERTIFICATE_VERIFY_FAILED; with the image's setting, TLS completes to logging.googleapis.com, secretmanager.googleapis.com, telemetry.googleapis.com and a run.app host (the probe RPC returns UNIMPLEMENTED from the server).
  • The eMudhra test endpoints close gRPC connections before certificate verification (they do not negotiate HTTP/2), so gRPC's rejection of them could not be observed directly; it follows from the two results above.
• On the old production base python:3.12-slim@sha256:a6e34c59…, apt-get install --no-install-recommends ca-certificates upgrades 20250419 to 20250419+deb13u1.
• otelcol-google:0.163.1 validate --config=…/config.yaml exits 0 (a missing config file exits 1 as a control).

Operator actions after merge

1. Before merge: open a tracking issue and put Fixes #ISSUE_NUMBER as the first line of this description.
2. Merging runs the Push workflow; its versioning job commits "Update PolicyEngine API", and that commit's Push run builds the Cloud Run image from this Dockerfile, deploys staging, promotes it, deploys production and rebuilds ghcr.io/policyengine/policyengine. Watch that run until production is deployed. (The release run already in progress for 392e653 builds from the refreshed python:3.12-slim, which fixes the operating-system store, but it still ships certifi 2026.4.22 and grpcio's bundled roots.)
3. Confirm production serves the new revision: gcloud run services describe policyengine-api --project policyengine-api --region us-central1 --format='value(status.latestReadyRevisionName,status.traffic)', and compare the revision's image with the one the run pushed. Do the same for policyengine-api-staging.
4. Merging also starts "Deploy observability collector" (its path filter covers the collector files and the workflow). On its last four runs the build-and-deploy step succeeded but the verification step failed (gcloud.auth.print-identity-token: Invalid account type for --audiences, run 39305297785), so confirm with gcloud run services describe on the collector service that its latest ready revision uses the image built from 0.163.1. If the workflow does not start, run it with workflow_dispatch.

Environment variables

This PR changes a Dockerfile, a container image and two workflows, so each variable the changed code reads or declares was traced:

Variable	Read or declared by	Provider	How it reaches the code	Status
GRPC_DEFAULT_SSL_ROOTS_FILE_PATH (new)	gcp/cloud_run/Dockerfile ENV; read by gRPC in the API process	The image itself (fixed value)	Image ENV. deploy_cloud_run_candidate.sh sets no variable of that name with --set-env-vars, so the image value applies at runtime. Confirmed in the local build.	verified
PYTHONDONTWRITEBYTECODE, PYTHONUNBUFFERED, UV_PROJECT_ENVIRONMENT, PATH, GATEWAY_AUTH_REQUIRED (unchanged)	gcp/cloud_run/Dockerfile ENV	The image itself (fixed values)	Image ENV	verified
OBSERVABILITY_PROJECT_ID, OBSERVABILITY_METRIC_LOCATION	Collector config validation step (changed image tag); collector config.yaml at runtime	GitHub Actions variables in the production environment (OBSERVABILITY_PROJECT_ID also at repository level)	Job env: from vars.*; the job declares environment: production; passed to the validation container with --env and to the Cloud Run service with --set-env-vars	verified (variables exist; the 2026-10-19 run's validation and deploy steps succeeded with them)
OBSERVABILITY_COLLECTOR_REGION, OBSERVABILITY_COLLECTOR_SERVICE, OBSERVABILITY_COLLECTOR_ARTIFACT_REPOSITORY, OBSERVABILITY_COLLECTOR_IMAGE_NAME, OBSERVABILITY_COLLECTOR_SERVICE_ACCOUNT (unchanged)	deploy_observability_collector.sh	GitHub Actions variables in the production environment	Job env: from vars.*	verified (variables exist; deploy step succeeded on 2026-10-19)
GCP_WORKLOAD_IDENTITY_PROVIDER, GCP_DEPLOY_SERVICE_ACCOUNT (unchanged)	google-github-actions/auth in the collector job	GitHub Actions repository secrets	with: inputs	verified (secrets exist; authentication step succeeded on 2026-10-19)
APP_ID, APP_PRIVATE_KEY (unchanged)	actions/create-github-app-token in the versioning job	GitHub Actions organization secrets available to this repository	with: inputs	verified (secrets listed for the repository; versioning job succeeded in run 39629179225)
OLD_SIMULATION_GATEWAY_URL, ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION (unchanged)	push.yml workflow-level env:	Repository variable; fixed value in the workflow	Workflow env:	verified

The setup-uv steps read no variables of their own beyond the default github.token that GitHub Actions provides.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.00%. Comparing base (392e653) to head (62de73a).
⚠️ Report is 85 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3999       +/-   ##
===========================================
+ Coverage   49.24%   86.00%   +36.76%     
===========================================
  Files         243      259       +16     
  Lines       14398    15777     +1379     
  Branches     2343     2628      +285     
===========================================
+ Hits         7090    13569     +6479     
+ Misses       6682     1456     -5226     
- Partials      626      752      +126     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.