[anth-volk]

Fixes #3989

Merge on or after 2026-10-23

Do not merge before 2026-10-23. The first live report must cover a full US Eastern day that started after #3984 reached production. Before #3984, an unhandled Flask 500 has no exception record, so its failures would group as "HTTP 500 without recorded exception" and open issues that stop matching once #3984 is deployed. #3984 merged on 2026-10-20. If it is not in production by the end of 2026-10-21 Eastern, move this date later by the same number of days.

Right after merging, before 00:45 Eastern that night:

1. gh workflow run telemetry-error-issues.yml -f dry_run=true. This first run confirms the value of the existing GCP_WORKLOAD_IDENTITY_PROVIDER secret, which can't be read. Check that its log and job summary contain no message or stack text.
2. gh workflow run telemetry-error-issues.yml -f dry_run=false, then review the issues it opens.

From the next night on, the schedule runs it. Scheduled runs are never dry runs.

Summary

• .github/scripts/sync_telemetry_error_issues.py (standard library only):

  • Window: reports one US Eastern calendar day, midnight to midnight; days when the clocks change last 23 or 25 hours. --report-date YYYY-MM-DD reports another day, and refuses a day that hasn't ended.

  • Queries: reads the linked dataset's _AllLogs view with two named-parameter queries, each limited to 5 GiB billed.

    • ERROR records with the policyengine.api-v1 namespace, for the day and the seven days before it, excluding health-check routes, with duplicate deliveries removed.
    • request.completed counts per route for the day, for error rates.
    • Only code-defined or normalized values leave BigQuery: stack frames (path", line N, in function headers) from the stack's first traceback, a truncation flag, a message's first three whole words outside quoted values, and SHA-256 hashes of request IDs, observability IDs and traces. A single quote opens a value only at the start or after a character that isn't a letter or digit, so apostrophes don't. The patterns are query parameters, and the tests apply the same ones.

  • One record per failure: records of one service that share a request ID (or a trace) are one failure. The failure is represented by its exception record, else its error message, else its failed request.completed record.

  • Fingerprints: 16 hex characters of HMAC-SHA256, algorithm 1, keyed by TELEMETRY_ERROR_FINGERPRINT_KEY.

    Kind	Key parts
    exception	Service family, exception class, three innermost frames in policyengine_api, policyengine_fastapi, policyengine_simulation_* or policyengine_stage12_* (path and function, no line), from the stack's first traceback: for a chained exception, its first cause. The logging library keeps only the first 16,384 characters of a stack, so the final traceback can be dropped but the first is always kept. The route is left out, so one shared failure opens one issue.
    message	Service family, route or operation, message template, error_type and failure_category attributes
    failed_request	Service family, route, HTTP status

    Versioned simulation services (policyengine-simulation-py<x>-<y>-<z>) form one family, and the Modal gateway's main environment counts as production.

  • Measures, for production and staging: occurrences, distinct affected requests, observability IDs, error rate on the affected routes (affected requests that have a route, over completed requests on those routes), occurrences per hour in the day and over the seven days before, first and last seen, and up to three related fingerprints that share traces.

  • Issues:

    • Opening: a production group opens an issue at 3 occurrences in at least 2 requests, at most 5 per run, largest first.
    • Matching: by the fingerprint in a hidden state comment, or in the title, among open and closed issues by github-actions[bot].
    • Updating: each run replaces only the marked section, and only when it changed. Titles and text outside the section are never changed, and a newer day's section is never overwritten by a rerun of an earlier day.
    • Reopening: an issue closed as completed reopens with regression and a comment when production failures that start 24 hours after it closed meet the same thresholds. Issues closed as not planned or duplicate stay closed.
    • Labels: telemetry-error, service: <family> and regression; missing labels are created, and existing labels match regardless of case, as GitHub matches them.
    • Writes go one at a time, at least a second apart. Rate-limited 403 and 429 responses are retried as GitHub advises: after retry-after, until x-ratelimit-reset when no requests remain, or after a minute.

  • Public output:

    • Titles, bodies, the job summary and logs contain only fingerprints, counts, validated code-defined names and a Logs Explorer link. Every name is checked against a pattern and shown as <unrecognized> if it doesn't match.
    • A failed stage prints only the stage, the exception class and an API's HTTP status and reason.
• .github/scripts/sync_telemetry_error_issues.sh: requires its variables, masks the access token and key, and adds --dry-run and --report-date when set.
• .github/workflows/telemetry-error-issues.yml:
  • Runs at 00:45 America/New_York. That time is never skipped or repeated when the clocks change, and it avoids GitHub's delays for schedules at the start of an hour. It can also be run manually, defaulting to a dry run.
  • The job runs only in this repository, from master, in the telemetry-error-issues environment, one run at a time, with contents: read, issues: write and id-token: write.
  • It authenticates through Workload Identity Federation without writing a credentials file.
• Documentation: gcp/observability/README.md describes the job, how to run it by hand, the infrastructure, why the fingerprint key must never be rotated on its own, and the Cloud Run platform-failure gap.
• Changelog: fragment added.

Migration scope: no route groups, migration metadata or user-visible API contracts change. This adds a workflow and its script.

Infrastructure (created 2026-10-20)

Resource	Details
Linked BigQuery dataset observability_logs	On the policyengine-observability bucket, in location US, with the _AllLogs view
Service account telemetry-error-issues@policyengine-observability.iam.gserviceaccount.com	roles/bigquery.jobUser on the project; roles/bigquery.dataViewer on the dataset only (granted with GRANT ... ON SCHEMA). It needs no Cloud Logging role.
Workload Identity binding	roles/iam.workloadIdentityUser on the service account for principal://.../workloadIdentityPools/github/subject/repo:PolicyEngine/policyengine-api:environment:telemetry-error-issues only
GitHub environment telemetry-error-issues	Deployments limited to master

Environment variables

Name	Provider	Wiring	Status
GCP_WORKLOAD_IDENTITY_PROVIDER	Repository secret	secrets. in the auth step	Exists; value unreadable, so confirmed by the first manual dry run
TELEMETRY_ERROR_ISSUES_SERVICE_ACCOUNT	telemetry-error-issues environment variable	vars. in the auth step	Set
GOOGLE_OAUTH_ACCESS_TOKEN	steps.auth.outputs.access_token	Step env: to the wrapper (required, masked) to the script	Created at run time from the binding above
TELEMETRY_ERROR_FINGERPRINT_KEY	telemetry-error-issues environment secret	Step env: to the wrapper (required, masked) to the script	Set; random 32 bytes, stored only in GitHub
OBSERVABILITY_PROJECT_ID	Repository variable (policyengine-observability)	Step env: to the wrapper to --project-id	Set at repository level, so the environment job can read it
GITHUB_TOKEN	github.token, with issues: write	Step env:	Built into Actions
GITHUB_REPOSITORY, GITHUB_STEP_SUMMARY	Actions defaults	Read by the wrapper and script	Built into Actions
DRY_RUN, REPORT_DATE	Manual-run inputs; false and empty on scheduled runs	Step env: to wrapper flags	Covered by tests

Review fixes

A review found these, each now fixed with a test written first:

1. Chained exceptions with a long cause got two fingerprints. Shortening the stack could drop the final traceback, so frames came from different tracebacks. Frames now come from the first traceback only.
2. An apostrophe paired with the next quote and put the quoted value into the message template, making one group per value.
3. The error rate could exceed 100%, because the numerator included failures without a route.
4. Label names were compared exactly. An existing label that differs only in case would have made label creation fail with HTTP 422.
5. A secondary-rate-limit 403 without retry-after failed the run instead of waiting.

Testing

• Dry runs against live data, for 2026-10-19, first with operator credentials and then as the service account:
  • They read 1,648 ERROR records across eight days and found 15 groups, 5 of them above the issue thresholds.
  • The queries scanned about 0.18 GiB and 0.02 GiB.
  • A generated Logs Explorer link opened the group's 12 records.
• tests/unit/test_telemetry_error_issues.py: 212 tests, each commit written test-first.
  • Covered:
    • the window and its 23- and 25-hour days;
    • frame extraction from standard, chained, exception group and truncated tracebacks;
    • value validation;
    • choosing one record per failure;
    • fingerprints;
    • measures;
    • the BigQuery requests and SQL text, including that stacks and messages are read only through their expressions and identifiers only as hashes;
    • rendering;
    • the issue sync, against an in-memory GitHub fake;
    • the end-to-end run;
    • the workflow file;
    • the wrapper, run with subprocess and a fake python3 first on PATH.
  • Marker strings placed in every record field never appear in any title, body, link, log line or job summary.
  • The file also passes under Python 3.12, the version the workflow uses.
  • Running the template and frame patterns in BigQuery on the tests' strings gave the same results as the tests' Python version of them.
• Workflow suites: with tests/unit/test_alembic_workflows.py and tests/unit/test_cloud_run_deploy_scripts.py, 354 passed.
• Repository checks:
  • python scripts/run_quality_guards.py passed, including ci-scripts with shellcheck.
  • make format and ruff check are clean on the changed files.
  • There are no inline suppressions.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.71%. Comparing base (be0c586) to head (bf9ea6b).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3990       +/-   ##
===========================================
+ Coverage   49.31%   84.71%   +35.39%     
===========================================
  Files         242      258       +16     
  Lines       14299    15677     +1378     
  Branches     2354     2639      +285     
===========================================
+ Hits         7052    13281     +6229     
+ Misses       6622     1626     -4996     
- Partials      625      770      +145     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.