Skip to content

Report each day's production telemetry error groups as GitHub issues #3989

Closed

Description

@anth-volk

Problem

The Public API and the simulation API write structured telemetry to the central Cloud Logging bucket policyengine-observability, but nothing reviews their errors systematically. The six alert policies have no notification channels. A dry run of this change against 2026-10-19 found 15 error groups in production and staging. The largest, one exception on /<country_id>/report, failed about 14% of that route's requests that day, and nobody was told.

Change

A scheduled GitHub Actions workflow reports each US Eastern calendar day's errors as GitHub issues in this repository:

  • At 00:45 US Eastern it reads the previous day's ERROR records, plus the seven days before for comparison, from the bucket's linked BigQuery dataset.
  • It chooses one record per failed request and service, groups failures by a keyed fingerprint, and measures each group in production and staging: occurrences, affected requests, error rate on the affected routes, hourly rates, and first and last seen.
  • It opens one issue per production group with at least 3 occurrences in at least 2 requests, at most 5 per run, and updates existing issues. An issue closed as completed reopens with a regression label when failures return. Staging counts appear in the issues but never open one.

This repository is public, so issues, logs and job summaries may contain only fingerprints, counts, code-defined names (service, exception class, route template) and a Logs Explorer link, never error messages, stack text or identifiers.

Out of scope

  • Cloud Run platform failures (timeouts, out-of-memory kills, crashes before the application logs) are not forwarded to the bucket, so they are not reported. This is documented as a known gap.
  • Opening pull requests automatically from these issues is a later change.

Merge timing

Merge on or after 2026-10-23, once #3984 has run in production for a full Eastern day. Before that, unhandled 500s from before #3984 have no exception record, and would open issues for groups that stop matching once it is deployed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions