Skip to content

Unhandled exceptions in Flask routes return a 500 without recording the exception in telemetry #3983

Closed

Description

@anth-volk

Problem

When a Flask route raises an exception that no other handler catches, response_generic_error in policyengine_api/routes/error_routes.py returns a 500 JSON response but never passes the exception to the observability runtime. The central log bucket then has a request.completed record with outcome=error and no record of which exception caused it: no error.type and no stack.

In a 7-day sample of 400 ERROR records from the central bucket, only 50 carried error.type. At least 134 were request.completed records with outcome=error and no exception details. Those failures can't be grouped by exception type, which a planned daily error report needs.

Change

  • Add record_unhandled_exception() to policyengine_api.observability. It calls runtime.record_exception(error, handled=False, status_code=...) and suppresses any exception the telemetry call raises, so recording can never change the response.
  • Call it from response_generic_error before the 500 response is built. HTTP exceptions (404, 405 and others) go through response_http_exception and are not recorded.

Effects on telemetry:

  • Each Flask 500 adds one exception.recorded ERROR record with error.type, the message and the stack.
  • That request's request.completed record carries the same error.* fields. This needs policyengine-observability 3.0.3 (Keep the exception recorded by an error handler on request.completed policyengine-observability#36); in 3.0.2 the shared Flask integration's teardown_request callback discarded the recorded exception.
  • Each such 500 adds two policyengine.error.count points: the new one with operation.kind=unhandled and the existing one with operation.kind=request. The error-rate alert and dashboard now count only operation.kind values request and operation, so each failure counts once there. The alert has no notification channels, so no one is paged.

The response body and status code don't change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions