[anth-volk]

Fixes #3983

Summary

• policyengine_api/observability/runtime.py: new record_unhandled_exception(error, *, status_code=500). It calls runtime.record_exception(error, handled=False, status_code=status_code) inside contextlib.suppress(Exception), the same way set_runtime_context protects the response from telemetry failures. policyengine_api.observability exports it.
• policyengine_api/routes/error_routes.py: response_generic_error, the catch-all handler for exceptions that aren't HTTPException, calls it before building the 500 response. response_http_exception and the status-code handlers (404, 405, abort(500) and other HTTP errors) are unchanged and record nothing.
• pyproject.toml and uv.lock: require policyengine-observability 3.0.3, which keeps the recorded exception on a failed request (Keep the exception recorded by an error handler on request.completed policyengine-observability#36). The lock file was regenerated with uv 0.12.1, the version CI uses. gcp/observability/README.md names the new version.
• Changelog fragment added.

Effects on telemetry, for each exception that reaches response_generic_error:

• One exception.recorded ERROR record, with error.type, error.handled=false, http.response.status_code=500, http.route, request.id, the message and the stack. Its request.id matches the request's request.completed record, so the two records can be joined.
• The request's request.completed record carries the same error.type, error.message and error.stack. With policyengine-observability 3.0.2 it did not: instrument_flask ends the request from Flask's teardown_request callback, which receives None once an error handler has handled the exception, and end_request replaced the attached error with that None. 3.0.3 keeps the attached error when the request has already failed, and leaves requests that record a handled exception and still succeed unchanged.
• policyengine.error.count gains one point with operation.kind=unhandled, in addition to the existing operation.kind=request point that end_request adds for the same 500. The "API v1 elevated error rate" alert and the "PolicyEngine API v1 observability" dashboard used to add up every series, so they would have counted each of these failures twice. Both now count only operation.kind values request and operation (changed in Cloud Monitoring on 2026-10-20), so each failed request or background operation counts once. The alert policy has no notification channels, so nobody is paged.
• The simulation and report-output creation routes log the exception with logger.exception before re-raising it. Their failures now produce two ERROR records with the same error.type: the existing log record and the new exception.recorded record. A report that counts failures by exception type should count only exception.recorded records.

Not covered by this change:

• Routes that catch an exception and build their own 500 response, such as the household calculation routes, which already log it with logger.exception.
• Native FastAPI routes. Their request.completed record already carries error.* fields, because the ASGI middleware passes the exception to end_request.

The response status code and body don't change.

Migration scope: no route groups, migration metadata or user-visible API contracts change.

Testing

• Written test-first. tests/unit/test_observability_runtime.py has 2 new tests: the helper calls record_exception once with handled=False and the status code, and it returns normally when record_exception raises.
• New tests/unit/routes/test_error_routes.py (5 tests) builds a Flask app with error_bp:
  • a route raising RuntimeError calls the helper once with that error and status 500, and still returns the existing 500 body;
  • with the app instrumented by a real runtime that keeps its records in memory, the same route produces exactly one exception.recorded record with the exception type, message, stack, error.handled=false, status 500 and route, and the same request.id as the request.completed record. The other tests replace record_exception with a stand-in that accepts any keyword argument. If a policyengine-observability release renamed one of its keywords, recording would stop without failing them; this test fails in that case;
  • the request's request.completed record has outcome=error, status 500 and the exception's type, message and stack. This test failed with policyengine-observability 3.0.2 and passes with 3.0.3;
  • a 404 does not call the helper;
  • when the runtime call raises, the 500 response is unchanged.
• FLASK_DEBUG=1 python -m pytest tests/unit/routes tests/unit/test_asgi_factory.py tests/unit/test_request_context.py tests/contract tests/unit/test_observability_runtime.py: 446 passed, with 3.0.3 installed.
• Every unit test file that refers to observability (623 tests): passed with 3.0.3.
• python scripts/run_quality_guards.py: passed.
• make format and ruff check are clean on the changed files. No inline suppressions are added.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.62%. Comparing base (1163955) to head (4fb038b).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3984       +/-   ##
===========================================
+ Coverage   46.17%   84.62%   +38.45%     
===========================================
  Files         239      258       +19     
  Lines       13987    15689     +1702     
  Branches     2272     2643      +371     
===========================================
+ Hits         6458    13277     +6819     
+ Misses       6958     1637     -5321     
- Partials      571      775      +204     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.