[anth-volk]

Fixes #962
Fixes PolicyEngine/policyengine-api#4069

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

The gateway no longer calls runtime.record_exception(...) for requests it answers with a client error. policyengine-observability writes each recorded exception at ERROR severity, and the request's own request.completed record already reports these responses with outcome client_error. The visible symptom is PolicyEngine/policyengine-api#4069. Most of these records come from the deploy pipeline's authentication test, which sends the country __authentication_probe__ and expects 400.

Calls removed, with status codes and response bodies unchanged and no replacement log:

Endpoint	Response	Cause
POST /simulate/economy/comparison	400	ValueError from route resolution or the breakdown capability check
POST /simulate/economy/comparison	400 500	ValueError from the bundle and SPM lookup that has no SPM error code; now a recorded server error, see the update below
POST /simulate/economy/budget-window	400	ValueError from route resolution
POST /simulate/economy/budget-window	400 500	ValueError from the bundle and SPM lookup that has no SPM error code; now a recorded server error, see the update below
GET /jobs/{job_id}	404	the gateway has no metadata for the job ID
GET /budget-window-jobs/{batch_job_id}	404	no batch state or seed for the batch ID
GET /versions/{kind}	404	unknown kind

The last three went through routes_common.record_not_found, which has no callers left and is removed.

Still recorded:

• the 500 when Modal's FunctionCall.from_id fails during GET /jobs/{job_id};
• failed-job responses, through log_and_redact_exception;
• the 404 from GET /jobs/{job_id} when the gateway has metadata for the job but Modal reports the call missing or its output expired. The gateway submitted that job, so the 404 is not clearly a client error. One of the two calls is also in the failed-job branch that another change is editing.

The entry service has no record_exception calls. Its 4xx responses (403 for a missing permission, 422 and 404 on the Stage 12 report routes) record no exception, and it relays the gateway's responses with an INFO event.

docs/engineering/skills/observability.md states no rule about recording exceptions, so it is unchanged.

Testing

Written test-first: the first commit adds the tests, and seven of them failed with the expected reason. The status code and body assertions passed, and the assertion that nothing was recorded failed.

• New tests/test_exception_records.py: an unknown country on both submission endpoints returns 400 and records nothing. (The bundle lookup tests originally expected the same; the update below changes them.) A Modal lookup failure on GET /jobs/{job_id} still records its exception as unhandled with status 500. That test passes before and after the fix.
• The existing 404 tests for an unknown job ID, unknown batch ID and unknown version kind now assert that nothing is recorded.
• uv run --frozen pytest tests/ in the gateway: 273 passed, 1 deselected (the opt-in Modal integration marker), before the rebase described below.
• Executor tests that build the gateway app (test_canonical_spm.py, test_budget_window_scheduler.py): 67 passed.
• ruff 0.15.22 format check and lint, scripts/run_quality_guards.py, python -m scripts.rules guard --base-ref origin/main, python -m scripts.rules render (no diff), and python -m scripts.rules pyright --sync projects/policyengine-simulation-gateway all pass. The gateway's OpenAPI golden file was unchanged by the first two commits.

Update, 2026-10-28: bundle-lookup errors return 500

The change

In both submission handlers, an error without an SPM error code raised while describing the route's bundle or resolving SPM settings (ValueError or HuggingFaceDatasetReferenceError) used to return 400 with the exception text as detail. It now follows the gateway's convention for server failures:

• log_and_redact_exception logs the exception with its traceback and records it as a handled error with status 500.
• failed_job_response returns 500 with status: "failed", result: null, no structured errors, and error: "Simulation failed (correlation_id=<hex>)". The observability ID header is set.
• The caller no longer sees the internal exception text. The log scopes are simulation_submission and budget_window_submission.

Unchanged:

• A coded SPM error still returns 400 with structured errors.
• A route-resolution ValueError, such as an unknown country, still returns 400 without an exception record, as do the three 404s above.

Why this is a server error

No request input can reach this branch. Every SPM error the request can cause has a code and takes the structured 400 response. The bundle step's only request inputs are the country, already checked during route resolution, and the version, which only selects a manifest. What can raise there is bad data in our own deployed bundle manifest, such as a malformed dataset address or reference. In the last 30 days the branch never fired: all 200 gateway 400 exception records came from resolve_route.

Schema and client changes

• Both submission routes declare 500: "Server failure (redacted error with a correlation ID)" next to 200, 400 and 422. The declaration is in the gateway's OpenAPI generator, with tests/golden/openapi.json regenerated (only the two 500 entries change), and in the entrypoint's route declarations. The entrypoint's contract test compares its document with the gateway's golden file. The 500 has no response model, like the 500 on the status routes, because an unhandled server error still returns a plain-text body.
• The generated client is not committed. scripts/generate-clients.sh builds it from the entrypoint's OpenAPI document, and client publication checks out the deployed commit. Regenerated locally, only the two submission modules change: a 500 is now a declared status that parses to None, instead of raising UnexpectedStatus when raise_on_unexpected_status is set. Against the regenerated client, the local apis-integ run (-m "not requires_gcp and not beta_only", with the Docker Compose executor service) has 33 passed, 2 skipped and 6 deselected, and Pyright for projects/policyengine-apis-integ passes.

Downstream

No downstream change is needed:

• The entrypoint forwards the gateway's status unchanged.
• The Public API already treats a single-year 400 and 500 the same. The exception is a request with a local-area breakdown, where a 500 now takes the ordinary error path.
• For budget-window submissions, the Public API no longer stores a 500 as a validation failure, so a later request retries.

Commits and tests (red, then green)

• 4b86bf50 changes the bundle-lookup tests, for both handlers, to expect a 500 with the redacted body, no exception text, the observability header, one exception record (handled, status 500) and no Modal spawn. Both failed before the fix with assert 400 == 500.
• 0715a7a5 changes the two handlers. The tests pass.
• 0965c748 declares the 500 in the gateway's OpenAPI generator, its golden file and the entrypoint.
• 97412692 replaces changelog_entry.yaml, which main no longer uses, with the changelog fragment changelog.d/962.changed.md.
• Gateway: 274 passed, 1 deselected. Entrypoint: 840 passed. (After the rebase onto main; 273 and 742 before it.)
• ruff 0.15.22 format check and lint, scripts/run_quality_guards.py, python -m scripts.rules guard --base-ref origin/main, python -m scripts.rules render (no diff), and Pyright for the gateway, the entrypoint and apis-integ all pass.

Update, 2026-10-29: rebased onto main

• Rebased onto main at f6d48051. The only conflict was changelog_entry.yaml, which main deleted when it moved to changelog fragments, so the branch no longer touches it. tests/golden/openapi.json and version_routes.py merged without conflict, and regenerating the OpenAPI document reproduces the merged golden file byte for byte.
• changelog.d/962.changed.md (type changed) describes both changes: the exception records removed for client errors, and the 500 for bundle-lookup errors.
• Gateway: 274 passed, 1 deselected (the opt-in Modal integration marker). Entrypoint: 840 passed. Rules tests: 263 passed. The changelog fragment check, ruff 0.15.22 format check and lint, scripts/run_quality_guards.py, python -m scripts.rules guard --base-ref origin/main and python -m scripts.rules render (no diff) pass. The higher counts include tests added on main.

🤖 Generated with Claude Code