Enable ruff structure and import rules (INP, PLC, SLF, N, A, FAST, E)

changelog.d/ruff-structure-imports.changed.md

@@ -0,0 +1 @@
+Enabled ruff's structure and import rules (INP, PLC, SLF, N, A, FAST, E): long lines are wrapped, FastAPI dependencies are declared with Annotated, deferred imports that avoid heavy loads, cycles or import-time I/O carry a reason unless the module is in ruff's banned-module-level-imports allowlist of lazily loaded modules (policyengine, spm_calculator, the v2 database, session and services modules, Auth0 Management and the Google Secret Manager client, which TID253 keeps out of module-level imports), and directories without an __init__.py on purpose are declared to ruff as namespace packages; tests are also exempt from SLF001, INP001, PLC0415 and N802, and .github scripts from N999. Three private-member reads now use public names: the in-memory cache's operation_lock, SimulationService.ensure_simulation_dual_write_state and constants.get_bundle_manifest(). Published schemas and the HTTP API are unchanged.

docs/engineering/skills/typing-only-imports.md

@@ -36,6 +36,21 @@ where `package/__init__.py` re-exports it) does not match its listed definition
 path. Ruff does not follow inheritance across modules, so every package class
 that models in other modules inherit from is listed.
 
+### Route modules with per-router dependencies
+
+FastAPI resolves an endpoint's string annotations against its module's globals
+only. A route that declares a dependency with `Annotated` therefore needs every
+name in that annotation at module level. Query dependencies are module-level
+aliases, such as `CountryQueryDependency`. The signed-in user routes also depend
+on `require_user`, the authenticator `NativeRouteDependencies` injects into each
+router, which exists only inside the router builder. Their modules
+(`fastapi_routes/v2/me`, `user_households`, `user_policies`, `user_reports` and
+`user_simulations`) don't use `from __future__ import annotations`, so
+`Annotated[AccountPrincipal, Depends(require_user)]` is evaluated where the
+handler is defined, and the names their other annotations use are imported at
+runtime. With the future import, FastAPI would silently treat `principal` as a
+required query parameter.
+
 ## Adding a model base
 
 When you add a model class that models in other modules inherit from, add the

docs/generated/rules.md

@@ -290,23 +290,23 @@ flowchart LR
 | Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |
 | --- | --- | --- | --- | --- | --- | --- | --- |
 | `api.quality.alembic-revisions-unused-imports` | Ruff doesn't lint Alembic revision files (lint.exclude in pyproject.toml), because they must stay exactly as Alembic generates them. | mechanical | advisory | no | nothing | — | — |
-| `api.quality.ci-cd-scripts-ruff-exemptions` | CI/CD scripts under .github may print their results and fetch fixed URLs (T201, S310); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of S310, T201 for `.github/**` | `pr.yml:lint` | — |
+| `api.quality.ci-cd-scripts-ruff-exemptions` | CI/CD scripts under .github may print their results, fetch fixed URLs, have hyphenated file names and import the lazily loaded packages at module level (T201, S310, N999, TID253); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of N999, S310, T201, TID253 for `.github/**` | `pr.yml:lint` | — |
 | `api.quality.command-line-modules-ruff-exemptions` | Application modules that also run as command-line programs (if __name__ == "__main__") may print their output (T201); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of T201 for `policyengine_api/{data/v2/catalog/initialization,data/v2/household_migration_qualification,data/v2/policy_migration_qualification,data/v2/privilege_reconciliation,data/v2/user_association_migration_qualification,worker_spm_release}.py` | `pr.yml:lint` | — |
 | `api.quality.command-line-scripts-ruff-exemptions` | Command-line scripts under scripts/ may print their output, run other programs, fetch fixed URLs, stamp today's date and pick random samples (T201, S603, S607, S310, DTZ011, S311); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of DTZ011, S310, S311, S603, S607, T201 for `scripts/**` | `pr.yml:lint` | — |
 | `api.quality.environment-variables-load` | Required environment variables and credentials load in a deployed-like environment. | mechanical | enforced | yes | CI job `test_env_vars` | `pr.yml:test_env_vars` | — |
 | `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, and the user sign-in and account-deletion libraries pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:typecheck-v2` | — |
 | `api.quality.ruff-format` | All Python files are formatted with ruff format. | mechanical | enforced | yes | ruff format | `pr.yml:lint` | — |
-| `api.quality.ruff-lint` | All Python code passes ruff check with every rule family selected in pyproject.toml, with no ratchet; families are added one themed group at a time until the selection is ALL. | mechanical | enforced | yes | ruff check (ANN, BLE, D, DTZ, E4, E7, E9, EM, ERA, EXE, F, FIX, FLY, FURB, G, I, LOG, PIE, PYI, RET, RSE, S, T20, TC, TD, TRY, UP, W) | `pr.yml:lint` | — |
+| `api.quality.ruff-lint` | All Python code passes ruff check with every rule family selected in pyproject.toml, with no ratchet; families are added one themed group at a time until the selection is ALL. | mechanical | enforced | yes | ruff check (A, ANN, BLE, D, DTZ, E, E4, E7, E9, EM, ERA, EXE, F, FAST, FIX, FLY, FURB, G, I, INP, LOG, N, PIE, PLC, PYI, RET, RSE, S, SLF, T20, TC, TD, TID253, TRY, UP, W) | `pr.yml:lint` | — |
 | `api.quality.runtime-evaluated-models-resolve` | Every package model is one ruff treats as runtime-evaluated (a base resolves, through the import in its own module, to a name in runtime-evaluated-base-classes, or is a same-module class that does), and every Pydantic model and SQLAlchemy mapper resolves its annotations once the package is imported. | mechanical | enforced | yes | pytest `tests/unit/test_runtime_evaluated_annotations.py` | `pr.yml:test` | **unproven** |
 | `api.quality.tests-pass` | The unit, contract and to-refactor test suites pass. | mechanical | enforced | yes | CI job `test` | `pr.yml:test` | — |
-| `api.quality.tests-ruff-exemptions` | Test files may use pytest's bare assert and need no docstrings, annotations or named constants for magic numbers (S101, D1, ANN, PLR2004). They may also use fake credentials, naive datetimes, subprocess calls to repository scripts, SQL built from strings and prints (S105, S106, DTZ001, DTZ011, S603, S607, S608, T201). Every other ruff rule applies to tests. | mechanical | enforced | yes | ruff per-file ignore of ANN, D1, DTZ001, DTZ011, PLR2004, S101, S105, S106, S603, S607, S608, T201 for `**/tests/**` | `pr.yml:lint` | — |
+| `api.quality.tests-ruff-exemptions` | Test files may use pytest's bare assert, need no docstrings, annotations or named constants for magic numbers, and may reach into private members, live outside packages and import inside a test (S101, D1, ANN, PLR2004, SLF001, INP001, PLC0415). Their fakes may name methods after the interface they replace, such as a gRPC stub's Export (N802). They may also use fake credentials, naive datetimes, subprocess calls to repository scripts, SQL built from strings and prints (S105, S106, DTZ001, DTZ011, S603, S607, S608, T201), and may import the lazily loaded packages at module level (TID253). Every other ruff rule applies to tests. | mechanical | enforced | yes | ruff per-file ignore of ANN, D1, DTZ001, DTZ011, INP001, N802, PLC0415, PLR2004, S101, S105, S106, S603, S607, S608, SLF001, T201, TID253 for `**/tests/**` | `pr.yml:lint` | — |
 
 ```mermaid
 flowchart LR
   r_api_quality_alembic_revisions_unused_imports("api.quality.alembic-revisions-unused-imports (advisory)")
   gap_unenforced>"not enforced"]
   r_api_quality_ci_cd_scripts_ruff_exemptions("api.quality.ci-cd-scripts-ruff-exemptions (enforced)")
-  e_ruff_per_file_ignore_of_S310__T201_for__github___["ruff per-file ignore of S310, T201 for .github/**"]
+  e_ruff_per_file_ignore_of_N999__S310__T201__TID253_for__github___["ruff per-file ignore of N999, S310, T201, TID253 for .github/**"]
   j_CI_pr_yml_lint["CI pr.yml:lint"]
   r_api_quality_command_line_modules_ruff_exemptions("api.quality.command-line-modules-ruff-exemptions (enforced)")
   e_ruff_per_file_ignore_of_T201_for_policyengine_api__data_v2_catalog_initialization_data_v2_household_migration_qualification_data_v2_policy_migration_qualification_data_v2_privilege_reconciliation_data_v2_user_association_migration_qualification_worker_spm_release__py["ruff per-file ignore of T201 for policyengine_api/{data/v2/catalog/initialization,data/v2/household_migration_qualification,data/v2/policy_migration_qualification,data/v2/privilege_reconciliation,data/v2/user_association_migration_qualification,worker_spm_release}.py"]
@@ -321,17 +321,17 @@ flowchart LR
   r_api_quality_ruff_format("api.quality.ruff-format (enforced)")
   e_ruff_format["ruff format"]
   r_api_quality_ruff_lint("api.quality.ruff-lint (enforced)")
-  e_ruff_check__ANN__BLE__D__DTZ__E4__E7__E9__EM__ERA__EXE__F__FIX__FLY__FURB__G__I__LOG__PIE__PYI__RET__RSE__S__T20__TC__TD__TRY__UP__W_["ruff check (ANN, BLE, D, DTZ, E4, E7, E9, EM, ERA, EXE, F, FIX, FLY, FURB, G, I, LOG, PIE, PYI, RET, RSE, S, T20, TC, TD, TRY, UP, W)"]
+  e_ruff_check__A__ANN__BLE__D__DTZ__E__E4__E7__E9__EM__ERA__EXE__F__FAST__FIX__FLY__FURB__G__I__INP__LOG__N__PIE__PLC__PYI__RET__RSE__S__SLF__T20__TC__TD__TID253__TRY__UP__W_["ruff check (A, ANN, BLE, D, DTZ, E, E4, E7, E9, EM, ERA, EXE, F, FAST, FIX, FLY, FURB, G, I, INP, LOG, N, PIE, PLC, PYI, RET, RSE, S, SLF, T20, TC, TD, TID253, TRY, UP, W)"]
   r_api_quality_runtime_evaluated_models_resolve("api.quality.runtime-evaluated-models-resolve (enforced)")
   e_pytest_tests_unit_test_runtime_evaluated_annotations_py["pytest tests/unit/test_runtime_evaluated_annotations.py"]
   j_CI_pr_yml_test["CI pr.yml:test"]
   r_api_quality_tests_pass("api.quality.tests-pass (enforced)")
   e_CI_job_test["CI job test"]
   r_api_quality_tests_ruff_exemptions("api.quality.tests-ruff-exemptions (enforced)")
-  e_ruff_per_file_ignore_of_ANN__D1__DTZ001__DTZ011__PLR2004__S101__S105__S106__S603__S607__S608__T201_for____tests___["ruff per-file ignore of ANN, D1, DTZ001, DTZ011, PLR2004, S101, S105, S106, S603, S607, S608, T201 for **/tests/**"]
+  e_ruff_per_file_ignore_of_ANN__D1__DTZ001__DTZ011__INP001__N802__PLC0415__PLR2004__S101__S105__S106__S603__S607__S608__SLF001__T201__TID253_for____tests___["ruff per-file ignore of ANN, D1, DTZ001, DTZ011, INP001, N802, PLC0415, PLR2004, S101, S105, S106, S603, S607, S608, SLF001, T201, TID253 for **/tests/**"]
   r_api_quality_alembic_revisions_unused_imports -.-> gap_unenforced
-  r_api_quality_ci_cd_scripts_ruff_exemptions --> e_ruff_per_file_ignore_of_S310__T201_for__github___
-  e_ruff_per_file_ignore_of_S310__T201_for__github___ --> j_CI_pr_yml_lint
+  r_api_quality_ci_cd_scripts_ruff_exemptions --> e_ruff_per_file_ignore_of_N999__S310__T201__TID253_for__github___
+  e_ruff_per_file_ignore_of_N999__S310__T201__TID253_for__github___ --> j_CI_pr_yml_lint
   r_api_quality_command_line_modules_ruff_exemptions --> e_ruff_per_file_ignore_of_T201_for_policyengine_api__data_v2_catalog_initialization_data_v2_household_migration_qualification_data_v2_policy_migration_qualification_data_v2_privilege_reconciliation_data_v2_user_association_migration_qualification_worker_spm_release__py
   e_ruff_per_file_ignore_of_T201_for_policyengine_api__data_v2_catalog_initialization_data_v2_household_migration_qualification_data_v2_policy_migration_qualification_data_v2_privilege_reconciliation_data_v2_user_association_migration_qualification_worker_spm_release__py --> j_CI_pr_yml_lint
   r_api_quality_command_line_scripts_ruff_exemptions --> e_ruff_per_file_ignore_of_DTZ011__S310__S311__S603__S607__T201_for_scripts___
@@ -342,14 +342,14 @@ flowchart LR
   e_mypy --> j_CI_pr_yml_typecheck_v2
   r_api_quality_ruff_format --> e_ruff_format
   e_ruff_format --> j_CI_pr_yml_lint
-  r_api_quality_ruff_lint --> e_ruff_check__ANN__BLE__D__DTZ__E4__E7__E9__EM__ERA__EXE__F__FIX__FLY__FURB__G__I__LOG__PIE__PYI__RET__RSE__S__T20__TC__TD__TRY__UP__W_
-  e_ruff_check__ANN__BLE__D__DTZ__E4__E7__E9__EM__ERA__EXE__F__FIX__FLY__FURB__G__I__LOG__PIE__PYI__RET__RSE__S__T20__TC__TD__TRY__UP__W_ --> j_CI_pr_yml_lint
+  r_api_quality_ruff_lint --> e_ruff_check__A__ANN__BLE__D__DTZ__E__E4__E7__E9__EM__ERA__EXE__F__FAST__FIX__FLY__FURB__G__I__INP__LOG__N__PIE__PLC__PYI__RET__RSE__S__SLF__T20__TC__TD__TID253__TRY__UP__W_
+  e_ruff_check__A__ANN__BLE__D__DTZ__E__E4__E7__E9__EM__ERA__EXE__F__FAST__FIX__FLY__FURB__G__I__INP__LOG__N__PIE__PLC__PYI__RET__RSE__S__SLF__T20__TC__TD__TID253__TRY__UP__W_ --> j_CI_pr_yml_lint
   r_api_quality_runtime_evaluated_models_resolve --> e_pytest_tests_unit_test_runtime_evaluated_annotations_py
   e_pytest_tests_unit_test_runtime_evaluated_annotations_py --> j_CI_pr_yml_test
   r_api_quality_tests_pass --> e_CI_job_test
   e_CI_job_test --> j_CI_pr_yml_test
-  r_api_quality_tests_ruff_exemptions --> e_ruff_per_file_ignore_of_ANN__D1__DTZ001__DTZ011__PLR2004__S101__S105__S106__S603__S607__S608__T201_for____tests___
-  e_ruff_per_file_ignore_of_ANN__D1__DTZ001__DTZ011__PLR2004__S101__S105__S106__S603__S607__S608__T201_for____tests___ --> j_CI_pr_yml_lint
+  r_api_quality_tests_ruff_exemptions --> e_ruff_per_file_ignore_of_ANN__D1__DTZ001__DTZ011__INP001__N802__PLC0415__PLR2004__S101__S105__S106__S603__S607__S608__SLF001__T201__TID253_for____tests___
+  e_ruff_per_file_ignore_of_ANN__D1__DTZ001__DTZ011__INP001__N802__PLC0415__PLR2004__S101__S105__S106__S603__S607__S608__SLF001__T201__TID253_for____tests___ --> j_CI_pr_yml_lint
 ```
 
 ## Observability

policyengine_api/constants.py

@@ -93,6 +93,11 @@ def _resolve_bundle_package_versions(bundle: dict | None) -> dict[str, str]:
     return package_versions
 
 
+def get_bundle_manifest() -> dict | None:
+    """Return the installed bundle manifest read at import, or None if it is missing."""
+    return _policyengine_bundle
+
+
 def get_bundle_data_release(country_id: str) -> dict:
     """Return the bundle's data release entry for a country, or {} if it has none."""
     if not isinstance(_policyengine_bundle, dict):
@@ -246,13 +251,15 @@ def get_report_output_cache_version(country_id: str) -> str:
     "national",  # National level (e.g., "us")
     "state",  # US states (e.g., "state/ca", "state/ny")
     "place",  # US Census places (e.g., "place/NJ-57000")
-    "congressional_district",  # US congressional districts (e.g., "congressional_district/CA-37")
+    # US congressional districts (e.g., "congressional_district/CA-37")
+    "congressional_district",
 )
 
 UK_REGION_TYPES = (
     "national",  # National level (e.g., "uk")
     "country",  # UK countries (e.g., "country/england", "country/scotland")
-    "constituency",  # UK parliamentary constituencies (e.g., "constituency/Aldershot")
+    # UK parliamentary constituencies (e.g., "constituency/Aldershot")
+    "constituency",
     "local_authority",  # UK local authorities (e.g., "local_authority/Maidstone")
 )
 
@@ -262,11 +269,13 @@ def get_report_output_cache_version(country_id: str) -> str:
     "us": [
         "state/",  # US states (e.g., "state/ca", "state/ny")
         "place/",  # US Census places (e.g., "place/NJ-57000")
-        "congressional_district/",  # US congressional districts (e.g., "congressional_district/CA-37")
+        # US congressional districts (e.g., "congressional_district/CA-37")
+        "congressional_district/",
     ],
     "uk": [
         "country/",  # UK countries (e.g., "country/england", "country/scotland")
-        "constituency/",  # UK parliamentary constituencies (e.g., "constituency/Aldershot")
+        # UK parliamentary constituencies (e.g., "constituency/Aldershot")
+        "constituency/",
         "local_authority/",  # UK local authorities (e.g., "local_authority/Maidstone")
     ],
 }

policyengine_api/country.py

@@ -179,7 +179,8 @@ def _json_safe(
 
     def build_microsimulation_options(self) -> dict:
         """Build the regions, time periods and datasets offered for economy runs."""
-        # { region: [{ name: "uk", label: "the UK" }], time_period: [{ name: 2022, label: "2022", ... }] }
+        # Shape: { region: [{ name: "uk", label: "the UK" }],
+        #   time_period: [{ name: 2022, label: "2022", ... }] }
         options = dict()
         if self.country_id == "uk":
             constituency_names_path = (
@@ -396,7 +397,8 @@ def build_parameters(self) -> dict:
                 }
             elif isinstance(parameter, ParameterScaleBracket):
                 bracket_index = int(parameter.name[parameter.name.index("[") + 1 : -1])
-                # Set the label to 'first bracket' for the first bracket, 'second bracket' for the second, etc.
+                # Set the label to 'first bracket' for the first bracket, 'second
+                # bracket' for the second, etc.
                 bracket_label = f"bracket {bracket_index + 1}"
                 parameter_data[parameter.name] = {
                     "type": "parameterNode",
@@ -681,7 +683,9 @@ def modify_parameters(parameters: ParameterNode) -> ParameterNode:
                 start, end = period.split(".")
                 node_type = type(node.values_list[-1].value)
                 if node_type is int:
-                    node_type = float  # '0' is of type int by default, but usually we want to cast to float.
+                    # '0' is of type int by default, but usually we want to cast to
+                    # float.
+                    node_type = float
                 if node.values_list[-1].value is None:
                     node_type = float
                 node.update(
@@ -692,7 +696,7 @@ def modify_parameters(parameters: ParameterNode) -> ParameterNode:
 
         return parameters
 
-    class reform(Reform):
+    class reform(Reform):  # noqa: N801  # policyengine-core uses the class name as the reform key
         def apply(self) -> None:
             self.modify_parameters(modify_parameters)
 

policyengine_api/data/v2/catalog/extraction.py

@@ -746,7 +746,9 @@ def extract_catalog(
 def extract_installed_catalog() -> NormalizedCatalog:
     """Load and normalize the installed PolicyEngine.py certified catalog."""
     import policyengine
-    from policyengine.bundle import get_current_bundle
+    from policyengine.bundle import (
+        get_current_bundle,
+    )
 
     policyengine_version = importlib_metadata.version("policyengine")
     return extract_catalog(