Enforce shared assertions in e2e tests (#3108)

* Enforce shared assertions in e2e tests

* Use e2e fixtures for shared assertions

* Ignore expected fixtures in code scanning

* Share HTTP mocks in e2e tests

* Cover HTTP assertion helper branch

* Address review feedback on assert helpers

* Cover async assertion guard branch

* Refine shared e2e assertion helpers

* Tighten assert helper usage guard

* Bind HTTP mocks to expected URLs

* Use expected fixture for collapsed external model

* Tighten HTTP request assertions

* Use shared HTTP mock fixture

* Use shared HTTP mock in JSON URL test

* Tighten assert guard coverage

* Use output fixtures for e2e asserts

* Type HTTP mock helpers

* Avoid CodeQL protocol body noise

* Address review assertion helper feedback

* Restore native deferred expected output

* Limit HTTP mock assertions to requested options

* Document e2e assertion helper policy

* Update llms docs for assertion policy

* Clarify e2e snapshot update workflow

Author: koxudaxi

.github/workflows/codeql.yaml

@@ -38,6 +38,9 @@ jobs:
         with:
           languages: ${{ matrix.language }}
           queries: +security-and-quality
+          config: |
+            paths-ignore:
+              - tests/data/expected/**
 
       - name: Autobuild
         uses: github/codeql-action/autobuild@ce64ddcb0d8d890d2df4a9d1c04ff297367dea2a # v3

docs/development-contributing.md

@@ -51,6 +51,39 @@ $ tox run -e cli-docs
 ## 11. Commit and Push...
 ```
 
+## 🧪 E2E test assertions
+
+E2E tests that validate generated output must use the shared assertion helpers
+instead of direct `assert` statements. The helpers compare complete generated
+files, generated modules, warnings, errors, and HTTP request behavior with
+consistent diffs and update hints.
+
+Use helpers such as `run_main_and_assert`, `run_main_url_and_assert`,
+`create_assert_file_content`, `assert_output`, `assert_directory_content`,
+`assert_generated_modules_output`, `assert_generated_file_matches_output`,
+`assert_httpx_get_kwargs`, and `assert_warnings_contain` where they fit. Prefer
+full expected-file or inline snapshot comparisons for generated output over
+substring checks.
+
+The file-output helpers use
+[`inline-snapshot`](https://15r10nk.github.io/inline-snapshot/latest/) and
+`external_file()` internally. When an expected output file is missing, the
+failure message includes the command to create it, such as
+`tox run -e <version> -- --inline-snapshot=create`. When an existing expected
+file differs from the generated output, the failure includes a diff and the
+command to update it, such as
+`tox run -e <version> -- --inline-snapshot=fix`. Review the generated files and
+the resulting `git diff` before committing. See the inline-snapshot
+[`--inline-snapshot` pytest options](https://15r10nk.github.io/inline-snapshot/latest/pytest/#-inline-snapshotcreatefixtrimupdate)
+for the meaning of `create` and `fix`.
+
+Direct `assert` statements are blocked in the guarded E2E test modules by
+`tests/test_assert_helper_usage.py`. If you add a new E2E test file that checks
+generated output, add it to `E2E_TEST_PATHS` in that guard test so new direct
+assertions are also rejected. Only use `@pytest.mark.allow_direct_assert` for
+cases that cannot reasonably be expressed with the shared helpers, such as
+mock setup internals or narrow intermediate-state checks.
+
 ## ➕ Adding a New CLI Option
 
 When adding a new CLI option to `datamodel-code-generator`, follow these steps:

docs/llms-full.txt

@@ -30847,6 +30847,39 @@ $ tox run -e cli-docs
 ## 11. Commit and Push...
 ```
 
+## 🧪 E2E test assertions
+
+E2E tests that validate generated output must use the shared assertion helpers
+instead of direct `assert` statements. The helpers compare complete generated
+files, generated modules, warnings, errors, and HTTP request behavior with
+consistent diffs and update hints.
+
+Use helpers such as `run_main_and_assert`, `run_main_url_and_assert`,
+`create_assert_file_content`, `assert_output`, `assert_directory_content`,
+`assert_generated_modules_output`, `assert_generated_file_matches_output`,
+`assert_httpx_get_kwargs`, and `assert_warnings_contain` where they fit. Prefer
+full expected-file or inline snapshot comparisons for generated output over
+substring checks.
+
+The file-output helpers use
+[`inline-snapshot`](https://15r10nk.github.io/inline-snapshot/latest/) and
+`external_file()` internally. When an expected output file is missing, the
+failure message includes the command to create it, such as
+`tox run -e <version> -- --inline-snapshot=create`. When an existing expected
+file differs from the generated output, the failure includes a diff and the
+command to update it, such as
+`tox run -e <version> -- --inline-snapshot=fix`. Review the generated files and
+the resulting `git diff` before committing. See the inline-snapshot
+[`--inline-snapshot` pytest options](https://15r10nk.github.io/inline-snapshot/latest/pytest/#-inline-snapshotcreatefixtrimupdate)
+for the meaning of `create` and `fix`.
+
+Direct `assert` statements are blocked in the guarded E2E test modules by
+`tests/test_assert_helper_usage.py`. If you add a new E2E test file that checks
+generated output, add it to `E2E_TEST_PATHS` in that guard test so new direct
+assertions are also rejected. Only use `@pytest.mark.allow_direct_assert` for
+cases that cannot reasonably be expressed with the shared helpers, such as
+mock setup internals or narrow intermediate-state checks.
+
 ## ➕ Adding a New CLI Option
 
 When adding a new CLI option to `datamodel-code-generator`, follow these steps:

pyproject.toml

@@ -232,6 +232,7 @@ verbosity_assertions = 2
 markers = [
   "perf: marks tests as performance tests (excluded from CI benchmarks)",
   "benchmark: marks tests as benchmark tests",
+  "allow_direct_assert: marks e2e tests where direct assert statements are intentionally allowed",
 ]
 
 [tool.coverage]