Skip to content

refactor: reduce e2e test duplication with parameterization and enhance CLI docs#2675

Merged
koxudaxi merged 5 commits intomainfrom
refactor/parameterize-e2e-tests-with-model-outputs
Dec 18, 2025
Merged

refactor: reduce e2e test duplication with parameterization and enhance CLI docs#2675
koxudaxi merged 5 commits intomainfrom
refactor/parameterize-e2e-tests-with-model-outputs

Conversation

@koxudaxi
Copy link
Copy Markdown
Owner

@koxudaxi koxudaxi commented Dec 17, 2025

Summary

  • Parameterize duplicate e2e tests to reduce code duplication
  • Add model_outputs to cli_doc markers for multi-model tab display in documentation
  • Add missing cli_doc markers to existing tests

Summary by CodeRabbit

  • Documentation

    • Expanded CLI reference with separate per-target examples (Pydantic v1/v2, dataclasses, TypedDict, msgspec); clarified and reworded options (--output-model-type, --use-frozen-field, --enum-field-as-literal, --allof-merge-mode) and added field-description guidance.
  • Tests

    • Converted many tests to parameterized multi-model checks covering multiple output types; added/updated tests for enum-as-Literal behaviors, field-description, reserved-field-name handling, use-frozen-field, and dataclass discriminator scenarios.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Dec 17, 2025

Warning

Rate limit exceeded

@koxudaxi has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 2 minutes and 19 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between d1d6b72 and 541b73e.

📒 Files selected for processing (1)
  • tests/main/graphql/test_main_graphql.py (3 hunks)

Walkthrough

Documentation examples were expanded to show per-target-model outputs (Pydantic v1/v2, dataclasses, TypedDict, msgspec) and help text was clarified. Tests were refactored to use @pytest.mark.cli_doc with model_outputs mappings and parameterized to validate outputs across multiple model targets and Python-version paths.

Changes

Cohort / File(s) Summary
CLI reference docs
docs/cli-reference/field-customization.md, docs/cli-reference/model-customization.md, docs/cli-reference/typing-customization.md, docs/cli-reference/quick-reference.md
Split single output examples into per-target subsections (Pydantic v1 / Pydantic v2 / dataclass / TypedDict / msgspec), updated generated-code example blocks and help text (e.g., --output-model-type, --use-frozen-field, --use-field-description, --allof-merge-mode, --enum-field-as-literal).
JSON Schema tests — multi-model parametrization
tests/main/jsonschema/test_main_jsonschema.py
Replaced single golden_output usages with model_outputs mappings; converted many tests to parameterized forms accepting output_model/expected_file; consolidated reserved-field and frozen-field tests into multi-model parameterized suites.
OpenAPI tests — multi-target and py-version cases
tests/main/openapi/test_main_openapi.py
Converted a single-model discriminator-default test into a parameterized multi-model test; removed the old dataclass-only variant; added dataclass-specific tests for Python 3.10 and a Python 3.9 warning path; added @pytest.mark.cli_doc mappings.
GraphQL & other tests — CLI doc markers and new case
tests/main/graphql/test_main_graphql.py, tests/test_main_kr.py
Added @pytest.mark.cli_doc decorators and model_outputs mappings in GraphQL tests; added new test test_main_use_field_description with CLI-doc marker to validate --use-field-description output.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

  • Pay attention to per-version/generated example accuracy in docs/cli-reference/*.
  • Verify model_outputs mappings, parameter lists, and expected-file paths in tests/main/jsonschema/test_main_jsonschema.py and tests/main/openapi/test_main_openapi.py.
  • Confirm the new CLI-doc decorators and added test test_main_use_field_description wiring in tests/test_main_kr.py and tests/main/graphql/test_main_graphql.py.

Possibly related PRs

Poem

🐰 I hopped through docs and tests with cheer,

Split outputs by model far and near,
Parametrized runs in tidy arrays,
Dataclasses and v2 brighten my days,
A little rabbit stamps the changes here 🥕

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: refactoring e2e tests to reduce duplication via parameterization and enhancing CLI documentation with model_outputs.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@koxudaxi koxudaxi marked this pull request as ready for review December 17, 2025 07:52
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 17, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.52%. Comparing base (7df80ab) to head (541b73e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2675      +/-   ##
==========================================
- Coverage   99.52%   99.52%   -0.01%     
==========================================
  Files          79       79              
  Lines       11178    11172       -6     
  Branches     1348     1348              
==========================================
- Hits        11125    11119       -6     
  Misses         32       32              
  Partials       21       21              
Flag Coverage Δ
unittests 99.52% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Dec 17, 2025

CodSpeed Performance Report

Merging #2675 will not alter performance

Comparing refactor/parameterize-e2e-tests-with-model-outputs (541b73e) with main (7df80ab)

Summary

✅ 45 untouched
🆕 7 new
⏩ 10 skipped1

Benchmarks breakdown

Benchmark BASE HEAD Change
🆕 test_main_allof_root_model_constraints_none N/A 286.9 ms N/A
🆕 test_main_jsonschema_reserved_field_name[typing.TypedDict-reserved_field_name_schema_typed_dict.py] N/A 29 ms N/A
🆕 test_main_jsonschema_reserved_field_name[pydantic_v2.BaseModel-reserved_field_name_schema_pydantic.py] N/A 45.6 ms N/A
🆕 test_main_jsonschema_reserved_field_name[dataclasses.dataclass-reserved_field_name_schema_dataclass.py] N/A 29.8 ms N/A
🆕 test_main_use_frozen_field[pydantic_v2.BaseModel-use_frozen_field_v2.py] N/A 54 ms N/A
🆕 test_main_use_frozen_field[pydantic.BaseModel-use_frozen_field_v1.py] N/A 48.6 ms N/A
🆕 test_main_use_frozen_field[dataclasses.dataclass-use_frozen_field_dataclass.py] N/A 35.8 ms N/A

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
docs/cli-reference/model-customization.md (1)

18-29: Broaden --output-model-type summary beyond GraphQL-only wording

The table entry now says “Generate data models from GraphQL schema definitions,” but --output-model-type applies to all supported inputs (JSON Schema, OpenAPI, GraphQL, etc.). Consider rephrasing to something like “Select the backend for generated data models (e.g., Pydantic v1/v2, dataclasses, TypedDict, msgspec)” so the summary matches the actual CLI behavior; the GraphQL focus can stay in the detailed section. The --use-frozen-field description is clear and accurate.

tests/main/jsonschema/test_main_jsonschema.py (1)

1963-1971: Path prefix inconsistency in model_outputs.

Lines 1968-1970 use "main/jsonschema/" prefix while lines 2016-2018 use "jsonschema/" prefix. While this may match actual file locations, consider standardizing the path structure for consistency across test expectations.

Also applies to: 2015-2018

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f42c49f and c42d5e3.

📒 Files selected for processing (7)
  • docs/cli-reference/field-customization.md (2 hunks)
  • docs/cli-reference/model-customization.md (6 hunks)
  • docs/cli-reference/quick-reference.md (4 hunks)
  • tests/main/graphql/test_main_graphql.py (1 hunks)
  • tests/main/jsonschema/test_main_jsonschema.py (6 hunks)
  • tests/main/openapi/test_main_openapi.py (1 hunks)
  • tests/test_main_kr.py (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
tests/main/openapi/test_main_openapi.py (2)
tests/test_main_kr.py (1)
  • output_file (44-46)
tests/main/conftest.py (2)
  • output_file (94-96)
  • run_main_and_assert (196-352)
tests/main/jsonschema/test_main_jsonschema.py (2)
tests/test_main_kr.py (1)
  • output_file (44-46)
tests/main/conftest.py (2)
  • output_file (94-96)
  • run_main_and_assert (196-352)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: 3.9 on Windows
  • GitHub Check: benchmarks
🔇 Additional comments (10)
tests/main/graphql/test_main_graphql.py (1)

30-38: Nice cli_doc metadata and docstring alignment

The cli_doc marker (options, input_schema, cli_args, model_outputs) lines up cleanly with the parametrized outputs, and the updated docstring clearly explains --output-model-type in this GraphQL context. No issues from my side.

Also applies to: 44-48

tests/test_main_kr.py (1)

281-287: Well-structured coverage for --use-field-description

The new cli_doc marker, golden output wiring, and explanatory docstring are consistent with the existing schema/inline description tests and exercise the option cleanly.

Also applies to: 289-295

tests/main/openapi/test_main_openapi.py (1)

2864-2896: Parametrized discriminator default tests and cli_doc look consistent

The new parametrized test plus cli_doc metadata correctly exercise --use-one-literal-as-default for both Pydantic v2 and dataclass outputs, with expected paths built via EXPECTED_OPENAPI_PATH / expected_file. The dedicated 3.10/3.9 dataclass tests add clear coverage of version-specific behavior. No changes requested.

docs/cli-reference/field-customization.md (2)

1342-1393: Pydantic v1/v2 split for --field-extra-keys looks consistent and clear

The new examples correctly distinguish v1 (extra keys as Field kwargs) from v2 (extra keys in json_schema_extra with examples), aligning with the described behavior for extra field metadata while keeping the schema-driven values visible.


1455-1523: --field-extra-keys-without-x-prefix multi-model output is well-aligned

The v1 example shows sanitized keyword arguments (e.g., field_comment, register_, schema_, x_abc) while the v2 example moves the raw extension keys into json_schema_extra and maps x-repr to repr, which matches the intent of the option and keeps invalid identifiers out of the signature.

docs/cli-reference/model-customization.md (2)

2950-3440: GraphQL-based --output-model-type examples are coherent for Pydantic v1 and dataclasses

The updated section clearly demonstrates how a single GraphQL schema can be rendered into Pydantic v1 models (with Field(..., alias='__typename')) and into dataclasses (with typename__ defaults) while keeping scalar aliases (Boolean, ID, Int, String) consistent. Cross-references between Film/Person/Planet/Species/Starship/Vehicle line up correctly, and the forward-ref updates for Pydantic are in place for the circular references.


4726-4839: --use-frozen-field description and multi-backend outputs accurately capture behavior

The new bullets correctly describe:

  • Pydantic v1 using Field(allow_mutation=False) plus validate_assignment=True,
  • Pydantic v2 using Field(..., frozen=True) and AwareDatetime for date-time,
  • Dataclasses ignoring this flag (no frozen fields).

The corresponding examples match that behavior and clearly show that only readOnly properties (id, created_at) become frozen while password (writeOnly) stays mutable.

tests/main/jsonschema/test_main_jsonschema.py (3)

435-439: LGTM! Multi-model output support added correctly.

The transition from golden_output to model_outputs enables the documentation to display outputs for both Pydantic v1 and v2 in tabbed format, while the test continues to verify both variants through parameterization.


4099-4139: Excellent parameterization and documentation.

The test consolidation is well-executed with clear parameterization across three model types. The docstring (lines 4122-4126) effectively documents the behavioral differences for the schema field across TypedDict, dataclass, and Pydantic models, making this an excellent example test.


4198-4233: Well-structured parameterization with comprehensive documentation.

The parameterization successfully consolidates tests for frozen field behavior across three model types. The docstring (lines 4221-4225) provides valuable documentation about the different implementations, noting that dataclasses silently ignore the flag—important information for users.

Comment thread docs/cli-reference/quick-reference.md Outdated
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
tests/main/jsonschema/test_main_jsonschema.py (2)

4103-4143: Consider clarifying the CLI args in the documentation marker.

The cli_args in the @pytest.mark.cli_doc decorator on line 4114 shows ["--target-python-version", "3.11"], but the actual test execution on lines 4137-4142 includes ["--output-model-type", output_model, "--target-python-version", "3.11"].

Since this test demonstrates behavior across different model types (TypedDict, dataclass, Pydantic v2), consider either:

  1. Adding a representative --output-model-type to the cli_args (e.g., for pydantic_v2)
  2. Or documenting in a comment that the CLI args vary per model type

This helps ensure the documentation accurately reflects the command needed to reproduce the documented output.


4202-4237: Similar CLI args documentation issue as reserved_field_name test.

The cli_args on line 4213 shows ["--use-frozen-field"], but the actual test execution on line 4236 includes ["--output-model-type", output_model, "--use-frozen-field"]. Since this test demonstrates different behavior across model types, consider adding a representative --output-model-type to the cli_args for documentation clarity.

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c42d5e3 and b3a8ce2.

📒 Files selected for processing (4)
  • docs/cli-reference/model-customization.md (6 hunks)
  • docs/cli-reference/quick-reference.md (4 hunks)
  • tests/main/graphql/test_main_graphql.py (1 hunks)
  • tests/main/jsonschema/test_main_jsonschema.py (6 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/main/graphql/test_main_graphql.py
  • docs/cli-reference/quick-reference.md
🧰 Additional context used
🧬 Code graph analysis (1)
tests/main/jsonschema/test_main_jsonschema.py (2)
tests/test_main_kr.py (1)
  • output_file (44-46)
tests/main/conftest.py (2)
  • output_file (94-96)
  • run_main_and_assert (196-352)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
  • GitHub Check: py312-isort6 on Ubuntu
  • GitHub Check: py312-isort7 on Ubuntu
  • GitHub Check: py312-black23 on Ubuntu
  • GitHub Check: 3.11 on Windows
  • GitHub Check: 3.10 on Windows
  • GitHub Check: 3.12 on Windows
  • GitHub Check: 3.9 on Windows
  • GitHub Check: 3.14 on Windows
  • GitHub Check: 3.13 on macOS
  • GitHub Check: 3.13 on Windows
  • GitHub Check: benchmarks
  • GitHub Check: Analyze (python)
🔇 Additional comments (7)
docs/cli-reference/model-customization.md (3)

18-18: Table descriptions updated for clarity; minimal changes.

Lines 18 and 29 clarify the supported model types in the table summaries. The changes are minor but improve consistency with the expanded documentation below.

Also applies to: 29-29


2948-3040: Multi-version output examples added to --output-model-type section.

The section now provides tabbed examples showing Pydantic v1 and v2 output differences. The examples correctly highlight version-specific behavior:

  • Pydantic v1 (line 3013): items: Optional[List[Any]] without explicit default
  • Pydantic v2 (line 3035): items: Optional[List[Any]] = None with explicit default

This demonstrates real API differences between versions and helps users understand version-specific output.


4324-4439: Multi-framework output examples added to --use-frozen-field section.

The section now includes tabbed examples for Pydantic v1, Pydantic v2, and dataclass outputs, demonstrating how --use-frozen-field affects each framework:

  • Pydantic v1 (lines 4390-4391): Config with validate_assignment = True; fields use allow_mutation=False
  • Pydantic v2 (lines 4413-4418): Uses frozen=True in Field definitions
  • Dataclass (lines 4433-4439): Fields remain unchanged (silently ignored per documentation at line 4331)

The examples are accurate and consistent with each framework's design. The documentation note at lines 4328-4331 correctly explains per-framework behavior.

tests/main/jsonschema/test_main_jsonschema.py (4)

435-456: LGTM! Good refactoring for multi-model documentation support.

The addition of model_outputs to the CLI doc marker enables multi-model tab display in documentation, and the updated docstring clearly explains the purpose of the test across different model types.


967-999: LGTM! Consistent multi-model documentation pattern.

The model_outputs addition follows the same pattern as other parameterized tests and correctly maps to the existing test parameters.


2019-2046: LGTM! Consistent implementation.

The multi-model outputs are properly mapped to the existing parameterization.


4250-4250: Good catch adding the missing flag!

Adding "--use-frozen-field" to the extra_args on line 4250 ensures this test actually validates the frozen field behavior when no readOnly fields are present.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2e0d5ed and d1d6b72.

📒 Files selected for processing (4)
  • docs/cli-reference/quick-reference.md (7 hunks)
  • docs/cli-reference/typing-customization.md (4 hunks)
  • tests/main/graphql/test_main_graphql.py (3 hunks)
  • tests/main/jsonschema/test_main_jsonschema.py (7 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
tests/main/jsonschema/test_main_jsonschema.py (1)
tests/main/conftest.py (2)
  • output_file (94-96)
  • run_main_and_assert (196-352)
🪛 markdownlint-cli2 (0.18.1)
docs/cli-reference/quick-reference.md

30-30: Strong style
Expected: asterisk; Actual: underscore

(MD050, strong-style)


30-30: Strong style
Expected: asterisk; Actual: underscore

(MD050, strong-style)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
  • GitHub Check: 3.10 on Windows
  • GitHub Check: 3.11 on Windows
  • GitHub Check: 3.10 on macOS
  • GitHub Check: 3.11 on macOS
  • GitHub Check: 3.14 on Windows
  • GitHub Check: 3.9 on Windows
  • GitHub Check: 3.9 on macOS
  • GitHub Check: 3.12 on Windows
  • GitHub Check: 3.13 on Windows
  • GitHub Check: Analyze (python)
  • GitHub Check: benchmarks
🔇 Additional comments (9)
docs/cli-reference/quick-reference.md (1)

29-29: ✓ Addressed previous feedback on --output-model-type description.

The updated description now correctly reflects that the option applies globally across input types (OpenAPI, JSON Schema, GraphQL), no longer misleadingly appearing GraphQL-specific. All four updated descriptions are clear and accurate:

  • --allof-merge-mode: Emphasizes constraint merging from root model references
  • --enum-field-as-literal: Clarifies the conversion from Enum classes to Literal types
  • --output-model-type: Now properly general: "Select the output model type (Pydantic v1/v2, dataclasses, TypedDict, msgspec)"
  • --use-frozen-field: Specifies the readOnly property context

Descriptions are consistently updated in both the category table and alphabetical index.

Also applies to: 31-31, 86-86, 97-97, 158-158, 180-180, 207-207, 235-235

docs/cli-reference/typing-customization.md (3)

7-9: Enhanced option descriptions and comprehensive behavior documentation.

The updated descriptions for --allof-merge-mode and --enum-field-as-literal are now more precise:

  • --allof-merge-mode: Clarifies that it "merges only constraint properties (minLength, maximum, etc.) from parent schemas referenced in allOf," explicitly distinguishing constraint-only merging from full property inheritance.
  • --enum-field-as-literal: More direct phrasing: "Convert all enum fields to Literal types instead of Enum classes."

Also applies to: 28-33


45-256: Thorough input schema and comparative output examples.

The extended test schema for --allof-merge-mode covers critical edge cases:

  • Root model constraints (ConstrainedStringDatatype, NonNegativeIntegerDatatype)
  • Non-root-model allOf (ObjectWithAllOf, RefToArrayAllOf)
  • Incompatible types and conflicting formats
  • Nested allOf and constraints-only schemas

The "With Option" vs "Without Option" output comparison clearly demonstrates the semantic difference in constraint merging. This level of documentation is excellent for users understanding the feature's scope and limitations.

Also applies to: 260-592


682-789: Appropriate GraphQL schema for enum examples.

Using GraphQL as the input format for --enum-field-as-literal examples is fitting, as GraphQL enums have a direct mapping to Literal types. The dual output comparison effectively shows the Literal-based vs Enum-based approaches.

tests/main/jsonschema/test_main_jsonschema.py (4)

4111-4120: cli_args may need to include --output-model-type for accurate documentation.

The cli_args array (line 4114) doesn't specify --output-model-type, but the test is parameterized across three different output model types (TypedDict, dataclass, Pydantic v2). For the documentation to accurately reflect the commands for each model type, consider whether cli_args should be parameterized or if the model_outputs approach handles this automatically.

Additionally, verify that the model_outputs keys ("typeddict", "dataclass", "pydantic_v2") align with the documentation system's expected naming convention.


4210-4219: cli_args missing --output-model-type specification.

Similar to the previous test, the cli_args (line 4213) don't include --output-model-type, even though this test is parameterized across three model types. The model_outputs mapping may handle this, but it's worth verifying that documentation generation correctly shows the appropriate model type for each tab.


4269-4292: LGTM - Clear documentation of new allof-merge-mode feature.

The addition of the --allof-merge-mode option with clear documentation is well implemented. The docstring effectively explains the behavior, and the use of comparison_output (line 4274) will help users understand the difference between merge modes.


435-439: No changes needed. The code is correct as-is.

tests/main/graphql/test_main_graphql.py (1)

30-38: LGTM! Well-structured CLI documentation metadata.

The cli_doc marker correctly maps the parameterized test cases to model_outputs for documentation purposes, and the cli_args provides a clear default example.

Comment thread tests/main/graphql/test_main_graphql.py
@koxudaxi koxudaxi merged commit 0857ae7 into main Dec 18, 2025
69 of 70 checks passed
@koxudaxi koxudaxi deleted the refactor/parameterize-e2e-tests-with-model-outputs branch December 18, 2025 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant