Skip to content

test(cli): skip invalid-command snapshot on Python 3.12.0-3.12.6#1982

Open
bearomorphism wants to merge 1 commit intocommitizen-tools:masterfrom
bearomorphism:fix/1864-skip-old-3.12-argparse-snapshot
Open

test(cli): skip invalid-command snapshot on Python 3.12.0-3.12.6#1982
bearomorphism wants to merge 1 commit intocommitizen-tools:masterfrom
bearomorphism:fix/1864-skip-old-3.12-argparse-snapshot

Conversation

@bearomorphism
Copy link
Copy Markdown
Collaborator

@bearomorphism bearomorphism commented May 9, 2026

Description

Closes #1864.

Why

tests/test_cli.py::test_invalid_command is a snapshot test that captures the stderr output of cz <invalid>. The output is generated by argparse, and CPython 3.13 changed the format of the invalid choice: error from quoted choices ('init', 'commit', ...) to unquoted (init, commit, ...) — see CPython gh-129019. The change was backported to 3.12.7. The repo's reference snapshots reflect the no-quote format, so:

Python Quoted? Snapshot match?
3.10.x, 3.11.x yes yes (snapshots use quotes)
3.12.0 - 3.12.6 yes no — test fails
3.12.7+, 3.13.x, 3.14.x no yes (snapshots use no quotes)

A user on 3.12.3 hits AssertionError: FILES DIFFER running pytest tests/test_cli.py::test_invalid_command. The CI matrix runs on a recent 3.12.x, so CI doesn't catch it.

What changed

File Change
tests/test_cli.py Add pytest.mark.skipif(sys.version_info[:2] == (3, 12) and sys.version_info < (3, 12, 7), reason=...) to the test_invalid_command parametrised test. The reason text references CPython gh-129019 and this issue.

How it works

  • The first guard sys.version_info[:2] == (3, 12) scopes the skip to Python 3.12.x only — older 3.10/3.11 (which still print quoted choices) and newer 3.13/3.14 (which print unquoted) are unaffected.
  • The second guard sys.version_info < (3, 12, 7) uses Python's lexicographic tuple comparison: (3, 12, 6, ...) < (3, 12, 7) is True, (3, 12, 7, ...) < (3, 12, 7) is False. So 3.12.0 through 3.12.6 skip; 3.12.7+ run.
  • The --invalid-arg parametrised case shares the same decorator. Its argparse path also formats choices, so the same skip logic applies even though the diff message is slightly different.
  • 3.10 / 3.11 snapshots use quotes, so they would also break if argparse's no-quote change is back-ported to those branches. However, both are in security-only maintenance (3.10 is EOL, 3.11 since April 2024), and a formatting-only change won't be merged under that policy. We don't need to skip 3.10/3.11.

Why a skipif instead of duplicate snapshots?

A skipif is cheaper to maintain (no new fixture file, no risk of the duplicate snapshot drifting). It's also more honest: argparse's behaviour on 3.12.0-3.12.6 is a known upstream issue, fixed in 3.12.7. Users on those patches can upgrade; we don't need commitizen to keep accumulating fixtures for upstream's old behaviour.

Backward compatibility

  • Test-only change. No behaviour change for cz users.
  • The CI matrix (Ubuntu/macOS/Windows × 3.10/3.11/3.12/3.13/3.14) continues to exercise test_invalid_command on every supported Python because the GitHub-hosted runners use a recent patch within each minor (3.12.10+ at the time of writing).

Checklist

Was generative AI tooling used to co-author this PR?

  • Yes (please specify the tool below)

Generated-by: Claude following the guidelines

Code Changes

  • Add test cases to all the changes you introduce (this PR is a test-only change; the regression itself is the existing snapshot, the fix is the skip)
  • Run uv run poe all locally to ensure this change passes linter check and tests (poe lint clean; verified the test runs on a recent 3.12.x and skips on an old one — see "Steps to Test")
  • Manually test the changes (see "Steps to Test" below)
  • Update the documentation for the changes (no doc change required — this is a test-fixture compatibility fix)

Expected Behavior

Python interpreter pytest tests/test_cli.py -k test_invalid_command
3.10.x runs and passes (snapshot uses quotes, argparse uses quotes)
3.11.x runs and passes (same)
3.12.0 - 3.12.6 skipped with the gh-129019 rationale
3.12.7+ runs and passes (snapshot has no quotes, argparse has no quotes)
3.13.x runs and passes (same)
3.14.x runs and passes (same)

Steps to Test This Pull Request

git fetch fork fix/1864-skip-old-3.12-argparse-snapshot
git checkout fork/fix/1864-skip-old-3.12-argparse-snapshot

# 1. On a recent 3.12 patch (any 3.12.7+) — both parametrised cases run and pass.
uv run --python 3.12 pytest tests/test_cli.py::test_invalid_command -v
# expected: 2 passed

# 2. On an old 3.12 patch (where argparse still quotes) — both are skipped with the gh-129019 reason.
uv run --python 3.12.3 pytest tests/test_cli.py::test_invalid_command -v
# expected: 2 skipped (with the rationale visible in -v output)

# 3. On every other supported Python — same as before, no skip.
for py in 3.10 3.11 3.13 3.14 ; do
  uv run --python "$py" pytest tests/test_cli.py::test_invalid_command -v
done
# expected: 2 passed for each

# 4. Full test_cli.py suite (sanity check that nothing else broke).
uv run pytest tests/test_cli.py -q
# expected: 22 passed, 1 skipped (on old 3.12) OR 23 passed (on a recent 3.12 / other versions)

Additional Context

Surfaced while triaging open issues in #1976 (round 3). The original reporter and @noirbizarre (the assignee) had already discussed skipping the test on 3.12 — this PR scopes that skip to the affected patch range so the test still runs on every patch where the snapshot is valid. Reviewed by an internal claude-sonnet-4.6 code-review pass and by GitHub Copilot; both confirmed the version-tuple comparison and that the 3.10/3.11 snapshots are safe (security-only maintenance precludes the same back-port there).

Closes commitizen-tools#1864.

The reference snapshots for `test_invalid_command` reflect the
no-quote argparse error format that landed in CPython 3.13 and was
backported to 3.12.7 (gh-129019). On Python 3.12.0-3.12.6 argparse
still prints quoted choices, so the snapshot diff fails for those
patch releases.

Skip the test on 3.12.0-3.12.6 with a precise `skipif` rather than
maintaining a duplicate snapshot. The CI matrix already runs on
recent 3.12.x where the test is exercised; users on older patches
just see the test as `s` instead of a hard failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.23%. Comparing base (4b93a50) to head (831b8a8).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1982   +/-   ##
=======================================
  Coverage   98.23%   98.23%           
=======================================
  Files          61       61           
  Lines        2779     2779           
=======================================
  Hits         2730     2730           
  Misses         49       49           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the CLI snapshot test suite to account for an argparse error-message formatting difference present in Python 3.12.0–3.12.6, where choices are rendered with quotes, conflicting with the existing reference snapshots (which match the 3.12.7+/3.13+ “no-quote” format).

Changes:

  • Skip test_invalid_command on Python 3.12.0–3.12.6 using a precise pytest.mark.skipif patch-version check.
  • Add an explanatory skip reason referencing the upstream argparse behavior change and the related issue.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: test_invalid_command fails on Python 3.12

2 participants