Skip to content

test(tests): let the packaging guards accept a release candidate - #346

Merged
danielplohmann merged 1 commit into
masterfrom
test/accept-prerelease-version
Sep 14, 2026
Merged

danielplohmann merged 1 commit into
masterfrom
test/accept-prerelease-version

Conversation

@danielplohmann

Copy link
Copy Markdown
Owner

Second blocker found setting up #343's first rehearsal, after #345. The suite forbids the version string the rehearsal needs.

What fails today

RELEASING.md tags a candidate vX.Y.Zrc1, and release_guard.py refuses a tag that does not match __version__, so the version carries the suffix for as long as the candidate is the tip. Setting __version__ = "4.8.0rc1" on master fails three guards:

test_the_version_is_a_three_part_release          ^\d+\.\d+\.\d+$ vs '4.8.0rc1'
test_the_escaper_compatibility_marker_...         ValueError: invalid literal for int(): '0rc1'
test_the_changelog_documents_the_current_version  '4.8.0rc1' not in the documented list

The third is the changelog heading pattern ## \[v([\d.]+)\], whose character class carries no letters.

That is not a cosmetic failure. The release workflow gates on ci.yml concluding success for the tagged commit, so a red suite means the tag can never publish — the pre-release path documented across six repositories cannot be walked in any of them.

The change

_version_tuple returns (release numbers, 1 for a release / 0 for a candidate) — the same ordering #345 gave report import, so a candidate sorts below its own release in both places rather than each having its own idea.

  • the version guard asserts the string is a release or a candidate for one, and that the release part is still three-part;
  • the changelog heading pattern accepts a candidate's own section;
  • a new case pins the ordering and the rejections.

Candidates of one release are deliberately not ordered against each other. 4.8.0a1 against 4.8.0b1 compares equal. Nothing here needs that order, and inventing one would be a rule with no reader.

What still fails, verified rather than assumed: 4.8, 4.8.0-dev, 4.8.0.1, v4.8.0 and "" all raise from _version_tuple, and setting __version__ = "4.8" still fails the same three guards — so this relaxes the shape by exactly one suffix rather than accepting anything.

Verification

  • tests/testPackageMetadata.py → 11 passed, 5 subtests, on master's 4.7.0
  • the same, with __version__ = "4.8.0rc1" and a ## [v4.8.0rc1] section applied locally → 11 passed
  • the whole fast tier under that pre-release version → 2,048 passed, 1 skipped, 1,690 subtests. That is the check that matters: it says the rehearsal's tree is green, not just this file.
  • ruff check and ruff format --check clean

Test-only, so no changelog entry — nothing under src/smda/ or pyproject.toml moves.

The v4.8.0rc1 bump comes next and is held until this merges.

🤖 Generated with Claude Code

RELEASING.md tags a candidate vX.Y.Zrc1 and the release workflow refuses
a tag that does not match __version__, so the version carries the suffix
while a candidate is the tip. Three guards rejected that outright: the
three-part regex, _version_tuple's int() on each dotted part, and the
changelog heading pattern.

They accept a PEP 440 pre-release now and order it below its own release,
matching how SmdaFunction orders a report's version. A two-part version,
a -dev suffix and a four-part version are still rejected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

📊 SMDA Performance Evaluation Benchmark Results

Generated on: 2026-09-14 13:20:56

Summary

Metric Result
Correctness ✅ PASS — 0 / 155 common file(s) differ
Determinism ✅ PASS — base 3 run(s), PR 3 run(s)
Verdict inconclusive — within run-to-run noise (±5.7%)
Median paired speedup +0.52% (95% CI [-0.77%, +1.24%])
Timing noise band ±5.7%

Per-side Timing Context (best of 3/3 runs per file)

Side Files Functions Median best time/file (s) Sum of best times (s) Throughput estimate (func/s)
base 155 173496 0.7853 292.23 ~594
pr 155 173496 0.7380 292.22 ~594

These context rows sum each file's best observed time across repeated runs; they are normalized comparison estimates, not single-run CI wall-clock times.

Paired per-file timing (positive speedup = PR faster):

Statistic Value
Files compared 155
Median paired speedup +0.52% (95% CI [-0.77%, +1.24%])
Mean speedup -2.69% (95% CI [-6.72%, +0.73%])
Std dev / IQR 23.88% / 10.91%
Wilcoxon signed-rank p 0.9779 (n=155)

ℹ️ base and PR are timed in the same job on the same runner, with their passes interleaved base/PR/PR/base/base/PR so that drift over the job cancels rather than landing on whichever side ran last. The noise band above is therefore run-to-run variation on one machine, which is what it is able to measure; a difference inside it is reported as inconclusive. Correctness and determinism are not timing-based and are unaffected either way.

Determinism (self-check across repeated runs)

Side Runs Files Deterministic Median timing CV
base 3 155 ✅ 5.7%
pr 3 155 ✅ 5.7%
Pairwise run matrix (individual run medians, diagnostic)

Diagnostic only: each row compares one raw PR run against one raw base run. The headline verdict above uses paired per-file best-of-runs timings.

Comparison Run PR Files Base Files Common Function Set Matches Med PR (s) Med Base (s) Med Diff (s) Speedup %
pr_0 vs base_0 155 155 155 155/155 0.9893s 1.0070s -0.0177s +1.76%
pr_0 vs base_1 155 155 155 155/155 0.9893s 0.9410s +0.0484s -5.14%
pr_0 vs base_2 155 155 155 155/155 0.9893s 1.0234s -0.0340s +3.33%
pr_1 vs base_0 155 155 155 155/155 0.9341s 1.0070s -0.0730s +7.25%
pr_1 vs base_1 155 155 155 155/155 0.9341s 0.9410s -0.0069s +0.73%
pr_1 vs base_2 155 155 155 155/155 0.9341s 1.0234s -0.0893s +8.73%
pr_2 vs base_0 155 155 155 155/155 0.8959s 1.0070s -0.1112s +11.04%
pr_2 vs base_1 155 155 155 155/155 0.8959s 0.9410s -0.0451s +4.79%
pr_2 vs base_2 155 155 155 155/155 0.8959s 1.0234s -0.1275s +12.46%

@danielplohmann
danielplohmann merged commit 9a6becb into master Sep 14, 2026
35 checks passed
@danielplohmann
danielplohmann deleted the test/accept-prerelease-version branch September 14, 2026 13:25
@danielplohmann danielplohmann added this to the v4.8.0 milestone Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant