Skip to content

Check the whole forced report, and skip visibly when it cannot be checked - #16

Merged
emrefbulut merged 1 commit into
mainfrom
fix/force-width-test
Sep 13, 2026
Merged

emrefbulut merged 1 commit into
mainfrom
fix/force-width-test

Conversation

@emrefbulut

Copy link
Copy Markdown
Owner

What changed

test_the_report_is_78_columns_of_ascii now checks every line of the --force output instead of stopping at the line ending in MEASUREMENT, and asserts the exit code and a minimum line count first.

The one assertion that genuinely cannot hold without torch — that the forced MEASUREMENT block is present — moves into its own test behind an explicit skipif.

Why

The old test looked like it covered the quotable report. It did not:

for line in text.splitlines():
    if line.strip().endswith("MEASUREMENT"):
        break                      # <- everything after this was never checked
    block_lines.append(line)

Its coverage silently depended on the environment:

  • with torch — the forced measurement block exists, and the break skipped it
  • without torch — the block does not exist at all, so there was nothing to skip

Either way the output --force uniquely produces had never been width- or ASCII-checked in any environment, and the test reported passed in both. Dumping a real run shows exactly what was escaping (lines 35–40):

32 [checked    ] started       no. This version of the command stops before training
33 [checked    ] ==============================================================================
34 [checked    ]
35 [AFTER-BREAK] FORCED MEASUREMENT
36 [AFTER-BREAK] stride=512  seed pairs=1  split=42  train=0
37 [AFTER-BREAK] inflation=+0.0 pp (uncertainty not estimated, n=1)
38 [AFTER-BREAK] recording-level: test 50.00%, window-level: test 50.00%
39 [AFTER-BREAK] environment: device cpu | torch 2.13.0+cpu | numpy 2.5.1 | scipy 1.18.0
40 [AFTER-BREAK]              sigmf 1.13.0 | cuda none

The exit-code and line-count assertions are there for the same class of reason: without them a run producing no output at all would satisfy every loop by iterating zero times.

How it was verified

The skip is now visible. Forcing _torch_installed() to return False — the state CI's test (3.11) / test (3.12) jobs are in:

SKIPPED [1] tests/test_preflight.py:323: the forced MEASUREMENT block only exists
once training can run; install the torch extra to exercise it
1 passed, 1 skipped, 17 deselected

skipped, with a reason, rather than a silently narrowed passed.

Mutations — and one of them corrected my own assumption:

mutation result
unbreakable 95-character token in the measurement block caught by both tests
non-ASCII character in the measurement block caught by both tests
80-column breakable line in the measurement block not caught

The third is the honest caveat and I am not going to paper over it. The measurement block is printed through rich, which wraps at the console width, so an over-long sentence is wrapped before it ever reaches the output and cannot violate the contract. The width assertion on that block is therefore load-bearing for content rich cannot wrap — paths, hashes, long numbers — and for ASCII. That is narrower than "this block is 78 columns", and claiming the wider thing would repeat the mistake this PR is fixing.

The preflight block above it is built with _field_lines and plain print, so there the width check is load-bearing in full.

Conventions it touches

  • 2 A passing test does not prove it can fail — the whole PR. The old test had never been red on the block it appeared to cover, and my first mutation attempt showed my replacement was weaker than I assumed until I found one rich cannot wrap.
  • 3 Do not claim what you did not measure — the caveat above, and the skip reason naming what is missing.
  • 1 Never fall back silently — applied to tests: a check that cannot run must say so, not quietly check less.

Noticed, not fixed here

Line 32 above says started no. This version of the command stops before training and lines 35–40 are the measurement it then performed. That is the self-contradiction tracked as Bölüm C item 5; this PR only stops the test from ignoring that region.

…cked

test_the_report_is_78_columns_of_ascii stopped at the line ending in
MEASUREMENT and checked only what came before it. That made its coverage
depend on the environment without saying so. With torch installed the forced
measurement block exists and the break skipped it; without torch the block
does not exist at all. Either way the output --force uniquely produces had
never been width- or ASCII-checked in any environment, and the test reported
a pass in both.

It now checks every line of the output, and asserts the exit code and a
minimum line count first: without those, a run that produced no output at
all would satisfy both loops by iterating zero times.

The assertion that cannot hold without torch -- that the forced MEASUREMENT
block is actually present -- moves to its own test behind an explicit
skipif. A CI job that cannot run it now reports "skipped" with a reason
instead of quietly passing a narrower check. Verified by forcing
_torch_installed() False: 1 passed, 1 skipped, reason printed. That test
uses one seed pair, because it reads line lengths rather than measuring
anything, and the default 15 would cost 30 training runs to do it.

What the width assertion does and does not catch, measured rather than
assumed. The measurement block is printed through rich, which wraps at the
console width, so an over-long sentence is wrapped before it reaches the
output and cannot violate the contract. Mutations run: an unbreakable
95-character token is caught, a non-ASCII character is caught, an 80-column
breakable line is NOT caught because rich wraps it first. The check is
therefore load-bearing for content rich cannot wrap -- paths, hashes, long
numbers -- and for ASCII. That is narrower than "the block is 78 columns"
and is the honest description.
@emrefbulut
emrefbulut merged commit 7d0a7aa into main Sep 13, 2026
5 checks passed
emrefbulut added a commit that referenced this pull request Sep 14, 2026
The stacked PRs #18, #19 and #20 were merged at the same moment, so only #18
reached main -- #19 merged into fix/gate-semantics-and-provenance and #20 into
docs/refresh, their own bases. Everything from those two is therefore sitting
on this branch and main is still on 0.4.0. That was my mistake in stacking
three deep instead of retargeting; this merge is the fix.

One conflict, in CHANGELOG.md. main carried the entry describing
docs/release-notes/v0.5.0.md as an unpublished draft, which commit 6491f27 on
this branch had already rewritten into "the four places that carry a version
agree again" -- the draft label was the state this release ends. Kept this
branch's side, which also carries three entries main does not have.

Checked rather than assumed, because a merge across branches that both edited
the test files could silently drop one side: #17's CUDA comparability tests
and #16's forced-measurement width test are both present afterwards, and the
suite goes from 385 to 388 passing, which is main's three additions arriving
rather than anything being lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant