Skip to content

Every negation in the bats suites asserts wherever it stands - #16

Merged
marcos-mendez merged 1 commit into
masterfrom
test/negations-that-assert
Sep 29, 2026
Merged

marcos-mendez merged 1 commit into
masterfrom
test/negations-that-assert

Conversation

@marcos-mendez

Copy link
Copy Markdown
Collaborator

Every negation in this suite now asserts wherever it stands. Twenty-two of the thirty-one did not.

Closes #15. Part of Keel-Linux/tracker#19, which has the organization-wide sweep.

Why a bare ! is inert

A bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command — that exemption is in the shell by design, so a script can test a command without dying — so ! cmd decides the test in final position and is inert everywhere else. The line reads as a refutation either way.

shellcheck grades the two cases apart, which is what made an exact sweep possible:

shellcheck -f gcc --shell=bash tests/*.bats | grep SC2314

error: is an inert one. note: is one in final position, which asserts today and stops asserting the moment a line is added after it.

The headline: twelve of thirteen IPv6 rejection cases

tests/test-ipconfig.bats, "ip6_syntax rejects what is not an IPv6 address": thirteen cases, lines 324–336. Twelve were inert. Only 1:2:3:4:5:6:7::8 decided anything.

Before changing them, each of the thirteen was run against ipconfig_ip6_syntax by hand, so that a green suite afterwards would be evidence rather than a coincidence. All thirteen are refused, and all eight of the acceptance cases are accepted. The validator is correct; it was the tests that were not running.

That matters for what this PR claims: nothing shipped broken here. It is also luck, not evidence, and it is the reason the gate in Keel-Linux/.github exists rather than a note in a review.

The rest

File Inert What was not being checked
tests/test-ipconfig.bats 17 the twelve above, plus valid_config refusing bogus and '', unchanged refusing an absent stanza, the lxc build leaving no eth0, unchanged refusing a mismatched inet6 config
tests/test-init-fence.bats 3 nat_available for a family with no nat table — the case the predicate exists for — and two processes that must be dead after stop_mini_server
tests/test-tagid.bats 2 an untagged page is not tagged, and @APP_NAME@ does not survive the copy into the writable htdocs

What changed

Every SC2314 site becomes run ! cmd, the nine that were already in final position included, and each file declares bats_require_minimum_version 1.5.0 as the other suites in the organization do. Leaving half of them in the old shape would leave a reader to work out which half is real, and the guard reports severity error only, so a final-position ! is not a failure — it is simply a line that will go inert when somebody appends to the body.

One site was a pipeline and could not be prefixed:

-    ! printf '%s\n' "${lines[@]}" | grep -q WARNING
+    run ! grep -q WARNING <<<"$output"

run takes a command, not a pipeline. The here-string is expanded before run executes, so it reads the output of run "$FENCE" start above it.

Test plan

  • The thirteen IPv6 rejection cases verified against the validator by hand, before the conversion. All refused.

  • shellcheck -f gcc --shell=bash --severity=error --include=SC2314 tests/*.bats reports nothing.

  • Each converted assertion proven to bite: replacing one case with a valid address fails the test at that line.

  • bats tests/ green, 112 tests. No assertion changed its verdict.

  • tests/coverage.sh: 99.55 percent of measured lines, threshold 98. Unchanged.

    100.00    19/19    firstboot.d/29tagid              ok
    100.00     8/8     lib/tagid.sh                     ok
    100.00    29/29    firstboot.d/01ipconfig           ok
    100.00    72/72    lib/ipconfig.sh                  ok
    100.00    28/28    bin/turnkey-init-fence           ok
     98.44    63/64    lib/init-fence.sh                ok
     99.55  total
    
  • Tests ship nothing, so package / changelog passes by reporting that nothing that ships changed. No changelog entry and no version bump.

`! cmd` on its own line asserts nothing in a bats body unless it happens to
be the last command of that body: bash does not apply errexit to a negated
command, so the test's verdict is the exit status of the final command and
every earlier `! cmd` is inert. shellcheck grades the two cases apart, error
for the inert ones and note for the rest, which is how these were found:

    shellcheck -f gcc --shell=bash tests/*.bats | grep SC2314

Twenty-two of the thirty-one negations in this suite were inert, the largest
group in the organization. Seventeen of them are in test-ipconfig.bats, and
thirteen of those are the whole of "ip6_syntax rejects what is not an IPv6
address": only the last of the thirteen decided anything, in an IPv6
validator, in an IPv6-first distribution. The other five are
test-init-fence.bats (nat_available for a family whose nat table is missing,
and two processes that must be dead) and test-tagid.bats (an untagged page,
and the placeholder that must not survive the copy).

Every site becomes `run ! cmd`, the form that asserts wherever it stands,
including the nine that were in final position and did assert: a line whose
meaning depends on its position is the trap itself, and leaving half of them
in the old shape leaves a reader to work out which half. That form needs
bats 1.5.0, declared at the top of each file as the other suites in the
organization already declare it.

One site was a pipeline, `! printf ... | grep -q WARNING`, which `run` cannot
take; it reads the captured output of the preceding run instead.

No assertion changed its verdict: all thirteen IPv6 rejection cases reject,
and the suite is green with the assertions live, 112 tests. Coverage is
unchanged, 99.55 percent of measured lines (threshold 98).
@marcos-mendez

marcos-mendez commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Reviewed: all 31 converted refutations go red when the code under test accepts that call (the 13 ip6_syntax cases included), and none reads $output or $status from the run it replaced. One fails with status 3, from the stub, by design. No changes needed. Note: #18 and #14 still carry the old bare negations and will need the same conversion after a merge with master.

@marcos-mendez
marcos-mendez merged commit 371a9f8 into master Sep 29, 2026
3 checks passed
marcos-mendez added a commit that referenced this pull request Sep 29, 2026
…-skip

master gained #16, which rewrote the bare negations in
tests/test-init-fence.bats. The one hunk both branches touched is the
"both nat tables" test, where #16 made 'grep -q WARNING' a 'run !' and
this branch had replaced it with counts of the inserts and of every
prefixed journal line; the counts are kept, as review of both asked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfWQScmDZ94KCS5DMYrfe6
marcos-mendez added a commit that referenced this pull request Sep 29, 2026
…script

master gained #16, which turned two bare negations in the old
tests/test-tagid.bats into 'run !'. This branch rewrote that file and uses
'run !' throughout, so its version is kept whole.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfWQScmDZ94KCS5DMYrfe6
marcos-mendez added a commit that referenced this pull request Sep 29, 2026
Every negation in the bats suites asserts wherever it stands
marcos-mendez added a commit that referenced this pull request Sep 29, 2026
…script

master gained #16, which turned two bare negations in the old
tests/test-tagid.bats into 'run !'. This branch rewrote that file and uses
'run !' throughout, so its version is kept whole.
marcos-mendez added a commit that referenced this pull request Sep 29, 2026
…-skip

master gained #16, which rewrote the bare negations in
tests/test-init-fence.bats. The one hunk both branches touched is the
"both nat tables" test, where #16 made 'grep -q WARNING' a 'run !' and
this branch had replaced it with counts of the inserts and of every
prefixed journal line; the counts are kept, as review of both asked.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

22 of the 31 negations in the bats suites assert nothing, including the IPv6 validator's whole rejection set

1 participant