Skip to content

A negated command only asserts in final position - #11

Merged
marcos-mendez merged 2 commits into
masterfrom
test/negations-that-assert
Sep 29, 2026
Merged

marcos-mendez merged 2 commits into
masterfrom
test/negations-that-assert

Conversation

@marcos-mendez

Copy link
Copy Markdown
Collaborator

Every negation in these suites now asserts wherever it stands. Ten did not assert at all, the largest group in one file in the organization.

Closes #10. Part of Keel-Linux/tracker#19, which has the organization-wide sweep.

Why a bare ! is inert

A bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command, so ! cmd decides the test in final position and is inert everywhere else. shellcheck grades the two apart, error: for the inert ones and note: for the rest:

shellcheck -f gcc --shell=bash tests/*.bats | grep SC2314

What was not being checked

Where Was not asserting
115–119 five of the six cases of "is_global_ipv6 refuses link local, loopback, multicast and IPv4". Only the empty string ran, so fe80::1, FE80::1, ::1, ff02::1 and an IPv4 address were unchecked. This is the predicate that picks the address the boot test then talks to.
100, 101 bt_is_container_name refusing a leading dot and an empty string, the two LXC refuses
103 bt_is_appliance_name 9wordpress
196 bt_is_ssh_banner "220 ready" — the answer a mail server gives. The test is named "accepts an OpenSSH banner and nothing else"; the "nothing else" half did not run.
202 bt_firstboot_done_in with the flag still RUN_FIRSTBOOT=true, which is the whole point of the predicate
407 bt_password_in_config with the wrong password. The test read as though a mismatch was proven and only the empty file was.

All of them are run ! now, together with the seven that were in final position and did assert: a line whose meaning depends on its position is the trap itself.

What was found

Nothing broken. Every predicate was run against every one of its cases by hand before the conversion, and each answers as its test believed: bt_is_global_ipv6 refuses all five refusal cases and accepts all three acceptance cases, bt_is_ssh_banner refuses 220 ready, bt_password_in_config refuses the wrong password, and so on. The assertions were dead, not wrong.

That is luck rather than evidence, which is why the sweep comes with a gate in Keel-Linux/.github and a docs/traps.md entry.

Test plan

  • Every predicate verified against every one of its cases by hand, before the conversion.

  • shellcheck -f gcc --shell=bash --severity=error --include=SC2314 tests/*.bats reports nothing.

  • Every converted site audited for run clobbering $output/$status/$lines; none here does.

  • bats tests/*.bats green, 196 tests. No assertion changed its verdict.

  • tests/coverage.sh, threshold 97, unchanged:

    100.00  31/31    zz-project-packages
    100.00  26/26    zzz-keel-archive
     99.00  99/100   wordpress.sh
     97.73  43/44    40wordpress
     98.95  282/285  boot-test-lib.sh
    100.00  54/54    keel-archive-check
    
  • Tests ship nothing, so package / changelog passes by reporting that nothing that ships changed. No changelog entry and no version bump.

`! cmd` on its own line asserts nothing in a bats body unless it happens to
be the last command of that body: bash does not apply errexit to a negated
command, so the verdict is the exit status of the final command and every
earlier `! cmd` is inert. shellcheck names the class SC2314 and grades the
two cases apart, error for the inert ones and note for the rest.

Ten negations here were inert, the largest group in one file in the
organization:

- "is_global_ipv6 refuses link local, loopback, multicast and IPv4": five of
  the six cases, so only the empty string ran. This is the predicate that
  picks the address the boot test then talks to, and `fe80::1`, `::1`,
  `ff02::1` and an IPv4 address were all unchecked.
- "container_name and the name predicates": a leading dot and an empty
  string, the two LXC refuses.
- "is_ssh_banner accepts an OpenSSH banner and nothing else": `220 ready`,
  which is the answer a mail server gives, so the test that says only SSH
  counts did not say it.
- "firstboot_done_in reads the flag 98finalize clears": the case where the
  flag is still `true`, meaning the whole point of the predicate.
- "password_in_config compares the declared password with the one in the
  file": the wrong password. The test read as though a mismatch was proven
  and only the empty file was.

All of them become `run !`, together with the seven that were in final
position and did assert, because a line whose meaning depends on its
position is the trap itself.

No assertion changed its verdict; every predicate answers as the test
believed. Suite green, 196 tests. Coverage unchanged, threshold 97:
zz-project-packages 100.00, zzz-keel-archive 100.00, wordpress.sh 99.00,
40wordpress 97.73, boot-test-lib.sh 98.95, keel-archive-check 100.00.
@marcos-mendez

Copy link
Copy Markdown
Collaborator Author

Reviewed: every converted refutation goes red when the code under test accepts that call (all six is_global_ipv6 cases included), and each fails with status 1. No changes needed; merged main so the renamed appliance check runs on this head. Note: #6 adds tests/wrappers.bats with three new bare negations, which the gate will refuse.

@marcos-mendez
marcos-mendez merged commit 090a03c into master Sep 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ten negations in tests/boot-test.bats assert nothing, including five of the six is_global_ipv6 refusals

1 participant