Repository navigation
A negated command only asserts in final position - #11
Merged
Merged
Conversation
`! cmd` on its own line asserts nothing in a bats body unless it happens to be the last command of that body: bash does not apply errexit to a negated command, so the verdict is the exit status of the final command and every earlier `! cmd` is inert. shellcheck names the class SC2314 and grades the two cases apart, error for the inert ones and note for the rest. Ten negations here were inert, the largest group in one file in the organization: - "is_global_ipv6 refuses link local, loopback, multicast and IPv4": five of the six cases, so only the empty string ran. This is the predicate that picks the address the boot test then talks to, and `fe80::1`, `::1`, `ff02::1` and an IPv4 address were all unchecked. - "container_name and the name predicates": a leading dot and an empty string, the two LXC refuses. - "is_ssh_banner accepts an OpenSSH banner and nothing else": `220 ready`, which is the answer a mail server gives, so the test that says only SSH counts did not say it. - "firstboot_done_in reads the flag 98finalize clears": the case where the flag is still `true`, meaning the whole point of the predicate. - "password_in_config compares the declared password with the one in the file": the wrong password. The test read as though a mismatch was proven and only the empty file was. All of them become `run !`, together with the seven that were in final position and did assert, because a line whose meaning depends on its position is the trap itself. No assertion changed its verdict; every predicate answers as the test believed. Suite green, 196 tests. Coverage unchanged, threshold 97: zz-project-packages 100.00, zzz-keel-archive 100.00, wordpress.sh 99.00, 40wordpress 97.73, boot-test-lib.sh 98.95, keel-archive-check 100.00.
Collaborator
Author
|
Reviewed: every converted refutation goes red when the code under test accepts that call (all six is_global_ipv6 cases included), and each fails with status 1. No changes needed; merged main so the renamed appliance check runs on this head. Note: #6 adds tests/wrappers.bats with three new bare negations, which the gate will refuse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every negation in these suites now asserts wherever it stands. Ten did not assert at all, the largest group in one file in the organization.
Closes #10. Part of Keel-Linux/tracker#19, which has the organization-wide sweep.
Why a bare
!is inertA bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command, so
! cmddecides the test in final position and is inert everywhere else. shellcheck grades the two apart,error:for the inert ones andnote:for the rest:What was not being checked
fe80::1,FE80::1,::1,ff02::1and an IPv4 address were unchecked. This is the predicate that picks the address the boot test then talks to.bt_is_container_namerefusing a leading dot and an empty string, the two LXC refusesbt_is_appliance_name 9wordpressbt_is_ssh_banner "220 ready"— the answer a mail server gives. The test is named "accepts an OpenSSH banner and nothing else"; the "nothing else" half did not run.bt_firstboot_done_inwith the flag stillRUN_FIRSTBOOT=true, which is the whole point of the predicatebt_password_in_configwith the wrong password. The test read as though a mismatch was proven and only the empty file was.All of them are
run !now, together with the seven that were in final position and did assert: a line whose meaning depends on its position is the trap itself.What was found
Nothing broken. Every predicate was run against every one of its cases by hand before the conversion, and each answers as its test believed:
bt_is_global_ipv6refuses all five refusal cases and accepts all three acceptance cases,bt_is_ssh_bannerrefuses220 ready,bt_password_in_configrefuses the wrong password, and so on. The assertions were dead, not wrong.That is luck rather than evidence, which is why the sweep comes with a gate in
Keel-Linux/.githuband adocs/traps.mdentry.Test plan
Every predicate verified against every one of its cases by hand, before the conversion.
shellcheck -f gcc --shell=bash --severity=error --include=SC2314 tests/*.batsreports nothing.Every converted site audited for
runclobbering$output/$status/$lines; none here does.bats tests/*.batsgreen, 196 tests. No assertion changed its verdict.tests/coverage.sh, threshold 97, unchanged:Tests ship nothing, so
package / changelogpasses by reporting that nothing that ships changed. No changelog entry and no version bump.