diff --git a/.github/workflows/drift-guard.yml b/.github/workflows/drift-guard.yml index b4b8be8e5..e0c946bca 100644 --- a/.github/workflows/drift-guard.yml +++ b/.github/workflows/drift-guard.yml @@ -234,7 +234,7 @@ jobs: - name: Self-test the duplicate-ctest-name checker run: bash scripts/test_check_ctest_name_collisions.sh - # ── The scheduled mutation gate's own self-test (morph#408) ────────────── + # ── The mutation gates: a self-test (morph#408) and a citation gate (morph#608) ── # # scripts/check_mutation_regression.py is the gate .github/workflows/mutation.yml # runs after scripts/mutation.sh, on a schedule rather than per-PR (that @@ -253,7 +253,7 @@ jobs: # successful run -- would be indistinguishable from a healthy one without # this self-test. mutation-regression-lint: - name: Mutation-regression checker self-test + name: Mutation gates (regression self-test, survivor citations) runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 @@ -261,6 +261,34 @@ jobs: - name: Self-test the mutation-regression checker run: python3 scripts/check_mutation_regression.py --self-test + # scripts/check_mutation_survivors.py, and unlike everything else in this + # file this one is the *gate itself*, not only its self-test. It reads no + # coverage data and no Mull report -- its inputs are + # scripts/mutation_survivors.json and the headers in the tree -- so there + # is nothing to defer it to a build leg or to the scheduled campaign for. + # + # It belongs per-PR precisely because what it audits rots per-PR. The + # triage record cites each equivalent mutant by `{file, line, source}`, + # and an ordinary edit anywhere above one of those lines moves it. By the + # time morph#608 measured, 5 of the 7 entries pointed at the wrong line + # and include/morph/core/backend.hpp's was off by 479 -- landing in the + # middle of an unrelated function, for a reader the mutation gate's own + # failure message had sent there. Nothing failed, because nothing looked: + # check_mutation_regression.py compares survivor *counts* against + # scripts/mutation_baseline.json and never opens the triage file at all. + # + # The resolution is check_branch_coverage.py's + # resolve_allowlist_source_line(), shared rather than reimplemented -- + # scripts/branch_partial_allowlist.json and + # scripts/error_path_allowlist.json are hardened by that same function, + # against the same defect found three times over (morph#349, morph#355, + # morph#419). + - name: Audit the mutation-survivor citations + run: python3 scripts/check_mutation_survivors.py + + - name: Self-test the mutation-survivor citation gate + run: python3 scripts/check_mutation_survivors.py --self-test + # ── The coverage-object gate's own self-test ─────────────────────────── # The gate itself (scripts/check_coverage_objects.sh) runs inside # scripts/coverage.sh, in ci.yml's linux-coverage job, because it compares the diff --git a/docs/spec/testing_charter.md b/docs/spec/testing_charter.md index c1bcf5283..79594902b 100644 --- a/docs/spec/testing_charter.md +++ b/docs/spec/testing_charter.md @@ -74,6 +74,7 @@ floor means anything. | Compile-time contract checks | `tests/compile_checks/` | Every configure that reaches `tests/CMakeLists.txt` | `FATAL_ERROR` at configure time | | Multiple independent harnesses | Catch2 suite + `scripts/scenario/` (wire-level scenario corpus against real rung servers) | `tests/`, local scenario runs | Catch2 assertions; `scripts/scenario/scenario_coverage.py`'s drift gate (self-tested in `drift-guard.yml`) | | Error-path instrumentation | `scripts/check_error_path_coverage.py` (morph#406) | Local / on demand; self-test only in `drift-guard.yml` today | Its own self-test; not yet wired as a build-blocking gate (see "What is unenforced") | +| Mutation-triage citations still point at the code they triage | `scripts/check_mutation_survivors.py` (morph#608) | `drift-guard` (the gate itself, per-PR — it needs no build and no Mull report) | The gate: every `{file, line, source}` entry in `scripts/mutation_survivors.json` must resolve to the line it names, via `check_branch_coverage.py`'s `resolve_allowlist_source_line()`. Audits the structured entries only; the same file's free-text citations are counted and reported, not gated (morph#613) | ## Declined techniques, and why @@ -196,6 +197,6 @@ Named honestly rather than folded into the table above as if a check existed: | [testing_strategy.md](testing_strategy.md) | The opt-in test categories this charter's table cites in detail — fuzz harness, soak tests, load benchmark, adversarial cross-socket run. | | [error_handling.md](error_handling.md) | The propagation design morph#406's error-path instrument measures test coverage of. | | `codecov.yml` | The per-subsystem coverage targets and the artifact-audit allowlists (`branch_partial_allowlist.json`, `error_path_allowlist.json`) this charter's guarantee is enforced through. | -| `scripts/mutation_survivors.json` | The triaged survivor list behind this charter's 64.06% mutation-score figure. | +| `scripts/mutation_survivors.json` | The triaged survivor list behind this charter's 64.06% mutation-score figure. Its structured `{file, line, source}` entries are audited per-PR by `scripts/check_mutation_survivors.py` (morph#608); its free-text citations are not (morph#613). | | `tests/oom_injector.hpp` | The OOM-injection limitation this charter states under "Instrument reach". | | `examples/common/testkit/fault_proxy.hpp` | The ladder-only fault-injection seam this charter states has no `include/morph`-side equivalent. | diff --git a/scripts/check_mutation_survivors.py b/scripts/check_mutation_survivors.py new file mode 100644 index 000000000..c19fc4537 --- /dev/null +++ b/scripts/check_mutation_survivors.py @@ -0,0 +1,430 @@ +#!/usr/bin/env python3 +"""Audit scripts/mutation_survivors.json's citations against the code they name. + +Usage: + python3 scripts/check_mutation_survivors.py [SURVIVORS_JSON] + python3 scripts/check_mutation_survivors.py --self-test + +Why this exists (morph#608) +--------------------------- +scripts/mutation_survivors.json records each triaged equivalent mutant as +`{file, line, mutator, source, reason}`. Nothing audited the `line` field, and +5 of its 7 line-carrying entries pointed at the wrong line by the time morph#608 +measured them -- `include/morph/core/backend.hpp:749` was off by 479 lines and +landed in the middle of an unrelated function. Every `source` text still existed +and was still unique, so no *disposition* was wrong; only the coordinates were. +Re-checking the file's own history shows no `line` hint in it has ever been +edited since the file was created, so the two entries that are still right are +right by luck -- nothing above them happened to move -- not because anyone +refreshed them. + +That is the same defect this repository has already found in +scripts/branch_partial_allowlist.json and scripts/error_path_allowlist.json +(morph#349, morph#355, morph#419): a citation keyed by line number alone rots +silently, because nothing reads it until a human follows it and finds +unrelated code. Both of those files are hardened by +check_branch_coverage.py's resolve_allowlist_source_line(), which resolves an +entry by its `source` text and fails the gate when the `line` hint has drifted. +mutation_survivors.json carried the same shape of citation with none of the +auditing. This script closes that, by calling the same resolver rather than +writing a second one -- it is check_error_path_coverage.py (morph#406) and this +script, three callers of one implementation, because an independently +maintained second copy of the fix is how this defect class gets a fourth +chance. + +Why this gate is cheap, and where it runs +----------------------------------------- +Unlike the other two callers, this one reads no coverage data at all: its +inputs are a JSON file and the headers in the tree. So it runs as a *gate*, not +just as a self-test, in .github/workflows/drift-guard.yml's +`mutation-regression-lint` job -- on every pull request, in seconds, with no +build. The mutation campaign itself (.github/workflows/mutation.yml) is +scheduled rather than per-PR, but the citations in this file rot with ordinary +edits to include/morph, which happen per-PR, so that is where the audit belongs. + +What it does NOT audit, stated rather than left to be discovered +---------------------------------------------------------------- +Only the *structured* entries -- dicts carrying `file`, `line` and `source`. +The same document also carries citations as free text, inside prose strings: +`"include/morph/core/backend.hpp:613 -- emitMetric(Metric::registerCount, 1.0)"`, +`"site": "remote.hpp:1031"`, and 29 more (31 in all, as this script's own +count_free_text_citations() reports on every run). Those have no verbatim +`source` text to resolve against -- the text after the `--` is a paraphrase, +and several use a bare filename with no directory -- so this resolver cannot +see them, and several of them are measurably stale (morph#613). The count is +printed on every run so this gate's coverage of the file is visible rather than +assumed: auditing the structured half of a file while the prose half rots is a +narrower claim than a green tick looks. + +Vacuity +------- +The failure mode AGENTS.md names first is a check that reports success while +measuring nothing, so two things are refused rather than skipped: + + * **Finding no auditable entries at all** fails. The entries are located by + walking the whole document for dicts carrying `file`/`line`/`source`, + rather than by a hard-coded `classes.equivalent.entries` path, so a future + triage pass that adds a new class gets audited for free -- but if a + restructure ever moved them all out of reach, this gate would otherwise go + green over zero entries, which is exactly the shape of failure it exists to + prevent. + * **An entry with `file` and `line` but no `source`** fails, rather than + being skipped as "not one of mine". An entry with no source text is + un-auditable by construction, and silently ignoring it would let the next + contributor add exactly the unauditable citation this gate was written to + forbid. +""" + +from __future__ import annotations + +import json +import os +import re +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import check_branch_coverage # noqa: E402 -- resolve_allowlist_source_line(), see above + +SURVIVORS = "scripts/mutation_survivors.json" + +# `path/to/file.hpp:123` or `file.hpp:123` appearing inside a prose string -- +# the free-text citations this resolver cannot audit. Counted and reported, not +# gated; see the module docstring and morph#613. +FREE_TEXT_CITATION = re.compile(r"[\w./-]+\.(?:hpp|cpp|h|cc):\d+") + + +def find_entries(document): + """[(json path, entry dict)] for every dict carrying `file` and `line`. + + Walks the whole document rather than indexing a fixed key path, so a class + added by a future triage pass is audited without editing this script, and + a restructure that moved the entries cannot silently empty the gate (the + caller fails on an empty result). + """ + found = [] + + def walk(node, where): + if isinstance(node, dict): + if "file" in node and "line" in node: + found.append((where, node)) + for key, value in node.items(): + walk(value, f"{where}.{key}") + elif isinstance(node, list): + for index, value in enumerate(node): + walk(value, f"{where}[{index}]") + + walk(document, "") + return found + + +def count_free_text_citations(document): + """How many `file.hpp:123` citations live in prose strings, not entries. + + Reported so this gate's own coverage of the file is visible. Strings that + belong to an audited entry (its `file`, `source` or `reason`) are not + counted -- only citations this resolver structurally cannot reach. + """ + audited = {id(entry) for _, entry in find_entries(document)} + count = 0 + + def walk(node): + nonlocal count + if isinstance(node, dict): + if id(node) in audited: + # An audited entry's own fields are accounted for; only its + # prose (`reason`) can still carry an unauditable citation. + node = {"reason": node.get("reason", "")} + for value in node.values(): + walk(value) + elif isinstance(node, list): + for value in node: + walk(value) + elif isinstance(node, str): + count += len(FREE_TEXT_CITATION.findall(node)) + + walk(document) + return count + + +def check(survivors_path, repo_root, out=sys.stdout): + if not os.path.exists(survivors_path): + print(f"error: {survivors_path} does not exist. It is the triage record for " + f"every mutation survivor this repository has dispositioned; without it " + f"a survivor with a reason and one with none are indistinguishable.", + file=sys.stderr) + return 1 + + with open(survivors_path, encoding="utf-8") as handle: + document = json.load(handle) + + entries = find_entries(document) + failures = [] + audited = 0 + + for where, entry in entries: + path, hint = entry["file"], entry["line"] + location = f"{survivors_path}{where}" + + if not isinstance(hint, int): + failures.append(f"{location} ({path}) has a non-integer `line` " + f"{hint!r}, which no resolver can check.") + continue + + wanted = str(entry.get("source", "")).strip() + if not wanted: + failures.append( + f"{path}:{hint} ({location}) cites a line with no `source` text. A " + f"bare line number is the citation that rots (morph#349, morph#419) -- " + f"add the verbatim source line so this entry can be re-resolved when " + f"the code moves." + ) + continue + + reason = str(entry.get("reason", "")).strip() + if not reason: + failures.append(f"{path}:{hint} ({location}) is recorded with no reason. " + f"A bare disposition is not a triage.") + continue + + resolved = check_branch_coverage.resolve_allowlist_source_line( + repo_root, path, hint, wanted, survivors_path, failures) + if resolved is None: + continue + audited += 1 + + free_text = count_free_text_citations(document) + + if not entries: + failures.append( + f"{survivors_path} carries no structured entries at all (nothing with " + f"`file`, `line` and `source`). Either the file was restructured and this " + f"gate now audits nothing -- which is the failure it exists to prevent -- " + f"or the triage record was emptied. Neither is a state to pass." + ) + + if failures: + print(f"{survivors_path}: {len(failures)} citation(s) no longer match the " + f"code they name.\n", file=out) + for failure in failures: + print(f" - {failure}", file=out) + print( + "\nThese are triage dispositions, not suppressions: the reasoning is " + "recorded\nbecause someone checked it. Re-read the code at the resolved " + "line, confirm the\nreason still holds, and then update the entry's " + "`line` to the number printed\nabove. Do not delete an entry to silence " + "this gate, and do not move a `source`\ntext to make it match -- either " + "is a triage nobody performed.", file=out) + return 1 + + print(f"ok: {audited} structured citation(s) in {survivors_path} resolve to the " + f"line they name.", file=out) + print(f"note: {free_text} further citation(s) in this file are free text inside " + f"prose\n strings, which carry no verbatim `source` and so are NOT " + f"audited here (morph#613).", file=out) + return 0 + + +def self_test(): + import io + import shutil + import tempfile + + failures = 0 + + def note(message): + print(message) + + def fail(message, output=""): + nonlocal failures + failures += 1 + print(f"error: {message}", file=sys.stderr) + if output: + print(output, file=sys.stderr) + + work = tempfile.mkdtemp() + try: + root = os.path.join(work, "repo") + os.makedirs(os.path.join(root, "include", "morph", "core")) + header = os.path.join(root, "include", "morph", "core", "x.hpp") + + def write_header(text): + with open(header, "w", encoding="utf-8") as handle: + handle.write(text) + + survivors = os.path.join(root, "survivors.json") + + def write_survivors(document): + with open(survivors, "w", encoding="utf-8") as handle: + json.dump(document, handle) + + def run(): + buf = io.StringIO() + return check(survivors, root, out=buf), buf.getvalue() + + def entry(**overrides): + base = { + "file": "include/morph/core/x.hpp", + "line": 3, + "mutator": "cxx_replace_scalar_call", + "source": "out.reserve(text.size());", + "reason": "fixture: a capacity hint, equivalent by construction", + } + base.update(overrides) + return {"classes": {"equivalent": {"entries": [base]}}} + + original = ( + "void f() {\n" + " std::string out;\n" + " out.reserve(text.size());\n" # line 3 + "}\n" + ) + write_header(original) + + # 1. a citation that matches -> passes, and says how many it audited. + write_survivors(entry()) + rc, output = run() + if rc != 0: + fail("a citation matching its line did not pass the gate", output) + elif "ok: 1 structured citation" not in output: + fail("a passing run did not report how many citations it audited", output) + else: + note("ok: a citation that matches its line passes, and is counted") + + # 2. the code moves, the hint does not -> fails, printing the new line. + # This is morph#608's own shape, and the case that proves this gate + # is not vacuous: nothing about the JSON changes between 1 and 2. + write_header("// a new leading comment shifts everything down\n" + original) + rc, output = run() + if rc == 0: + fail("a drifted `line` hint passed the gate", output) + elif "has moved to line 4" not in output: + fail("a drifted hint failed, but did not print the corrected line", output) + elif "update the entry's `line`" not in output: + fail("a drifted hint failed without telling the reader how to refresh it", + output) + else: + note("ok: a drifted `line` hint fails, and the corrected line is printed") + + # 3. the cited text is gone entirely -> fails differently, because the + # disposition itself may no longer apply. + write_header("void f() {\n std::string out;\n}\n") + rc, output = run() + if rc == 0: + fail("a citation whose source text no longer exists passed the gate", output) + elif "nowhere in the file any more" not in output: + fail("a vanished source text failed with the wrong message", output) + else: + note("ok: a citation whose source text is gone fails, and says the code changed") + + # 4. the cited text now appears twice, at neither the hinted line -> + # ambiguous, and refused rather than guessed at. + write_header( + "void f() {\n" + " out.reserve(text.size());\n" + " out.reserve(text.size());\n" + "}\n" + ) + write_survivors(entry(line=9)) + rc, output = run() + if rc == 0: + fail("an ambiguous citation passed the gate", output) + elif "not decidable" not in output: + fail("an ambiguous citation failed with the wrong message", output) + else: + note("ok: a citation matching several lines is refused as ambiguous") + + # 5. an entry with no `source` -> refused, not skipped. Skipping it is + # how the gate would quietly stop covering new entries. + write_header(original) + document = entry() + del document["classes"]["equivalent"]["entries"][0]["source"] + write_survivors(document) + rc, output = run() + if rc == 0: + fail("an entry with no `source` text passed the gate", output) + elif "no `source` text" not in output: + fail("a source-less entry failed with the wrong message", output) + else: + note("ok: an entry with no `source` text is refused, not skipped") + + # 6. an entry with no reason -> refused, on the same terms as every + # other allowlist in this repository. + write_survivors(entry(reason="")) + rc, output = run() + if rc == 0: + fail("an entry with no reason passed the gate", output) + elif "not a triage" not in output: + fail("a reason-less entry failed with the wrong message", output) + else: + note("ok: an entry with no reason is refused") + + # 7. a document with no entries at all -> refused. A gate that audits + # nothing must not report success (AGENTS.md's first failure mode). + write_survivors({"classes": {"equivalent": {"entries": []}}}) + rc, output = run() + if rc == 0: + fail("a document with no auditable entries passed the gate", output) + elif "audits nothing" not in output: + fail("an empty document failed with the wrong message", output) + else: + note("ok: a document this gate can audit nothing in is refused") + + # 8. entries found outside `classes.equivalent` are audited too, so a + # future class does not have to remember to edit this script. + write_survivors({"some_future_class": {"rows": [ + {"file": "include/morph/core/x.hpp", "line": 99, + "source": "out.reserve(text.size());", + "reason": "fixture: in a class this script has never heard of"}]}}) + rc, output = run() + if rc == 0: + fail("an entry outside classes.equivalent was not audited", output) + elif "has moved to line 3" not in output: + fail("an entry in an unknown class failed with the wrong message", output) + else: + note("ok: an entry in a class this script does not know about is audited") + + # 9. a file that no longer exists -> named, not ignored. + write_survivors(entry(file="include/morph/core/gone.hpp")) + rc, output = run() + if rc == 0: + fail("a citation naming a deleted file passed the gate", output) + elif "does not exist" not in output: + fail("a deleted file failed with the wrong message", output) + else: + note("ok: a citation naming a file that is gone fails, and names it") + + # 10. the free-text counter sees prose citations and does not gate on + # them -- the narrowness this gate reports about itself. + write_header(original) + document = entry() + document["notes"] = ["include/morph/core/y.hpp:613 -- a prose citation", + "and remote.hpp:1031 as well"] + write_survivors(document) + rc, output = run() + if rc != 0: + fail("a prose citation wrongly failed the gate", output) + elif "note: 2 further citation(s)" not in output: + fail("the free-text citation count was wrong or missing", output) + else: + note("ok: free-text citations are counted and reported, not gated") + + finally: + shutil.rmtree(work, ignore_errors=True) + + if failures: + print(f"\n{failures} self-test check(s) failed", file=sys.stderr) + return 1 + + print("\nall self-test checks passed") + return 0 + + +def main(argv): + if "--self-test" in argv: + return self_test() + survivors = argv[0] if argv else SURVIVORS + return check(survivors, os.getcwd()) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/mutation_survivors.json b/scripts/mutation_survivors.json index cb4147561..0122e40ea 100644 --- a/scripts/mutation_survivors.json +++ b/scripts/mutation_survivors.json @@ -29,6 +29,16 @@ "side channels are recorded separately so a future decision about logs is made about logs.", "But the residual class is an upper bound on missing assertions, not a set of them.", "", + "The `line` on each entry below points at current code, not at where the survivor", + "sat when it was triaged. scripts/check_mutation_survivors.py resolves every entry by", + "its `source` text on every pull request and fails when the hint has drifted (morph#608,", + "which found 5 of 7 already wrong and one off by 479 lines). When it fails, re-read the", + "code at the line it prints, confirm the reason still holds, and update `line`. Deleting", + "an entry, or editing its `source` to match some other line, is a triage nobody did.", + "Note what that gate does NOT cover: the citations written as prose in this file (the", + "`representative_sites` lists, `site` fields, and the narrative in `runs`) carry no", + "verbatim source text, so nothing audits them and several are known stale (morph#613).", + "", "Re-measure with: MULL_PREFIX=... bash scripts/mutation.sh core-forms" ], "classes": { @@ -46,7 +56,7 @@ }, { "file": "include/morph/core/bridge.hpp", - "line": 119, + "line": 121, "mutants": 3, "mutator": "cxx_add_to_sub", "source": "return modelHash ^ (key.sharing.hash_code() + 0x9e3779b9U + (modelHash << 6) + (modelHash >> 2));", @@ -54,7 +64,7 @@ }, { "file": "include/morph/core/backend.hpp", - "line": 749, + "line": 1228, "mutants": 1, "mutator": "cxx_replace_scalar_call", "source": "aware.reserve(_changeAware.size());", @@ -66,31 +76,31 @@ "mutants": 1, "mutator": "cxx_replace_scalar_call", "source": "out.reserve(msg.size() + 8);", - "reason": "A capacity hint on a std::string, equivalent for the same reason as backend.hpp:749." + "reason": "A capacity hint on a std::string, equivalent for the same reason as the `aware.reserve(_changeAware.size());` entry in core/backend.hpp above." }, { "file": "include/morph/core/wire.hpp", - "line": 302, + "line": 307, "mutants": 1, "mutator": "cxx_replace_scalar_call", "source": "out.reserve(text.size());", - "reason": "A capacity hint on a std::string, equivalent for the same reason as backend.hpp:749." + "reason": "A capacity hint on a std::string, equivalent for the same reason as the `aware.reserve(_changeAware.size());` entry in core/backend.hpp above." }, { "file": "include/morph/forms/views.hpp", - "line": 264, + "line": 271, "mutants": 1, "mutator": "cxx_replace_scalar_call", "source": "ordered.reserve(propsObj.size());", - "reason": "A capacity hint on a std::vector, equivalent for the same reason as backend.hpp:749." + "reason": "A capacity hint on a std::vector, equivalent for the same reason as the `aware.reserve(_changeAware.size());` entry in core/backend.hpp above." }, { "file": "include/morph/forms/views.hpp", - "line": 430, + "line": 437, "mutants": 1, "mutator": "cxx_replace_scalar_call", "source": "ids.reserve(_providers.size());", - "reason": "A capacity hint on a std::vector, equivalent for the same reason as backend.hpp:749." + "reason": "A capacity hint on a std::vector, equivalent for the same reason as the `aware.reserve(_changeAware.size());` entry in core/backend.hpp above." } ] },