diff --git a/docs/CRITIC.md b/docs/CRITIC.md index 69804f1..deda7c0 100644 --- a/docs/CRITIC.md +++ b/docs/CRITIC.md @@ -79,6 +79,7 @@ In the factory, test code is candidate code, and it runs inside the process whos | `exit_at_import` | calls `sys.exit`, `exit`, `quit`, `os._exit` or `unittest.main`, or raises `SystemExit`, `KeyboardInterrupt`, `GeneratorExit` or `BaseException`, at module level (not under `if __name__ == "__main__":`) or in `setUpModule`, `tearDownModule`, `setUpClass` or `tearDownClass` | | `harness_override` | defines or assigns `run`, `__call__`, `debug`, `id`, `_callTestMethod`, `_callSetUp`, `_callTearDown`, `_callCleanup`, `_addExpectedFailure`, `_addUnexpectedSuccess`, `countTestCases` or `__getattribute__`, or a method that registers or runs a test's cleanups or runs its subtests (`addCleanup`, `enterContext`, `doCleanups`, `addClassCleanup`, `enterClassContext`, `doClassCleanups`, `subTest`), in a class that derives from a name ending in `TestCase` or from `unittest`, or that defines a `test*` method | | `load_tests` | defines or assigns `load_tests` | +| `test_replaced` | assigns, deletes or `setattr`s an attribute named `test...` on anything other than `self` (`Victim.test_a = ...`, `cls.test_a = ...`, `del type(t).test_a`), or patches one with `mock.patch("pkg.Cls.test_a")`, `patch.object(Cls, "test_a")` or `patch.multiple(Cls, test_a=...)` | | `unittest_patch` | assigns or deletes an attribute of anything imported from `unittest`; assigns a result callback (`startTest`, `stopTest`, `addSuccess`, ..., `wasSuccessful`, `testsRun`, `shouldStop`, `stop`), a test's identity (`_testMethodName`, `__qualname__`, `__module__`) or one of the cleanup and subtest methods above on anything; uses `setattr` or `delattr` for those; patches a `unittest.*` or `sys.modules` target with `mock.patch`; reads `_outcome`, `_resultForDoCleanups` or `shouldStop` | | `sys_modules` | writes `sys.modules['unittest...']` or `sys.modules['__main__']` (assignment, `del`, `pop`, `setdefault`, `__setitem__`), or calls `sys.modules.update` or `sys.modules.clear` | | `runner_reach` | imports `__main__`, `gc`, `ctypes`, `msvcrt` or `_winapi`; reads `sys.modules['__main__']`; calls `gc.get_objects`, `gc.get_referrers`, `gc.get_referents`, `sys._getframe`, `sys._current_frames`, `inspect.currentframe`, `inspect.stack`, `inspect.getouterframes`, `inspect.getinnerframes`, `inspect.trace`, `os.dup`, `os.dup2`, `os.write`, `os.fdopen`, `os.set_inheritable`, `os.set_handle_inheritable`, `sys.settrace`, `sys.setprofile`, `sys.addaudithook`, `threading.settrace` or `threading.setprofile`; reads a frame attribute (`f_back`, `f_locals`, `f_globals`, `tb_frame`, `gi_frame`, `cr_frame`, `ag_frame`); or holds a string naming the shard runner's directories (`syberlabs-shard...`) | @@ -86,7 +87,7 @@ In the factory, test code is candidate code, and it runs inside the process whos Why these: each is a way for test code to end the process with an exit code it chooses (the shard runner then sees a crash, but serial `unittest` sees success), to replace how a test is started or counted, to choose the test set at discovery, or to reach the runner's channel and nonce in its own memory and write a false report, which the runner cannot detect (`spec/GAPS.md` 58). -**Beyond test files.** A test can import any module, and the module's code then runs in the shard's process. So moving a forgery verbatim out of the test file and into a helper the test calls passed both layers under revision 1 (the verification of #116, finding 1). Every Python file (`.py`, `.pyw`) the candidate adds or changes is now held to the families that tamper with the harness from inside that process: `harness_override`, `unittest_patch`, `sys_modules`, `runner_reach` and `dynamic_code` (`HARNESS_RULES`). Four families stay test-scoped (`TEST_ONLY_RULES`): `process_exit`, `signal_handler`, `exit_at_import` and `load_tests`. Production code legitimately exits, kills child processes and installs handlers, and `load_tests` only means something in a module that discovery loads. Two parts of `runner_reach` also stay test-scoped: +**Beyond test files.** A test can import any module, and the module's code then runs in the shard's process. So moving a forgery verbatim out of the test file and into a helper the test calls passed both layers under revision 1 (the verification of #116, finding 1). Every Python file (`.py`, `.pyw`) the candidate adds or changes is now held to the families that tamper with the harness from inside that process: `harness_override`, `unittest_patch`, `sys_modules`, `runner_reach`, `dynamic_code` and `test_replaced` (`HARNESS_RULES`). Four families stay test-scoped (`TEST_ONLY_RULES`): `process_exit`, `signal_handler`, `exit_at_import` and `load_tests`. Production code legitimately exits, kills child processes and installs handlers, and `load_tests` only means something in a module that discovery loads. Two parts of `runner_reach` also stay test-scoped: - The write side: `os.write`, `os.dup`, `os.dup2`, `os.fdopen`, `os.set_inheritable`, `os.set_handle_inheritable`, `msvcrt` and `_winapi`. This is ordinary I/O; `syberlabs/journal.py` locks with `msvcrt`, and the runner itself uses the rest. A false report needs the channel's nonce. Without it, a write through these is `channel_tampered`. Only reading the process's memory gets the nonce (`gc`, frames, `__main__`, `ctypes`, trace and audit hooks), and that read side applies to every file. - The runner's directory names: since results moved to the channel, those directories hold only logs. @@ -109,7 +110,7 @@ What it does not catch: It is a pattern list, not an analysis; a candidate that knows the list can write around it. Its precision against human review is unmeasured. -The reviewer declares revision `4` (`Reviewed-by: rules@4`). Its verdicts changed four times: with the test-integrity rule, again when the rule's scope grew to every changed Python file, again when `harness_override` came to name `_addExpectedFailure` and `_addUnexpectedSuccess`, and again when it came to name the cleanup and subtest methods. Through the first two `TestCase.run` reports an expected failure and an expected failure that passed; a test class that defines `_addUnexpectedSuccess` to call `addSuccess` turned an outcome that fails the run into a pass, and revision 2 did not see it (`scripts/run_shards.py` now refuses it too, `docs/TESTING.md`). Through the others a test runs its own cleanups and subtests; a test class that defines `subTest` to suppress what its block raises, or `doCleanups` to drop its cleanups, turned its own failure into a pass, and revision 3 did not see it, nor `self.subTest = ...` in `setUp` (`unittest_patch` now names that assignment; the runner refuses the first two and fails the shard on the third). `tests/test_publish.py` pins the line. +The reviewer declares revision `5` (`Reviewed-by: rules@5`). Revision 5 adds `test_replaced`: a test that replaced another test's method (in its body, at import, or in a class fixture) made the replaced test report whatever the replacement did, and revision 4 did not see it (`scripts/run_shards.py` now fails the shard too, `docs/TESTING.md`). It matches names, not objects, so it also flags a test that builds its own inner `TestCase` and replaces that one's method; `setattr(Cls, f"test_{n}", ...)` with a computed name is not matched, which is how tests are generated at import. Before that, its verdicts changed four times: with the test-integrity rule, again when the rule's scope grew to every changed Python file, again when `harness_override` came to name `_addExpectedFailure` and `_addUnexpectedSuccess`, and again when it came to name the cleanup and subtest methods. Through the first two `TestCase.run` reports an expected failure and an expected failure that passed; a test class that defines `_addUnexpectedSuccess` to call `addSuccess` turned an outcome that fails the run into a pass, and revision 2 did not see it (`scripts/run_shards.py` now refuses it too, `docs/TESTING.md`). Through the others a test runs its own cleanups and subtests; a test class that defines `subTest` to suppress what its block raises, or `doCleanups` to drop its cleanups, turned its own failure into a pass, and revision 3 did not see it, nor `self.subTest = ...` in `setUp` (`unittest_patch` now names that assignment; the runner refuses the first two and fails the shard on the third). `tests/test_publish.py` pins the line. ## Model-backed critic diff --git a/docs/TESTING.md b/docs/TESTING.md index 95c3318..51460d2 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -132,7 +132,17 @@ otherwise: the change (`test code changed X while T ran`); so a test whose `setUp` or body gives its instance one of the refused methods (`self.subTest = ...`), after the check at its start, also fails the shard. A change made and - undone within one test, before it stops, is not seen. + undone within one test, before it stops, is not seen. The same checks + compare every discovered test's method, on its class and its instance, + with the one its module defined when the loader loaded it (recorded per + module, before the next module is imported): test code that replaces or + deletes another test's method (`Victim.test_x = lambda self: None` in a + test body, at import from a module loaded later, or in `setUpClass`) fails + the shard, naming the replaced test. Every discovered test is compared, + not only the shard's own, so the verdict does not depend on packing. A + test that replaces its own method on its instance is refused too + (conservatively: it has already fetched it). Tests a module generates at + import, before the loader reads it, are what was loaded. - **After running:** every shard's stream is intact and ends with `done`; every test it started reported an outcome before it stopped (a success, failure, error, skip, expected failure, unexpected success or failed @@ -267,6 +277,9 @@ critic finding, and the honest rows still pass: | A `TestCase` defines `subTest` to suppress what its block raises; its subtest fails (after #123) | **0** | 1, `harness_override` | before: none; now high (`harness_override`, rules@4) | | A `TestCase` defines `doCleanups` to drop its cleanups; a cleanup fails (after #123) | **0** | 1, `harness_override` | before: none; now high (`harness_override`, rules@4) | | A test's `setUp` sets `self.subTest` to suppress exceptions; its subtest fails (after #123) | **0** | 1, `harness_changed` (seen when the test stops) | before: none; now high (`unittest_patch`, rules@4) | +| A test replaces another test's method in its body; the replaced test would fail (after #130) | **0** | 1, `harness_changed` (seen when the replacing test stops) | before: none; now high (`test_replaced`, rules@5) | +| A test module, loaded later, replaces an earlier module's test method at import (after #130) | **0** | 1, `harness_changed` (seen when the next test starts) | before: none; now high (`test_replaced`, rules@5) | +| A `setUpClass` replaces another class's test method with `setattr` (after #130) | **0** | 1, `harness_changed` | before: none; now high (`test_replaced`, rules@5) | | A test patches `TestCase._callTestMethod` at run time (new) | **0** | 1, `harness_changed` | high (`unittest_patch`) | | A test writes a line without the nonce to the channel (new) | n/a | 1, `channel_tampered` | high (`runner_reach`) | | A test finds the runner in its own process and writes a consistent, false stream (new) | n/a | **0: not caught** | high (`runner_reach`, `process_exit`) | @@ -286,7 +299,10 @@ The three rows marked "after #119" ran the same way on `main` at `296e3ca` (before) and on this change (after); serial `unittest discover` says `OK` for each of them as well. The three rows marked "after #123" ran the same way on #123's head at `2a8bdeb` (before) and on its follow-up (after); serial -`unittest discover` says `OK` for each. `tests/test_run_shards.py` holds every +`unittest discover` says `OK` for each. The three rows marked "after #130" +ran the same way on #130's head restacked onto `main` (`c0e58bc`, before) and +on its follow-up (after), with `--shards 1`; serial `unittest discover` says +`OK` for each. `tests/test_run_shards.py` holds every row of the table that the runner catches. The guarantee, exactly: with the factory's `tests` check (the host @@ -306,8 +322,10 @@ host already trusts it. Two further gaps are named in `spec/GAPS.md` 58: - Test code can keep every test in the set but make tests vacuous: its own in the diff, or another's at run time by replacing its method. No count can see this. Swallowing its own failures through the cleanup and subtest - methods is refused (above); a `try`/`except` around its own assertions, or - an assertion removed, is not. + methods is refused (above), and so is replacing a discovered test's method + (above), unless it is made and undone within one test; a `try`/`except` + around its own assertions, an assertion removed, or a helper or assertion + method a test calls replaced at run time, is not. - Outside test files, the patterns that end the process or write to raw descriptors do not apply, because production code does both. diff --git a/examples/critic_rules.py b/examples/critic_rules.py index b7076cc..51bb8e8 100644 --- a/examples/critic_rules.py +++ b/examples/critic_rules.py @@ -45,7 +45,9 @@ # also names the two methods through which a test reports an expected failure or an unexpected success. # Revision 4: harness_override also names the methods that register and run a test's cleanups and run its # subtests, and unittest_patch an assignment of one of them on anything. -REVIEWER = {"name": "rules", "revision": "4"} +# Revision 5: test_replaced, an assignment, deletion or patch of a test method (an attribute named test...) on +# anything other than ``self``. +REVIEWER = {"name": "rules", "revision": "5"} TEST_MARKERS = ("tests/", "test/", "/test_", "_test.", "spec/") SOURCE_SUFFIXES = (".py", ".js", ".ts", ".go", ".rs", ".java", ".rb", ".c", ".h", ".cs", ".mjs") MAX_INTEGRITY_FINDINGS = 40 # per candidate; the summary counts the rest @@ -61,6 +63,8 @@ "...): it can report a test that never ran; or one that runs its cleanups or subtests " "(doCleanups, subTest, ...): it can swallow its own failures", "load_tests": "defines load_tests: the module chooses its own test set at discovery", + "test_replaced": "replaces, deletes or patches a test method (an attribute named test... on anything other than " + "self): the replaced test then reports whatever its replacement does", "unittest_patch": "changes unittest's own objects (assigns, deletes or patches unittest attributes or a result's " "callbacks, or reaches self._outcome / shouldStop)", "sys_modules": "writes sys.modules['unittest...'] or sys.modules['__main__']", @@ -100,7 +104,7 @@ # Where each family applies. Test code gets every family. Any other changed Python file gets the # families that tamper with the harness from inside the test process: a test can import it. -HARNESS_RULES = ("harness_override", "unittest_patch", "sys_modules", "runner_reach", "dynamic_code") +HARNESS_RULES = ("harness_override", "unittest_patch", "sys_modules", "runner_reach", "dynamic_code", "test_replaced") TEST_ONLY_RULES = ("process_exit", "signal_handler", "exit_at_import", "load_tests") # Two parts of ``runner_reach`` stay test-scoped. A forged report needs the channel's nonce, and only # reading the process's memory gets it: gc, frames, ``__main__``, ctypes, trace and audit hooks. @@ -111,6 +115,8 @@ TEST_ONLY_REACH_CALLS = {"os.dup", "os.dup2", "os.write", "os.fdopen", "os.set_inheritable", "os.set_handle_inheritable"} TEST_ONLY_REACH_MODULES = {"msvcrt", "_winapi"} +# The prefix ``unittest``'s loader collects test methods by (``TestLoader.testMethodPrefix``). +TEST_PREFIX = "test" PYTHON_SOURCE = (".py", ".pyw") PYTHON_COMPILED = (".pyc", ".pyd", ".so") @@ -330,6 +336,27 @@ def _test_case_like(self, cls: ast.ClassDef) -> bool: return any(isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)) and item.name.startswith("test") for item in cls.body) + def _patched_attributes(self, name: str, node: ast.Call) -> list: + """The attribute names a ``mock.patch`` call replaces, when the source names them (none on ``self``). + + ``patch("pkg.Cls.attr")`` replaces ``attr``; ``patch.object(obj, "attr")`` and ``patch.multiple(obj, + attr=...)`` replace theirs on ``obj``; ``patch.dict`` replaces keys, not attributes. + """ + args = node.args + if name.endswith("patch.dict"): + return [] + if name.endswith("patch.object"): + if len(args) >= 2 and isinstance(args[1], ast.Constant) and isinstance(args[1].value, str) \ + and not self._on_self(args[0]): + return [args[1].value] + return [] + if name.endswith("patch.multiple"): + return [] if args and self._on_self(args[0]) else [k.arg for k in node.keywords if k.arg] + first = args[0] if args else next((k.value for k in node.keywords if k.arg == "target"), None) + if isinstance(first, ast.Constant) and isinstance(first.value, str) and "." in first.value: + return [first.value.rsplit(".", 1)[1]] + return [] + # statements --------------------------------------------------------------------------- def visit_Raise(self, node): exc = node.exc.func if isinstance(node.exc, ast.Call) else node.exc @@ -338,7 +365,35 @@ def visit_Raise(self, node): self.hit(node, "exit_at_import", f"raise {name}") self.generic_visit(node) + @staticmethod + def _leaves(target): + """The names a target binds: the target itself, or each element of a tuple, list or starred target.""" + if isinstance(target, (ast.Tuple, ast.List)): + for element in target.elts: + yield from _Scan._leaves(element) + elif isinstance(target, ast.Starred): + yield from _Scan._leaves(target.value) + else: + yield target + + @staticmethod + def _on_self(node) -> bool: + return isinstance(node, ast.Name) and node.id == "self" + + def _test_method_targets(self, targets, verb): + """``X.test_... = ...`` or ``del X.test_...`` when X is not ``self``: a test's method replaced at run time. + + ``self.test_...`` is left out: a test instance has already fetched its own method when its code runs, + and the attribute cannot reach another test's instance. + """ + for target in targets: + for leaf in self._leaves(target): + if isinstance(leaf, ast.Attribute) and leaf.attr.startswith(TEST_PREFIX) and not self._on_self(leaf.value): + owner = self.name(leaf.value) or ast.unparse(leaf.value) + self.hit(leaf, "test_replaced", f"{verb} {owner}.{leaf.attr}") + def _targets(self, targets, verb): + self._test_method_targets(targets, verb) for target in targets: for node in ast.walk(target): if isinstance(node, ast.Attribute): @@ -443,6 +498,9 @@ def visit_Call(self, node): elif name != "getattr" and (self._rooted(target, ("unittest",)) or args[1].value in RESULT_CALLBACKS or args[1].value in HARNESS or args[1].value in IDENTITY_ATTRIBUTES): self.hit(node, "unittest_patch", f"{name}({target}, {args[1].value!r})") + if name != "getattr" and isinstance(args[1], ast.Constant) and isinstance(args[1].value, str) \ + and args[1].value.startswith(TEST_PREFIX) and not self._on_self(args[0]): + self.hit(node, "test_replaced", f"{name}({target or ast.unparse(args[0])}, {args[1].value!r})") if name and name.split(".")[-1] in ("pop", "update", "setdefault", "clear", "__setitem__", "__delitem__") \ and self.name(node.func.value if isinstance(node.func, ast.Attribute) else None) == "sys.modules": method = name.split(".")[-1] @@ -456,6 +514,9 @@ def visit_Call(self, node): if isinstance(target, str) and self._rooted(target, ("unittest", "__main__", "sys.modules")) \ and not target.startswith("unittest.mock"): self.hit(node, "unittest_patch", f"{name}({target!r})") + for attribute in self._patched_attributes(name, node): + if attribute.startswith(TEST_PREFIX): + self.hit(node, "test_replaced", f"{name}(..., {attribute!r})") self.generic_visit(node) diff --git a/scripts/run_shards.py b/scripts/run_shards.py index 0a66b94..cb01b6d 100644 --- a/scripts/run_shards.py +++ b/scripts/run_shards.py @@ -30,6 +30,9 @@ in two shards, and no discovered test takes the ``unittest`` machinery that starts and reports it (``run``, ``__call__``, ``debug``, ``id``, ...), or that runs its cleanups and subtests (``doCleanups``, ``subTest``, ...), from outside ``unittest``; +* while running, the method of every discovered test stays the one its module defined when + it was loaded: test code that replaces a test's method (on its class or its instance) + fails the shard when the next test starts or this one stops; * after running, every shard's stream is intact and finished, it started every id it was assigned (or names the class or module fixture that skipped or failed before it could start, as ``unittest`` itself does), and the totals add up; @@ -112,6 +115,8 @@ # Also watched at run time: the functions of ``unittest.case`` through which ``TestCase.run`` reports a skip # or an error. A test that replaced ``_addError`` with a no-op would have every later failure vanish. CASE_FUNCTIONS = ("_addSkip", "_addError") +# What a test's own method is, by identity: absent from the instance and the class (``_test_method``). +_ABSENT = object() # Files Python could import: one of these, untracked, anywhere in the worktree, is a change a test run sees. IMPORTABLE = (".py", ".pyc", ".pyo", ".pyd", ".pth", ".so", ".dll") @@ -127,14 +132,42 @@ def _flatten(suite): yield item -def discover(start: str, pattern: str, top_level: str | None) -> list: +def _test_method(test) -> tuple: + """Where ``test``'s own test method comes from now, by identity: the instance's attribute of that name, the + first class along the MRO that defines it, and what that class holds (each ``_ABSENT`` when there is none).""" + name = getattr(test, "_testMethodName", None) + if not isinstance(name, str): + return _ABSENT, _ABSENT, _ABSENT + owner = next((klass for klass in type(test).__mro__ if name in vars(klass)), None) + return (getattr(test, "__dict__", {}).get(name, _ABSENT), _ABSENT if owner is None else owner, + _ABSENT if owner is None else vars(owner)[name]) + + +class _Loader(unittest.TestLoader): + """A ``TestLoader`` that records each test's method (``_test_method``) as soon as its module is loaded. + + Recorded per module, before the next test module is imported: a module imported later that replaces an + earlier module's test method at import is a change from what was loaded, not part of it. + """ + + def __init__(self): + super().__init__() + self.loaded: list = [] # (test, its _test_method when its module was loaded) + + def loadTestsFromModule(self, module, *args, **kwargs): + suite = super().loadTestsFromModule(module, *args, **kwargs) + self.loaded += [(test, _test_method(test)) for test in _flatten(suite) if isinstance(test, unittest.TestCase)] + return suite + + +def discover(start: str, pattern: str, top_level: str | None, loader: unittest.TestLoader | None = None) -> list: """The test cases ``python -m unittest discover -s START -p PATTERN [-t TOP]`` would run, in its order. A fresh ``TestLoader``, which is what ``unittest.defaultTestLoader`` is in a process that has not discovered before: on Python 3.11 the shared loader keeps the top-level directory of an earlier ``discover`` call, so a second call in the same process could resolve a different start directory. """ - return list(_flatten(unittest.TestLoader().discover(start, pattern, top_level))) + return list(_flatten((loader or unittest.TestLoader()).discover(start, pattern, top_level))) def _fixture_key(test) -> str | None: @@ -464,16 +497,20 @@ def _harness_snapshot() -> dict: class _ShardResult(unittest.TextTestResult): - def __init__(self, *args, channel=None, pristine=None, **kwargs): + def __init__(self, *args, channel=None, pristine=None, loaded=(), **kwargs): super().__init__(*args, **kwargs) self.channel = channel self.pristine = pristine + # (test, _test_method) for every discovered test, as its module was loaded. Every one, not only this shard's: + # whether a replacement fails the run must not depend on which shard the replaced test was packed into. + self.loaded = list(loaded) self.changed = False self.durations: dict = {} self._mark = time.perf_counter() def _check_harness(self, test, at="start"): - """Report, once, a test that starts after test code changed how tests are run or reported. + """Report, once, a test that starts after test code changed how tests are run or reported, or replaced + a discovered test's method (by id: what ``unittest`` would call is no longer what its module defined). Checked again when the test stops (``at``), which names the test that made the change: its ``setUp`` or body can also give the instance a method that a class could not take (``self.subTest = ...``). @@ -484,6 +521,8 @@ def _check_harness(self, test, at="start"): what = sorted(f"{cls}.{name}" for (cls, name), value in now.items() if self.pristine.get((cls, name)) is not value) what += sorted(f"result.{name}" for name in RESULT_METHODS if name in vars(self)) what += [f"{type(test).__qualname__}.{name}" for name in _overrides(test)] + what += sorted({case.id() for case, loaded in self.loaded + if any(a is not b for a, b in zip(_test_method(case), loaded))}) if what: self.changed = True self._send("harness_changed", id=test.id(), what=what, at=at) @@ -562,10 +601,11 @@ def run_child() -> int: wanted = set(spec["ids"]) pristine = _harness_snapshot() # before any test module is imported - tests = [t for t in discover(spec["start"], spec["pattern"], spec["top_level"]) if t.id() in wanted] + loader = _Loader() + tests = [t for t in discover(spec["start"], spec["pattern"], spec["top_level"], loader) if t.id() in wanted] runner = unittest.TextTestRunner( stream=sys.stderr, verbosity=2, warnings=None if sys.warnoptions else "default", - resultclass=lambda *a, **k: _ShardResult(*a, channel=channel, pristine=pristine, **k)) + resultclass=lambda *a, **k: _ShardResult(*a, channel=channel, pristine=pristine, loaded=loader.loaded, **k)) result = runner.run(unittest.TestSuite(tests)) def described(pairs): diff --git a/spec/GAPS.md b/spec/GAPS.md index ba6b195..5c13063 100644 --- a/spec/GAPS.md +++ b/spec/GAPS.md @@ -114,7 +114,7 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e 57. **The guard's record is the hook's by a MAC and an exact id, its files are confined by `--restricted`, and what is still open is named.** `guard-hardening` (2026-10-23) answers the independent verification of #82 (`92074e1cd0`). Built: the hook judges every string of the input, keys included, `WebFetch`'s `prompt` too, as typed and Unicode-folded (NFKC, `Cf` removed, a fixed Cyrillic and Greek lookalike table); a decision counts only with a verifying HMAC-SHA256 under a random per-call key and the exact `tool_use_id`, tool and digest of the stream's lookup (no fallback match); `policy.json` and `tree_lines.txt` are read-only and digest-checked by the hook, the policy digest on `signal.guard.policy_digest`; every call passes `--restricted` and `--tools`; Claude Code below 2.1.295 is refused and the version read is `signal.harness.installed_version`; a failed or timed-out call journals its lookups first; the login lanes also drop `CLAUDE_CONFIG_DIR` and the proxy variables; a path named `.claude` and a symbolic link resolving to an instruction file count as instruction files. Open or not solved: (1) the key and digests reach the hook through Claude Code's environment, because the hooks reference documents no per-hook `env`; every process Claude Code starts with that environment can read them, stdio MCP servers (GitHits) included, and a model can if the operator allows a code-running tool (`Bash`, `PowerShell`); the default tool list has none and `--restricted` removes them unless named, but allowing one weakens the MAC and the policy check (`docs/CLAUDE_CODE.md`, the query guard); (2) `--restricted` is read from 2.1.309's `--help` and the CLI reference (2.1.248 and later), not observed: no real Claude Code ran, so whether it confines a path-less `--allowedTools Write` under `dontAsk`, whether `--settings` hooks run under `--bare`, and whether the hook's `tool_input` digests equal the stream's (a mismatch reads `unguarded` and stops the search) are the live canary's to settle; `--restricted` also stops user settings from loading, which changes what a login-lane run reads (its `apiKeyHelper` among them, which closes part of entry 56 (8) if the flag holds); (3) the encodings and splits `docs/CLAUDE_CODE.md` lists (base64, hex and the rest, a credential split across lookups or by spaces, paraphrase, lookalikes outside the table) still pass; (4) `CLAUDE_CODE_SUBPROCESS_ENV_SCRUB=1` may strip `SYBERLABS_GUARD_KEY` from the hook, which then blocks every lookup (fail closed, not open); (5) an `@path` import in an unchanged `CLAUDE.md` that names a file the model edited is not caught, nor a link to a directory that merely holds a `CLAUDE.md`; (6) `ALL_PROXY` is dropped although the env-var reference does not list it, on the verifier's advice; (7) the guard `VERSION` is `2`, so a reader comparing `signal.guard` across #82 and later candidates sees the version change; (8) `checks._stop` is still imported by its private name (entry 56 (10)); `syberlabs/checks.py` was allowed only for a public alias and was not changed. *2026-10-23, (1) narrowed by `pre-pilot-cleanup`.* With `--mcp-config` naming an MCP server, `ClaudeCodeProvider` refuses at construction (`invalid_provider`) an allowed code-running built-in (`Bash` and `PowerShell`, the two `--restricted`'s `--help` names; `claude_code.CODE_TOOLS`, compared case-insensitively, a rule's specifier dropped), because every child of Claude Code inherits the MAC key and a model with a shell beside a stdio MCP server could sign a decision; `--allow-code-tools-with-mcp` overrides and is recorded as `signal.guard.allow_code_tools_with_mcp: true`. An `--mcp-config` file that is not a JSON object with an empty or absent `mcpServers` counts as naming a server. Still open: the MCP server itself holds the key (the MAC shows a decision was written by a process holding the key, not that the hook wrote it), and a code-running built-in the help does not name is not refused. Also from #84's verification: `claude --version` is read only in the documented `X.Y.Z (Claude Code)` form, and a query with a lone surrogate is digested with `surrogatepass` on both sides (`query_guard.utf8`). -58. **A passing `tests` check is the shard runner's account of test code that ran in its own process, not an observation from outside it; the floor and the critic's integrity rule narrow what that code can hide, and only a sandbox closes it.** `test-integrity` (2026-10-26) answers the independent verification of #114, which showed test code defeating a `tests` check judged by exit code alone (`syberlabs/checks.py` `run_check`): an `os._exit(0)` at import, a forged shard result file, a faked `run()`. Built, in `scripts/run_shards.py`: the runner imports no test module (discovery runs in a child); each child reports over a private inherited pipe with a per-child random nonce and a sequence number, opened before any test module is imported; the verdict is counted from a live stream of `start`, `outcome` and `stop` events that the child's `done` summary must match; a test whose class or instance takes a method that starts or reports it from outside `unittest` is refused before anything runs, and a test that starts after test code changed those methods fails its shard; and, when the host sets `SYBERLABS_BASE` (now in every check's environment, from the candidate's recorded base) or `--floor BASE` is given, the working tree must be the committed tree and every test discovered on the base must still be discovered unless the diff `BASE..HEAD` deletes or renames its file, removes its `def`, or renames its class (`test_set_shrank`). In `examples/critic_rules.py`, a high finding for added test code that exits, kills, signals, overrides or patches the `unittest` harness, defines `load_tests`, writes `sys.modules` for `unittest` or `__main__`, reaches the runner (`__main__`, `gc`, frames, raw descriptors) or runs code the rule cannot read; `critic.no_high_finding` turns it into a refusal. What is not closed: (a) test code in the shard's process can read the nonce and the runner's objects and write a stream that is consistent and false; `tests/test_run_shards.py` shows such a forgery passing the runner, and the critic's patterns are what flag it, so a forgery written in a form the patterns do not list passes both; (b) a test can stay in the set and be vacuous, its own assertions emptied in the diff or another test's method replaced at run time, which no count can see; (c) the patterns reach beyond test files only in part: since `test-integrity-2` (2026-10-26, reviewer `rules@2`) every Python file the candidate adds or changes is held to the harness families (`harness_override`, `unittest_patch`, `sys_modules`, `runner_reach`'s memory-reading side, `dynamic_code`), so the forgery moved verbatim into a helper the test imports is now a high finding, but ending the process, signal handlers, exiting at import, `load_tests` and raw-descriptor writes stay test-scoped, because production code does them, and a file the candidate did not change is not read; (d) the floor counts discovered tests, so a newly skipped test is not missing, and a test that is never discovered on the base is not protected. Also since `test-integrity-2`: a shard past its time budget (`--shard-timeout`, by default five times its planned seconds and at least 300 s) is stopped with its process tree and fails the run with `shard_timeout`, naming the last test it started; and CI floors every pull request against the merge base with its target branch (a push has no floor, and CI runs the candidate's own runner). Since the follow-up to #119 (2026-10-26, reviewer `rules@3`): a test that starts and stops with no outcome on the stream fails its shard, the runner also watches `unittest.case._addSkip` and `_addError` and `TestCase.doCleanups` and `subTest` at run time, and both the runner and the critic refuse a test class that takes `_addExpectedFailure` or `_addUnexpectedSuccess`, so a test can no longer silence a failure by muting how `unittest` reports it, or report its own unexpected success as a pass, without either layer seeing it; (a) to (d) stand. Since the follow-up to #123 (2026-10-26, reviewer `rules@4`): both layers also refuse a test class that takes a method that registers or runs its cleanups or runs its subtests (`addCleanup`, `enterContext`, `doCleanups`, `addClassCleanup`, `enterClassContext`, `doClassCleanups`, `subTest`), by where the method comes from rather than what it does; the critic's `unittest_patch` names an assignment of one of them on anything; and the runner checks the harness again when each test stops, so an instance given `subTest` in `setUp` fails the shard. A `subTest` that suppresses what its block raises, or a `doCleanups` that drops the cleanups, no longer turns a test's own failure into a pass. Of (b), what stays open is a test made vacuous in its own body (an assertion removed, a `try`/`except` around it) and another test's method replaced at run time; a harness change made and undone within one test is not seen by the runner; (a), (c) and (d) stand. The evaluation record, the schemas, the golden traces and the admission rules are unchanged: `SYBERLABS_BASE` is host behaviour like the cache allowance of entry 32 and the pass-through of entry 39, and nothing about it enters the record. The eventual answer is a container or VM sandbox per shard (research register item N4), so that the outcome is read from outside the process that produced it. +58. **A passing `tests` check is the shard runner's account of test code that ran in its own process, not an observation from outside it; the floor and the critic's integrity rule narrow what that code can hide, and only a sandbox closes it.** `test-integrity` (2026-10-26) answers the independent verification of #114, which showed test code defeating a `tests` check judged by exit code alone (`syberlabs/checks.py` `run_check`): an `os._exit(0)` at import, a forged shard result file, a faked `run()`. Built, in `scripts/run_shards.py`: the runner imports no test module (discovery runs in a child); each child reports over a private inherited pipe with a per-child random nonce and a sequence number, opened before any test module is imported; the verdict is counted from a live stream of `start`, `outcome` and `stop` events that the child's `done` summary must match; a test whose class or instance takes a method that starts or reports it from outside `unittest` is refused before anything runs, and a test that starts after test code changed those methods fails its shard; and, when the host sets `SYBERLABS_BASE` (now in every check's environment, from the candidate's recorded base) or `--floor BASE` is given, the working tree must be the committed tree and every test discovered on the base must still be discovered unless the diff `BASE..HEAD` deletes or renames its file, removes its `def`, or renames its class (`test_set_shrank`). In `examples/critic_rules.py`, a high finding for added test code that exits, kills, signals, overrides or patches the `unittest` harness, defines `load_tests`, writes `sys.modules` for `unittest` or `__main__`, reaches the runner (`__main__`, `gc`, frames, raw descriptors) or runs code the rule cannot read; `critic.no_high_finding` turns it into a refusal. What is not closed: (a) test code in the shard's process can read the nonce and the runner's objects and write a stream that is consistent and false; `tests/test_run_shards.py` shows such a forgery passing the runner, and the critic's patterns are what flag it, so a forgery written in a form the patterns do not list passes both; (b) a test can stay in the set and be vacuous, its own assertions emptied in the diff or another test's method replaced at run time, which no count can see; (c) the patterns reach beyond test files only in part: since `test-integrity-2` (2026-10-26, reviewer `rules@2`) every Python file the candidate adds or changes is held to the harness families (`harness_override`, `unittest_patch`, `sys_modules`, `runner_reach`'s memory-reading side, `dynamic_code`), so the forgery moved verbatim into a helper the test imports is now a high finding, but ending the process, signal handlers, exiting at import, `load_tests` and raw-descriptor writes stay test-scoped, because production code does them, and a file the candidate did not change is not read; (d) the floor counts discovered tests, so a newly skipped test is not missing, and a test that is never discovered on the base is not protected. Also since `test-integrity-2`: a shard past its time budget (`--shard-timeout`, by default five times its planned seconds and at least 300 s) is stopped with its process tree and fails the run with `shard_timeout`, naming the last test it started; and CI floors every pull request against the merge base with its target branch (a push has no floor, and CI runs the candidate's own runner). Since the follow-up to #119 (2026-10-26, reviewer `rules@3`): a test that starts and stops with no outcome on the stream fails its shard, the runner also watches `unittest.case._addSkip` and `_addError` and `TestCase.doCleanups` and `subTest` at run time, and both the runner and the critic refuse a test class that takes `_addExpectedFailure` or `_addUnexpectedSuccess`, so a test can no longer silence a failure by muting how `unittest` reports it, or report its own unexpected success as a pass, without either layer seeing it; (a) to (d) stand. Since the follow-up to #123 (2026-10-26, reviewer `rules@4`): both layers also refuse a test class that takes a method that registers or runs its cleanups or runs its subtests (`addCleanup`, `enterContext`, `doCleanups`, `addClassCleanup`, `enterClassContext`, `doClassCleanups`, `subTest`), by where the method comes from rather than what it does; the critic's `unittest_patch` names an assignment of one of them on anything; and the runner checks the harness again when each test stops, so an instance given `subTest` in `setUp` fails the shard. A `subTest` that suppresses what its block raises, or a `doCleanups` that drops the cleanups, no longer turns a test's own failure into a pass. Of (b), what stays open is a test made vacuous in its own body (an assertion removed, a `try`/`except` around it) and another test's method replaced at run time; a harness change made and undone within one test is not seen by the runner; (a), (c) and (d) stand. Since the follow-up to #130 (2026-10-31, reviewer `rules@5`): the runner records every discovered test's method as its module is loaded and fails the shard when one is no longer what `unittest` would call (replaced or deleted on its class or instance, in a test, at import from a later module, or in a fixture), and the critic's `test_replaced` names an assignment, deletion or patch of a `test...` attribute on anything other than `self`. Of (b), a replacement made and undone within one test, and a replaced helper or assertion method, stay open. The evaluation record, the schemas, the golden traces and the admission rules are unchanged: `SYBERLABS_BASE` is host behaviour like the cache allowance of entry 32 and the pass-through of entry 39, and nothing about it enters the record. The eventual answer is a container or VM sandbox per shard (research register item N4), so that the outcome is read from outside the process that produced it. 59. **The GitHits API client is wired into Claude Code as arm B's only lookup route; its contamination and license checks are filters on what returns, and its lookups are counted from the client's own records.** `githits-client` (2026-10-26) adds `syberlabs/githits.py` and `syberlabs githits search|example|read|list|grep` (`docs/GITHITS.md`): a standard-library client for five read-only GitHits REST operations, with a SyberLabs exclusion in three layers (a target refused before any request, a result whose identity is under an excluded owner dropped and counted, an example or read body that cites one dropped whole as `contamination_blocked`), a permissive-only license policy (`license_mode: strict` on examples; client-side, a result whose SPDX license is not on the allow list dropped and counted as `results_dropped_license`), and an HMAC-signed record per call (`query_guard.sign`, now shared). `githits-wiring` (2026-10-26) closes the first two deferred items: (1) `ClaudeCodeProvider(githits=True)` (`--githits`) writes a per-call, read-only launcher outside the worktree that runs the host checkout's client (`syberlabs/githits_cli_entry.py` under `python -I`), allows `Bash(:*)` and nothing else that runs code, matches `Bash` in the query guard (only the launcher with plain arguments passes, and its arguments are judged by the guard's rules), and journals one `lookup` per authentic client record (tool `githits.`), `signal.tool_usage.githits` being their number and `githits_lookups` reading those facts; (2) GitHits' shipped skills (`githits-com/githits-cli`, Apache-2.0) are adapted into `skills/githits/SKILL.md` with their license and notice, and appended to arm B's system prompt (`signal.harness.skill`). The same packet fixes #129's three low findings (nested SPDX expressions, the token scrubbed before truncation and out of `X-Request-ID`, `SYBERLABS_GITHITS_EXCLUDE` adding to `SyberLabs` instead of replacing it) and adds `syberlabs githits doctor` (one `GET /v1/languages`). `githits-pilot-hardening` (2026-10-26) narrows the launcher's entry to the five lookups (`doctor` and `--license-mode` stay the operator's, `example` always sends `strict`, and a `githits.doctor` fact is never counted in `githits_lookups`), records the skill's digest with CRLF normalized to LF (`.gitattributes` checks `skills/**` out with LF), tests that `SYBERLABS_GITHITS_EXCLUDE_REPLACE` reaches neither arm, and, under `--allow-code-tools-with-mcp`, matches `PowerShell` in the query guard and judges it by the launcher-only rule. Deferred: (3) the other operations of the document are not covered: `search_status` (polling a `search_ref` when a search returns incomplete; the client prints the ref and stops), `research` and the rest of `/v1/experimental/*`, package intelligence, `code_diff`, `resolve_target`, `feedback`; (4) no retry or back-off on `429`/`503`: `Retry-After` is not read by the lookup operations (only `doctor` reports it) and the call fails. Not solved: (5) layer (c) is detection, not prevention: an example can blend code from an excluded repository without citing it, and a mention spelled outside the listed host patterns passes; for the pilot, contamination is resolved on SyberLabs' side (unpublished work items, both arms run locally, a pinned base) with the client as the backstop, and an index-side exclusion by GitHits is optional; (6) a package target whose name does not name the owner reaches GitHits, and only its served repository, after the call, can drop it; (7) the license check depends on license data: in contract 0.1.0 only examples carry licenses (in Markdown), so search, grep, list and read results pass unjudged, and what `strict` admits server-side is not documented; (8) the fixture pins the five operations' closed subset of the document (215,618 bytes), and `doctor`'s operation in a second fixture, not the whole 1,364,570-byte document, which is over the contract's change-size limit; the upstream SHA-256 is recorded in `docs/GITHITS.md` for comparison by hand; (9) the record key and the guard's key sit in Claude Code's environment, which the launcher's client inherits, so a verifying MAC shows a key holder wrote the line, not which process; (10) no call to the real API was made, and no real Claude Code ran: every behaviour is tested against a loopback fake that follows the pinned schemas and a stand-in `claude` whose Bash tool is emulated (a command no `Bash(:*)` rule names is denied, the launcher runs as `sh` runs it); whether Claude Code 2.1.309 matches `Bash(:*)` against a Windows path written with forward slashes, and runs the launcher through Git Bash, is for the first live call (`syberlabs githits doctor` first, then one guarded lookup); (11) a record cannot be tied to the tool use that ran it (the client has no tool use id), so the host reconciles counts: authentic records against launcher calls the guard passed; fewer records is `signal.githits.mismatch: githits_unrecorded` (a client that failed before recording, a usage error, a denied permission), more is `githits_unexpected_records`, every record of the call is then `unguarded` and the search stops `provider_guard_inactive`; (12) closed by the two-arm accounting repair: failed calls journal unreconciled records as `unguarded` or `githits.unrecorded` when none exist, and a `no_change` refusal with missing records journals a synthetic gap; the metric remains unknown without a candidate signal; (13) `X-GitHits-Session-ID` is `-`, not `/`, because GitHits accepts only `[A-Za-z0-9_-]{1,64}`; (14) closed by the two-arm accounting repair: when client-record facts exist, `githits_lookups` also includes the host's MCP lookup attempts once; mixed routes still raise `two_routes`; (15) arm A (no `--githits` and no `--mcp-config`) receives no `GITHITS_*` or `SYBERLABS_GITHITS_*` variable since `githits-pilot-hardening`, the token included: least privilege was chosen over the byte-identical environment `githits-wiring` kept, so arm A's environment differs from before this route by exactly those variables (its argv does not); an `--mcp-config` run keeps them, because a GitHits MCP server may read its token from that environment; (16) the skill rides in `--append-system-prompt`, so it is part of every arm B request's system prompt, and arm A's harness note has no skill. diff --git a/tests/test_critic.py b/tests/test_critic.py index 225e749..e7c3df8 100644 --- a/tests/test_critic.py +++ b/tests/test_critic.py @@ -421,6 +421,19 @@ def _added(path, source, scope=None): "import ctypes\n", "LOGS = 'syberlabs-' 'shards-'\n", ], + "test_replaced": [ + "import unittest\n\nclass T(unittest.TestCase):\n def test_0(self):\n T.test_1 = lambda self: None\n" + " def test_1(self):\n self.fail('x')\n", + "from tests.test_other import Victim\nVictim.test_fails = lambda self: None\n", + "def helper(victim):\n setattr(victim, 'test_fails', None)\n", + "def helper(test):\n del type(test).test_fails\n", + "def helper(test):\n test.__class__.test_fails, x = None, 1\n", + "import unittest\n\nclass T(unittest.TestCase):\n @classmethod\n def setUpClass(cls):\n" + " cls.test_a = lambda self: None\n", + "from unittest import mock\n\ndef helper(Victim):\n return mock.patch.object(Victim, 'test_fails')\n", + "from unittest import mock\n\ndef helper(Victim):\n return mock.patch.multiple(Victim, test_fails=None)\n", + "from unittest.mock import patch\nSTOP = patch('tests.test_other.Victim.test_fails').start()\n", + ], "dynamic_code": [ "exec('import os')\n", "def helper(text):\n return eval(text)\n", @@ -442,6 +455,13 @@ def _added(path, source, scope=None): "import unittest\n\nclass T(unittest.TestCase):\n def test_a(self):\n self.addCleanup(print)\n" " with self.subTest(i=0):\n self.doCleanups()\n", "class Steps:\n def subTest(self):\n return None\n", + # an instance's own attributes named test... (a test fetches its own method before its code runs) + "import unittest\n\nclass T(unittest.TestCase):\n def setUp(self):\n self.test_dir = 'x'\n" + " setattr(self, 'testing', True)\n def test_a(self):\n self.test_dir = None\n", + "from unittest import mock\n\ndef helper(obj):\n obj.tests_dir.path = 'x'\n" + " return mock.patch.dict('os.environ', {'test_mode': '1'}), mock.patch.object(obj, 'path')\n", + "import unittest\n\ndef make(n):\n return lambda self: None\n\nclass T(unittest.TestCase):\n pass\n\n" + "for n in range(3):\n setattr(T, f'test_{n}', make(n))\n", ] @@ -568,7 +588,7 @@ def test_the_relocated_forgery_is_high_on_the_helper(self): self.assertEqual(self.rules(document, "tests/test_forge.py"), set()) self.assertEqual(sorted(f["line"] for f in document["findings"]), [1, 2, 6, 7]) self.assertTrue(all("a test can import this file" in f["claim"] for f in document["findings"])) - self.assertEqual(document["reviewer"], {"name": "rules", "revision": "4"}) + self.assertEqual(document["reviewer"], {"name": "rules", "revision": "5"}) def test_the_harness_families_apply_to_any_python_file(self): write_side = ("os.write", "syberlabs-") # the runner_reach snippets that stay test-scoped (next test) diff --git a/tests/test_publish.py b/tests/test_publish.py index afd20ad..73333a8 100644 --- a/tests/test_publish.py +++ b/tests/test_publish.py @@ -317,7 +317,7 @@ def test_reviewed_by_names_the_critic_from_the_observation(self): kit, thread = self.accepted(checks=CRITIC) lines = self.pull_request_lines(thread) self.assertEqual([line for line in lines if line.startswith("Reviewed-by:")], - ["Reviewed-by: rules@4 (critic check critic: high 0, medium 0, low 0)"]) + ["Reviewed-by: rules@5 (critic check critic: high 0, medium 0, low 0)"]) self.assertEqual([line for line in lines if line.startswith("Evidence:")], ["Evidence: tested"]) self.assertIn("Host checks on that tree: critic passed, tests passed.", lines) self.assertFalse([line for line in lines if line.startswith("Approved-by:")]) @@ -330,7 +330,7 @@ def test_reviewed_by_lists_both_the_approver_and_the_critic(self): self.assertEqual(thread.accept("c1").status, "succeeded") lines = self.pull_request_lines(thread) self.assertEqual([line for line in lines if line.startswith("Reviewed-by:")], - ["Reviewed-by: lead@example.test (maintainer), rules@4 (critic check critic: high 0, medium 0, low 0)"]) + ["Reviewed-by: lead@example.test (maintainer), rules@5 (critic check critic: high 0, medium 0, low 0)"]) self.assertIn("Approved-by: lead@example.test (maintainer)", lines) def test_reviewer_name_and_revision_cannot_forge_a_field(self): diff --git a/tests/test_run_shards.py b/tests/test_run_shards.py index ce2d202..97fe0b1 100644 --- a/tests/test_run_shards.py +++ b/tests/test_run_shards.py @@ -647,6 +647,76 @@ def test_1_never_runs(self): raise AssertionError("must run") """ +# Test code that replaces another test's method, so that the replaced test reports whatever the replacement does. +# Serial unittest reports OK for each; the runner records every test's method as its module is loaded. +REPLACES_IN_A_TEST = """ + import unittest + + class Victim(unittest.TestCase): + def test_0_replace(self): + Victim.test_1_must_fail = lambda self: None + def test_1_must_fail(self): + self.fail("must fail") +""" +REPLACES_AT_IMPORT = """ + import unittest + from test_r_victim import Victim + + Victim.test_must_fail = lambda self: None + + class Other(unittest.TestCase): + def test_ok(self): pass +""" +REPLACED_VICTIM = """ + import unittest + + class Victim(unittest.TestCase): + def test_must_fail(self): + self.fail("must fail") +""" +REPLACES_IN_A_CLASS_FIXTURE = """ + import unittest + + class Replacer(unittest.TestCase): + @classmethod + def setUpClass(cls): + setattr(Victim, "test_must_fail", lambda self: None) + def test_a(self): pass + + class Victim(unittest.TestCase): + def test_must_fail(self): + self.fail("must fail") +""" +# The negative control: what looks like replacing a test method and is not one the loader found. Tests generated +# at import before their module is loaded, an instance attribute of another name, and a TestCase a test builds and +# runs itself, whose method it replaces. All pass under both runners. +NOT_A_REPLACEMENT = """ + import io, unittest + + def make(n): + def test(self): + self.assertGreaterEqual(n, 0) + return test + + class Generated(unittest.TestCase): + def setUp(self): + self.test_dir = "scratch" + def test_uses_an_attribute(self): + self.test_dir = None + + for n in range(3): + setattr(Generated, f"test_generated_{n}", make(n)) + + class Builds(unittest.TestCase): + def test_runs_its_own_inner_case(self): + class Inner(unittest.TestCase): + def test_x(self): + self.fail("replaced before it runs") + Inner.test_x = lambda self: None + result = unittest.TextTestRunner(stream=io.StringIO()).run(Inner("test_x")) + self.assertTrue(result.wasSuccessful()) +""" + class Integrity(unittest.TestCase): """Test code runs in the shard's process; these are the ways it can try to lie, and what catches each.""" @@ -782,6 +852,64 @@ def test_the_critic_flags_each_way_of_swallowing_a_failure(self): self.assertEqual({f["claim"].split("(")[1].split(")")[0] for f in findings}, {rule}) self.assertTrue(all(f["severity"] == "high" for f in findings)) + def test_a_test_that_replaces_another_test_s_method_fails_the_shard(self): + """Replaced in a test's body, at import by a module loaded later, or in a class fixture: what ``unittest`` + would call is no longer the method its module defined when it was loaded. Every discovered test is compared, + so which shard the replaced test was packed into does not matter.""" + rows = (({"test_replace.py": REPLACES_IN_A_TEST}, + "test code changed test_replace.Victim.test_1_must_fail while test_replace.Victim.test_0_replace ran"), + ({"test_r_victim.py": REPLACED_VICTIM, "test_s_replacer.py": REPLACES_AT_IMPORT}, + "after test code changed test_r_victim.Victim.test_must_fail"), + ({"test_fixture.py": REPLACES_IN_A_CLASS_FIXTURE}, + "ran test_fixture.Replacer.test_a after test code changed test_fixture.Victim.test_must_fail")) + for extra, says in rows: + with self.subTest(files=sorted(extra)): + self.assertTrue(_summary(_serial_suite(extra))[1].startswith("OK")) # serial unittest is fooled + for shards in ("1", "2"): + ran = self.run_suite(extra, "--shards", shards) + self.assertEqual(ran.returncode, 1, ran.stdout) + self.assertIn(says, ran.stdout) + self.assertIn("shard_incomplete", _summary(ran.stdout)[1]) + + def test_a_test_that_replaces_its_own_method_on_its_instance_fails_the_shard(self): + """Conservative: the test has already fetched its own method, but the instance no longer holds what was + loaded, and the runner does not judge what a replacement could do.""" + body = "import unittest\n\nclass Own(unittest.TestCase):\n def test_a(self):\n self.test_a = None\n" + ran = self.run_suite({"test_own.py": body}) + self.assertEqual(ran.returncode, 1, ran.stdout) + self.assertIn("test code changed test_own.Own.test_a while test_own.Own.test_a ran", ran.stdout) + + def test_what_only_looks_like_replacing_a_test_method_passes(self): + """The negative control: tests generated before their module is loaded, an instance attribute named + test..., and a test case a test builds and runs itself. The run is OK and counts what serial unittest + counts.""" + base = _summary(_serial_suite({}))[0] + with tempfile.TemporaryDirectory() as folder: + _write_suite(Path(folder), {"test_lookalike.py": NOT_A_REPLACEMENT}) + serial = _summary(_serial(Path(folder)).stderr) + self.assertEqual(serial[0], base + 5) # the three generated tests are discovered and run + for shards in ("1", "2"): + ran = _shards(Path(folder), "--shards", shards) + self.assertEqual(ran.returncode, 0, ran.stdout) + self.assertNotIn("test code changed", ran.stdout) + self.assertEqual(_summary(ran.stdout), serial) + # The critic matches names, not objects: it flags the inner case's replaced method too (docs/CRITIC.md). + findings = critic_rules.integrity_findings("tests/test_lookalike.py", + textwrap.dedent(NOT_A_REPLACEMENT).lstrip(), None) + self.assertEqual([f["claim"].split(": ")[1] for f in findings], ["assigns Inner.test_x"]) + + def test_the_critic_flags_each_replacement(self): + for name, body, line in (("test_replace.py", REPLACES_IN_A_TEST, 5), + ("test_s_replacer.py", REPLACES_AT_IMPORT, 4), + ("test_fixture.py", REPLACES_IN_A_CLASS_FIXTURE, 6)): + with self.subTest(name=name): + findings = critic_rules.integrity_findings(f"tests/{name}", textwrap.dedent(body).lstrip(), None) + self.assertEqual([(f["line"], f["claim"].split("(")[1].split(")")[0]) for f in findings], + [(line, "test_replaced")]) + self.assertTrue(all(f["severity"] == "high" for f in findings)) + self.assertEqual(critic_rules.integrity_findings("tests/test_r_victim.py", + textwrap.dedent(REPLACED_VICTIM).lstrip(), None), []) + def test_bytes_on_the_channel_without_the_nonce_discard_the_shard(self): ran = self.run_suite({"test_garbage.py": CHANNEL_GARBAGE}) self.assertEqual(ran.returncode, 1, ran.stdout)