Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/CRITIC.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,15 @@ In the factory, test code is candidate code, and it runs inside the process whos
| `exit_at_import` | calls `sys.exit`, `exit`, `quit`, `os._exit` or `unittest.main`, or raises `SystemExit`, `KeyboardInterrupt`, `GeneratorExit` or `BaseException`, at module level (not under `if __name__ == "__main__":`) or in `setUpModule`, `tearDownModule`, `setUpClass` or `tearDownClass` |
| `harness_override` | defines or assigns `run`, `__call__`, `debug`, `id`, `_callTestMethod`, `_callSetUp`, `_callTearDown`, `_callCleanup`, `_addExpectedFailure`, `_addUnexpectedSuccess`, `countTestCases` or `__getattribute__`, or a method that registers or runs a test's cleanups or runs its subtests (`addCleanup`, `enterContext`, `doCleanups`, `addClassCleanup`, `enterClassContext`, `doClassCleanups`, `subTest`), in a class that derives from a name ending in `TestCase` or from `unittest`, or that defines a `test*` method |
| `load_tests` | defines or assigns `load_tests` |
| `test_replaced` | assigns, deletes or `setattr`s an attribute named `test...` on anything other than `self` (`Victim.test_a = ...`, `cls.test_a = ...`, `del type(t).test_a`), or patches one with `mock.patch("pkg.Cls.test_a")`, `patch.object(Cls, "test_a")` or `patch.multiple(Cls, test_a=...)` |
| `unittest_patch` | assigns or deletes an attribute of anything imported from `unittest`; assigns a result callback (`startTest`, `stopTest`, `addSuccess`, ..., `wasSuccessful`, `testsRun`, `shouldStop`, `stop`), a test's identity (`_testMethodName`, `__qualname__`, `__module__`) or one of the cleanup and subtest methods above on anything; uses `setattr` or `delattr` for those; patches a `unittest.*` or `sys.modules` target with `mock.patch`; reads `_outcome`, `_resultForDoCleanups` or `shouldStop` |
| `sys_modules` | writes `sys.modules['unittest...']` or `sys.modules['__main__']` (assignment, `del`, `pop`, `setdefault`, `__setitem__`), or calls `sys.modules.update` or `sys.modules.clear` |
| `runner_reach` | imports `__main__`, `gc`, `ctypes`, `msvcrt` or `_winapi`; reads `sys.modules['__main__']`; calls `gc.get_objects`, `gc.get_referrers`, `gc.get_referents`, `sys._getframe`, `sys._current_frames`, `inspect.currentframe`, `inspect.stack`, `inspect.getouterframes`, `inspect.getinnerframes`, `inspect.trace`, `os.dup`, `os.dup2`, `os.write`, `os.fdopen`, `os.set_inheritable`, `os.set_handle_inheritable`, `sys.settrace`, `sys.setprofile`, `sys.addaudithook`, `threading.settrace` or `threading.setprofile`; reads a frame attribute (`f_back`, `f_locals`, `f_globals`, `tb_frame`, `gi_frame`, `cr_frame`, `ag_frame`); or holds a string naming the shard runner's directories (`syberlabs-shard...`) |
| `dynamic_code` | calls `exec`, `eval` or `compile`; imports a computed name (`__import__`, `importlib.import_module`, `importlib.reload`), unless the name's package is fixed in the source (`f"pkg.{name}"` or `"pkg." + name`, where `pkg` is not one of the modules named in this table); or gets, sets or deletes a computed attribute of `os`, `sys`, `signal`, `unittest`, `gc`, `inspect` or `__main__` |

Why these: each is a way for test code to end the process with an exit code it chooses (the shard runner then sees a crash, but serial `unittest` sees success), to replace how a test is started or counted, to choose the test set at discovery, or to reach the runner's channel and nonce in its own memory and write a false report, which the runner cannot detect (`spec/GAPS.md` 58).

**Beyond test files.** A test can import any module, and the module's code then runs in the shard's process. So moving a forgery verbatim out of the test file and into a helper the test calls passed both layers under revision 1 (the verification of #116, finding 1). Every Python file (`.py`, `.pyw`) the candidate adds or changes is now held to the families that tamper with the harness from inside that process: `harness_override`, `unittest_patch`, `sys_modules`, `runner_reach` and `dynamic_code` (`HARNESS_RULES`). Four families stay test-scoped (`TEST_ONLY_RULES`): `process_exit`, `signal_handler`, `exit_at_import` and `load_tests`. Production code legitimately exits, kills child processes and installs handlers, and `load_tests` only means something in a module that discovery loads. Two parts of `runner_reach` also stay test-scoped:
**Beyond test files.** A test can import any module, and the module's code then runs in the shard's process. So moving a forgery verbatim out of the test file and into a helper the test calls passed both layers under revision 1 (the verification of #116, finding 1). Every Python file (`.py`, `.pyw`) the candidate adds or changes is now held to the families that tamper with the harness from inside that process: `harness_override`, `unittest_patch`, `sys_modules`, `runner_reach`, `dynamic_code` and `test_replaced` (`HARNESS_RULES`). Four families stay test-scoped (`TEST_ONLY_RULES`): `process_exit`, `signal_handler`, `exit_at_import` and `load_tests`. Production code legitimately exits, kills child processes and installs handlers, and `load_tests` only means something in a module that discovery loads. Two parts of `runner_reach` also stay test-scoped:
- The write side: `os.write`, `os.dup`, `os.dup2`, `os.fdopen`, `os.set_inheritable`, `os.set_handle_inheritable`, `msvcrt` and `_winapi`. This is ordinary I/O; `syberlabs/journal.py` locks with `msvcrt`, and the runner itself uses the rest. A false report needs the channel's nonce. Without it, a write through these is `channel_tampered`. Only reading the process's memory gets the nonce (`gc`, frames, `__main__`, `ctypes`, trace and audit hooks), and that read side applies to every file.
- The runner's directory names: since results moved to the channel, those directories hold only logs.

Expand All @@ -109,7 +110,7 @@ What it does not catch:

It is a pattern list, not an analysis; a candidate that knows the list can write around it. Its precision against human review is unmeasured.

The reviewer declares revision `4` (`Reviewed-by: rules@4`). Its verdicts changed four times: with the test-integrity rule, again when the rule's scope grew to every changed Python file, again when `harness_override` came to name `_addExpectedFailure` and `_addUnexpectedSuccess`, and again when it came to name the cleanup and subtest methods. Through the first two `TestCase.run` reports an expected failure and an expected failure that passed; a test class that defines `_addUnexpectedSuccess` to call `addSuccess` turned an outcome that fails the run into a pass, and revision 2 did not see it (`scripts/run_shards.py` now refuses it too, `docs/TESTING.md`). Through the others a test runs its own cleanups and subtests; a test class that defines `subTest` to suppress what its block raises, or `doCleanups` to drop its cleanups, turned its own failure into a pass, and revision 3 did not see it, nor `self.subTest = ...` in `setUp` (`unittest_patch` now names that assignment; the runner refuses the first two and fails the shard on the third). `tests/test_publish.py` pins the line.
The reviewer declares revision `5` (`Reviewed-by: rules@5`). Revision 5 adds `test_replaced`: a test that replaced another test's method (in its body, at import, or in a class fixture) made the replaced test report whatever the replacement did, and revision 4 did not see it (`scripts/run_shards.py` now fails the shard too, `docs/TESTING.md`). It matches names, not objects, so it also flags a test that builds its own inner `TestCase` and replaces that one's method; `setattr(Cls, f"test_{n}", ...)` with a computed name is not matched, which is how tests are generated at import. Before that, its verdicts changed four times: with the test-integrity rule, again when the rule's scope grew to every changed Python file, again when `harness_override` came to name `_addExpectedFailure` and `_addUnexpectedSuccess`, and again when it came to name the cleanup and subtest methods. Through the first two `TestCase.run` reports an expected failure and an expected failure that passed; a test class that defines `_addUnexpectedSuccess` to call `addSuccess` turned an outcome that fails the run into a pass, and revision 2 did not see it (`scripts/run_shards.py` now refuses it too, `docs/TESTING.md`). Through the others a test runs its own cleanups and subtests; a test class that defines `subTest` to suppress what its block raises, or `doCleanups` to drop its cleanups, turned its own failure into a pass, and revision 3 did not see it, nor `self.subTest = ...` in `setUp` (`unittest_patch` now names that assignment; the runner refuses the first two and fails the shard on the third). `tests/test_publish.py` pins the line.

## Model-backed critic

Expand Down
26 changes: 22 additions & 4 deletions docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,17 @@ otherwise:
the change (`test code changed X while T ran`); so a test whose `setUp` or
body gives its instance one of the refused methods (`self.subTest = ...`),
after the check at its start, also fails the shard. A change made and
undone within one test, before it stops, is not seen.
undone within one test, before it stops, is not seen. The same checks
compare every discovered test's method, on its class and its instance,
with the one its module defined when the loader loaded it (recorded per
module, before the next module is imported): test code that replaces or
deletes another test's method (`Victim.test_x = lambda self: None` in a
test body, at import from a module loaded later, or in `setUpClass`) fails
the shard, naming the replaced test. Every discovered test is compared,
not only the shard's own, so the verdict does not depend on packing. A
test that replaces its own method on its instance is refused too
(conservatively: it has already fetched it). Tests a module generates at
import, before the loader reads it, are what was loaded.
- **After running:** every shard's stream is intact and ends with `done`;
every test it started reported an outcome before it stopped (a success,
failure, error, skip, expected failure, unexpected success or failed
Expand Down Expand Up @@ -267,6 +277,9 @@ critic finding, and the honest rows still pass:
| A `TestCase` defines `subTest` to suppress what its block raises; its subtest fails (after #123) | **0** | 1, `harness_override` | before: none; now high (`harness_override`, rules@4) |
| A `TestCase` defines `doCleanups` to drop its cleanups; a cleanup fails (after #123) | **0** | 1, `harness_override` | before: none; now high (`harness_override`, rules@4) |
| A test's `setUp` sets `self.subTest` to suppress exceptions; its subtest fails (after #123) | **0** | 1, `harness_changed` (seen when the test stops) | before: none; now high (`unittest_patch`, rules@4) |
| A test replaces another test's method in its body; the replaced test would fail (after #130) | **0** | 1, `harness_changed` (seen when the replacing test stops) | before: none; now high (`test_replaced`, rules@5) |
| A test module, loaded later, replaces an earlier module's test method at import (after #130) | **0** | 1, `harness_changed` (seen when the next test starts) | before: none; now high (`test_replaced`, rules@5) |
| A `setUpClass` replaces another class's test method with `setattr` (after #130) | **0** | 1, `harness_changed` | before: none; now high (`test_replaced`, rules@5) |
| A test patches `TestCase._callTestMethod` at run time (new) | **0** | 1, `harness_changed` | high (`unittest_patch`) |
| A test writes a line without the nonce to the channel (new) | n/a | 1, `channel_tampered` | high (`runner_reach`) |
| A test finds the runner in its own process and writes a consistent, false stream (new) | n/a | **0: not caught** | high (`runner_reach`, `process_exit`) |
Expand All @@ -286,7 +299,10 @@ The three rows marked "after #119" ran the same way on `main` at `296e3ca`
(before) and on this change (after); serial `unittest discover` says `OK` for
each of them as well. The three rows marked "after #123" ran the same way on
#123's head at `2a8bdeb` (before) and on its follow-up (after); serial
`unittest discover` says `OK` for each. `tests/test_run_shards.py` holds every
`unittest discover` says `OK` for each. The three rows marked "after #130"
ran the same way on #130's head restacked onto `main` (`c0e58bc`, before) and
on its follow-up (after), with `--shards 1`; serial `unittest discover` says
`OK` for each. `tests/test_run_shards.py` holds every
row of the table that the runner catches.

The guarantee, exactly: with the factory's `tests` check (the host
Expand All @@ -306,8 +322,10 @@ host already trusts it. Two further gaps are named in `spec/GAPS.md` 58:
- Test code can keep every test in the set but make tests vacuous: its own
in the diff, or another's at run time by replacing its method. No count can
see this. Swallowing its own failures through the cleanup and subtest
methods is refused (above); a `try`/`except` around its own assertions, or
an assertion removed, is not.
methods is refused (above), and so is replacing a discovered test's method
(above), unless it is made and undone within one test; a `try`/`except`
around its own assertions, an assertion removed, or a helper or assertion
method a test calls replaced at run time, is not.
- Outside test files, the patterns that end the process or write to raw
descriptors do not apply, because production code does both.

Expand Down
Loading
Loading