Skip to content

feat: surface test-fidelity findings in the gate, dashboard, brief and mod - #529

Open
AraneaDev wants to merge 32 commits into
feat/test-fidelity-tsfrom
feat/test-fidelity-surfaces
Open

AraneaDev wants to merge 32 commits into
feat/test-fidelity-tsfrom
feat/test-fidelity-surfaces

Conversation

@AraneaDev

Copy link
Copy Markdown
Owner

Summary

Phase 2 of test fidelity shows the findings that phase 1 (#528) stores. It is stacked on #528: the PR targets feat/test-fidelity-ts and should be retargeted to main once #528 merges.

  • Keys and suppression: each finding gets a key that ignores line numbers, sha1(rule_id|sub_rule|file|target_canonical|message). knossos.json can suppress findings with test_finding_suppressions entries of the form {rule, path?, reason?}. A rule can be a category, CATEGORY/sub_rule, or a full rule id. Suppression applies when findings are read, so no rescan is needed.
  • Snapshots: each snapshot archives its keyed findings (migration 028). A scan the test rules never judged has no record, so a gate against it reports indeterminate with baseline_has_no_test_findings instead of failing.
  • Gate and SARIF: quality_gate gains a new_definite_test_findings budget, computed as a set difference of distinct keys. Findings also appear in SARIF: ruleId is rule_id/sub_rule, level is error for definite and warning for warning.
  • Dashboard: a new test_findings section. Test doubles and test cases no longer count toward health, dead code, the agent brief or the boundary counts. Spies on lib globals (console, Date.now) are treated as external.
  • Turn brief: lists the definite findings a turn introduced. It follows doubles edges back from the edited production files, so a stale mock in a test file the turn never touched still counts.
    • Turns the live watcher already scanned use per-file test baselines stored in the scan ledger. One-scan and multi-scan turns both work.
    • Deleting or renaming a module, including through Bash rm / git mv, introduces the mocks it strands. Up to 10 deletions per turn count; a turn that deletes more is treated as a checkout.
  • Mod:
    • A Read note: knossos: 2 test doubles of PaymentGateway no longer match it (charge(): arity).
    • A turn-end note: knossos: this turn introduced 1 stale mock. Fix it before finishing:
    • Both notes contain definite findings only, and each finding is said once per session.
    • A new setting, testChecks (default on), turns these notes off.
    • The band has a chip, N stale mocks · M test findings.
    • The Issues tab lists findings, definite first.
  • Stale mocks vs test findings: "stale mock" means GHOST_MEMBER, ARITY_MISMATCH or RETURN_DRIFT. Other definite findings, such as tautologies, are "test findings".
  • History: snapshot_diff gains a test_findings section, and architecture_trends gains per-snapshot counts. A snapshot with no record reads as not recorded, never as 0.
  • Phase 1 fix: vi.mock('../src/X.js') now resolves to X.ts, as TypeScript does. NodeNext/ESM projects use .js specifiers, and they were getting definite missing-module findings.

The output schema stays 1.0; every new field is additive.

Review

Each task got its own review. A final whole-branch review, run on the most capable model, found two bugs:

  • With the watcher on (the default), the turn-end note never fired.
  • An upgrade baseline failed the gate instead of reporting indeterminate.

Both were fixed and re-reviewed. Tasks 9–12 came out of that review and had a whole-branch review of their own, which found no Critical or Important issues.

Known costs of design rulings:

  • A PHP class that moves between two edited files under the same FQCN can have its existing stale mock reported as introduced once.
  • Another process deleting 10 or fewer files mid-turn can have its stranded mocks attributed to the turn.

Test plan

  • vendor/bin/phpunit: 6,838 tests. The only failures are the 3 Rust-worker tests that also fail on main (ProjectUnitVerificationTest ×2, ReturnedReceiverTest ×1).
  • Mod vitest (2,520) and the staged claude plugin test (218) pass. TypeScript worker vitest: 519 pass.
  • phpstan (as CI runs it), php-cs-fixer, eslint, format:check, lint:md, the file-length and maintainability checks, docstring checks, and generate-reference --check.
  • Benchmarks: large incremental 18.84s against a 20s budget; test_fidelity_rescan_one_file 2.10s against 8s.
  • JS coverage floors: CI measures them in the container. The new worker branches are exercised by a PHPUnit scan.
  • Dogfood on a real Vitest/Jest repo, including a NodeNext one.

AraneaDev and others added 30 commits November 19, 2026 16:09
… test facts from the boundary sample, tolerate a missing folder in audit_tests
Co-Authored-By: Claude <noreply@anthropic.com>
… ledger baselines

Every ledgered scan now records, for each file it changes, the keys of the
definite, unsuppressed test findings that file touched before it, capped at
100 a file with a truncated flag. ScanLedger::since composes them as
test_baselines, the earliest record winning, and the turn brief judges a
reported file the watcher already took in by them instead of leaving
test_findings not_evaluated. A missing, cut or merged baseline still does.

Also tests TurnTestFindings' cap, truncated flag and file-then-line order,
and drops its unused suppressionsOf fallback.

Co-Authored-By: Claude <noreply@anthropic.com>
A scan now carries test_findings_judged, set when its test findings are
written, even when there were none. The archive copies it into
test_findings_recorded instead of the constant 1, so the scan active at an
upgrade from before the test rules, archived with no rows, reads as no
record and a quality gate against it is indeterminate rather than failing
on every existing finding. The column goes in migration 028, which this
unreleased branch adds.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…wants

Co-Authored-By: Claude <noreply@anthropic.com>
…ndings call

Keeps LedgeredScanner's dependency fan-out within the maintainability
budget of 15.

Co-Authored-By: Claude <noreply@anthropic.com>
A stale mock is a double that no longer matches its target (ghost, arity, return drift); every other definite finding is a test finding. The dashboard gains stale, the brief gains stale per item and total_stale, and the band, Read note and turn-end head say each in its own words.
A double whose module resolved to nothing now touches the files its raw relative specifier can name, read with the TypeScript worker's TYPESCRIPT_SOURCES and REQUIRE_SUFFIXES; a reported, deleted one of them (a renamed module's old path) makes its finding affected.

Ruling R13: a key counts as before only when every taken-in reported file that holds it now held it in its ledger baseline, so a later watcher scan's baseline no longer hides a ghost an earlier one made. A baseline read from a scan the test rules never judged leaves the brief not_evaluated.
…t strands

Ruling R14: an edit tool cannot delete, so the old path of an rm or git mv is never reported. The deletions the turn's own diff saw (including those a ledgered scan since the turn began took in) now count as the turn's when there are at most ten; past that the diff is checkout-sized and deletions introduce nothing, with the turn still judged. The before-set is judged over the reported and those deleted files, once the diff names them.
snapshot_diff gains a test_findings section (added and removed by key, not_recorded when a side kept none) and architecture_trends a definite_test_findings and stale_test_findings figure per snapshot, null without a record. Both apply the project's current suppressions to every side.
A relative mock specifier with a .js, .jsx, .mjs or .cjs extension now resolves to the TypeScript source it compiles from first, as NodeNext and bundler resolution do, instead of being reported as a definite missing module. The mapping moves to syntax-predicates.js and a PHP test pins MockedModulePaths to it.
…introduced by deleting it

A unit test over a stored missing_module whose .js specifier names the deleted .ts source, with a twin whose baseline lacks the key. The brief-level case is renamed to what it checks.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 67309d85-3a75-4abe-ab05-4d79da1d8da5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant