Skip to content

fix(reporter): keep internal grabber steps and console noise out of reports - #226

Merged
DavertMik merged 7 commits into
mainfrom
fix/report-internal-steps
Sep 17, 2026
Merged

DavertMik merged 7 commits into
mainfrom
fix/report-internal-steps

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

Closes the report noise reported in the session: I.grabBrowserLogs() calls showing up in the Testomat.io stack, and console/network errors landing as failed steps.

Root cause

Every grabber filter in the codebase was dead code.

CodeceptJS 4's Step has title, not nameactor.js:60 builds new Step(helper, action), and HelperStep sets this.title = action. There is no name property on the class at all (only Section has one). So:

if (step.name?.startsWith('grab')) return;   // step.name is ALWAYS undefined

…never fired, in src/action.ts or src/explorer.ts. It failed open silently, which is why I.grabBrowserLogs() reached both:

  • the Testomat.io stack field — test.stepstestData.logsformatLogs() in the reporter client
  • Langfuse step spans — attachStepLoggertag('step')SpanDestination

Changes

Internal steps — one predicate, isInternalStep() in src/utils/step-analyzer.ts, reading step.title, replacing three hand-rolled checks that had drifted apart (action.ts filtered grab only, explorer.ts filtered grab+save, rerunner.ts filtered nothing):

  • src/action.tsattachStepLogger
  • src/explorer.ts — reported test steps
  • src/ai/rerunner.ts — three step handlers that had no filter
  • src/utils/logger.ts — Langfuse spans were all named I.step for the same reason; now I.click, I.fillField, …

Console/network errorsaddObservation() notes were becoming category: 'user', status: 'failed' steps in the report, so a page logging errors in the background made a passing test read as broken. They are now filtered out of combineStepsAndNotes and out of extractLastNoteMessage (an observation landing last could become the test's headline message), and travel in the test logs instead. Pilot, Analyst and Historian still read them through notesToString() / getLog() unchanged.

Notes for review

Two judgement calls worth a look:

  1. Observations were moved to the report's logs/stack rather than dropped, on the grounds that they are real evidence. Easy to drop entirely if a clean stack is preferred.
  2. save* is now filtered in attachStepLogger too (it was grab-only there; explorer.ts already filtered both). Harmless — the tester prompt forbids saveScreenshot — but it does change executed-step recording.

Left out deliberately

src/action.ts:589 has the identical dead-property bug:

if (assertionsTarget && ASSERTION_STEP_NAMES.has(step.name)) {

step.name is undefined, so no assertion has ever been captured, meaning output.framework: 'playwright' has been generating tests with zero expect() lines since the CodeceptJS 4 migration. The fix is step.namestep.title on both lines, but it changes generated-test output, so it is not in this PR. Happy to add it here or split it out.

Also unchanged: getPrintableNotes() still renders these to Pilot as FAILED Console error: ….

Verification

bun test tests/unit/ — 1439 pass, 0 fail. Format and lint clean. Three new cases: isInternalStep unit coverage, an explorer test asserting grab/save steps are not recorded while I.click is, and a reporter test asserting observations stay out of report steps.

🤖 Generated with Claude Code

DavertMik and others added 7 commits September 14, 2026 14:46
Extracts src/utils/markdown-query.ts into src/utils/mdq/ as a
publish-ready package: MarkdownDoc + Selection, insert/remove verbs
alongside query, a comment selector, frontmatter handling, JS-value
matchers, and a planned jq-like CLI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhJNfxLquFknSLJ6V8J9iD
Frontmatter uses yaml's Document API (comment-preserving) rather than a
hand-rolled parser; MarkdownDoc gains append/prepend; leading '.' is
accepted in the CLI grammar; documents a fourth migration breakage class
where MarkdownDoc === string silently stops a guard from firing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhJNfxLquFknSLJ6V8J9iD
11 tasks, 68 steps. Ordering keeps the repo green at every boundary:
port behind a shim first, migrate the 11 write-return-type breaks second,
then add features additively.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhJNfxLquFknSLJ6V8J9iD
…eports

Every grabber filter in the codebase was dead code. CodeceptJS 4's Step has
`title`, not `name` (actor.js builds `new Step(helper, action)` -> title), so
`step.name?.startsWith('grab')` was always undefined and failed open. That let
`I.grabBrowserLogs()` and friends through into the Testomat.io stack field and
into Langfuse step spans.

- add `isInternalStep()` to step-analyzer, reading `step.title`, and use it in
  the three places that filtered steps by hand (action, explorer, rerunner —
  rerunner had no filter at all)
- name Langfuse step spans from `step.title` instead of the missing `step.name`
- keep `addObservation()` notes (console and network errors) out of reported
  steps and out of the test summary message; they now travel in the test log,
  and Pilot/Analyst still read them from the notes as before

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DavertMik
DavertMik merged commit b4ccba9 into main Sep 17, 2026
2 checks passed
@DavertMik
DavertMik deleted the fix/report-internal-steps branch September 17, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant