fix(reporter): keep internal grabber steps and console noise out of reports - #226
Merged
Merged
Conversation
Extracts src/utils/markdown-query.ts into src/utils/mdq/ as a publish-ready package: MarkdownDoc + Selection, insert/remove verbs alongside query, a comment selector, frontmatter handling, JS-value matchers, and a planned jq-like CLI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhJNfxLquFknSLJ6V8J9iD
Frontmatter uses yaml's Document API (comment-preserving) rather than a hand-rolled parser; MarkdownDoc gains append/prepend; leading '.' is accepted in the CLI grammar; documents a fourth migration breakage class where MarkdownDoc === string silently stops a guard from firing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhJNfxLquFknSLJ6V8J9iD
11 tasks, 68 steps. Ordering keeps the repo green at every boundary: port behind a shim first, migrate the 11 write-return-type breaks second, then add features additively. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhJNfxLquFknSLJ6V8J9iD
…eports
Every grabber filter in the codebase was dead code. CodeceptJS 4's Step has
`title`, not `name` (actor.js builds `new Step(helper, action)` -> title), so
`step.name?.startsWith('grab')` was always undefined and failed open. That let
`I.grabBrowserLogs()` and friends through into the Testomat.io stack field and
into Langfuse step spans.
- add `isInternalStep()` to step-analyzer, reading `step.title`, and use it in
the three places that filtered steps by hand (action, explorer, rerunner —
rerunner had no filter at all)
- name Langfuse step spans from `step.title` instead of the missing `step.name`
- keep `addObservation()` notes (console and network errors) out of reported
steps and out of the test summary message; they now travel in the test log,
and Pilot/Analyst still read them from the notes as before
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the report noise reported in the session:
I.grabBrowserLogs()calls showing up in the Testomat.io stack, and console/network errors landing as failed steps.Root cause
Every grabber filter in the codebase was dead code.
CodeceptJS 4's
Stephastitle, notname—actor.js:60buildsnew Step(helper, action), andHelperStepsetsthis.title = action. There is nonameproperty on the class at all (onlySectionhas one). So:…never fired, in
src/action.tsorsrc/explorer.ts. It failed open silently, which is whyI.grabBrowserLogs()reached both:stackfield —test.steps→testData.logs→formatLogs()in the reporter clientattachStepLogger→tag('step')→SpanDestinationChanges
Internal steps — one predicate,
isInternalStep()insrc/utils/step-analyzer.ts, readingstep.title, replacing three hand-rolled checks that had drifted apart (action.tsfilteredgrabonly,explorer.tsfilteredgrab+save,rerunner.tsfiltered nothing):src/action.ts—attachStepLoggersrc/explorer.ts— reported test stepssrc/ai/rerunner.ts— three step handlers that had no filtersrc/utils/logger.ts— Langfuse spans were all namedI.stepfor the same reason; nowI.click,I.fillField, …Console/network errors —
addObservation()notes were becomingcategory: 'user',status: 'failed'steps in the report, so a page logging errors in the background made a passing test read as broken. They are now filtered out ofcombineStepsAndNotesand out ofextractLastNoteMessage(an observation landing last could become the test's headline message), and travel in the testlogsinstead. Pilot, Analyst and Historian still read them throughnotesToString()/getLog()unchanged.Notes for review
Two judgement calls worth a look:
logs/stack rather than dropped, on the grounds that they are real evidence. Easy to drop entirely if a clean stack is preferred.save*is now filtered inattachStepLoggertoo (it wasgrab-only there;explorer.tsalready filtered both). Harmless — the tester prompt forbidssaveScreenshot— but it does change executed-step recording.Left out deliberately
src/action.ts:589has the identical dead-property bug:step.nameis undefined, so no assertion has ever been captured, meaningoutput.framework: 'playwright'has been generating tests with zeroexpect()lines since the CodeceptJS 4 migration. The fix isstep.name→step.titleon both lines, but it changes generated-test output, so it is not in this PR. Happy to add it here or split it out.Also unchanged:
getPrintableNotes()still renders these to Pilot asFAILED Console error: ….Verification
bun test tests/unit/— 1439 pass, 0 fail. Format and lint clean. Three new cases:isInternalStepunit coverage, an explorer test asserting grab/save steps are not recorded whileI.clickis, and a reporter test asserting observations stay out of report steps.🤖 Generated with Claude Code