Skip to content

fix(screenshot-diff): compare saved screenshots on any platform - #658

Open
filip131311 wants to merge 2 commits into
mainfrom
filip/screenshot-diff-platforms
Open

fix(screenshot-diff): compare saved screenshots on any platform#658
filip131311 wants to merge 2 commits into
mainfrom
filip/screenshot-diff-platforms

Conversation

@filip131311

Copy link
Copy Markdown
Collaborator

Fixes #635.

Before / after

$ argent run screenshot-diff --udid chromium-cdp-9222 --baselinePath A.png --currentPath B.png
Tool 'screenshot-diff' is not supported on chromium app (no chromium support declared).
$ argent run screenshot-diff --udid chromium-cdp-9222 --baselinePath A.png --currentPath B.png
{ "summary": "...status: unchanged...", "diffPath": "...", "contextDiffPath": "..." }

Comparing two saved PNGs touches no device, yet the tool refused by platform — so visual regression
testing was unavailable for Electron entirely. Two platforms the issue doesn't mention were refused
the same way: remote iOS (remote:*) and Vega.

Why the capability is now shared rather than widened

The list was a second copy of the one screenshot keeps, and they had drifted: a screenshot can be
captured from five platforms, a diff was allowed on two. It is the same question — this tool's only
device interaction is a screenshot — so they are now one object. That is what stops it drifting
again, and it is why there's no hand-written platform allow-list anywhere in this diff.

The narrower thing is live capture, so that is where the limit sits now: requesting one on a
platform whose screenshots come from a backend this tool doesn't drive says so and points at the
two-file form. Compare the old error, which is what the issue objected to:

before: Tool 'SimulatorServer' is not supported on ios simulator (this is an Apple TV (tvOS)
        simulator — touch, paste and rotate input are not available...)      ← never says "screenshot"
after:  Cannot capture a screenshot for a diff on chromium. Capture it separately with the
        `screenshot` tool, then pass both files as baselinePath and currentPath — comparing saved
        images works on every platform.

Those platforms also stop requesting the simulator-server, which would otherwise fail while
resolving the service
— before any of this tool's code runs — which is exactly how the caller ended
up seeing a blueprint error instead of one about screenshots.

What I deliberately did NOT do

An earlier draft extracted screenshot's capture into a shared helper so live capture would work on
Chromium/tvOS/Vega too, and made services() return {} unconditionally. I dropped it:

  • It would have silently removed the simulator-server self-heal. _recoverFailedServices iterates
    the refs from services(), so returning {} makes recovery unreachable — and
    simulator-server.ts:293-295 documents that the wedged-server case emits no exit, so that retry
    is the only recovery. A user rebooting a simulator would have gone from a transparent retry to a
    manual stop-simulator-server. That is a regression on this tool's primary documented flow, traded
    for a file↔file fix.
  • rotation would have been silently dropped on tvOS and Vega (neither capture function takes it)
    and on Chromium without the optional sharp. For screenshot that means a wrongly-oriented image;
    for a diff it means a wrong verdict against a rotated baseline.

Live capture on those platforms is worth doing — as its own PR, with rotation, abort-signal and
topMask handled deliberately. This PR keeps screenshot untouched.

Also noted, not fixed here

topMask defaults to "status-bar" and blanks the top 6%. On an Electron window or a TV screen there
is no status bar, so a real change in that band is ignored. It is not exposed in the schema. Now that
those platforms can diff, it's worth a follow-up — but changing it would alter the already-working
path, so I left it.

Verification

Live, against the built CLI:

before after
chromium, two saved PNGs refused real diff
chromium + captureCurrent refused as unsupported tool error naming screenshots + the workaround
tvOS, two saved PNGs worked still works
vega, two saved PNGs refused (unreported) real diff

tool-server 3096 passed. The existing screenshot-diff-tool.test.ts passes unmodified, including
its guard that the simulator-server is declared only for live captures — that guarantee is preserved
rather than deleted. New tests assert at the capability gate (assertSupported), because that is
where the refusal happened; a test driving execute cannot see this bug at all.

🤖 Generated with Claude Code

Comparing two saved PNGs touches no device, but the tool refused outright on any
platform missing from its own capability list — so visual regression testing was
unavailable for Electron apps entirely, and for remote iOS and Vega too, even
though the comparison never goes near a device.

The list was a second copy of the one the `screenshot` tool keeps, and the two
had drifted: a screenshot can be captured from five platforms, a diff was allowed
on two. They are the same question — this tool's only device interaction is a
screenshot — so they are now one shared object, which is what stops them drifting
again.

Live capture is the part that really is narrower, so that is where the limit now
sits. Asking for one where the screenshot comes from a backend this tool does not
drive says so, and points at the two-file form, instead of reporting that the
whole tool is unsupported here.

Those platforms also no longer request the simulator-server, which would have
failed while resolving the service — before any of this tool's own code runs, so
the caller would have seen the blueprint's error rather than one about
screenshots. Where the simulator-server IS the backend it is still declared, so a
wedged server can still be disposed and retried.

Fixes #635
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

screenshot-diff refuses Chromium and tvOS targets even when both inputs are saved PNGs and no device capture is needed

1 participant