fix(screenshot-diff): compare saved screenshots on any platform - #658
Open
filip131311 wants to merge 2 commits into
Open
fix(screenshot-diff): compare saved screenshots on any platform#658filip131311 wants to merge 2 commits into
filip131311 wants to merge 2 commits into
Conversation
Comparing two saved PNGs touches no device, but the tool refused outright on any platform missing from its own capability list — so visual regression testing was unavailable for Electron apps entirely, and for remote iOS and Vega too, even though the comparison never goes near a device. The list was a second copy of the one the `screenshot` tool keeps, and the two had drifted: a screenshot can be captured from five platforms, a diff was allowed on two. They are the same question — this tool's only device interaction is a screenshot — so they are now one shared object, which is what stops them drifting again. Live capture is the part that really is narrower, so that is where the limit now sits. Asking for one where the screenshot comes from a backend this tool does not drive says so, and points at the two-file form, instead of reporting that the whole tool is unsupported here. Those platforms also no longer request the simulator-server, which would have failed while resolving the service — before any of this tool's own code runs, so the caller would have seen the blueprint's error rather than one about screenshots. Where the simulator-server IS the backend it is still declared, so a wedged server can still be disposed and retried. Fixes #635
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #635.
Before / after
Comparing two saved PNGs touches no device, yet the tool refused by platform — so visual regression
testing was unavailable for Electron entirely. Two platforms the issue doesn't mention were refused
the same way: remote iOS (
remote:*) and Vega.Why the capability is now shared rather than widened
The list was a second copy of the one
screenshotkeeps, and they had drifted: a screenshot can becaptured from five platforms, a diff was allowed on two. It is the same question — this tool's only
device interaction is a screenshot — so they are now one object. That is what stops it drifting
again, and it is why there's no hand-written platform allow-list anywhere in this diff.
The narrower thing is live capture, so that is where the limit sits now: requesting one on a
platform whose screenshots come from a backend this tool doesn't drive says so and points at the
two-file form. Compare the old error, which is what the issue objected to:
Those platforms also stop requesting the simulator-server, which would otherwise fail while
resolving the service — before any of this tool's code runs — which is exactly how the caller ended
up seeing a blueprint error instead of one about screenshots.
What I deliberately did NOT do
An earlier draft extracted
screenshot's capture into a shared helper so live capture would work onChromium/tvOS/Vega too, and made
services()return{}unconditionally. I dropped it:_recoverFailedServicesiteratesthe refs from
services(), so returning{}makes recovery unreachable — andsimulator-server.ts:293-295documents that the wedged-server case emits noexit, so that retryis the only recovery. A user rebooting a simulator would have gone from a transparent retry to a
manual
stop-simulator-server. That is a regression on this tool's primary documented flow, tradedfor a file↔file fix.
rotationwould have been silently dropped on tvOS and Vega (neither capture function takes it)and on Chromium without the optional
sharp. Forscreenshotthat means a wrongly-oriented image;for a diff it means a wrong verdict against a rotated baseline.
Live capture on those platforms is worth doing — as its own PR, with rotation, abort-signal and
topMaskhandled deliberately. This PR keepsscreenshotuntouched.Also noted, not fixed here
topMaskdefaults to"status-bar"and blanks the top 6%. On an Electron window or a TV screen thereis no status bar, so a real change in that band is ignored. It is not exposed in the schema. Now that
those platforms can diff, it's worth a follow-up — but changing it would alter the already-working
path, so I left it.
Verification
Live, against the built CLI:
captureCurrenttool-server 3096 passed. The existing
screenshot-diff-tool.test.tspasses unmodified, includingits guard that the simulator-server is declared only for live captures — that guarantee is preserved
rather than deleted. New tests assert at the capability gate (
assertSupported), because that iswhere the refusal happened; a test driving
executecannot see this bug at all.🤖 Generated with Claude Code