fix(screenshot): say when scale or rotation was not applied - #673
Open
filip131311 wants to merge 1 commit into
Open
fix(screenshot): say when scale or rotation was not applied#673filip131311 wants to merge 1 commit into
filip131311 wants to merge 1 commit into
Conversation
screenshot takes scale and rotation on every target, but several backends cannot honour them: Chromium needs the optional sharp package, which is not shipped, and the Apple TV and Vega captures have no rotation step at all. The caller got a full-size, un-rotated PNG that looks like a successful transform. The only existing signal was a stderr line the caller never sees — and it is guarded by a once-per-process flag, so every screenshot after the first was silent even there. Live on Electron: scale 0.25 and rotation LandscapeLeft both returned the untouched 1600x1136 capture with nothing attached. The note rides the response envelope rather than the result body, because clients render an image result as image blocks plus a Saved: line and discard every other field — a note inside the result would never have been seen. The tool returns a reserved key that http.ts pops off and pushes into the note it already builds, which mcp-server, the CLI and the tools-client all render today. That keeps this to the tool-server: no client change, and an older client works with a newer server on day one. A note is raised only when the caller explicitly asked for something AND the backend dropped it. Both halves matter: a visual snapshot passes scale 1 on every step and rotation Portrait is the identity, so gating on the parameter alone would have attached a note to captures where nothing was lost, and gating on the effective value would have reported the ARGENT_SCREENSHOT_SCALE default the caller never asked for. Auto-screenshot passes no geometry at all and stays silent. Two further drop sites are covered while the plumbing is here: Chromium also loses scale when the PNG header cannot be read, and that path's comment claimed a sharp-metadata fallback the caller does not have. Rotation's schema description promised Chromium rotation unconditionally while scale already carried the honest caveat; both now match the behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #626.
Reproduced live (Electron testbed)
Both returned nothing but the saved path.
sharpreally is absent from the published package (nonode_modules/sharp, not inpackage.json), so this is every user's default. And the existing stderr line is guarded by a once-per-process flag — after the first Chromium screenshot there was no signal anywhere, not even on stderr.The suggested one-liner wouldn't have worked
The issue proposes attaching a
noteto the tool result. I checkedargent-mcp/src/content.ts:75-83: for an image-output tool the renderer returns only image blocks plus aSaved: <path>line and discards every other result field. Anoteon the result would have been silently dropped by the very client the issue is about.The delivery that does work is still tool-server-only.
http.tsalready builds a response-envelopenote(update reminders, active recordings) and has the tool'sdatain scope where it does it. So the tool returns a reserved key,http.tspops it off and pushes it into that note — andmcp-server.ts:369,argent-cli/run.ts:310andargent-tools-clientall render it today. No client change, and an older client works against a newer server immediately.The gate: explicitly asked and actually dropped
Both halves matter, and each alone is a bug:
flow-visual.tspassesscale: 1.0, which is a no-op, not a loss.rotation: "Portrait"is likewise the identity.ARGENT_SCREENSHOT_SCALEdefault the caller never asked for.Verified live, all five cases:
scale: 1.0(visual-snapshot shape)rotation: "Portrait"rotation: "LandscapeLeft"scale: 0.3on Android (applied server-side)And
datastays{ image }— the reserved key is stripped before it reaches the client, so--jsonoutput and non-image results are unchanged.Scope
The issue names Chromium and tvOS. Vega has the identical gap —
captureVegaScreenshotPng({ scale })never receives rotation, same astvScreenshot(udid, scale, signal)— so a Chromium-only fix would leave two silent droppers behind. Both TV notes are worded so they don't read as retryable.Two further drop sites covered while the plumbing was here:
scalewhen the PNG header can't be read — and that path's comment claimed a "falls back to sharp metadata" that the caller doesn't do. Comment corrected.rotation's schema description promised Chromium rotation unconditionally whilescalealready carried the honest caveat.Deliberately not done
sips -r; Vega already decodes withpngjs), but rotation on a fixed-landscape TV is close to meaningless — an agent asking for it is confused, and saying so is more useful than a sideways image.sharpa real dependency — ~30 MB of native binary for every consumer, most of whom never touch Chromium.optionalDependenciesisn't a middle ground either (npm installs them by default). The real fix is to move the Chromium post-process onto the in-repopngjs+resizeDecodedPngand delete the sharp branch — worth a follow-up, with the caveat thatresizeDecodedPngis lanczos3-only, so the other threedownscalerkernels would need writing or trimming.screenshotstep won't surface the note, since the envelope is request-scoped. Accepted: a flow step passing explicit geometry to a TV/Chromium target is a corner case, and a result-field path can be layered on later without undoing this.Checks
http-recording-note.test.ts) that passed in isolation and on re-run — not from this change.