Skip to content

fix(screenshot): say when scale or rotation was not applied - #673

Open
filip131311 wants to merge 1 commit into
mainfrom
filip/screenshot-dropped-geometry
Open

fix(screenshot): say when scale or rotation was not applied#673
filip131311 wants to merge 1 commit into
mainfrom
filip/screenshot-dropped-geometry

Conversation

@filip131311

Copy link
Copy Markdown
Collaborator

Fixes #626.

Reproduced live (Electron testbed)

screenshot { scale: 0.25 }               → 1600×1136   (unscaled)
screenshot { rotation: "LandscapeLeft" } → 1600×1136   (unrotated)

Both returned nothing but the saved path.

sharp really is absent from the published package (no node_modules/sharp, not in package.json), so this is every user's default. And the existing stderr line is guarded by a once-per-process flag — after the first Chromium screenshot there was no signal anywhere, not even on stderr.

The suggested one-liner wouldn't have worked

The issue proposes attaching a note to the tool result. I checked argent-mcp/src/content.ts:75-83: for an image-output tool the renderer returns only image blocks plus a Saved: <path> line and discards every other result field. A note on the result would have been silently dropped by the very client the issue is about.

The delivery that does work is still tool-server-only. http.ts already builds a response-envelope note (update reminders, active recordings) and has the tool's data in scope where it does it. So the tool returns a reserved key, http.ts pops it off and pushes it into that note — and mcp-server.ts:369, argent-cli/run.ts:310 and argent-tools-client all render it today. No client change, and an older client works against a newer server immediately.

The gate: explicitly asked and actually dropped

Both halves matter, and each alone is a bug:

  • Gating on the parameter alone would fire on every visual-snapshot step — flow-visual.ts passes scale: 1.0, which is a no-op, not a loss. rotation: "Portrait" is likewise the identity.
  • Gating on the effective value would report the ARGENT_SCREENSHOT_SCALE default the caller never asked for.

Verified live, all five cases:

call note
no geometry param (auto-screenshot shape)
scale: 1.0 (visual-snapshot shape)
rotation: "Portrait"
rotation: "LandscapeLeft"
scale: 0.3 on Android (applied server-side)

And data stays { image } — the reserved key is stripped before it reaches the client, so --json output and non-image results are unchanged.

note: scale was not applied — this is the unmodified capture. Chromium image post-processing
needs the optional `sharp` package: run `npm install sharp` in the tool-server's environment
and retry, or work with the full-size image.

Scope

The issue names Chromium and tvOS. Vega has the identical gapcaptureVegaScreenshotPng({ scale }) never receives rotation, same as tvScreenshot(udid, scale, signal) — so a Chromium-only fix would leave two silent droppers behind. Both TV notes are worded so they don't read as retryable.

Two further drop sites covered while the plumbing was here:

  • Chromium also loses scale when the PNG header can't be read — and that path's comment claimed a "falls back to sharp metadata" that the caller doesn't do. Comment corrected.
  • rotation's schema description promised Chromium rotation unconditionally while scale already carried the honest caveat.

Deliberately not done

  • Implementing rotation for tvOS/Vega. Both are cheap (sips -r; Vega already decodes with pngjs), but rotation on a fixed-landscape TV is close to meaningless — an agent asking for it is confused, and saying so is more useful than a sideways image.
  • Making sharp a real dependency — ~30 MB of native binary for every consumer, most of whom never touch Chromium. optionalDependencies isn't a middle ground either (npm installs them by default). The real fix is to move the Chromium post-process onto the in-repo pngjs + resizeDecodedPng and delete the sharp branch — worth a follow-up, with the caveat that resizeDecodedPng is lanczos3-only, so the other three downscaler kernels would need writing or trimming.
  • A flow-nested screenshot step won't surface the note, since the envelope is request-scoped. Accepted: a flow step passing explicit geometry to a TV/Chromium target is a corner case, and a result-field path can be layered on later without undoing this.

Checks

  • 3094 tests pass; 9 new. All 11 pre-existing screenshot tests pass unmodified.
  • One unrelated flake on the first full run (http-recording-note.test.ts) that passed in isolation and on re-run — not from this change.
  • SpiderShield 9.10; extract-tools 46/46; prettier, eslint, both typechecks clean; lock untouched.

screenshot takes scale and rotation on every target, but several backends
cannot honour them: Chromium needs the optional sharp package, which is not
shipped, and the Apple TV and Vega captures have no rotation step at all. The
caller got a full-size, un-rotated PNG that looks like a successful transform.
The only existing signal was a stderr line the caller never sees — and it is
guarded by a once-per-process flag, so every screenshot after the first was
silent even there.

Live on Electron: scale 0.25 and rotation LandscapeLeft both returned the
untouched 1600x1136 capture with nothing attached.

The note rides the response envelope rather than the result body, because
clients render an image result as image blocks plus a Saved: line and discard
every other field — a note inside the result would never have been seen. The
tool returns a reserved key that http.ts pops off and pushes into the note it
already builds, which mcp-server, the CLI and the tools-client all render
today. That keeps this to the tool-server: no client change, and an older
client works with a newer server on day one.

A note is raised only when the caller explicitly asked for something AND the
backend dropped it. Both halves matter: a visual snapshot passes scale 1 on
every step and rotation Portrait is the identity, so gating on the parameter
alone would have attached a note to captures where nothing was lost, and gating
on the effective value would have reported the ARGENT_SCREENSHOT_SCALE default
the caller never asked for. Auto-screenshot passes no geometry at all and stays
silent.

Two further drop sites are covered while the plumbing is here: Chromium also
loses scale when the PNG header cannot be read, and that path's comment claimed
a sharp-metadata fallback the caller does not have.

Rotation's schema description promised Chromium rotation unconditionally while
scale already carried the honest caveat; both now match the behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

screenshot: scale and rotation are accepted and silently ignored on Chromium and tvOS (the warning never reaches the response)

1 participant