Skip to content

fix(gestures): reject off-screen finger positions instead of dispatching them - #671

Open
filip131311 wants to merge 1 commit into
mainfrom
filip/gesture-offscreen-fingers
Open

fix(gestures): reject off-screen finger positions instead of dispatching them#671
filip131311 wants to merge 1 commit into
mainfrom
filip/gesture-offscreen-fingers

Conversation

@filip131311

Copy link
Copy Markdown
Collaborator

Fixes #628.

Reproduced first try

gesture-pinch { centerX: 0.5, centerY: 0.35, startDistance: 0.75, endDistance: 0.2, angle: 90 }
→ { "pinched": true }        …and the notification shade is open

0.35 − 0.75/2 = −0.025. Android reads the off-screen touch as a status-bar pull, so the app never sees the pinch, and the tool reports success.

gesture-rotate has the identical gap (not in the issue) — verified live at y = −0.15, returning {rotated: true}. It didn't happen to open the shade on that path, which is what makes it worse to leave: the same invalid input is sometimes silently harmful.

Clamping — the issue's first suggestion — does not work

I tested the clamped outcome directly, by picking parameters that land a finger exactly where clamping would put it:

gesture-pinch { centerY: 0.375, startDistance: 0.75, angle: 90 }   ⇒ y1 = 0.0 exactly
→ { "pinched": true }        …and the notification shade opened again

A finger at exactly y = 0.0 — perfectly in bounds — still trips the system gesture. So clamping −0.025 → 0.0 converts an off-screen touch into an in-bounds touch with the same observable failure, while still reporting success. It would also change the effective separation (0.75 → 0.725) and shift the zoom anchor off centerY, so the caller silently gets a different gesture than it asked for.

The repo already knows [0,1] isn't safe: flow-pinch-geometry.ts:17 defines SCREEN_EDGE_INSET = 0.02 and :37-40 encodes system-gesture zones (top: 0.08, from a Pixel measurement).

So: reject, and additionally say in both tool descriptions that a finger merely near an edge can still be taken by a system gesture. I did not promote those inset constants into the tool layer — the flow layer that owns them deliberately treats them as "prefer-not-reject no-start zones" and demotes in ranking rather than rejecting, and hard-rejecting them here would refuse deliberate edge gestures.

Implementation

Both tools build every frame, validate, and only then dispatch — a rejected call sends zero touch events. Rejecting after the first Down would strand a synthetic finger with no matching Up, which gesture-rotate's abort path already guards against and gesture-pinch has no protection for.

The check runs over the frames that will actually be sent, not the endpoints. A pinch is affine in t so endpoints would bound it — but an arc is not: 0°→180° starts and ends on the centre line while passing a full radius away at 90°. There's a test for exactly that. Checking swept frames also keeps a legitimate short arc near an edge valid, which a "whole circle must fit" check would refuse (also tested).

Bounds are inclusive with a small epsilon: a full-width gesture legitimately produces exactly 0 or 1, and a real rotate frame already yields 0.09999999999999998.

Live output:

gesture-pinch: finger y = -0.025 at the start of the gesture is off-screen (must be 0–1).
centerX 0.5 / centerY 0.35 with startDistance 0.75 at angle 90 puts a finger 0.025 past the
top edge — the OS reads an off-screen touch as a system gesture (on Android, the notification
shade) instead of a pinch, so the app never sees it. Reduce the distance, move the center, or
set endCenterX/endCenterY to drift the centroid inward as the fingers spread.   [HTTP 400]

Uses InvalidToolInputError, which surfaces the bare message at HTTP 400. A zod .refine() would have been JSON-serialized into zod's issue array, burying the coordinate — and refinements are dropped from the advertised JSON Schema anyway, as gesture-rotate/index.ts:65-67 already notes.

Scope

Only the two tools that derive positions the caller can't see. gesture-tap/swipe/custom take coordinates directly — there the value is the caller's stated intent, nothing is computed and hidden.

Worth a follow-up, noted while checking: gesture-tap's Chromium path clamps to the viewport while its native path doesn't. And run-sequence catches a step throw and returns HTTP 200 with partial results, so a rejected gesture inside a sequence stops the remainder — the message is written to read standalone.

Verified live on a Pixel_9

call result
reported pinch HTTP 400, coordinate named, no shade, nothing dispatched
rotate at y = −0.15 HTTP 400, coordinate named
valid pinch (0.5/0.5, 0.2→0.6) {pinched: true}, gesture reached the app

Checks

  • 3095 tests pass; 9 new. All 8 pre-existing gesture tests pass unmodified, including rotate's abort test.
  • SpiderShield 9.08 avg (both tools 9.2, up from the description gaining explicit failure guidance); extract-tools 46/46; prettier, eslint, both typechecks clean; lock untouched.

…ing them

gesture-pinch computes two finger positions from a center, a distance and an
angle, then dispatches them with no bounds check. centerY 0.35 with
startDistance 0.75 at 90 degrees puts a finger at y = -0.025; Android reads the
off-screen touch as a status-bar pull, opens the notification shade, and the
tool returns { pinched: true }. gesture-rotate has the same shape and the same
gap — verified at y = -0.15.

Both now build every frame first, check it, and only then dispatch, so a
rejected call sends nothing. Rejecting after the first Down would leave a
synthetic finger held on the glass with no matching Up, which gesture-rotate's
abort path already guards against.

The check runs over the frames that will actually be sent rather than the
endpoints. A pinch is affine in time so its endpoints would bound it, but an arc
is not: 0 to 180 degrees starts and ends on the centre line while passing a full
radius away at 90. Checking swept frames also keeps a short arc near an edge
valid, which a whole-circle check would refuse.

Bounds are inclusive: a full-width gesture legitimately lands exactly on 0 or 1,
and the arithmetic that gets there drifts a few ulps, so the comparison carries
a small epsilon.

Clamping was the issue's first suggestion and does not work. Clamping -0.025 to
0 produces a finger at exactly y = 0 — which I dispatched, and it opened the
notification shade just the same. Clamping would also change the effective
separation and move the zoom anchor off centerY, so the caller would get a
different gesture than it asked for and still be told it succeeded. The tool
descriptions now say that a finger merely NEAR an edge can still be taken by a
system gesture, since enforcing the inset the flow layer uses would refuse
deliberate edge gestures — that layer treats those zones as prefer-not, never
reject.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gesture-pinch does not clamp finger positions to the screen, so an off-screen finger triggers system gestures

1 participant