Skip to content

fix(describe): flag an Android read that saw only system chrome - #672

Open
filip131311 wants to merge 1 commit into
mainfrom
filip/android-describe-blind-read
Open

fix(describe): flag an Android read that saw only system chrome#672
filip131311 wants to merge 1 commit into
mainfrom
filip/android-describe-blind-read

Conversation

@filip131311

Copy link
Copy Markdown
Collaborator

Fixes #627.

The false pass, measured

With the display off, describe returns a bare ROOT Screen (0,0,1,1) — no error, no hint. The same element, never moved:

screen call result
on await-ui-element visible "Go to home screen" {"success": true, "elapsed": 5}
off await-ui-element hidden "Go to home screen" {"success": true, "elapsed": 5}

An agent gating an action on "the dialog is gone" gets a green light against a screen nobody can read. await-ui-element's isBlindRead has no flag to key on here — its only Android guard is everMatched, which is false when the wait starts after the screen is already dark. The code comment said as much: "Android / Chromium never set these flags."

The signal was already on the wire

getHierarchy() returns nodeCount alongside xml, and describe destructured only xml. nodeCount > 0 with zero rendered children means every node on screen belonged to com.android.systemui and pruned away — which is exactly what a powered-off display and a lock screen look like.

Measured across four states on a Pixel_9 / API 36:

state nodeCount rendered children blind?
app visible 62 2 no
home screen (launcher) 64 7 no
screen off 28 0 yes
keyguard, display on 73 0 yes

So: no adb probe, no new module, no extra round trip, no dumpsys parsing to go stale across OEMs, and nothing added to the hot path.

windowCount is deliberately not the signal. The obvious guess is "no windows ⇒ blind", but it stays 1 with the display off, because the keyguard window is still there — it would never fire. Worth measuring rather than assuming.

I verified the keyguard case separately (set a PIN and disabled the screen timeout so the device couldn't sleep mid-test, since an earlier attempt was unattributable): mWakefulness=Awake, SCREEN_STATE_ON, mIsShowing=true, lock screen visibly on the display — and the tree still comes back empty. A display-power-only check would have missed it entirely.

Fails open

Unknown or zero nodeCount ⇒ no hint. An older device helper that reports no count must not turn every sparse screen into a blind read — that is worse than the bug. This is not hypothetical: the existing hidden succeeds when the node is gone test uses a stub with no nodeCount, and my first draft flagged it, because undefined <= 0 is false. It now passes unmodified.

hint, not a throw

hint already flows into both isBlindRead (so hidden can no longer resolve) and the timeout note — so the false pass closes with zero consumer changes, and the six callers of describeAndroid keep working. On a sleeping device an empty tree plus "wake it" is more useful than a hard error.

Live, after the fix:

describe (screen off) → hint: "This read is BLIND, not empty: every node on screen belonged to the
system UI… An element missing from this tree is NOT evidence that it is hidden or gone… Wake the
device (`button` with button `power`) and dismiss the lock screen, then read again."

await-ui-element hidden → success: false
   note: "could not confirm the element is hidden — the UI tree was empty or unreadable at timeout
          (This read is BLIND, not empty: …)"

Screen on: no hint, full tree, unchanged.

Behaviour change to know about

A hidden wait that previously returned instantly now runs to its timeout and fails, and an unmet wait halts a run-sequence. That is the intent, but it will change any recorded sequence that (wrongly) depended on the instant pass.

Interaction with #591

#591 (draft) touches the same two files. No textual collision: its android/index.ts hunk is @@ -52,7 +52,34 @@ and mine is the destructure at :54 plus a new helper above the TV hint — the one adjacent line resolves trivially either merge order. In await-ui-element/index.ts its hunks are at 187-195 / 217-223 / 354-360; my only edit is the now-false comment at :151-165, which had to be corrected since it documented precisely the gap this closes. The new await-ui-element test is a separate file because #591 also extends test/await-ui-element.test.ts.

#519 is the iOS half of this class; it throws inside flow-ios-tree.ts, whose contract is "helpers throw". The true Android mirror is flows/flow-android-tree.ts, which does not route through describeAndroid — so flow assert hidden still false-passes there. Left as a follow-up so it can be reviewed next to #519.

Checks

  • 3094 tests pass; 8 new, of which 5 fail against the pre-fix source (the 3 that pass on both are the fail-open invariance checks).
  • All pre-existing await-ui-element / describe-tv / await-screen-idle tests pass unmodified.
  • extract-tools 46/46; prettier, eslint, both typechecks clean; lock untouched.

With the display off, or a lock screen up, describe returns a bare
ROOT Screen (0,0,1,1) — every node on screen belongs to com.android.systemui
and prunes away. Nothing marks it, so an empty screen and an unreadable one are
indistinguishable, and await-ui-element's isBlindRead has no flag to key on: its
only Android guard is everMatched, which is false if the wait starts after the
screen is already dark.

Measured on a device: the same element, unmoved, reported visible with the
screen on and hidden with it off, both succeeding in 5ms. An agent gating an
action on 'the dialog is gone' got a green light against a screen nobody
could read.

The signal is two counts the helper already returns and describe threw away, so
this costs no extra round trip and no probe: nodeCount is what the
accessibility layer handed over, and the rendered child count is what survived
pruning. Everything pruned means the only window was system chrome. Measured
across four states — app 62 nodes/2 children, launcher 64/7, screen off 28/0,
keyguard 73/0 — the rule separates them exactly.

windowCount is deliberately NOT the signal: it stays 1 with the display off,
because the keyguard window is still there, so a check for zero windows would
never fire. That was worth measuring rather than assuming.

Unknown or zero nodeCount fails open. An older helper that reports no count
must not turn every sparse screen into a blind read — that would be worse than
the bug, and the existing 'hidden succeeds when the node is gone' test covers
exactly that shape.

hint rather than a throw: it already flows into isBlindRead, which stops hidden
resolving, and into the timeout note, so no consumer changes. Six callers of
describeAndroid keep working, and describe on a sleeping device stays more
useful as an empty tree plus 'wake it' than as an error.

Note this makes a previously instant hidden wait run to its timeout and fail,
which halts a run-sequence — that is the point, but it is a behaviour change
for any sequence that depended on the false pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

describe on Android returns an empty tree with no degraded-read hint when the screen is off

1 participant