feat(describe-ui): Add --include-web-content for WKWebView page content

[underscoretang]

For Humans

Fixes #65 — opened as a draft to propose the approach.

describe-ui can't see inside a WKWebView or SFSafariViewController, because the page renders in a separate WebContent process the in-process accessibility walk never reaches. With the new opt-in --include-web-content flag, it hit-tests a grid of screen points across that process boundary and returns the page's headings, links, buttons, and fields as ordinary elements with real tappable frames 🔍. It reuses remote-content options FBControlCore's serializer already supports at the pinned idb rev — no dependency changes, and the default output stays byte-identical when the flag is off.

Visuals

Before — default walk on the playground's WKWebView fixture screen (page content invisible):

$ axe describe-ui --udid <udid> | grep -c AXE_WEB
0

After — same screen with the flag; every page element is discovered, typed, framed, and tagged:

$ axe describe-ui --include-web-content --udid <udid>
...
{ "AXLabel": "AXE_WEB_HEADING", "type": "Heading",   "frame": {"x": 28, "y": 208, "width": 235, "height": 30}, "is_remote": "point_grid" }
{ "AXLabel": "AXE_WEB_LINK",    "type": "Link",      "frame": {"x": 28, "y": 307, "width": 123, "height": 23}, "is_remote": "point_grid" }
{ "AXLabel": "AXE_WEB_BUTTON",  "type": "Button",    "frame": {"x": 28, "y": 353, "width": 172, "height": 41}, "is_remote": "point_grid" }
{ "AXLabel": "AXE_WEB_FIELD",   "type": "TextField", "frame": {"x": 28, "y": 417, "width": 226, "height": 41}, "is_remote": "point_grid" }

(Real output from a booted iPhone 17 Pro simulator, iOS 26.5; trimmed to the discovered elements.)

Conversation Highlights

Agent Question	Steven Answer
(unprompted direction on the CLI declarations)	"i feel like these options are too verbose"

That call shaped the final surface: the option help strings are one-liners matching the file's idiom, and the measurement evidence behind the two non-obvious defaults lives in this description instead of inline comments.

Stack

Not stacked.

Created with: claude-fable-5

For Agents

Reviewer question

Should describe-ui gain an opt-in point-grid discovery path for out-of-process WebKit content, shaped as one enabling flag plus two tuning options (--web-content-grid-step, --web-content-max-points)?

Diff size: 73 changed production lines (Sources/); 251 total (+245/−6) including the playground fixture and tests.

Load-bearing code

1. DescribeUI.swift:22-64 — the three CLI declarations and scoped validation (guards apply only when the flag is on; tuning options document their inertness otherwise).
2. AccessibilityFetcher.swift:262-272 — the serializer call: is_remote is unioned into the request keys only in remote mode, which is what keeps the default JSON schema unchanged; the collectFrameCoverage comment marks the decision that looks like a mistake.
3. DescribeUITests.swift:167-224 — both E2E legs share a readiness poll (WKWebView paints asynchronously) and assert label, type, is_remote, and a positive frame.
4. WebContentTestView.swift — deterministic fixture: static HTML, no network, scrolling disabled, native sibling label so tests can tell "screen is up" from "web content discovered".

Design decisions (the two that look like mistakes until explained)

1. collectFrameCoverage stays off. Turning it on looks like a free optimization — skip hit-testing points the native walk already covered. In practice the serializer fills the coverage grid from every non-Application frame, and real screens are full of full-screen containers (Safari alone contributes 28), so the grid saturates to 100% during the walk and the skip check suppresses every probe: zero web nodes discovered, silently, exit 0. With coverage off, every grid point is probed; duplicate hits are collapsed by PID and frame de-duplication, and --web-content-max-points bounds the cost.
2. Grid step defaults to 25 pt, not the library's 50 pt. Measured against a page of ordinary web controls, a 50 pt grid leaves ~17 pt bands between probe rows and silently missed a standard 20 pt-tall link while still returning a complete-looking tree. 25 pt found every element on the same page; anything smaller only added duplicate hits at quadratically higher cost.

Other behavior worth knowing

• --include-web-content + --point is rejected with a clear message: a point lookup already resolves web content across the process boundary.
• Discovered elements are appended to the root Application element's children — the process boundary means there is no ancestor chain to nest them under the hosting web view. Consumers matching by label/type/frame are unaffected.
• The cost is why it's opt-in, and it's predictable: each probe point costs ~1.44 ms, so added_ms ≈ 1.44 × ceil(width/step) × ceil(height/step). On a 402x874 pt screen the 25 pt default is 595 probes, measured at +530 ms to +860 ms per call depending on the screen (2.3x-4.8x baseline). Halving the step quadruples that. --web-content-max-points caps the worst case: on a screen with no web content it cuts the overhead from +530 ms to +52 ms with no loss in what's found.
• Only content currently on screen is discovered. Hit-testing samples screen points, so one call sees exactly the current viewport; scroll and call again to reach the rest. Measured on a 40-item page: ~6 items per call at any grid step, and the set tracks the viewport as it scrolls.
• With the flag on, natively walked elements also carry is_remote: "recursive"; only point_grid marks grid-discovered content. Whole-screen probing can also return other out-of-process UI, such as the status bar. FBAccessibilityRemoteContentOptions.region is the unused knob that would bound both.
• The fixture exercises WKWebView; SFSafariViewController shares the same out-of-process WebKit mechanism, which is why the help names both.
• Help goldens are deliberately untouched. Tests/Goldens/** is capture-time release evidence whose provenance.json pins a stable_contract_sha256; hand-editing the help stdout would break that contract, and scripts/regenerate-goldens.sh recaptures the new help output at the next release regeneration.

Mechanical / safe to skim

ContentView screen routing (+4), the launch-helper wait key (+2, keyed on a native element since web content is invisible to a plain describe-ui), and the 3-line is_remote addition to the test JSON decoder.

Verification

• swift build clean; full non-E2E suite 221/221.
• All 14 DescribeUI tests pass, including both E2E legs on a booted iPhone 17 Pro (iOS 26.5) simulator. The E2E tests are gated behind AXE_E2E=1 like the existing simulator tests.
• Manual verification output shown in Visuals above.

Risk & rollback

Flag-off behavior is byte-identical (asserted end-to-end by describeUIOmitsWebContentByDefault), no dependency changes, single commit — revert is one clean git revert.

[cameroncooke]

@underscoretang This is an interesting approach, I would like to understand more from a performance point of view and also how this handles scrolling content and content that is currently offscreen.

[underscoretang]

[AI: ] 🤖 Both good questions, and the second one turned out to be the more interesting of the two. I went and measured rather than guess. Everything below is a release build against an iPhone 17 Pro simulator (iOS 26.5, 402x874 pt), 12 runs per configuration after warm-ups, median wall clock for the whole describe-ui call. I've pushed a small follow-up commit documenting what this turned up.

Performance

The cost model is simpler than I expected. Each probe point costs about 1.44 ms, and the number of probes is just the screen area divided by the grid step squared, so:

added_ms ≈ 1.44 × ceil(width / step) × ceil(height / step)

On this screen that's 162 probes at 50 pt, 595 at 25 pt and 2310 at 12.5 pt, and the measured per-probe cost barely moves across them (1.41 / 1.44 / 1.53 ms). Halving the step quadruples the bill. In absolute terms at the 25 pt default:

Screen	Flag off	Flag on	Added
Native screen, no web view	401 ms	931 ms	+530 ms
WKWebView fixture	216 ms	1032 ms	+816 ms
Safari, dense page	355 ms	1212 ms	+857 ms

With the flag off nothing changes, since it's the same path and the same output.

While I was in here I corrected my own PR description: it claimed "+660 ms", which was one measurement generalised further than it deserved. The honest number is a range, and it's in the description now.

If that's too expensive to be comfortable, --web-content-max-points is the lever that actually works. On a screen with no web content it takes the overhead from +530 ms down to +52 ms and still finds the same elements, because the cap stops the walk long before the grid is exhausted.

On the default itself, the numbers back 25 pt but not strongly on either side. Counting distinct page elements found on a dense page: 50 pt finds 17, 25 pt finds 19, 12.5 pt finds 19. So 50 pt genuinely drops two real links while still returning a tree that looks complete, which is the failure mode I wanted to avoid, and 12.5 pt costs four times as much for nothing at all. 25 pt is where the yield flattens out.

Scrolling and offscreen content

Only what's on screen is discovered, and scrolling needs no special handling.

I put 40 evenly spaced marks on a page, about six to a viewport, and alternated describe-ui with a swipe:

flag off        -> nothing
pass 1 scroll 0 ->  6 marks: 01..06
pass 2 scroll 1 ->  7 marks: 07..13
pass 3 scroll 2 ->  6 marks: 15..20
pass 4 scroll 3 ->  7 marks: 22..28
pass 5 scroll 4 ->  7 marks: 29..35

So a single call sees exactly the current viewport and never reaches below the fold, no matter how fine the grid. Scroll and call again and you get the new viewport, with frames in real screen coordinates, so anything discovered stays tappable by coordinate. For a caller this behaves like the native tree already does with virtualised lists: you see what's rendered.

Worth flagging that the two gaps above (14 and 21) are inertial scroll overshoot between passes rather than the grid missing anything. I checked by holding one scroll position and sweeping the step: 50, 25, 12.5 and 6.25 pt all return the identical six marks.

Two things the measurements turned up

Neither is fatal, but both were undocumented and both are now in the follow-up commit.

With the flag on, natively walked elements also come back carrying is_remote: "recursive". The count of native elements doesn't change, so nothing is duplicated, but the tree does gain a field it didn't have before. Only point_grid marks content the grid actually found. Flag off is still byte-identical.

The grid covers the whole screen, so on a screen with no web view at all it still returns three things: a status bar group, the clock, and "Dynamic Island, Empty". They're genuinely out-of-process elements, so the mechanism is behaving correctly, but "web content" is a narrower promise than what the flag delivers.

That second one has an obvious fix I deliberately haven't written: FBAccessibilityRemoteContentOptions already carries a region you can bound the probes with, and it's unused here. Bounding to the app window would drop the system chrome and cut the probe count with the area, roughly in half on my fixture screen. I left it out because it's a design call about how much surface this flag should have, and you're better placed to make it than I am. Happy to add it, or to fold the two tuning options into constants and ship just the one flag, if you'd rather this landed smaller.

The benchmark harness is three short scripts and a couple of generated pages, no network. Glad to include them in the PR if you'd want the numbers reproducible in-tree, or to leave them out to keep the diff tight.