feat: multi-browser web watcher via Accessibility Service#151
feat: multi-browser web watcher via Accessibility Service#151FractalMachinist wants to merge 7 commits into
Conversation
Picks up PR #547 (merged Dec 2025) which fixes a panic in jstring_to_string when JNI passes modified UTF-8 strings (e.g. app names containing emoji). The old CStr::to_str().unwrap() path would abort on surrogate pairs; the fix uses jstr.into() which handles the JNI encoding correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Firefox ~v130+ uses a Jetpack Compose toolbar where the URL lives in the content-desc of a node with viewIdResourceName "ADDRESSBAR_URL_BOX". Two issues prevented the original approach from working: 1. findAccessibilityNodeInfosByViewId silently returns empty for IDs that don't contain ":" (requires "package:id/name" format); bare Compose testTag names like ADDRESSBAR_URL_BOX are rejected. 2. The toolbar is a sibling of the WebView content area, so searching from event.source never reaches it. Fix: use rootInActiveWindow as search root and traverse the tree manually with findNodeByResourceName (compares viewIdResourceName directly). The view-based fallback IDs (url_bar_title, mozac_browser_toolbar_url_view) are retained for older Firefox versions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If findWebView(child) returns child itself (i.e. the child IS the
matching WebView), the previous .also { child.recycle() } destroyed the
node before the caller could use it. Only recycle when the returned node
is not the child being searched.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rowser When a supported browser's URL extractor returns null, dump the full accessibility tree to logcat at debug level (tag: WebWatcher), rate-limited to once per minute per browser package. This gives bug reporters actionable data — the actual view IDs and content descriptions present — without requiring manual instrumentation. Addresses the unresolved Samsung Internet and Chrome reports from the original PR, where no logs were available to diagnose the failure. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Greptile SummaryThis PR replaces the Chrome-only
Confidence Score: 3/5The core of the PR — WebWatcher's accessibility event handling — has two correctness issues in newly added code that will affect users on Android 7–12 devices. The mobile/src/main/java/net/activitywatch/android/watcher/WebWatcher.kt — specifically Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
AE[onAccessibilityEvent] --> SI{systemui?}
SI -->|yes| SKIP[return]
SI -->|no| PKG[resolve packageName]
PKG --> EX{urlExtractor exists?}
EX -->|no| WC{windowChanged?}
WC -->|yes| HU0[handleUrl null, null]
WC -->|no| SKIP
EX -->|yes| SRC[event.source != null?]
SRC -->|no| SKIP
SRC -->|yes| UE[urlExtractor event]
UE --> UNL{url == null?}
UNL -->|yes| DT[maybeDumpTree rate-limited 1/min]
UNL -->|no| HU[handleUrl newUrl, browser]
HU --> FWV[findWebView source]
FWV --> WT[handleWindowTitle]
WT --> LOG[logBrowserEvent heartbeatHelper]
DT --> RI[rootInActiveWindow dumpNode tree to logcat]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
AE[onAccessibilityEvent] --> SI{systemui?}
SI -->|yes| SKIP[return]
SI -->|no| PKG[resolve packageName]
PKG --> EX{urlExtractor exists?}
EX -->|no| WC{windowChanged?}
WC -->|yes| HU0[handleUrl null, null]
WC -->|no| SKIP
EX -->|yes| SRC[event.source != null?]
SRC -->|no| SKIP
SRC -->|yes| UE[urlExtractor event]
UE --> UNL{url == null?}
UNL -->|yes| DT[maybeDumpTree rate-limited 1/min]
UNL -->|no| HU[handleUrl newUrl, browser]
HU --> FWV[findWebView source]
FWV --> WT[handleWindowTitle]
WT --> LOG[logBrowserEvent heartbeatHelper]
DT --> RI[rootInActiveWindow dumpNode tree to logcat]
Reviews (2): Last reviewed commit: "fix: address greptile's identified bugs ..." | Re-trigger Greptile |
|
I think you mentioned wrong account :) |
- Fix memory leak in findNodeByResourceName: recycle child before returning found node - Fix NPE in exception handling: use ex.message ?: ex.toString() instead of ex.message!! - Fix Intent flag combination: use 'or' instead of '+' for proper bitwise operation - Fix queue sync bug: consume NAVIGATION_FINISHED event with poll() instead of peek() - Fix unclosed file descriptor: close ParcelFileDescriptor from executeShellCommand() Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
@ErikBjare I believe this is ready for review. I don't see how to request your review directly. |
|
@greptileai review @FractalMachinist I'll do my best to not forget about this, do ping me again if I do! Thanks a lot for contributing! |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| val found = findNodeByResourceName(child, name) | ||
| if (found != null) { | ||
| child.recycle() | ||
| return found | ||
| } |
There was a problem hiding this comment.
findNodeByResourceName has the same recycle-before-return bug that was explicitly fixed in findWebView. When child.viewIdResourceName == name, the recursive call returns child immediately. Back in the caller, found === child, and the code calls child.recycle() before returning found — handing the caller a recycled (invalid) AccessibilityNodeInfo. On API 24–32 devices this corrupts the reference used to read the Compose URL bar content-description, silently producing garbage or a crash.
| val found = findNodeByResourceName(child, name) | |
| if (found != null) { | |
| child.recycle() | |
| return found | |
| } | |
| val found = findNodeByResourceName(child, name) | |
| if (found != null) { | |
| if (found !== child) child.recycle() | |
| return found | |
| } |
| private fun extractFirefoxUrl(event: AccessibilityEvent): String? { | ||
| val root = rootInActiveWindow ?: return null | ||
| return findNodeByResourceName(root, "ADDRESSBAR_URL_BOX") | ||
| ?.contentDescription?.toString() | ||
| ?.let { FIREFOX_SUFFIX_PATTERN.find(it)?.groupValues?.get(1) } | ||
| ?.takeIf { it.isNotBlank() && !it.equals("Search or enter address", ignoreCase = true) } | ||
| } |
There was a problem hiding this comment.
rootInActiveWindow returns a new AccessibilityNodeInfo whose lifecycle the caller owns. On API 24–32 (the app's minSdk is 24; recycle() was only deprecated in API 33), failing to recycle root leaks one node reference per Firefox accessibility event. The node returned by findNodeByResourceName is a child inside this tree and must also be recycled separately once its content has been read.
| private fun extractFirefoxUrl(event: AccessibilityEvent): String? { | |
| val root = rootInActiveWindow ?: return null | |
| return findNodeByResourceName(root, "ADDRESSBAR_URL_BOX") | |
| ?.contentDescription?.toString() | |
| ?.let { FIREFOX_SUFFIX_PATTERN.find(it)?.groupValues?.get(1) } | |
| ?.takeIf { it.isNotBlank() && !it.equals("Search or enter address", ignoreCase = true) } | |
| } | |
| private fun extractFirefoxUrl(event: AccessibilityEvent): String? { | |
| val root = rootInActiveWindow ?: return null | |
| return try { | |
| val matched = findNodeByResourceName(root, "ADDRESSBAR_URL_BOX") | |
| val result = matched?.contentDescription?.toString() | |
| ?.let { FIREFOX_SUFFIX_PATTERN.find(it)?.groupValues?.get(1) } | |
| ?.takeIf { it.isNotBlank() && !it.equals("Search or enter address", ignoreCase = true) } | |
| if (matched !== root) matched?.recycle() | |
| result | |
| } finally { | |
| root.recycle() | |
| } | |
| } |
5 participants
adding a generic WebWatcher that tracks browser URLs via Android's Accessibility Service. Verified Firefox working on Android 14 (Fairphone 6).
What this adds
ChromeWatcherwith aWebWatchersupporting Chrome, Firefox, Samsung Internet, Opera, and Edgeurl,title,browser,audible, andincognitofields per theweb.tab.currentevent typeFirefox Compose toolbar fix
Firefox ~v130+ migrated its address bar to Jetpack Compose. The original PR's approach (
findAccessibilityNodeInfosByViewId("ADDRESSBAR_URL_BOX")) silently returns empty results because Android requires IDs in"package:id/name"format and rejects bare Compose testTag names. Additionally, the toolbar is a sibling of the WebView content area, so searching fromevent.sourcenever reaches it.Fix: search from
rootInActiveWindowand traverse the tree manually, comparingviewIdResourceNamedirectly. The view-based fallback IDs (url_bar_title,mozac_browser_toolbar_url_view) are retained for older Firefox versions.Fixes from code review
findWebViewwherechild.recycle()was called even whenchildwas the node being returned, leaving the caller with a recycled reference (flagged by ellipsis-dev in the original PR).Testing
Verified on one physical device (Fairphone 6, Android 14) with Firefox 138. Chrome, Edge, Samsung Internet, and Opera have not been tested by the authors of this revision — the view IDs for those browsers are carried over from the original PR unchanged. The diagnostic tree logging (described below) was added specifically because we lack broad device/browser coverage. If you or a reviewer can test on Chrome or Samsung Internet and URL extraction fails, enabling debug logging (
adb logcat -s WebWatcher:D) will produce a dump of the accessibility tree that should make it straightforward to identify the correct view IDs. We believe the Firefox fix is correct because it was verified end-to-end (URLs and page titles appearing in the ActivityWatch timeline). We believe the recycle fix is correct by code inspection. For the untested browsers, confidence rests entirely on the original PR author's view IDs being stable across the versions you'll encounter.Diagnostic logging for unresolved browser issues
The original PR had unresolved reports of Chrome stopping mid-session and Samsung Internet never producing data. The Chrome issue was almost certainly the JNI emoji crash described below — the service would crash silently and stop processing events. For future reports,
WebWatchernow dumps the full accessibility tree to logcat (tag:WebWatcher, debug level) when a known browser's URL extractor returns null, rate-limited to once per minute per browser. This gives reporters actionable data without manual instrumentation.aw-server-rust submodule bump
This PR bumps
aw-server-rustto current master to pick up PR #547, which fixes a panic injstring_to_stringwhen JNI passes modified UTF-8 strings (e.g. app names containing emoji). Without this fix the service crashes silently on any heartbeat where the browser has a non-BMP character in its app name, which likely explains the "Chrome stopped working" report in the original PR thread.Happy to separate the submodule bump into a standalone PR if you prefer to take it independently.
Credits
Original implementation by @KonradKrol (incubly-oss/aw-android#138).