10x: perception performance, secure defaults, and full module decomposition#25
Merged
Conversation
Add tree_cache.py: a thread-safe per-window_uid TTL+LRU cache of
accessibility-tree captures (TTL from tree.cache_ttl_s, default 2.0s;
LRU capped at 8 windows), attached to session.Session like tree_tokens.
ScreenObserver.get_element_tree() gains optional window_uid/use_cache
parameters (existing hwnd-only callers keep the uncached behavior); a
fresh cached capture is returned without an adapter walk, a miss walks
and stores. get_element_tree_with_meta() additionally reports
{cache: hit|miss|bypass, capture_ms, node_count} for later perf
telemetry. tools.dispatch() invalidates the target window's entry
after any input tool (bring_to_foreground included) — the single choke
point — and post-action re-reads (ActionReceipt after-state, wait_for/
wait_idle polling, select_option re-walk, trace after-hash) bypass the
cache so they always observe fresh state. load_scenario invalidates
the whole cache since it replaces the mock world.
MockAdapter gains test hooks: capture_count and tree_mutator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
New config key tree.default_depth (5): get_window_structure and observe_window now bound the RETURNED tree to that many levels when the caller passes no depth=; explicit depth= requests are honored up to the existing tree.max_depth hard cap. Nodes whose children were dropped are marked truncated:true with a child_count so agents know to drill in. Diff baselines (tree_tokens) keep the full capture, so since= diffs and tree hashes are unaffected by payload depth. New adapter method get_element_subtree(hwnd, element_path, max_depth): MockAdapter navigates the positional element-id path of a fresh capture; WindowsAdapter navigates raw-UIA FindAll child indices so only the requested branch is walked (falling back to full walk + extraction for non-positional ids or navigation failure). The _uia_walk internals are refactored into reusable _uia_handles/_uia_walk_element pieces with module-level UIA property-id constants. ScreenObserver. get_element_subtree() prefers a fresh cached capture (no walk), then the adapter's scoped walk, then full walk + extraction; extraction returns depth-pruned copies so cached trees are never mutated. Exposed as scope= (element id) + depth= on get_window_structure and depth= on observe_window across REST (/api/structure, /api/observe), MCP schemas, and dispatch. Scoped responses carry tree_token=null since a subtree is not a valid diff baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
New config key tree.strategy: "merged" (default, unchanged behavior: raw-UIA walk + pywinauto walk + synthesis) or "uia_only", which returns the raw-UIA tree directly and skips the second pywinauto _walk and the _synthesize_trees pass — roughly halving Windows capture time. When the UIA walk fails entirely, uia_only still falls through to the pywinauto path as a rescue. The raw-UIA walker now builds a CacheRequest over the ~15 property IDs it reads (plus BoundingRectangle) and fetches children per level with FindAllBuildCache, reading properties via GetCachedPropertyValue — one cross-process COM round trip per node's children instead of ~15 per node. Every stage is guarded: CacheRequest construction failure disables batching for the walk, a per-node FindAllBuildCache failure falls back to FindAll + GetCurrentPropertyValue for that branch, and cached bounds fall back CachedBoundingRectangle → BoundingRectangle SAFEARRAY → CurrentBoundingRectangle. get_element_subtree navigation reuses the same batched walker. Tests: tests/test_uia_walk.py drives WindowsAdapter against fake comtypes/pywinauto/pywin32 modules injected via sys.modules, covering the batched path (zero per-node GetCurrentPropertyValue round trips), both fallback layers, uia_only vs merged strategy, and scoped subtree navigation. Manual verification on real Windows required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
observe_window gains changed_only=true (REST /api/observe, MCP schema,
dispatch): the fresh capture's tree_hash is compared against the
window's last cached capture — unchanged windows return a tiny
{unchanged: true, tree_hash} response; changed windows return
diff.diff_custom(old, new) plus the fresh hash instead of the full
tree; a window with no baseline returns the depth-bounded full tree.
The baseline is looked up with TreeCache.peek() (TTL-independent)
before the fresh bypass capture refreshes the cache, and since= keeps
its existing token-based semantics (it takes precedence when both are
passed).
Every get_window_structure / observe_window result now carries
perf: {capture_ms, node_count, cache: hit|miss|bypass, depth_used},
sourced from ScreenObserver.get_element_tree_with_meta (scoped
captures report timing around the subtree resolution and whether a
fresh cache entry backed it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
get_window_structure now attaches
degraded: {reason: "sparse_accessibility_tree", named_node_count,
threshold, suggested_fallbacks: [get_ocr, get_screen_description]}
when the captured tree has fewer named non-root nodes than
tree.sparse_threshold (default 5), so agents pivot to pixel-based
perception instead of acting on a near-empty tree from an
accessibility-dark window. Scoped (subtree) requests are exempt since
a leaf branch being small is expected.
get_capabilities gains tree_stats: per-window last-capture statistics
(node_count, named_node_count, capture_ms, captured_at) sourced from
the TreeCache stats ledger, which survives input-driven cache
invalidation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
- web_ui.host built-in default flips 0.0.0.0 -> 127.0.0.1; the existing
--host CLI flag / config web_ui.host stay the explicit opt-in path to
0.0.0.0 for Docker deployments and testing (no authentication added,
per direction: bind policy is the control).
- Startup prints a prominent banner + logger.warning whenever the bind
is non-loopback ("unauthenticated action API exposed to the network").
- CORS(app) blanket permissive CORS removed: web_ui.cors_origins config
(default [] = no CORS headers, same-origin only; the inspector UI is
served same-origin so it keeps working). ["*"] or explicit origins
re-enable CORS for Docker/testing dashboards. /api/action never gets
permissive CORS by default.
- README "Security & Network Bind Defaults" rewritten for the new
reality (loopback default, opt-in flag, Docker note, CORS config);
config reference + config.json.example updated. Note: this repo ships
no Dockerfile/compose of its own (the Docker harness lives in the
AutoGUI repo), so the container opt-in is documented rather than
patched into an entrypoint.
- Tests: bind-default config resolution, non-loopback warning banner,
and CORS behavior via the Flask test client (no ACAO by default incl.
/api/action + preflight; present for ["*"] and matching explicit
origins; absent for non-matching origins).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
- redaction.py: new trust-boundary layer. Results of the nine
screen-text-carrying tools (list_windows, find_element,
get_window_structure, observe_window, wait_for, snapshot_get,
snapshot_diff, get_ocr, get_screen_description) are flagged
`untrusted: true` and all extracted strings are stripped of ANSI
escape sequences (CSI/OSC/2-byte) and non-whitespace control
characters. Opaque fields (base64 data, tokens) are skipped. Wired
unconditionally into tools.dispatch (independent of the opt-in
redactor). README gains a "Trust Boundary: Screen Content Is
Untrusted" section documenting the prompt-injection contract for
MCP clients.
- §21 confirmation audit: every element-targeted destructive verb
(click/focus/set_value/invoke/select/hover/right_click/double_click/
key_into/clear_text/click_element_and_observe) already routed through
_check_confirmation via _do_element_action. ONE unprotected verb
found: `drag` resolved elements from selector/element_id endpoints
but never consulted the gate. Fixed — a confirmation_required rule
matching either endpoint now demands a confirm_token
(propose_action(action="drag", args={selector}) issues it).
Coordinate/global verbs (click_at, type_text, press_key, scroll,
hover_at, bring_to_foreground + aliases/composites) carry no element
identity so §21 role/name rules cannot address them — documented
exclusion, with a completeness test asserting every input tool is in
exactly one group.
- REST<->MCP<->dispatch parity test (design doc §27): asserts every
REGISTRY tool has an MCP schema (documented exclusions:
right_click_at/double_click_at aliases), every MCP schema is
dispatchable (legacy-only: get_screen_sketch/get_full_screenshot),
every tool has a dedicated REST route registered in the Flask url_map,
/api/tools enumerates the registry exactly, and each tool answers a
structured envelope through the generic console.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
- tree_cache.py: session-cumulative counters — cache hits/misses, entry count, and a capture-latency summary (count/total/max/mean ms) fed by every put(); exposed via TreeCache.counters(). - /api/metrics (Prometheus text format kept): new oso_tree_cache_hits_total / oso_tree_cache_misses_total counters, oso_tree_cache_entries gauge, oso_tree_capture_ms summary (_sum/_count) + oso_tree_capture_ms_max gauge, alongside the existing step/uptime/budget/trace series. - /api/healthz (verified real + made cheap): now reports the same tree_cache counter block; the OCR diagnostic (which spawns a `tesseract --version` subprocess) is computed once per process and cached so the endpoint is safe to poll; swallowed diagnostics failures now logger.debug instead of silent pass. - README endpoint quick-reference updated for both endpoints. - Tests: healthz counter block + hit/miss movement + step-count increment + diagnose-once cheapness; metrics content-type, all new series present with HELP/TYPE lines, and counters move after a cold+warm structure read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
- pyproject.toml (new): mypy lenient global (ignore_missing_imports, follow_imports=silent) with strict-ish overrides for tree_cache.py, errors.py, session.py (disallow_untyped_defs/incomplete_defs, check_untyped_defs, warn_return_any, no_implicit_optional); ruff lint select pinned to the default E4/E7/E9/F so E722 (bare except) is an explicit, guaranteed gate; coverage config with source=["."] and omit=tests/* for a stable floor basis. - mypy: fixed all 43 pre-existing errors across 12 files (Optional narrowing in budgets/session/ocr_util/ollama_setup, implicit-Optional defaults in observer.perform_action, _resolve_element None contract in tools.py element actions + propose_action, Optional tree threading in get_window_structure/observe paths, PIL Image typing, duplicated `params`/`prefix` bindings in window_agent/ollama_setup, Playwright :has-text double-match in element_selectors, ctypes.windll inline ignores on the two Windows-only guard sites). `mypy .` is clean. - except audit: 26 silent `except …: pass` blocks triaged down to 7. 19 now log context via logger.debug (OCR configure/preprocess guards, UIA/AT-SPI partial-walk truncation, trace pre/post hash, screenshot crop fallbacks, selector-as-element_id fallback) and budget accounting failures escalate to logger.exception. The 7 kept are precise-typed cleanup guards (OSError tmp unlink, ValueError ring.remove, FileNotFoundError wmctrl->xdotool fallback, int-parse) or commented UIA cached-property fallbacks that legitimately degrade. - CI: new `mypy .` step; regression lane now runs with `--cov --cov-fail-under=53` (measured 56.25% source coverage on the lane; floor set 3 points below). mypy added to requirements-dev.txt; coverage/mypy artifacts gitignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Mechanical split of the 2,375-line tools.py into a package: tools/context.py ToolContext + window/element resolution helpers tools/receipts.py receipt building + confirmation gating tools/actions.py element-targeted, coordinate and composite verbs tools/observe.py tree observation, diffs, filter/page helpers tools/vision.py screenshots, OCR, screen descriptions tools/snapshots.py snapshot tools + wait_for / wait_idle tools/trace_replay.py tracing, replay, scenarios, oracles tools/meta.py windows/capabilities/status/propose_action tools/dispatch.py REGISTRY, allowlist gate, cross-cutting hooks tools/__init__.py re-exports the entire pre-split surface (including underscore helpers) so every existing import path keeps working; the only behavioral-code edit is a local dispatch import in replay_step to break the trace_replay ↔ dispatch registry cycle. No behavior changes; all tests pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Mechanical split of the 2,048-line observer.py into a package:
observer/models.py Bounds/UIElement/WindowInfo/WindowResolution
+ find_element_by_path / prune_tree_depth
observer/platform_info.py PLATFORM / IS_WSL / EFFECTIVE_PLATFORM
observer/adapters/ mock.py, windows.py (incl. UIA constants),
macos.py, linux.py, wsl.py
observer/core.py ScreenObserver + adapter selection + cache wiring
observer/activation.py bring_to_foreground + _activate_* +
title-bar targeting (ActivationMixin)
observer/occlusion.py get_visible_areas / is_element_occluded +
_intersect_bounds / _subtract_rect (OcclusionMixin)
ScreenObserver now inherits the activation/occlusion mixins; method bodies
are byte-identical. Top-level mac_adapter.py / linux_adapter.py (the
optional pyobjc/pyatspi runtime upgrades) are untouched and still lazily
imported by core. observer/__init__.py re-exports the entire pre-split
surface (adapters, models, UIA constants, platform flags, geometry
helpers) so every existing import path keeps working; all tests pass
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Mechanical split of the three remaining god-files into packages, each with an __init__.py that re-exports the pre-split surface: window_agent/ (was 1,978 lines; run with 'python -m window_agent') client.py ANSI colours, stdlib HTTP, REST wrappers, LLMClient dispatch.py LLM tool-name → REST endpoint dispatcher tool_schemas.py SCREEN_TOOLS catalogue, tiers, keyword groups, meta-tools prompts.py SYSTEM_PROMPT loop.py run_agent agentic loop + tool-result printing cli.py banner, window picker, interactive prompts, main() __main__.py entry point (preserves the KeyboardInterrupt handler) web_inspector/ (was 1,770 lines) assets.py inline single-page UI HTML template (_HTML) views.py register_routes() — every Flask route, moved verbatim server.py create_web_app() factory (Flask + CORS + ToolContext) mcp_server/ (was 1,237 lines) tool_schemas.py _TOOLS (tools/list schema payload) server.py MCPServer (JSON-RPC transport + dispatch + handlers) (named mcp_server, not mcp, to avoid colliding with the external MCP SDK distribution name) No behavior changes; from web_inspector import create_web_app, from mcp_server import MCPServer, mcp_server._TOOLS and all other existing import paths keep working; all tests pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
scripts/check_module_size.py fails when any git-tracked non-test .py exceeds the design doc's ~600-line module cap, with an explicit, justified allowlist for intentional remainders: web_inspector/assets.py inline SPA HTML template (data, not logic) window_agent/tool_schemas.py declarative LLM tool catalogue mcp_server/tool_schemas.py declarative MCP tools/list payload ascii_renderer.py single cohesive renderer (pre-existing) description.py single cohesive generator (pre-existing) observer/adapters/windows.py single cohesive UIA adapter (640 lines) The guard also flags stale allowlist entries (files that shrank under the cap or were deleted) so exceptions cannot silently accumulate. Wired into ci.yml right after ruff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
There was a problem hiding this comment.
Pull request overview
Large refactor and hardening pass across OSScreenObserver focused on (1) faster perception via caching + bounded responses, (2) safer network/CORS defaults plus prompt-injection boundary marking, and (3) decomposing prior “god files” into import-preserving packages while adding CI quality gates (ruff/mypy/coverage/module-size).
Changes:
- Added per-window accessibility-tree caching with TTL/LRU plus perf/telemetry counters and diff/changed-only observation paths.
- Hardened defaults (loopback bind, opt-in CORS) and introduced “untrusted screen content” sanitization/flagging for screen-derived text.
- Split major modules into packages with shims, and added CI guards (module LOC cap, mypy, coverage floor).
Reviewed changes
Copilot reviewed 65 out of 66 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| window_agent/prompts.py | Extracted agent system prompt text into a dedicated module. |
| window_agent/loop.py | Agent tool-calling loop with retries and tool-result printing. |
| window_agent/dispatch.py | Maps agent tool calls to REST endpoints (client-side dispatcher). |
| window_agent/client.py | REST + LLM client helpers (stdlib HTTP, OpenWebUI chat). |
| window_agent/cli.py | Interactive CLI for selecting windows and running the agent loop. |
| window_agent/main.py | python -m window_agent entry point wrapper. |
| window_agent/init.py | Import-preserving shim re-exporting the pre-split surface. |
| web_inspector/server.py | Flask app factory with config-driven CORS behavior. |
| web_inspector/init.py | Import-preserving shim for the web inspector package. |
| tree_cache.py | New per-window TTL/LRU cache for accessibility-tree captures + telemetry. |
| tools/vision.py | Screenshot/OCR/description tools, including combined description mode. |
| tools/trace_replay.py | Tracing/replay/scenario/oracle tools split from monolith. |
| tools/snapshots.py | Snapshot + wait_for/wait_idle implementations split from monolith. |
| tools/receipts.py | Action receipts + confirmation-token gating logic split from monolith. |
| tools/meta.py | Introspection tools (capabilities, windows, proposals, status). |
| tools/dispatch.py | Central dispatch table with allowlist gate + hooks (redaction/untrusted/telemetry). |
| tools/context.py | ToolContext and shared window/element resolution helpers. |
| tools/init.py | Import-preserving shim re-exporting the pre-split tools surface. |
| tests/test_untrusted_and_confirmation.py | Tests for untrusted marking + confirmation audit (incl. drag gating). |
| tests/test_uia_walk.py | Mocked Windows UIA walker tests (CacheRequest batching + fallbacks). |
| tests/test_tree_cache.py | Unit + integration tests for tree cache + invalidation semantics. |
| tests/test_tools_p3.py | Tests for bounded depth, scope drill-in, and structure/observe behavior. |
| tests/test_tools_p2.py | Tests for changed_only observe + perf telemetry reporting. |
| tests/test_telemetry.py | Tests for /healthz and /metrics telemetry exposure and cheap polling. |
| tests/test_surface_parity.py | REST↔MCP↔dispatch parity checks and registry reachability smoke tests. |
| tests/test_security_defaults.py | Tests for loopback bind default + opt-in CORS behavior. |
| session.py | Session state extended to include the per-window tree cache. |
| scripts/check_module_size.py | CI script enforcing ~600 LOC module cap with allowlist. |
| requirements-dev.txt | Added mypy to dev dependencies. |
| redaction.py | Added untrusted screen-content sanitization + untrusted: true marking. |
| README.md | Updated docs for secure bind defaults, CORS, perf features, trust boundary. |
| pyproject.toml | Ruff/mypy/coverage configuration for CI quality gates. |
| ollama_setup.py | Typing/clarity tweaks around runner option parsing. |
| ocr_util.py | Bugfix in configured_path_exists logic. |
| observer/platform_info.py | Split platform detection helper module. |
| observer/occlusion.py | Split occlusion/visibility logic into mixin module. |
| observer/models.py | Split data models + subtree helpers into dedicated module. |
| observer/adapters/wsl.py | Split WSL adapter with DISPLAY-aware fallback behavior. |
| observer/adapters/mock.py | Split mock adapter; added perf/testing hooks. |
| observer/adapters/macos.py | Split macOS adapter stub + screenshot support. |
| observer/adapters/linux.py | Split Linux adapter with wmctrl/Xlib fallback + screenshot options. |
| observer/adapters/init.py | Adapter package exports (incl. UIA constants). |
| observer/activation.py | Split activation/bring_to_foreground strategies and title-bar targeting. |
| observer/init.py | Import-preserving shim for the observer package. |
| mcp_server/server.py | Split MCP server implementation (stdio JSON-RPC transport + dispatch). |
| mcp_server/init.py | Import-preserving shim exporting MCPServer and tool schemas. |
| main.py | Secure-by-default bind, config merge tweaks, bind warning banner, new tree defaults. |
| linux_adapter.py | Improved debug logging and typing cleanup in AT-SPI adapter upgrade path. |
| element_selectors.py | Parsing fix for :has-text handling logic. |
| description.py | Logging improvements + Pillow resize API update + parsing robustness. |
| config.json.example | Updated secure defaults and new tree/cache/CORS configuration docs. |
| budgets.py | Correctness/clarity improvements for optional budget limits. |
| ascii_renderer.py | Logging improvements (avoid silent exceptions in OCR paths). |
| .gitignore | Ignored coverage and mypy cache artifacts. |
| .github/workflows/ci.yml | Added module-size guard, mypy, and coverage floor to CI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+119
to
+125
| elif tool_name == "find_element": | ||
| params: Dict[str, Any] = {"selector": args["selector"]} | ||
| if "window_uid" in args: | ||
| params["window_uid"] = args["window_uid"] | ||
| elif wi is not None: | ||
| params["window_index"] = wi | ||
| return _get(rest, "/api/find_element", params) |
Comment on lines
+170
to
+180
| elif tool_name == "observe_window_diff": | ||
| params = {} | ||
| if "window_uid" in args: | ||
| params["window_uid"] = args["window_uid"] | ||
| elif wi is not None: | ||
| params["window_index"] = wi | ||
| if "since" in args: | ||
| params["since"] = args["since"] | ||
| if "format" in args: | ||
| params["format"] = args["format"] | ||
| return _get(rest, "/api/observe", params) |
Comment on lines
+254
to
+278
| elif tool_name == "get_screenshot_cropped": | ||
| # Pixel data is huge — same omission policy as get_screenshot. | ||
| params = {} | ||
| if "window_uid" in args: | ||
| params["window_uid"] = args["window_uid"] | ||
| elif wi is not None: | ||
| params["window_index"] = wi | ||
| for k in ("element_id", "padding_px", "max_width"): | ||
| if k in args: | ||
| params[k] = args[k] | ||
| result = _get(rest, "/api/screenshot/cropped", params) | ||
| if "data" in result: | ||
| result = {k: v for k, v in result.items() if k != "data"} | ||
| result["note"] = "Screenshot captured (base64 data omitted)." | ||
| return result | ||
|
|
||
| elif tool_name == "get_ocr": | ||
| params = {} | ||
| if "window_uid" in args: | ||
| params["window_uid"] = args["window_uid"] | ||
| elif wi is not None: | ||
| params["window_index"] = wi | ||
| if "element_id" in args: | ||
| params["element_id"] = args["element_id"] | ||
| return _get(rest, "/api/ocr", params) |
Comment on lines
+65
to
+71
| def sanitize_screen_text(text: str) -> str: | ||
| """Strip ANSI escape sequences and non-whitespace control characters | ||
| from screen-extracted text. Idempotent; returns non-str input as-is.""" | ||
| if not isinstance(text, str) or not text: | ||
| return text | ||
| text = _ANSI_ESCAPE_RE.sub("", text) | ||
| return _CONTROL_CHARS_RE.sub("", text) |
…e args, sanitize signature - window_agent/dispatch.py: route find_element, observe_window_diff, get_screenshot_cropped, and get_ocr through the shared _win_params helper so resolved window_title / default selection is honored and a null window_uid key no longer suppresses index/title selectors. Also forward the bbox param to the cropped-screenshot and OCR REST routes. - tests/test_surface_parity.py: update SMOKE_ARGS[wait_for] from the stale predicate/kind shape to the current any_of/type contract (window_appears). - redaction.py: annotate sanitize_screen_text as Optional[str] -> Optional[str] to reflect the documented non-str passthrough behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three-phase overhaul of OSScreenObserver. Every commit kept CI green (
ruff+ regressionpytest+ user-subset +mypy+ coverage floor).Phase 1 — Perception performance (the core product metric)
Previously every tool call re-walked the full accessibility tree. Now:
tree_cache.py) keyed bywindow_uid, invalidated at thedispatch()choke point after any input tool orbring_to_foreground.tree.default_depth) with exacttruncated/child_countmarkers and ascope=parameter to drill into subtrees without re-capturing the whole window.uia_onlystrategy + UIA CacheRequest batching — one COM round-trip per level instead of ~15 property fetches per node (mock-verified; flagged for manual Windows verification).changed_onlyobservation returns a tiny{unchanged, tree_hash}payload or a diff instead of the full tree.perftelemetry (capture_ms,node_count, cache hit/miss, depth) on every observe result.Regression suite 205 → 250.
Phase 2 — Security hardening + quality gates
127.0.0.1;--host 0.0.0.0is an explicit opt-in for Docker/testing (no authentication added), with a startup warning on any non-loopback bind.CORS(app)replaced with config-drivenweb_ui.cors_origins(default same-origin only).dragbypassed the token gate)./api/healthzdiagnostics; cache-hit-rate metrics in Prometheus format.mypyclean; silentexceptblocks cut from 26 to 7 (survivors are typed cleanup guards); coverage floor added to CI.Phase 3 — Full decomposition
Each god-file became a package behind an import-preserving shim — all existing tests pass unchanged:
tools.py(2,375) →tools/(context/receipts/actions/observe/vision/snapshots/trace_replay/meta/dispatch)observer.py(2,048) →observer/(models/core/activation/occlusion +adapters/)window_agent.py,web_inspector.py,mcp_server.py→ packagesscripts/check_module_size.pyenforces the design doc's ~600-line cap in CI (with a documented allowlist for declarative schema/asset data).Notes
uia_onlyare mock-tested only (headless container) — manual verification on real Windows is flagged in the relevant commit.🤖 Generated with Claude Code
https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Generated by Claude Code