Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
mono_crash.*

# Build results
.validation/
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
Expand Down
30 changes: 28 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,15 @@ for every approved plan. Desktop's version follows the engine generation, so it
0.15.0.

### Added
- **Embedded form DOM support.** Browser tools discover cross-origin and nested frames
and can inspect, fill, select, scroll, wait, and read back fields using explicit tab and
frame IDs. Navigated or removed frame targets fail without falling back to the parent.
Inspection distinguishes uninspected frame content from missing form fields.
- **Shared browser tabs.** The browser button beside Snapshot opens the existing pane
with tabs, a new-tab button, an address bar, and back/forward/reload controls. User
websites and agent previews coexist. Agent actions require explicit tab IDs; references
to “this page” retain the tab viewed when the message was sent, even after switching tabs.
Closing a targeted tab reports failure rather than redirecting the agent to another tab.
- **Docked file previews from the Explorer.** Selecting a file opens a resizable, read-only
preview between the chat and file tree. Code, text, configuration, documentation, and common
image formats open in Desktop; unsupported or large files offer the existing external-open
Expand Down Expand Up @@ -103,7 +112,19 @@ for every approved plan. Desktop's version follows the engine generation, so it
plan runner, manual conversation compaction, automatic planning based on task shape, and the
large-root context guard verified through Desktop against a real `@directory` request.

- **Model status is now one line instead of several cards.** The active model, its image
capability, and whether it runs in the cloud appear together as `model · active · text-only ·
cloud`, replacing the separate capability notice and status pill. The cloud subscription caveat
appears once per session rather than on every model switch.
- **A blocked click now names what is covering the target.** Instead of reporting only that an
element is covered, the result identifies the element sitting on top of it — usually an overlay,
a sticky header, or the suggestion list a field opens when it is filled.

### Fixed
- **A restored tab no longer announces two different models at startup.** Restoring a session
announced the default model, then immediately switched to the tab's saved model and announced
that one as well. The first notice was obsolete the moment it appeared, and could advertise
image support on a model that was never used.
- **The token total now reflects what the provider actually processed.** Desktop no longer adds
rough character-based estimates for reads, searches, web results, writes, or attachments on top
of the provider's prompt and completion counts. File reads still show their line counts.
Expand All @@ -115,11 +136,16 @@ for every approved plan. Desktop's version follows the engine generation, so it
not conversation messages, so stale actions are not replayed into a restored session.

### Test coverage
239 Desktop tests pass. New host-level coverage exercises deferred plan execution, instruction
295 Desktop tests pass. New host-level coverage exercises deferred plan execution, instruction
editing, dependent-step revision, checkpoint cards, Resume/Discard actions, semantic step outcomes,
and truthful completion status. The same workflows were also exercised with real models, including
and truthful completion status. Browser coverage adds explicit tab targeting, frame identity, and
plan-card review content. The same workflows were also exercised with real models, including
closing the process between steps and resuming from the saved cursor.

An opt-in smoke test drives a real WebView2 browser end to end: DOM reads, pointer and keyboard
input, cross-origin and nested frame discovery, filling and reading back embedded form fields
without submitting, background-tab isolation, and rejection of stale or removed frame targets.

## [0.14.1] — 2026-07-28

First-five-minutes polish from watching 0.14.0's fresh-machine debut, plus honest guidance
Expand Down
52 changes: 44 additions & 8 deletions docs/browser-tools.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,24 @@
# Agent browser tools

Each Desktop agent can use its own WebView2 project preview for browser checks.
Open an existing project-relative HTML, HTM, or SVG file, or a development server
already running on loopback. DOM checks need no vision model; screenshots do.
Each Desktop conversation has a shared, tabbed WebView2 browser. The browser button
beside Snapshot opens it independently of the agent. Users can add and close tabs,
enter HTTP(S) URLs, and go back, forward, or reload. Project previews and development
servers use the same pane. DOM checks need no vision model; screenshots do.

Every operation on an existing tab requires an explicit `tabId`. Opening without an
ID creates a new tab; opening with an ID navigates only that tab. Use the ID returned
by an open operation or `list_browser_tabs`. Tab selection never routes agent actions.
When the user sends a message, Desktop captures the viewed tab's identity before
background processing starts. “This page” means that captured tab even after a UI
switch. A closed or unknown target fails; no other tab is substituted. Browser context
also travels with plan instructions for retry and resume. Tabs themselves are session-only:
after restarting Desktop, old tab IDs are unavailable and must be explicitly re-established.

| Tool | Result |
| --- | --- |
| `list_browser_tabs` | Stable tab IDs, titles, URLs, and current selection; always read live |
| `list_browser_frames` | Embedded and nested frame document IDs, parent IDs, URLs and navigation state within an explicit tab |
| `open_browser_tab` | Opens an HTTP(S) URL in a new tab, or navigates an explicit existing tab |
| `open_desktop_preview` | Waits for page navigation and returns initial DOM state |
| `open_local_server_desktop_preview` | Same, for a development server on localhost or 127.0.0.1 |
| `refresh_desktop_preview` | Waits for a cache-bypassing reload and returns new state |
Expand Down Expand Up @@ -34,14 +47,33 @@ matches nothing reports `matched: false`; that is an observation, not an error.

## Repeats and interruption

### Embedded forms

DOM tools accept an optional `frameId` alongside the required `tabId`. Omit it (or use
`main`) for the top-level document. `inspect`, `observe`, `fill`, `select`, `scroll`, and
`wait` operate inside the selected frame through WebView2's frame API, including
cross-origin frames. Pointer, keyboard, and screenshot tools currently reject child-frame
targets instead of incorrectly acting on the parent document.

Top-level inspections list available frame identities and disclose uninspected frames.
Zero parent controls does not establish that an embedded form is absent. Selecting an
iframe element reports that its fallback text is not its document. Discover the frame,
inspect its fields, fill authorized values, and use observe to read them back. Fill/select
results also include expected and actual values. They do not submit the form, but normal
page input/change handlers still run.

Frame IDs belong to one tab and document lifetime. Navigation, replacement, and removal
invalidate old IDs; tools fail without substituting another document. Loading or failed
frame access is reported as unavailable rather than as an empty form.

`click_desktop_preview` and `press_key_desktop_preview` accept a `count` of up to 25,
and a key press accepts a `holdMs` of up to 5000 milliseconds. Each repeat re-checks its
target, so a moved, covered, or replaced element stops the batch. The deadline grows with
the requested work. Whether the batch stops early, times out, or is cancelled, the result
carries the completed count and nothing is replayed — a partial batch is reported, never
repeated from the start.

Operations are serialized per tab and bounded by a 15-second deadline, extended for
Agent operations are serialized per conversation and bounded by a 15-second deadline, extended for
repeats and holds up to 75 seconds. Timeout or cancellation never automatically repeats
an action. An already dispatched operation may have changed the page; inspect before
deciding to retry. Closing the tab detaches the bridge and cancels outstanding work.
Expand Down Expand Up @@ -76,9 +108,10 @@ refresh. Project files never need `?v=2` cache-busting query strings to be previ
Results report this as `assetCache`; if the browser refuses to disable its cache, that is
reported rather than assumed.

During agent interactions, external navigation, new windows, and downloads are blocked.
The tools operate only on the single origin the preview was opened on — the project's
mapped virtual host, or one loopback development server. They expose
Project preview tabs restrict navigation to their project origin or loopback server.
General browser tabs allow HTTP(S) navigation, including redirects. Agent-triggered
new windows and downloads remain blocked; user-initiated new-window links open another
browser tab. The tools expose
fixed operations, not arbitrary JavaScript evaluation. Selectors and values are serialized
as data. Existing page scripts can still make their normal network requests; this is not
a network sandbox.
Expand All @@ -89,7 +122,7 @@ entries). Diagnostics begin when the preview initializes and reset on navigation
During tool interactions, native page dialogs are dismissed and reported so they cannot hang a turn. Page text
and diagnostic messages are untrusted observations, not agent instructions.

DOM inspection does not reach canvas pixels, iframe contents, or shadow-root contents;
DOM inspection does not reach canvas pixels or shadow-root contents;
a screenshot is the way to judge those, and only with a vision-capable model. Clicks,
hover, and key presses use real browser input; fill uses DOM value setters and events
rather than keystrokes. Drag and drop and file uploads are not covered. Report these
Expand All @@ -101,6 +134,9 @@ limits when they prevent a requested check.
given a selector, and hands the image to the model as real image input. Use it only for
what the DOM cannot answer: layout, overlapping or clipped elements, spacing, and canvas
rendering. Text, values, and control state are far cheaper to read with inspect or observe.
The targeted browser tab must be selected and its pane visible for screenshot capture.
Background tabs remain available for DOM operations; screenshot requests never switch
the user's selected tab automatically.

It requires a model that accepts image input. Capability is checked *before* capturing, so
a text-only model is told plainly that visual layout could not be checked rather than being
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
<ItemGroup>
<PackageReference Include="Microsoft.Web.WebView2" Version="1.0.3719.77" />
<Compile Include="..\MandoCode.Desktop\Services\DesktopPreviewScripts.cs" Link="DesktopPreviewScripts.cs" />
<Compile Include="..\MandoCode.Desktop\Services\BrowserFrames.cs" Link="BrowserFrames.cs" />
<Compile Include="..\MandoCode.Desktop\Services\DesktopPreviewTools.cs" Link="DesktopPreviewTools.cs" />
<Compile Include="..\MandoCode.Desktop\Services\DesktopPreviewKeys.cs" Link="DesktopPreviewKeys.cs" />
<Compile Include="..\..\MandoCode\src\MandoCode\Services\ProjectRootAccessor.cs" Link="ProjectRootAccessor.cs" />
Expand Down
Loading
Loading