Skip to content

Feature: shared tabbed browser with embedded form support - #56

Merged
DevMando merged 1 commit into
mainfrom
feature/shared-browser-tabs
Sep 7, 2026
Merged

DevMando merged 1 commit into
mainfrom
feature/shared-browser-tabs

Conversation

@DevMando

@DevMando DevMando commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Summary

Desktop's preview pane becomes a real browser. It gains multiple tabs, an address bar, and back/forward/reload, and a toolbar button opens it whether or not the assistant is involved. Websites the user opens and project previews the assistant opens now live side by side in the same pane instead of competing for one slot.

The assistant can also work with forms embedded in a page — the kind that live inside an iframe, such as a contact form hosted by a third party. Previously those were invisible to it, and it would report that a page had no form at all.

Why this matters

Two problems drove this work.

One preview slot was not enough. The pane could show a single page. Opening a project preview replaced whatever the user was looking at, and there was no way to keep a reference page open beside the work.

The assistant could not see inside embedded content, and did not know it. Asked to fill a contact form, it would inspect the page, find no fields, and confidently report that the form was missing or broken. The fields were there — one layer down, inside a frame it could not reach.

What is new

  • Tabbed browsing. Add and close tabs, type a web address, navigate history. Project previews and local development servers use the same pane.
  • Embedded form support. The assistant discovers frames on a page, including nested ones and ones served from another domain, and can read, fill, and verify fields inside them. It does not submit a form unless asked.
  • Honest reporting about what was not inspected. A page with zero visible fields now says so and discloses that it contains frames whose contents were not examined, so "no form here" can no longer be concluded from a partial look.
  • Actions are pinned to a specific tab. Every assistant action names the tab it targets. The tab the user was viewing when they sent a message is recorded at that moment, so "this page" keeps meaning that tab even if they switch tabs while the assistant works. If the targeted tab is closed, the action reports a failure rather than quietly acting on a different tab.

Also in this branch

Fixes and polish found while reviewing the above:

  • Frame identities survive a navigation that never happened. A page attempting a blocked navigation used to invalidate every frame reference on it, leaving the assistant told there was one frame it had not inspected and simultaneously that no frames existed.
  • Internal bookkeeping stays internal. The note recording which tab a request came from was being rendered into plan review cards, written into the plan summary that stays in conversation history for the rest of a session, and occasionally recited back to the user mid-answer. It is now stripped from everything the user reads.
  • A blocked click says what is blocking it. Instead of "element is covered", the result names the element on top of the target — commonly an overlay, a sticky header, or the suggestion list a search field opens when it is filled.
  • Failures carry their context. Errors raised part-way through an operation now include the tab and recent browser console output, which is what deciding whether to retry actually depends on.
  • Startup announces one model, not two. Restoring a session announced the default model, then switched to the tab's saved model and announced that one too. The first notice was obsolete on arrival and could advertise image support on a model that was never used. Model name, status, image capability, and cloud/local now appear as a single line.

Scope and risk

Medium, concentrated in the preview pane and the assistant's browser tools. Existing project previews and local development servers keep their behaviour and their restriction to the project's own content; the wider web is reachable only in tabs opened as browser tabs. Downloads and pop-ups triggered by the assistant remain blocked.

Two behaviours reviewers may want to weigh in on, both carried over rather than introduced here:

  • Assistant-initiated navigation can retarget a tab the user opened.
  • Actions on external sites are not gated by an approval prompt, unlike file writes.

Verification

  • 295 Desktop tests pass, including new coverage for explicit tab targeting, frame identity, and plan-card review content.
  • The opt-in WebView2 smoke test passes against a real browser: DOM reads, pointer and keyboard input, cross-origin and nested frame discovery, filling and reading back embedded fields without submitting, background-tab isolation, and rejection of stale or removed frame references.
  • Build is clean with no warnings.

Not covered by automated tests: the startup announcement ordering and the tab-attribution on failures both need a live window, which no test harness currently creates. Both are worth a manual pass — open a restored tab and confirm exactly one model line appears.

Dependency

Pins the engine to DevMando/MandoCode#94, which stops the new browser listing tools from returning cached results. Merge that first.

The preview pane becomes a real browser: multiple tabs, an address bar,
back/forward/reload, and a toolbar button that opens it independently of the
agent. User websites and agent project previews live side by side in one pane.

Agent actions require an explicit tab ID. The tab the user was viewing when
they sent a message is captured at submit time, so "this page" keeps meaning
that tab even after the UI selection changes, and a closed target reports
failure instead of redirecting the agent to a different tab. Browser tools also
discover cross-origin and nested frames and can inspect, fill, select, scroll,
and wait inside them, so embedded forms are no longer invisible.

Also in this branch:

- Frame identities survive a navigation that was blocked and never happened.
- Host browser bookkeeping stays out of plan cards, out of the plan manifest
  that persists in chat history, and out of the model's replies.
- A blocked click reports which element is covering the target.
- Failures raised mid-operation carry their tab and console output.
- Startup announces one model rather than two, with image capability on the
  same line as the model it describes.

Pins the engine to the matching tool-cache fix.
@DevMando
DevMando merged commit d4e7515 into main Sep 7, 2026
1 check passed
@DevMando
DevMando deleted the feature/shared-browser-tabs branch September 7, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant