Feature: shared tabbed browser with embedded form support - #56
Merged
Merged
Conversation
The preview pane becomes a real browser: multiple tabs, an address bar, back/forward/reload, and a toolbar button that opens it independently of the agent. User websites and agent project previews live side by side in one pane. Agent actions require an explicit tab ID. The tab the user was viewing when they sent a message is captured at submit time, so "this page" keeps meaning that tab even after the UI selection changes, and a closed target reports failure instead of redirecting the agent to a different tab. Browser tools also discover cross-origin and nested frames and can inspect, fill, select, scroll, and wait inside them, so embedded forms are no longer invisible. Also in this branch: - Frame identities survive a navigation that was blocked and never happened. - Host browser bookkeeping stays out of plan cards, out of the plan manifest that persists in chat history, and out of the model's replies. - A blocked click reports which element is covering the target. - Failures raised mid-operation carry their tab and console output. - Startup announces one model rather than two, with image capability on the same line as the model it describes. Pins the engine to the matching tool-cache fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Desktop's preview pane becomes a real browser. It gains multiple tabs, an address bar, and back/forward/reload, and a toolbar button opens it whether or not the assistant is involved. Websites the user opens and project previews the assistant opens now live side by side in the same pane instead of competing for one slot.
The assistant can also work with forms embedded in a page — the kind that live inside an
iframe, such as a contact form hosted by a third party. Previously those were invisible to it, and it would report that a page had no form at all.Why this matters
Two problems drove this work.
One preview slot was not enough. The pane could show a single page. Opening a project preview replaced whatever the user was looking at, and there was no way to keep a reference page open beside the work.
The assistant could not see inside embedded content, and did not know it. Asked to fill a contact form, it would inspect the page, find no fields, and confidently report that the form was missing or broken. The fields were there — one layer down, inside a frame it could not reach.
What is new
Also in this branch
Fixes and polish found while reviewing the above:
Scope and risk
Medium, concentrated in the preview pane and the assistant's browser tools. Existing project previews and local development servers keep their behaviour and their restriction to the project's own content; the wider web is reachable only in tabs opened as browser tabs. Downloads and pop-ups triggered by the assistant remain blocked.
Two behaviours reviewers may want to weigh in on, both carried over rather than introduced here:
Verification
Not covered by automated tests: the startup announcement ordering and the tab-attribution on failures both need a live window, which no test harness currently creates. Both are worth a manual pass — open a restored tab and confirm exactly one model line appears.
Dependency
Pins the engine to DevMando/MandoCode#94, which stops the new browser listing tools from returning cached results. Merge that first.