feat(browser): open a session browser the agent drives by DiegoSGbr · Pull Request #941 · lichdotdev/lich

DiegoSGbr · 2026-10-24T16:04:18Z

Summary

A session card's Browser item, above Open in, opens a tab beside the session: its terminal on one side and its page on the other, laid out like a split (drag the divider, rotate, zoom). The tab holds that session and its browser only; a session in another layout is asked about before it moves.
The page is a headless Chrome, Edge, Chromium, Brave, Vivaldi or Helium already installed, drawn in the pane from a CDP screencast that follows the pane's size. The pane passes the person's mouse, wheel, keys and input method text to the page, copies and pastes with the system clipboard (Ctrl+C, Ctrl+X, Ctrl+V; Cmd on macOS), and has back, forward, reload, an address field and Open in a window, which moves the page to a Chromium window of its own with its cookies.
The session's agent drives the same page with the browser_* tools and lich browser: open, read, click, type, press a named key, go back and forward, scroll and screenshot. When the agent opens a browser on its own, a toast in the bottom-right corner offers to show it. opencode does not list browser_*; lich browser still works in that session.
The agent has a cursor of its own in the pane: lich's arrow, in the session's card color or the theme's primary, with a badge naming the provider. It glides to each click, field and scroll on the page, and to back, forward, reload and the address field on the toolbar, plays parallel calls in turn, catches up with a fast agent, and stays while the agent's turn runs.
Page reads are compact lines rather than JSON, and browser_click and browser_press answer with the page they left, so an agent does not follow each action with a read. An element out of view is scrolled into view before its click; a failed action keeps the browser unless its Chrome is gone.
A link that opens a new tab loads in the page itself, and Back returns to where it was. Closing the page, the tab or the card closes the browser. A headless browser no pane shows and no one used for five minutes is closed.

Test plan

A card menu item opens one Chromium window for that session. The agent drives the same page with the browser_* tools and lich browser, and a navigation returns once the document is interactive instead of waiting out the load event. Co-authored-by: Cursor <cursoragent@cursor.com>

…aunch from holding every card The reaper closed a headed window after five idle minutes while a person was still using it; only headless browsers are reaped now. Promoting a headless browser to a window reopens it on the same profile and carries every cookie, session ones included, instead of starting from an empty one. Focus raises the window with Page.bringToFront, which window.focus() from the page cannot. A clear before typing goes through the editing commands, so the field's input listeners see it. Chrome launches and closes happen outside the service lock, and close errors are logged instead of dropped.

…input browser.Attach starts a screencast of the session's headless browser sized to the pane (starting one when there is none), and the frames reach the window as a per-session event family, acked only once sent so a slow window slows the stream. The pane's mouse, keys, committed text and toolbar go through their own methods, so only the agent's actions light up browser-agent. browser-state tells the window when a browser starts, closes, moves to a window or loads another page; browser.Running is the catch-up read. A pane showing a browser keeps the reaper away from it.

The card's Browser item, and the toast that offers a browser the agent opened, put the session in a tab of its own beside its page: a layout whose second pane is the page, so the divider, rotating and zoom are the layout's. The tab takes no other session, and a session in another layout is asked about before it moves. The page pane draws the backend's frames and passes on the pointer, keys, committed text and pasted text; its toolbar goes back, forward, reloads, loads a typed address, or moves the browser to its own window. A card with a browser shows a globe.

A session of the project at the home directory routes with Home's flag set, and the layout on screen was hidden whenever that flag was, so a split or a browser tab holding one stayed off screen. showHome already clears the layout it leaves, so the flag added nothing but that.

…iling up A pane that grew (zoom, a dragged divider) got frames at the size its screencast started with, stretched over it: resizing a watched page now restarts the screencast at the new size. Chrome answers each pointer event in about 15ms, slower than a touchpad fires them, so the pane's queue merged nothing and lagged further behind with every event; moves and resizes that pile up behind a call in flight now keep the last one, and scrolls add up. Frames go out at JPEG quality 85, where text shows no ringing.

…s the page The pane's mouse, keys, text, resize and toolbar waited up to 30 seconds on Chrome and, on any failure, dropped the browser as an agent action does. An input event dispatched while a navigation swaps the document can go unanswered, so one click on a link froze the pane for 30 seconds and then closed it. The window's calls now wait 2 seconds for input and leave the browser running unless its process is gone.

A link with target=_blank, like GitHub's About link, opened a second tab in the headless browser, which put the pane's page in the background: there it takes no input and draws no frames, so every click after timed out. The page now watches the tabs it opens, closes each one once it has a URL, and loads that URL in itself.

browser_open said the browser was headless unless the user opened Browser on the card, and the MCP instructions called it the Chromium window the card opens, so an agent that read headed:false told the user no window would appear and asked them to open one. The tool, the instructions in every prompt language, lich browser's help and docs/cli.md now say the page shows in a tab beside the session, which lich offers to open. The pt-BR pane says Navegador, as the card item does.

omartelo · 2026-10-26T23:57:42Z

@DiegoSGbr thank you for this. A browser the agent drives is something we wanted in lich for a while, and your implementation gave us the whole foundation: the per-session Chromium on the system browser, the browser_* tools and lich browser across providers, the URL allowlist, page info that never carries form values, and the provider gap written down. We are very grateful for it.

We pushed a few commits on top of your branch to fit it more tightly into the product. Your commit is untouched; ours come after it, plus a merge of main.

The browser now lives inside lich. Instead of a separate window, the card's Browser item opens a tab beside the session: its terminal on one side and the page on the other, using the same layout machinery as splits (divider, rotate, zoom). The page is the headless browser your service already runs, drawn from a CDP screencast, with the person's mouse, keys, input method text and paste passed back to it. Open in a window still moves it to a real Chromium window for DevTools and the like.

The agent and the person share the page. When the agent opens a browser on its own, a toast offers to show it, and the pane marks what the agent does (a chip, and a dot where it clicked). The tool descriptions and the MCP instructions now tell the agent about the tab, so it no longer asks the user to open a window.

Fixes found while testing:

The idle reaper closed a headed window someone was still using; it now only reaps headless browsers nobody watches.
Promoting a headless browser to a window kept its profile and cookies only by luck; it now reopens on the same profile and carries every cookie over.
Focus raises the window with Page.bringToFront; type --clear fires the input event frameworks listen to.
A slow Chrome launch no longer holds every other session's browser behind one lock, and close errors are logged instead of dropped.
A link with target="_blank" put the page in the background, where it takes no input; new tabs now load in the page itself, and Back returns to where you were.

Thanks again for the contribution. There is more we want to build on it, and this gives us a solid place to start.

The pane marked an agent's click with a small dot that was easy to miss. lich's own arrow now glides to each click, dips and rings, and fades two seconds after the agent's last action, with a badge naming the provider that clicked. It takes the session's card color, or the theme's primary for a card without one, outlined in primary-foreground so it reads over any page. The header's agent chip goes: the badge says who is acting.

An agent's scroll was window.scrollBy, with no point to show; it is now a wheel turn where the agent last acted, or the viewport's middle before it acted anywhere, which also scrolls what is under that point the way a person's wheel does. Typing into or pressing on an element reports the element it clicked first. browser-agent carries the action and a scroll's distance, and the pane's cursor glides to each of them: a click, a type or a press dips and rings, a scroll slides two chevrons in its direction.

…ndimmed The cursor left two seconds after each action, so between two tool calls of one turn it vanished and came back, reading as a stutter. It now stays while the agent's turn runs and five seconds after its last action or the turn's end. The page pane is no longer dimmed out of focus as a terminal is: the session's prompt keeps the focus nearly always, and the person watches the page beside it.

An agent that calls tools in parallel sends a click and a scroll within the same moment. The store kept only a session's last action, so a click followed by a scroll between two frames was never drawn, and one that was drawn had its ring cut short by the next glide. The store now keeps a session's latest actions and the cursor plays each one whole, in order, keeping the latest three when the agent outruns it.

…ng an address An agent's back, forward, reload and navigation changed the page with nothing to show for it. browser-agent now names each of them, with the address a navigation went to, and the pane's cursor plays them on its toolbar: it glides to the button and presses it, the button lit as a person's click lights it, or to the address field, where the address is typed in. The cursor is drawn over the whole pane rather than the page.

The page pane took the terminal's paste chord, Ctrl+Shift+V, so Ctrl+V sent the key to the headless browser, which pasted its own empty clipboard; and a copy in the page stayed in that clipboard too. Ctrl+V (Cmd+V on macOS) now types the system clipboard's text into the page, and Ctrl+C or Ctrl+X reads the page's selection through browser.Selection and writes it to the system clipboard, before the key reaches the page so a cut still has its text. A password field gives nothing to copy, as Chrome's own copy does.

…th every action browser_info answered with JSON, its element list 93% of a read: every element's keys and its box, four numbers an agent clicking by index has no use for. A page now reads as lines, one per element (index, tag, label, link), with links on the page's own origin cut to their path: half the text on a session's real reads. browser_click, browser_type and browser_press, and lich browser's click, type and press, answer with the page they left, read once a navigation they started has its document, so an agent no longer follows each action with a read of its own. A read that fails after an action that went through keeps the browser.

Typing into a field seldom changes the page, and answering each one with the page was a fifth of the page text a long session read. browser_type and lich browser type answer ok; browser_press Enter, which submits, still answers with the page, and browser_info reads it when typing did change it.

…id below it Typing into a form scrolls the field into view and can push its save button below the page's bottom; a click on the button, by index or selector, was sent at its box there, outside the viewport, where it hit nothing and carried the pane's cursor off the pane. An element out of view is now scrolled to the middle before its click, as Playwright does; a click at x and y outside the viewport is refused with the viewport's size; and the cursor stays inside the page whatever it is told.

An agent filling a form types into a field every half second, and the cursor played each action whole, eight tenths of a second, so it fell seconds behind and glided to fields the page had since scrolled away from. An action now plays at four tenths of its time while more wait behind it, and only the last one plays whole.

An agent filling a form types into a field every half second, about when the cursor finishes a click's glide, and the cursor waited out each ring, eight tenths of a second, before the next: it fell further behind with every field. A click's ring now fades on where it was while the cursor glides to the next action, which it may do once the click has dipped. TestAgentCursorPlaysParallelActionsInTurn changes with that contract: the scroll after a click waits for the click's dip, not its ring, and the ring is still drawn.

…Chrome is alive Any agent action that failed dropped the browser, as if the page had died: a click on an index from an older read, or one that landed while the page was between documents, answered no element matches and closed the page the person was looking at, logged in and all. A failed call now drops the browser only when its Chrome process is gone, as the pane's own input already did. TestFailedInfoDropsTheBrowser now fails on a page whose Chrome died, and TestOpenDropsWhenReuseNavigateFails becomes TestOpenKeepsALiveBrowserWhenReuseNavigateFails: the contract changed for a navigation that fails on a live page. A missed target's error now tells the agent to read the page again.

…mpact page reads

omartelo · 2026-10-27T01:50:04Z

@DiegoSGbr another round on top of your branch. We spent the evening driving real sessions through the browser (a production web app, GitHub, long form flows) and reading the transcripts and logs after each one. Here is what that turned up and what changed, in 12 commits plus a merge of main.

The agent's cursor

The pane used to mark an agent's click with a small dot that was easy to miss. The agent now has lich's own cursor: an arrow in the session's card color, or the theme's primary for a card without one, outlined so it reads over any page, with a badge naming the provider (788ecfd2b).
It follows every action that has a place: clicks, typing into a field, key presses on an element, scrolls (shown with chevrons in the scroll's direction), and back, forward, reload and a navigation, played on the pane's toolbar with the address typed into the field (3493d82fb, 89cbed69a). For that, browser_scroll became a real wheel turn where the agent last acted instead of window.scrollBy, which also scrolls the element under the pointer the way a person's wheel does.
Transcripts showed agents calling tools in parallel (a click, a scroll and a back in the same moment). The window kept only the last action, so clicks were never drawn; it now keeps the latest few and plays them in order (90455f1da).
A fast agent fills a field every half second, faster than a whole animation. The cursor first fell seconds behind and pointed at fields the page had scrolled away from; it now plays at a quicker pace while actions wait (ef3f3f771), and moves on once a click has dipped, leaving the ring to fade where it was (6adfad638).
It stays up while the agent's turn runs instead of vanishing between tool calls, and the page pane is no longer dimmed out of focus, since the prompt keeps the focus nearly always (3bba05375).

Token use
We measured sessions against the same app. browser_info was 93% element list in JSON, keys and boxes for every element, and agents followed every click with a read of their own.

Page reads are now one line per element (index, tag, label, link), half the text on the same real pages, and browser_click and browser_press answer with the page they left once a navigation they started has its document (68872e42c). browser_type answers ok, since filling a field seldom changes the page (672bc8cee).
Context growth per page action went from about 4,700 tokens to about 1,800 to 2,400, with roughly a third of the API calls per action.

Bugs found while testing

Ctrl+V pasted the headless browser's own, empty clipboard, and a copy never left it. The pane now pastes the system clipboard's text and copies the page's selection through a new browser.Selection; a password field gives nothing (5d4d2f66c).
Typing into a long form scrolls its save button below the fold, and a click on it by index or selector went to its box outside the viewport, hitting nothing. The element is now scrolled into view first, as Playwright does, and a click at x and y outside the viewport is refused with the viewport's size (1c9f8bfce).
Any failed agent action closed the browser: a click on an index from an older read, or one sent while the page was between documents, answered "no element matches" and took the page and its login with it. A failed call now drops the browser only when its Chrome is gone, and that error tells the agent to read the page again (29e71c81a).

Each fix came with a test that reproduced it first, against a real Chrome where the behavior lives there. The CHANGELOG.md entry and the PR description now describe the cursor, the clipboard and the compact reads. Thanks again for the foundation; it held up well under all of this.

…tests Ubuntu 24.04 runners restrict unprivileged user namespaces through AppArmor, and Chromium aborts with "No usable sandbox!" there, failing every internal/browser live test on linux while macOS and Windows pass.

omartelo · 2026-10-27T02:06:49Z

@DiegoSGbr merged, thank you! This started as your PR and it is now lich's embedded browser: every session gets a browser its agent drives, shown right beside it in the window. Thanks for the idea, the groundwork and for letting us build on your branch. It ships in the next release.

DiegoSGbr · 2026-10-27T18:15:01Z

It’s very rewarding to see this evolution happening so quickly. Congratulations.

- A production build's one or two letter component names (the minifier's) no longer reach the tag or the agent; a development build's source path is cut at the project's src/ even under a folder named app/. - A refused send says why in the tray, not in a toast over the toolbar. - Ctrl+Shift+E and Ctrl+Enter are Cmd on macOS. - The wheel over the popover or the tray no longer scrolls the page under it. - The ring's number tab sits on its straight top edge, past the corner. - docs/ceilings.md: the pane copies the page's selection since #941, and a Chromium without an AppArmor profile does not start on Ubuntu 23.10+.

* feat(browser): leave notes on page elements for the agent A crosshair on the session browser's toolbar stops the pointer reaching the page and outlines the element under it, named in a tag. A click opens a note beside it, in the review's comment box; each note keeps a numbered tab on a dashed ring around its element, following it as the page scrolls. Notes wait in a tray floating over the page's bottom, so the page never resizes, and Send types them into the session as one message: label, page, a selector browser_click takes, the React component in a development build, the markup without field values, and the note. A send that fails keeps the notes. The page only answers what is where (browser.ElementAt, browser.Locate); the outline, tabs, popover and tray are drawn by the window, so nothing is injected into the page's DOM. * fix(browser): tidy the page notes' loose ends - A production build's one or two letter component names (the minifier's) no longer reach the tag or the agent; a development build's source path is cut at the project's src/ even under a folder named app/. - A refused send says why in the tray, not in a toast over the toolbar. - Ctrl+Shift+E and Ctrl+Enter are Cmd on macOS. - The wheel over the popover or the tray no longer scrolls the page under it. - The ring's number tab sits on its straight top edge, past the corner. - docs/ceilings.md: the pane copies the page's selection since #941, and a Chromium without an AppArmor profile does not start on Ubuntu 23.10+. * perf(browser): march the note ring at 20 fps and keep it still under the popover Every frame of the marching dashes redraws the whole window: with a note open and idle that held 31 to 42% of a core, and a lit tray row 38 to 62% (measured 2026-10-27). The ring under an open note is now still, and the march elsewhere repaints 20 times a second at the same speed.

DiegoSGbr mentioned this pull request Oct 25, 2026

feat: open a session browser the agent can drive #585

Closed

3 of 6 tasks

omartelo added 11 commits October 26, 2026 19:16

docs(browser): the session browser is a tab of frames beside its session

d0b3a13

test(browser): back returns to the page a new-tab link left

61f9329

Merge remote-tracking branch 'origin/main' into browser-tab-lich

a6fbc79

omartelo added 14 commits October 26, 2026 21:15

docs(changelog): the session browser's agent cursor, clipboard and co…

0fda3ed

…mpact page reads

Merge remote-tracking branch 'origin/main' into browser-tab-lich

a191e9a

omartelo merged commit ee26f09 into lichdotdev:main Oct 27, 2026
7 checks passed

omartelo mentioned this pull request Oct 27, 2026

docs(site): bring the tour up to the native window, and add the browser #946

Merged

4 tasks done

Navigation Menu

Repository navigation

feat(browser): open a session browser the agent drives - #941

feat(browser): open a session browser the agent drives#941
omartelo merged 27 commits into
lichdotdev:mainfrom
DiegoSGbr:browser-tab-lich

DiegoSGbr commented Oct 24, 2026 •

edited by omartelo

Loading

Uh oh!

omartelo commented Oct 26, 2026

Uh oh!

omartelo commented Oct 27, 2026

Uh oh!

Uh oh!

omartelo commented Oct 27, 2026

Uh oh!

DiegoSGbr commented Oct 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

DiegoSGbr commented Oct 24, 2026 • edited by omartelo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

omartelo commented Oct 26, 2026

Uh oh!

omartelo commented Oct 27, 2026

Uh oh!

Uh oh!

omartelo commented Oct 27, 2026

Uh oh!

DiegoSGbr commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DiegoSGbr commented Oct 24, 2026 •

edited by omartelo

Loading

DiegoSGbr commented Oct 27, 2026 •

edited

Loading