Cloud browsing via Browser Use: admin key setting, browser_use tool, live view in web UI - #706
Open
time-attack wants to merge 4 commits into
Open
Cloud browsing via Browser Use: admin key setting, browser_use tool, live view in web UI#706time-attack wants to merge 4 commits into
time-attack wants to merge 4 commits into
Conversation
…live view An org admin pastes a Browser Use Cloud API key into a new admin card (org-scoped, encrypted with the connector key, write-only, live-validated against the provider on save, Postgres-backed). When the key is set, agents on every harness gain a browser_use tool that submits the task to Browser Use's hosted agent (POST /api/v4/runs), polls the run to completion, and returns the final answer through the external-content security screen. The tool is offered only on non-read-only turns in all-internal conversations with an internal actor, and strict-posture approval gating applies like any other tool. While a run is active the tool re-emits its tool_call entry with the provider's interactive live-view URL (validated to an https browser-use.com origin); the web UI renders it as an open, collapsible, sandboxed iframe on the running tool row, collapsing to a plain row once the run settles so stale URLs never reload on replay. The timeline now merges a re-emitted tool_call into its existing row by callId (rows with a result still get a fresh row, preserving approval-retry semantics). Client robustness: caller aborts cancel the remote run even when the abort lands mid-poll; transient poll failures are tolerated (three consecutive failures cancel and rethrow); unexpected run statuses resolve as failed instead of spinning; runs are cancelled at a 15-minute wall clock. The create call deliberately ignores the caller signal so an abort during creation cannot orphan an uncancellable run. Deliberate scope choices: no per-scope ACL grants or admin-tunable spend ceiling yet (the org key is the opt-in, the wall clock is the cap); unattended trigger fires may use the tool, matching the browse skill; the cloud browser egresses from Browser Use's network outside org egress policy, which the admin card copy discloses. Also swept per fix-every-instance: the two bare catch-void blocks in slack/delivery.ts now route through swallow().
…actually draw While a turn streams, assistant messages render with work=null and in-flight activity appears only in the bottom live-work dock, so the iframe added to toolRow() was unreachable exactly when the browser was live. The dock now embeds the sandboxed live-view iframe under the Browsing line whenever the active tool call carries an https live view URL — open by default, with the dock's chevron toggling it. The toolRow branch stays for the paused-on- approval case, and settled rows still never load the dead URL.
…ndings
The turn's already-granted env secrets (keychain own+standing
materializations and org env-delivery service credentials — exactly the
values the sandbox env already receives, captured at the same two
injection points) become bindable for browser_use. The tool takes
secrets: [{env_key, domains}]; core resolves the value and sends it to
Browser Use as an inline secretBinding the cloud agent can type on the
allowed registrable hostnames but never read. Only env key names and
domains enter the tape; every binding is written to the audit log.
An ungranted env key fails fast naming the granted keys and pointing at
the keychain flow (mint a grant for your own credential, send an ask for
someone else's), so detection -> owner approval -> retry rides the
existing keychain machinery. Bound values are scrubbed out of run
results and provider error echoes (FastAPI 422 bodies quote the
submitted binding) before anything reaches the transcript, and domains
must be registrable hostnames — a bare TLD no longer pins a secret to
the whole of .com.
|
sina, this is sick. happy to help test the live Browser Use path before merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Admins can enable cloud browsing for agents by pasting a single Browser Use Cloud API key in the admin UI. When set, agents get a
browser_usetool that runs web tasks on Browser Use's hosted browser agent, and the person watching in the web UI sees a collapsible live view of the actual browser (interactive — they can take over, e.g. to complete a sign-in) embedded on the running tool row.browser-use-key, org-scoped): encrypted with the connector key, Postgres-backed, write-only, live-validated against the provider on save, explicit Clear button (no save-empty footgun). Reads back only a configured/not-configured flag derived from decryptability, so key-material rotation can't show "Configured" while agents silently have no tool.browser_usetool (all four harnesses, threaded throughHarnessTurnInputlikecredentialExecServices):POST /api/v4/runs, polls to completion, 15-minute wall clock with remote cancel, abort-safe (mid-poll aborts still cancel the paid run; creation ignores the caller signal so an abort can't orphan an uncancellable run), tolerant of transient poll failures, defensive on unexpected statuses. Completed and failed outputs route through the external-content security screen. Offered only on non-read-only, all-internal turns with an internal actor; strict-posture approval gating applies.tool_callwith the live-view URL (origin-validated tohttps://*.browser-use.comin core); the timeline merges same-callId re-emits into one row; the web UI renders an open collapsible sandboxed iframe (sandbox,referrerpolicy=no-referrer, https-only guard) while running, plain row once settled so stale URLs never reload on replay.Deliberate scope choices (called out for review)
Review
Reviewed pre-PR by an 8-angle fresh-context pass; all confirmed findings fixed, including: the tool never registering on the pi harness (tool list is built before
ref.currentexists — now threaded through harness options), unscreened failure text, deferredtool_callbreaking replay/turn-resume accounting, iframe sandbox/scheme hardening, admin "Configured" vs undecryptable divergence, and orphaned paid runs on mid-flight aborts.Testing
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.