-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(integrations): add Flue integration #2774
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
antonvishal
wants to merge
4
commits into
browserbase:main
Choose a base branch
from
antonvishal:flue-facade
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
8470a87
feat(integrations): add Flue integration with native tools
antonvishal cd3b493
fix(integrations): update Flue documentation and enforce strict input…
antonvishal 0ef5885
Merge origin/main into flue-facade
antonvishal 942ad1e
docs(integrations): list flue in the package overview
antonvishal File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| --- | ||
| title: "Flue" | ||
| description: "Give a Flue agent native Stagehand browser tools backed by one persistent browser session." | ||
| --- | ||
|
|
||
| The Stagehand integration binds `run`, `snapshot`, and `screenshot` as native in-process [Flue](https://flueframework.com/) tools backed by one browser session, without starting an MCP server or bridge process. | ||
|
|
||
| <Note> | ||
| Stagehand ships this experimental integration from the repository rather than publishing it as a standalone adapter. | ||
| </Note> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Node.js 24 or newer | ||
| - pnpm 11.10.0 | ||
| - A model-provider credential for Flue | ||
| - A current Google Chrome installation for local browser mode | ||
|
|
||
| ## Quickstart | ||
|
|
||
| <Steps> | ||
| <Step title="Clone and build Stagehand"> | ||
| ```bash | ||
| git clone https://github.com/browserbase/stagehand.git | ||
| cd stagehand | ||
| pnpm install --frozen-lockfile | ||
| pnpm exec turbo run build \ | ||
| --filter @browserbasehq/stagehand-integrations | ||
| ``` | ||
| </Step> | ||
| <Step title="Configure the Flue model"> | ||
| The example uses an OpenAI model by default: | ||
|
|
||
| ```bash | ||
| export OPENAI_API_KEY="your-openai-api-key" | ||
| ``` | ||
|
|
||
| Set `FLUE_STAGEHAND_MODEL` to select another Flue model. | ||
| </Step> | ||
| <Step title="Choose the browser"> | ||
| The example defaults to Browserbase when `BROWSERBASE_API_KEY` is set, otherwise it uses local Chrome: | ||
|
|
||
| ```bash | ||
| export STAGEHAND_BROWSER="browserbase" | ||
| export BROWSERBASE_API_KEY="your-browserbase-api-key" | ||
| ``` | ||
| </Step> | ||
| <Step title="Run a browser task"> | ||
| ```bash | ||
| pnpm --filter @browserbasehq/stagehand-integrations-example-flue-facade start \ | ||
| "Open https://example.com and report the page title." | ||
| ``` | ||
|
|
||
| Flue uses the `run` tool and should report `Example Domain` before the process exits. | ||
| </Step> | ||
| </Steps> | ||
|
|
||
| ## Configuration | ||
|
|
||
| | Variable | Purpose | | ||
| | --- | --- | | ||
| | `FLUE_STAGEHAND_MODEL` | Flue agent model. Defaults to `openai/gpt-5.6-luna`. | | ||
| | `OPENAI_API_KEY` | Credential for the example's default Flue model. | | ||
| | `STAGEHAND_BROWSER` | Select `local` or `browserbase`. Defaults to `browserbase` when `BROWSERBASE_API_KEY` is set, otherwise `local`. | | ||
| | `BROWSERBASE_API_KEY` | Required for Browserbase. | | ||
| | `BROWSERBASE_PROJECT_ID` | Optional Browserbase project ID. | | ||
| | `STAGEHAND_MODEL_NAME` | Optional. The facade tools are deterministic and do not call a Stagehand model today. | | ||
|
antonvishal marked this conversation as resolved.
|
||
| | `STAGEHAND_MODEL_API_KEY` | Optional credential for `STAGEHAND_MODEL_NAME`; the native integration can infer a supported provider key. | | ||
|
|
||
| ## Screenshot behavior | ||
|
|
||
| Flue's current tool loop is text-only. This integration saves each screenshot to a temporary file and returns the path to the agent. Open the returned path to inspect the image. | ||
|
|
||
| ## Session lifecycle | ||
|
|
||
| One Flue CLI run owns one lazily-created browser session. All three tools reuse it, so navigation, authentication, and snapshot IDs survive between calls. The browser closes when the run finishes or receives an interrupt. | ||
|
|
||
| <Warning> | ||
| `run` executes model-authored JavaScript in the browser. Use Browserbase for untrusted tasks and review the [integration security boundary](/v4/integrations/overview#security-boundary). | ||
| </Warning> | ||
|
|
||
| <Card title="Flue integration source" icon="github" href="https://github.com/browserbase/stagehand/tree/main/packages/integrations/flue"> | ||
| Read the native tool bindings and browser lifecycle implementation. | ||
| </Card> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # Flue + Stagehand facade (native tools) | ||
|
|
||
| This example gives a Flue agent the native tools `run`, `snapshot`, and `screenshot`. The tools | ||
| share a Stagehand session directly; no MCP connection or bridge process is required. | ||
|
|
||
| ## Setup | ||
|
|
||
| Use Node.js 24 or later. From the repository root, build the integrations package before running | ||
| the example: | ||
|
|
||
| ```bash | ||
| pnpm exec turbo run build --filter @browserbasehq/stagehand-integrations | ||
| ``` | ||
|
|
||
| Configure the environment as needed: | ||
|
|
||
| | Variable | Purpose | | ||
| | ------------------------- | ------------------------------------------------------------------------------------------------ | | ||
| | `STAGEHAND_BROWSER` | Browser backend. Defaults to `browserbase` when `BROWSERBASE_API_KEY` is set, otherwise `local`. | | ||
| | `BROWSERBASE_API_KEY` | Browserbase API key. | | ||
| | `STAGEHAND_MODEL_NAME` | Optional Stagehand model name, such as `openai/gpt-5.6-luna`. | | ||
|
antonvishal marked this conversation as resolved.
|
||
| | `STAGEHAND_MODEL_API_KEY` | Optional explicit API key for `STAGEHAND_MODEL_NAME`; otherwise a supported provider key is inferred. | | ||
| | `FLUE_STAGEHAND_MODEL` | Flue agent model; defaults to `openai/gpt-5.6-luna`. | | ||
| | `OPENAI_API_KEY` | Used by the Flue agent model in the host process. | | ||
|
|
||
| ## Run | ||
|
|
||
| ```bash | ||
| pnpm --filter @browserbasehq/stagehand-integrations-example-flue-facade test | ||
| pnpm --filter @browserbasehq/stagehand-integrations-example-flue-facade typecheck | ||
| pnpm --filter @browserbasehq/stagehand-integrations-example-flue-facade start \ | ||
| "Open https://example.com and report the page title." | ||
| ``` | ||
|
|
||
| ## Screenshots | ||
|
|
||
| Flue's current tool loop is text-only. This example saves each screenshot to a temporary file and | ||
| returns its path and MIME type to the agent; open the file to inspect it. | ||
|
|
||
| ## Security model | ||
|
|
||
| `run(code)` executes model-authored JavaScript in the extension service worker: | ||
| it runs browser-side, never in the host process. Browserbase is the | ||
| recommended isolation boundary. The Flue process holds only the browser session | ||
| handle; model-authored JavaScript does not execute inside the host process. | ||
|
|
||
| ## Session lifecycle | ||
|
|
||
| One CLI run owns one lazily-created browser session. All three tools reuse that browser, so page | ||
| state and snapshot IDs survive between tool calls. The browser closes when the Flue run finishes | ||
| or is interrupted. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| "use agent"; | ||
|
|
||
| import { useModel, useTool } from "@flue/runtime"; | ||
|
|
||
| import { FACADE_AGENT_INSTRUCTIONS } from "@browserbasehq/stagehand-integrations/facade"; | ||
|
|
||
| import runTool from "./tools/run.js"; | ||
| import screenshotTool from "./tools/screenshot.js"; | ||
| import snapshotTool from "./tools/snapshot.js"; | ||
|
|
||
| export const FLUE_STAGEHAND_INSTRUCTIONS = `${FACADE_AGENT_INSTRUCTIONS} | ||
|
|
||
| The screenshot tool returns a temporary local file path and MIME type. Report that path so the user can inspect the image.`; | ||
|
|
||
| export function StagehandAgent() { | ||
| useModel(process.env.FLUE_STAGEHAND_MODEL ?? "openai/gpt-5.6-luna"); | ||
| useTool(runTool); | ||
| useTool(snapshotTool); | ||
| useTool(screenshotTool); | ||
| return FLUE_STAGEHAND_INSTRUCTIONS; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| import { defineTool } from "@flue/runtime"; | ||
| import * as v from "valibot"; | ||
|
|
||
| import { | ||
| CodeModeRunInputSchema, | ||
| RUN_TOOL_DESCRIPTION, | ||
| } from "@browserbasehq/stagehand-integrations/facade"; | ||
|
|
||
| import { discardFacadeToolsIfUnhealthy, getFacadeTools } from "../../src/session.js"; | ||
|
|
||
| const runInput = v.strictObject({ | ||
| code: v.optional(v.pipe(v.string(), v.minLength(1))), | ||
| actions: v.optional(v.pipe(v.array(v.record(v.string(), v.unknown())), v.minLength(1))), | ||
| }); | ||
|
|
||
| export default defineTool({ | ||
| name: "run", | ||
| description: RUN_TOOL_DESCRIPTION, | ||
| input: runInput, | ||
| async run({ data }) { | ||
| const input = CodeModeRunInputSchema.parse(data); | ||
| const tools = await getFacadeTools(); | ||
| try { | ||
| const result = | ||
| input.code !== undefined | ||
| ? await tools.run(input.code) | ||
| : await tools.runActions(input.actions!); | ||
| return stringifyResult(result); | ||
| } catch (error) { | ||
| await discardFacadeToolsIfUnhealthy(tools); | ||
| throw error; | ||
|
antonvishal marked this conversation as resolved.
|
||
| } | ||
| }, | ||
| }); | ||
|
|
||
| function stringifyResult(value: unknown): string { | ||
|
antonvishal marked this conversation as resolved.
|
||
| if (typeof value === "string") return value; | ||
| try { | ||
| return JSON.stringify(value, null, 2) ?? String(value); | ||
| } catch { | ||
| return String(value); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| import { defineTool } from "@flue/runtime"; | ||
| import * as v from "valibot"; | ||
|
|
||
| import { | ||
| SCREENSHOT_TOOL_DESCRIPTION, | ||
| ScreenshotInputSchema, | ||
| } from "@browserbasehq/stagehand-integrations/facade"; | ||
|
|
||
| import { writeScreenshotArtifact } from "../../src/artifacts.js"; | ||
| import { discardFacadeToolsIfUnhealthy, getFacadeTools } from "../../src/session.js"; | ||
|
|
||
| export default defineTool({ | ||
| name: "screenshot", | ||
| description: SCREENSHOT_TOOL_DESCRIPTION, | ||
| input: v.strictObject({ | ||
| fullPage: v.optional(v.boolean()), | ||
| type: v.optional(v.picklist(["png", "jpeg"])), | ||
| quality: v.optional(v.pipe(v.number(), v.minValue(0), v.maxValue(100))), | ||
|
antonvishal marked this conversation as resolved.
|
||
| }), | ||
| async run({ data }) { | ||
| const input = ScreenshotInputSchema.parse(data); | ||
| const tools = await getFacadeTools(); | ||
| try { | ||
| const image = await tools.screenshot(input); | ||
| return { output: await writeScreenshotArtifact(image) }; | ||
| } catch (error) { | ||
| await discardFacadeToolsIfUnhealthy(tools); | ||
| throw error; | ||
| } | ||
| }, | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| import { defineTool } from "@flue/runtime"; | ||
| import * as v from "valibot"; | ||
|
|
||
| import { | ||
| SNAPSHOT_TOOL_DESCRIPTION, | ||
| SnapshotInputSchema, | ||
| } from "@browserbasehq/stagehand-integrations/facade"; | ||
|
|
||
| import { discardFacadeToolsIfUnhealthy, getFacadeTools } from "../../src/session.js"; | ||
|
|
||
| export default defineTool({ | ||
| name: "snapshot", | ||
| description: SNAPSHOT_TOOL_DESCRIPTION, | ||
| input: v.strictObject({ | ||
| includeIframes: v.optional(v.boolean()), | ||
| }), | ||
| async run({ data }) { | ||
| const input = SnapshotInputSchema.parse(data); | ||
| const tools = await getFacadeTools(); | ||
| try { | ||
| return await tools.snapshot(input); | ||
| } catch (error) { | ||
| await discardFacadeToolsIfUnhealthy(tools); | ||
| throw error; | ||
| } | ||
| }, | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| { | ||
| "name": "@browserbasehq/stagehand-integrations-example-flue-facade", | ||
| "version": "4.0.1", | ||
| "private": true, | ||
| "type": "module", | ||
| "scripts": { | ||
| "start": "tsx src/run-agent.ts", | ||
| "test": "vitest run", | ||
| "test:unit": "vitest run", | ||
| "typecheck": "tsc --noEmit" | ||
| }, | ||
| "dependencies": { | ||
| "@browserbasehq/stagehand": "workspace:*", | ||
| "@browserbasehq/stagehand-integrations": "workspace:*", | ||
| "@flue/runtime": "catalog:", | ||
| "dotenv": "catalog:", | ||
| "valibot": "catalog:" | ||
| }, | ||
| "devDependencies": { | ||
| "@earendil-works/pi-ai": "catalog:", | ||
| "@types/node": "catalog:", | ||
| "tsx": "catalog:", | ||
| "typescript": "catalog:", | ||
| "vitest": "catalog:" | ||
| }, | ||
| "engines": { | ||
| "node": ">=24" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| import { mkdtemp, writeFile } from "node:fs/promises"; | ||
| import os from "node:os"; | ||
| import path from "node:path"; | ||
|
|
||
| let artifactDirectory: string | undefined; | ||
| let screenshotNumber = 0; | ||
|
|
||
| export async function writeScreenshotArtifact(image: { | ||
| data: string; | ||
| mimeType: string; | ||
| }): Promise<{ path: string; mimeType: string }> { | ||
| artifactDirectory ??= await mkdtemp(path.join(os.tmpdir(), "stagehand-flue-")); | ||
| screenshotNumber += 1; | ||
| const extension = image.mimeType === "image/jpeg" ? "jpg" : "png"; | ||
| const screenshotPath = path.join( | ||
| artifactDirectory, | ||
| `screenshot-${screenshotNumber}.${extension}`, | ||
| ); | ||
| await writeFile(screenshotPath, Buffer.from(image.data, "base64")); | ||
| return { path: screenshotPath, mimeType: image.mimeType }; | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.