Persistent Browser Profiles and Session Management (user_data_dir) for Stateful Web Tasks

[0x-Professor]

Summary
Introduce first-class support for persistent browser profiles via Playwright’s user data directories, enabling session reuse (cookies, localStorage) across runs. This reduces repeated logins, stabilizes authenticated flows, and lowers LLM/automation overhead. The feature wires existing environment variables and BrowserConfig parameters through to Playwright launch options and provides a simple profile-management interface.

User Story
As a user running Web-Agent on authenticated websites, I want my browser session to persist across runs so that I don’t have to re-login or re-bootstrap state every time, enabling faster, more reliable, and cheaper automated workflows.

Problem / Pain

• Today, sessions are not persisted by default; repeated runs must re-authenticate or rebuild state, increasing friction and cost.
• Evidence of intended support exists but is not wired end-to-end:
  • app.py reads environment variables but passes None to BrowserConfig:
    • app.py L9–16: reads GOOGLE_API_KEY, BROWSER_INSTANCE_DIR, USER_DATA_DIR
    • app.py L14: BrowserConfig(browser='edge', browser_instance_dir=None, user_data_dir=None, headless=False)
  • This suggests a natural integration point in src/agent/web/browser/config.py (not shown here) and Playwright’s launch settings to use userDataDir profiles.
• README shows interactive usage and demos, but doesn’t describe session persistence or profile reuse; authenticated use cases (e.g., posting to X/Twitter) would benefit significantly from stateful runs.

Proposed Solution

• Behavior
  • Add opt-in persistent profiles via a new BrowserConfig option user_data_dir (and optional profile name), defaulting to current behavior (no persistence).
  • If user_data_dir is set, Playwright launches the browser with that path so cookies/storage persist across runs.
  • Provide a simple profile name → directory mapping, e.g., user_data_dir = .profiles/{profile_name}.
  • Respect env vars if present:
    • USER_DATA_DIR for the absolute or relative path to store browser state.
    • BROWSER_INSTANCE_DIR for per-browser instance data if the project separates instance storage and user data.
• API/CLI/Config sketch
  • BrowserConfig fields:
    • browser: str
    • headless: bool
    • user_data_dir: Optional[pathlib.Path or str] // new or fully enabled
    • browser_instance_dir: Optional[pathlib.Path or str]
    • profile_name: Optional[str] // convenience alias that maps to user_data_dir = .profiles/{profile_name}
  • app.py wiring:
    • Use os.getenv('USER_DATA_DIR') and os.getenv('BROWSER_INSTANCE_DIR') when not provided programmatically.
  • Error handling and edge cases:
    • If the directory does not exist, create it with secure permissions.
    • If the path is not writable, log a clear error and fall back to ephemeral session.
    • Cross-browser behavior: if a non-Chromium browser does not support userDataDir semantics, document limitations and degrade gracefully.
  • Compatibility strategy:
    • Default remains ephemeral (no user_data_dir) to avoid breaking changes.
    • Persistence is opt-in via config or env.
• Security and privacy:
  • Warn users that cookies/tokens stored in the profile are sensitive; document filesystem permissions and rotation/cleanup guidance.
  • Provide an explicit “reset profile” flag/function to delete stored state.

Feasibility & Integration Points

• app.py
  • L9–16: env variables GOOGLE_API_KEY, BROWSER_INSTANCE_DIR, USER_DATA_DIR already read.
  • L14: BrowserConfig instantiated with None; wire these through to config.
• src/agent/web/browser/config.py
  • Add/confirm fields for user_data_dir, profile_name, and browser_instance_dir.
  • Ensure the Playwright launcher respects user_data_dir (Chromium/Edge support userDataDir).
• Playwright integration
  • Use userDataDir when launching the context/browser for Chromium/Edge; document browser differences.
• Backwards compatibility
  • Defaults unchanged; opt-in only. No breaking API changes required if fields are optional.

Quality Considerations

• Security: Persistent profiles store credentials/tokens. Document secure storage locations and permissions; provide a reset mechanism.
• Performance: Negligible overhead; profile reuse reduces login-flow latency and LLM/tool calls.
• Reliability: Fewer flaky login steps; more deterministic flows.
• Accessibility: Not applicable (no UI change).
• i18n: Not applicable.
• Observability: Log which profile path is used and whether persistence is enabled/disabled.
• Maintainability: Small, well-contained changes within BrowserConfig and Playwright launch path; easy to test.

Related Issues/PRs
#16 — Proposes scenario recording/replay for cost efficiency. Different focus: that feature replays action sequences; this proposal enables session persistence via Playwright profiles. They are complementary; session persistence reduces repeated auth and state rebuild, while recording addresses deterministic task replay.

Risks & Mitigations

• Risk: Storing sensitive session data on disk → Mitigation: clear security guidance, optional feature, reset/cleanup commands, and recommend secure filesystem permissions.
• Risk: Browser discrepancies in userDataDir behavior → Mitigation: prefer Chromium/Edge, document limitations, fallback to ephemeral for unsupported combos.
• Risk: Path misconfiguration causes failures → Mitigation: robust validation, auto-create directories, clear logs, safe fallback to ephemeral mode.