From 0bd60121832844b07e8a8a6bad4fd6b1d6ac8424 Mon Sep 17 00:00:00 2001 From: Austin Kidwell Date: Thu, 10 Sep 2026 06:26:04 -0700 Subject: [PATCH 1/3] sync: council-automation Aug fixes + browser-bridge coverage honesty, LF-normalised MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings origin up to date with work that had been sitting in eight unpushed local commits, plus today's browser-bridge fix. Deliberately excludes two things those local commits carried. Content - council-automation: the 2026-08-22 session/auth/CDP-wedge fixes to council_browser, council_query, extended_research_runner, refresh_session and session_keeper, plus cdp_health.py and three new test modules. - commands/setup-autonomous-triage.md and patterns/AUTONOMOUS_TRIAGE_PATTERN.md. - browser-bridge: lib/browser-discovery.js (new), the coverage envelope and navigate-retarget disclosure in server.js, the deferred session_cleanup in lib/websocket-bridge.js, sessionCleanupGrace in lib/config.js, and test-coverage-honesty.js (18 tests, all passing). Excluded on purpose - Ten .bak-2026-08-22-* snapshots (~14k lines of stale duplicates). Every one still exists on disk in ~/.claude, git already holds the prior versions, and a public repo does not need more unreviewed copies of old code. - The CRLF flip. Local commit 0ec3912 rewrote 211 of 218 tracked text files from LF to CRLF without changing one line of content. Verified 2026-09-10: every text blob on origin/master is LF-only, so these files are staged as LF and the diff is content only. .gitattributes now pins that so the next automated sweep cannot repeat it. Also: .gitignore gains a **/__pycache__/ catch-all — the per-directory rules kept missing each new test package. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01F7yhQQARuYRqGTdJJZWvZH --- .gitattributes | 24 + .gitignore | 7 + commands/setup-autonomous-triage.md | 225 +++++++++ council-automation/cdp_health.py | 314 ++++++++++++ council-automation/council_browser.py | 303 +++++++++++- council-automation/council_query.py | 22 +- .../extended_research_runner.py | 32 +- council-automation/refresh_session.py | 87 +++- council-automation/session_keeper.py | 118 ++++- .../test_browser_busy_attribution.py | 182 +++++++ council-automation/test_cdp_health.py | 193 ++++++++ council-automation/test_session_auth_probe.py | 176 +++++++ .../browser-bridge/lib/browser-discovery.js | 462 ++++++++++++++++++ mcp-servers/browser-bridge/lib/config.js | 23 +- .../browser-bridge/lib/websocket-bridge.js | 48 +- mcp-servers/browser-bridge/server.js | 194 +++++++- .../browser-bridge/test-coverage-honesty.js | 240 +++++++++ patterns/AUTONOMOUS_TRIAGE_PATTERN.md | 258 ++++++++++ 18 files changed, 2859 insertions(+), 49 deletions(-) create mode 100644 .gitattributes create mode 100644 commands/setup-autonomous-triage.md create mode 100644 council-automation/cdp_health.py create mode 100644 council-automation/test_browser_busy_attribution.py create mode 100644 council-automation/test_cdp_health.py create mode 100644 council-automation/test_session_auth_probe.py create mode 100644 mcp-servers/browser-bridge/lib/browser-discovery.js create mode 100644 mcp-servers/browser-bridge/test-coverage-honesty.js create mode 100644 patterns/AUTONOMOUS_TRIAGE_PATTERN.md diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..bb7bd21 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,24 @@ +# Normalise line endings in the repository to LF. +# +# Why this file exists: this repo had no .gitattributes and the machine that +# feeds it runs core.autocrlf=true. On 2026-08-10 an automated commit sweep +# rewrote 211 of the 218 tracked text files from LF to CRLF in the index +# (commit 0ec3912, "+61050/-61050") without changing a single line of content. +# Verified 2026-09-10: every text blob on origin/master is LF-only, so this +# file is a no-op against the current tree and exists purely to stop the churn +# from happening again. Without it, the next sweep produces another +# whole-repository diff that hides real changes and breaks blame. +# +# text=auto: git decides per file, stores LF in the index, and still checks out +# CRLF on Windows because of core.autocrlf. Nothing about the local working +# copy changes. +* text=auto + +# Never touch binaries. +*.png binary +*.jpg binary +*.jpeg binary +*.gif binary +*.ico binary +*.zip binary +*.db binary diff --git a/.gitignore b/.gitignore index e87cad0..f2bf846 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ # Include root files !.gitignore +!.gitattributes !CLAUDE.md.example !LICENSE !NOTICE @@ -45,6 +46,12 @@ !health-check/** # Exclude generated/runtime files within tracked dirs +# Catch-all first: the per-directory rules below predate it and are kept for +# clarity, but each new test package kept landing an untracked __pycache__ in +# git status (council-automation/tests/__pycache__, 2026-09-10). One rule covers +# every future one. +**/__pycache__/ +**/.pytest_cache/ council-automation/__pycache__/ mcp-servers/browser-bridge/node_modules/ mcp-servers/browser-bridge/package-lock.json diff --git a/commands/setup-autonomous-triage.md b/commands/setup-autonomous-triage.md new file mode 100644 index 0000000..714177a --- /dev/null +++ b/commands/setup-autonomous-triage.md @@ -0,0 +1,225 @@ +# /setup-autonomous-triage — Install the autonomous triage loop in a host app + +Stand up the always-on error + inefficiency triage pattern from `patterns/AUTONOMOUS_TRIAGE_PATTERN.md` inside the current working directory's host app. Interactive: asks about the host app's stack, prereqs, and admin identity, then walks through the 9 phases in order with checkpoints between each. + +**Time budget:** ~1-2 hours for a Next.js + Prisma + Vercel host app (the reference stack). ~3-4 hours for a stack that needs adaptation (Python, Rails, Go). + +**Cost per host app after setup:** ~$0.25-$5.60/month depending on user count (see the pattern doc's cost model). + +**Non-goal:** this command does not deploy or admin-merge anything. It writes code, drops schema, wires env stubs, and hands off for you to review + push. + +--- + +## Prerequisites — the receiving Claude MUST verify these before starting + +Skim the host app before writing anything. Report back to the user which prereqs are satisfied and which will need stubs. + +- [ ] **Per-user audit event stream** — any of: a `audit_events`-shaped table, Sentry SDK with `beforeSend` hook, Segment/Posthog with per-user identifiers, or a custom client-side logger. +- [ ] **Cron scheduler** — Vercel Cron, GitHub Actions cron, Cloudflare Cron Triggers, AWS EventBridge, or an external pinger. +- [ ] **Object storage** — Vercel Blob (private), S3 (with signed URL support), R2, or writeable local disk. +- [ ] **Admin auth gate** — some way to check "is this request from an admin?" server-side. +- [ ] **`GITHUB_PAT` env** — a Personal Access Token with `repo` scope. This is the least portable prereq and is required. +- [ ] **`ANTHROPIC_API_KEY` env** — Claude API access. +- [ ] **Per-user notification abstraction** — a function or endpoint that takes `(userId, message)` and delivers to that specific user. Bell feeds, Slack DMs, Pushover with per-user keys, email — any single channel is enough. + +If any of these is missing, note the gap in your Phase 1 output and use `TODO(prereq: X)` stubs in the affected file. Do NOT halt setup. + +--- + +## Read this first — the pattern doc + +Before any file writes: read `patterns/AUTONOMOUS_TRIAGE_PATTERN.md` end-to-end. It's ~700 lines, ~30 minutes of reading. The command below assumes you have that context. + +--- + +## Phase 1 — Scope + adaptation planning + +Ask the user (via `AskUserQuestion` or similar): + +1. **Host app path** — confirm the working directory. Report what you see: framework, ORM, DB, cron system, blob system, auth. +2. **User count + rough error volume** — needed to set the initial per-user error cluster threshold. Under 20 users, use `COUNT >= 3`. 20-100 users, `COUNT >= 5`. Over 100, `COUNT >= 10`. +3. **Admin identity for the mirror** — the user_id whose bell will receive a copy of every notification (matches the reference-impl's `ADMIN_MIRROR_USER_ID` in `notify.ts`). +4. **Notification channel choice** — which per-user pipe to wire (bell + Web Push, or email, or Slack, or Pushover). The pattern is channel-agnostic; the setup wires one. +5. **Vercel Cron plan tier** — Hobby caps at once-per-day; sub-daily needs Pro. If Hobby, either upgrade or fall back to GitHub Actions cron. + +Report the adaptation matrix — copy the rows from `patterns/AUTONOMOUS_TRIAGE_PATTERN.md` §"Adaptation matrix" that apply, then confirm with the user before proceeding. + +--- + +## Phase 2 — Schema + +Drop `reference-impl/autonomous-triage/schema/triage_tables.prisma` (or its adapted equivalent) into the host app's schema. Two additive models: +- `triage_tickets` — one row per unique detected pattern; state machine `open → in_progress → resolved | skipped | archived`. +- `system_state` — key/value store for cron watermarks. + +**Adaptation checkpoints:** +- Prisma → Drizzle/TypeORM/SQLAlchemy: regenerate from the Prisma DSL. +- Postgres arrays → JSON if the target is MySQL or SQLite. +- Rename `pplx_response` if you want — it's the historical column name from the initial Perplexity gate design. Keeping it saves migration effort. + +**Deploy step:** the receiving Claude runs the schema migration but DOES NOT pass `--accept-data-loss` (or the ORM equivalent) without listing what would drop first. This is a hard rule — the reference-impl session accidentally dropped 14 legacy rows this way; don't repeat. + +**Confirm with the user before running the migration on a production DB.** + +--- + +## Phase 3 — Env vars + +Set two new env vars on the host's Prod + Preview: + +```bash +# Random 32-byte hex, guards /admin/triage mutation endpoints +TRIAGE_ADMIN_KEY= + +# Existing gh CLI token or a fresh PAT with `repo` scope +GITHUB_PAT= +``` + +Two more are typically already there: +- `ANTHROPIC_API_KEY` — from the host app's existing Claude usage +- `CRON_SECRET` — cron endpoint bearer auth + +If the host app already had a Perplexity API key set, ignore it — this pattern's runtime does NOT use Perplexity (see the pattern doc for the reasoning). Perplexity via `research_query` remains valuable at design time. + +--- + +## Phase 4 — Detection cron (auto-triage Stage A) + +Drop these files under the host app's route structure: +- `src/lib/triage/signals.ts` — the two SQL detectors + `computeFingerprint()`. +- `src/app/api/cron/auto-triage/route.ts` — Stage A only for this phase; Stages B and C are added in Phase 5-6. + +**Adaptation checkpoints:** +- Postgres INTERVAL syntax + regex functions (`~*`, `regexp_replace(...)`) — if the target DB is MySQL/SQLite, port each raw SQL block. +- The `audit_events` column list (`event_type`, `route`, `element_text`, `fetch_url`, `fetch_method`, `fetch_status`, `error_msg`, `user_id`, `session_id`, `occurred_at`) — if the host app's audit stream uses different column names, rewrite the queries. +- If the host app has no session concept in its audit stream, the rage-click detector doesn't work — omit `detectInefficiency` and ship error clustering only. +- Threshold: use the number picked in Phase 1. + +Register the cron entry in `vercel.json` (or the host's cron config): `*/15 * * * *`. `maxDuration = 30`. + +**Test manually:** +```bash +curl -H "Authorization: Bearer $CRON_SECRET" https://.vercel.app/api/cron/auto-triage +# Expect: { stageA: { opened, updated, errors } } +``` + +--- + +## Phase 5 — Sonnet gate + blast-radius classifier (Stage B) + +Drop: +- `src/lib/triage/sonnet-gate.ts` — Sonnet 4.5 with structured JSON output. Never throws. +- `src/lib/triage/blast-radius.ts` — pure static rule table. First-match strictness wins. +- `tests/unit/lib/triage/blast-radius.test.ts` — 19 unit tests. + +Wire Stage B into the cron. Refer to `reference-impl/autonomous-triage/src/app/api/cron/auto-triage/route.ts` for the exact structure — Stage B is a bounded loop (10 tickets/run) that runs classifyTicket, applies classifyBlastRadius as authoritative veto, persists, and skips-or-continues. + +**Adaptation checkpoints:** +- The blast-radius rule table's paths are Next.js/Prisma-shaped. Swap for the target stack — the STRUCTURE (four levels, first-match, authoritative-veto semantics) stays; the paths change. +- Sonnet call uses raw `fetch`; works in any JS runtime. Python port: `httpx` + `pydantic`. + +**Test the classifier:** +```bash +npx jest tests/unit/lib/triage/blast-radius.test.ts +# Expect: 19 passed +``` + +**Test the gate end-to-end:** manually insert a fake `triage_tickets` row, hit the cron, verify `pplx_response` populates and the ticket transitions correctly. + +--- + +## Phase 6 — Patch generator (Stage C) + +Drop: +- `src/lib/triage/patch-generator.ts` — GitHub API + Sonnet + Vercel Blob upload. + +**Hardcodes to extract at drop time:** +```typescript +const REPO_OWNER = 'intellegix'; // → env var HOST_GITHUB_OWNER +const REPO_NAME = 'ASR-PO-System-Enterprise'; // → env var HOST_GITHUB_REPO +``` + +Set both env vars on Prod + Preview. + +**Adaptation checkpoints:** +- Vercel Blob → S3/R2/local: rewrite the `put()` call. The dashboard's blob proxy (Phase 8) needs a matching `head()` equivalent. +- If the host app can't fetch from GitHub Contents API (private repo without PAT, etc.), the patch generator can't work. This is a hard-block on the pattern. + +Wire Stage C into the cron similarly to Stage B — bounded to 3 tickets/run. + +--- + +## Phase 7 — Admin dashboard + +Drop the four files: +- `src/app/(admin)/admin/triage/page.tsx` — MUI Table + row-expand + actions. +- `src/app/api/admin/triage/route.ts` — list + counts. +- `src/app/api/admin/triage/[id]/action/route.ts` — approve/skip/archive. +- `src/app/api/admin/triage/[id]/blob/route.ts` — admin-authed blob proxy. + +**Adaptation checkpoints:** +- Auth: replace `getServerSession(authOptions)` + `isAdmin(role)` with the host app's admin check. +- MUI + React Query: if the host uses Tailwind or another UI kit, port the page.tsx — API endpoints stay the same. +- Add a nav link to the admin sidebar. Reference-impl uses `HealingIcon` from `@mui/icons-material`. + +**Test:** sign in as admin → sidebar has `/admin/triage` → any in_progress ticket shows an Approve button that copies the apply command block to clipboard. + +--- + +## Phase 8 — Close-loop + cleanup crons + +Drop: +- `src/lib/triage/close-loop.ts` — GH PR poll, watermark, resolve-and-notify. +- `src/app/api/cron/triage-close-loop/route.ts` — every 5 min. +- `src/app/api/cron/triage-cleanup/route.ts` — nightly 03:00 UTC. + +**Adaptation checkpoints:** +- The close-loop cron calls `notifyUser(userId, {...})`. Wire this to the host app's per-user notification channel picked in Phase 1. If the host doesn't have one yet, wire a simple email fallback. +- Extract the GH owner/name hardcodes to env vars (same as Phase 6). +- Register both cron entries in the host's cron config. + +**Test the close-loop:** +```bash +curl -H "Authorization: Bearer $CRON_SECRET" https://.vercel.app/api/cron/triage-close-loop +# Expect: { priorWatermark, newWatermark, seen: 0, resolved: 0, ... } +``` + +--- + +## Phase 9 — Wrap + hand off + +Update the host app's `CLAUDE.md` with a note pointing at the new dashboard and describing the operational shape. Suggested paragraph: + +```markdown +## Autonomous triage + +- `/admin/triage` — human-in-loop review of Sonnet-drafted patches for errors + rage-click inefficiency detected in `audit_events`. +- Cron `*/15` → detect + gate; cron `*/5` → close-loop; cron nightly → cleanup. +- Approve action copies a `git apply` command sequence for local execution. +- Cost: ~$0.25-$1/mo baseline for internal-app scale (see patterns/AUTONOMOUS_TRIAGE_PATTERN.md). +``` + +Update the host app's project MEMORY.md (if it uses the toolkit's memory system) with a topic entry pointing at the pattern doc. + +Open a single PR per phase, OR a single wrapped PR — user's preference. Reference-impl was 6 PRs (#49-54) across 9 phases for clean review. + +Hand off to the user: "Everything's shipped. Watch `/admin/triage` for the first ticket." + +--- + +## Failure modes to expect + recover from + +- **Sonnet responds with prose instead of JSON.** The Zod validator returns `sonnet_parse_fail`. Ticket stays open, retried next cron. If persistent, the prompt drifted — check what the host app's error patterns look like in Sonnet's context. +- **GitHub API rate-limited.** Watermark still advances (so we don't reprocess). Next 5-min cycle picks up. If persistent, the PAT hit its limit — get a fresh one. +- **Vercel Blob unavailable.** Patch generation skips with `blob_upload_failed`. Ticket transitions to `skipped`. Retryable manually via the dashboard's re-run action (not built in MVP; add if this becomes common). +- **Ticket table growing unbounded.** Check the nightly cleanup cron is firing. If it's not — check `CRON_SECRET` and the cron config. +- **Admin never approves anything.** Check Pushover / notification setup; the batched end-of-run ping is a soft nudge, not a required signal. Consider a stronger reminder if tickets pile up. + +--- + +## Related toolkit resources + +- `patterns/AUTONOMOUS_TRIAGE_PATTERN.md` — architecture, rationale, adaptation matrix. +- `patterns/API_PATTERNS.md` — API design patterns; useful when adapting the cron endpoints. +- `patterns/SECURITY_CHECKLIST.md` — apply to the blob proxy + admin routes. +- `patterns/TESTING_PATTERNS.md` — the pattern's test suite is minimal (19 tests, blast-radius only); expand per host app's testing norms. diff --git a/council-automation/cdp_health.py b/council-automation/cdp_health.py new file mode 100644 index 0000000..a48ffbb --- /dev/null +++ b/council-automation/cdp_health.py @@ -0,0 +1,314 @@ +"""Pre-attach health check for the keeper Chrome's CDP endpoint. + +WHY THIS EXISTS (2026-08-22 outage, see PERPLEXITY-ACTIVATION-EVIDENCE-2026-08-22.md) +------------------------------------------------------------------------------------ +Playwright's ``chromium.connect_over_cdp()`` does not merely open a websocket. After +the browser socket connects it issues ``Target.setAutoAttach`` and then *initialises +every attached page target* (Runtime.enable, Page.enable, ...). A single page whose +renderer process has hung answers none of that, and Playwright waits on it forever — +the whole connect blocks until its 180 s timeout, even though the browser process +itself is perfectly healthy and ``/json/list`` responds instantly. + +On 2026-08-22 exactly one wedged ``https://www.perplexity.ai/`` tab in the keeper +Chrome took down the entire research pipeline for six hours: + + session_keeper.py -> connect_over_cdp times out (53 consecutive attempts) + council_browser.py -> connect_over_cdp times out, falls back to a temp profile, + whose cookies are stale, so the fallback fires the keeper + to refresh -- which fails the same way -- and the run ends + on a not-logged-in browser where the /research slash + command does not exist, reported as the wholly misleading + "Failed to activate research mode". + +The browser-process-level HTTP endpoints (``/json/list``, ``/json/close``) are served +by the *browser* process, not the renderer, so they keep working when a renderer is +wedged. That is the escape hatch this module uses: probe each page target on its own +websocket with a short budget, close the ones that do not answer, and only then let +Playwright attach. + +Safe to call unconditionally and on every run: when the endpoint is healthy it costs +one HTTP GET plus a few concurrent websocket round-trips (~1 s) and closes nothing. +""" +from __future__ import annotations + +import asyncio +import json +import urllib.request +from dataclasses import dataclass, field +from typing import Callable, Iterable + +# A hung renderer answers nothing; a healthy one answers in single-digit ms even +# under load. 5 s is ~50x the observed healthy latency and still 36x cheaper than +# the 180 s Playwright timeout it exists to prevent. +LIVENESS_TIMEOUT_S = 5.0 + +# Total budget for the whole sweep. Never let the health check itself become the +# thing that makes a run slow -- if it cannot finish in time, give up and let +# Playwright try, which is exactly the old behaviour. +SWEEP_BUDGET_S = 30.0 + +# Above this many page targets, also reap known-leaked junk tabs. Chrome slows +# measurably past a few dozen targets and the 2026-08-22 wedge was found at 43. +PAGE_COUNT_REAP_THRESHOLD = 8 + +# Junk that accumulates in the keeper Chrome and is never load-bearing: the +# Perplexity news feed, blank tabs, and pages left behind by article-reading runs. +# A real query result lives under /search/ and is deliberately NOT in this list. +_JUNK_URL_MARKERS = ("/discover", "about:blank", "chrome://newtab") + +# A page holding a live query result. Its renderer is the ONE most likely to be +# legitimately busy (deep-research streaming pins the JS main thread), so a single +# missed liveness probe is not evidence of a wedge. These get a second probe after +# a pause and are only closed if they miss both -- a renderer that is unreachable +# for ~20s straight will hang Playwright's attach anyway, so closing it is strictly +# better than the 180s stall it would otherwise cause. +_PROTECTED_URL_MARKERS = ("/search/",) + +# Gap between the two probes given to a protected page before declaring it wedged. +PROTECTED_REPROBE_DELAY_S = 3.0 + + +@dataclass +class CdpHealthReport: + """Outcome of one pre-attach sweep.""" + + reachable: bool = False + page_targets: int = 0 + hung_closed: list[str] = field(default_factory=list) + junk_closed: list[str] = field(default_factory=list) + errors: list[str] = field(default_factory=list) + skipped_reason: str | None = None + + @property + def closed_any(self) -> bool: + return bool(self.hung_closed or self.junk_closed) + + def summary(self) -> str: + """One-line, log-friendly summary. This is the diagnostic signal.""" + if not self.reachable: + return f"cdp_health unreachable reason={self.skipped_reason or 'unknown'}" + if self.skipped_reason: + return ( + f"cdp_health skipped reason={self.skipped_reason} " + f"pages={self.page_targets}" + ) + return ( + f"cdp_health pages={self.page_targets} " + f"hung_closed={len(self.hung_closed)} junk_closed={len(self.junk_closed)} " + f"errors={len(self.errors)}" + ) + + +def _http_get_json(url: str, timeout: float = 5.0) -> object: + with urllib.request.urlopen(url, timeout=timeout) as response: + return json.loads(response.read().decode("utf-8", errors="replace")) + + +def _http_get_text(url: str, timeout: float = 5.0) -> str: + with urllib.request.urlopen(url, timeout=timeout) as response: + return response.read().decode("utf-8", errors="replace") + + +def _is_protected(url: str) -> bool: + """True for pages that get a second chance before being declared wedged.""" + lowered = (url or "").lower() + return any(marker in lowered for marker in _PROTECTED_URL_MARKERS) + + +def _is_junk(url: str) -> bool: + lowered = (url or "").lower() + if not lowered: + return True + return any(marker in lowered for marker in _JUNK_URL_MARKERS) + + +async def _target_answers(ws_url: str, timeout: float = LIVENESS_TIMEOUT_S) -> bool: + """Return True if this target's renderer answers a trivial Runtime.evaluate. + + Connects to the *target's own* websocket rather than the browser socket, so a + hung renderer cannot stall the probe of any other target. + """ + import websockets + + try: + async with websockets.connect( + ws_url, max_size=None, ping_interval=None, open_timeout=timeout + ) as socket: + await socket.send( + json.dumps({"id": 1, "method": "Runtime.evaluate", + "params": {"expression": "1"}}) + ) + deadline = asyncio.get_running_loop().time() + timeout + while True: + remaining = deadline - asyncio.get_running_loop().time() + if remaining <= 0: + return False + raw = await asyncio.wait_for(socket.recv(), timeout=remaining) + message = json.loads(raw) + if message.get("id") == 1: + return "result" in message + except asyncio.TimeoutError: + return False + except Exception: + # A target that refuses a websocket (already gone, or a type that does not + # accept one) is not evidence of a wedge. Leave it alone. + return True + + +def _close_target(http_endpoint: str, target_id: str) -> bool: + """Close one target via the browser-process HTTP endpoint. + + Uses HTTP rather than CDP-over-websocket precisely because the browser process + still answers when a renderer is wedged. + """ + try: + _http_get_text(f"{http_endpoint.rstrip('/')}/json/close/{target_id}", timeout=5.0) + return True + except Exception: + return False + + +def _pick_keepers(alive: Iterable[dict]) -> set[str]: + """Target ids that must never be reaped as junk. + + Keeps the first Perplexity page seen (the keeper's home tab) so the browser is + never left with zero Perplexity pages, which is its own known failure mode. + """ + keepers: set[str] = set() + for target in alive: + url = (target.get("url") or "").lower() + if "perplexity.ai" in url and not _is_junk(url): + keepers.add(target["id"]) + break + return keepers + + +async def sweep_cdp_endpoint( + http_endpoint: str, + log: Callable[[str], None] | None = None, + reap_junk: bool = True, +) -> CdpHealthReport: + """Close wedged (and optionally leaked) page targets before Playwright attaches. + + Args: + http_endpoint: CDP HTTP base, e.g. ``http://127.0.0.1:9223``. + log: Optional single-argument logging callable. + reap_junk: Also close known-leaked junk tabs once the page count exceeds + ``PAGE_COUNT_REAP_THRESHOLD``. Never closes ``/search/`` result pages. + + Returns: + A :class:`CdpHealthReport`. Never raises — a health check that fails must + degrade to the previous behaviour, not break the run it is protecting. + """ + emit = log or (lambda _message: None) + report = CdpHealthReport() + + try: + targets = _http_get_json(f"{http_endpoint.rstrip('/')}/json/list", timeout=5.0) + except Exception as exc: + report.skipped_reason = f"{type(exc).__name__}: {exc}" + emit(report.summary()) + return report + + report.reachable = True + if not isinstance(targets, list): + report.skipped_reason = "malformed_json_list" + emit(report.summary()) + return report + + pages = [ + t for t in targets + if isinstance(t, dict) + and t.get("type") == "page" + and t.get("webSocketDebuggerUrl") + and t.get("id") + ] + report.page_targets = len(pages) + if not pages: + emit(report.summary()) + return report + + async def classify(target: dict) -> tuple[dict, bool]: + ws_url = target["webSocketDebuggerUrl"] + if await _target_answers(ws_url): + return target, True + # Second chance for a page that may simply be busy streaming a result. + if _is_protected(target.get("url", "")): + await asyncio.sleep(PROTECTED_REPROBE_DELAY_S) + return target, await _target_answers(ws_url) + return target, False + + try: + results = await asyncio.wait_for( + asyncio.gather(*(classify(t) for t in pages), return_exceptions=True), + timeout=SWEEP_BUDGET_S, + ) + except asyncio.TimeoutError: + report.skipped_reason = "sweep_budget_exceeded" + emit(report.summary()) + return report + except Exception as exc: + report.skipped_reason = f"{type(exc).__name__}: {exc}" + emit(report.summary()) + return report + + alive: list[dict] = [] + for outcome in results: + if isinstance(outcome, BaseException): + report.errors.append(f"{type(outcome).__name__}: {outcome}") + continue + target, responded = outcome + if responded: + alive.append(target) + continue + url = target.get("url", "") + if _close_target(http_endpoint, target["id"]): + report.hung_closed.append(url) + emit(f"cdp_health CLOSED hung page url={url[:100]}") + else: + report.errors.append(f"close_failed hung {url[:100]}") + + # Leaked-tab reaping. Only above the threshold, only junk, and always keep at + # least one Perplexity page so ensure_perplexity_tab's invariant still holds. + if reap_junk and len(alive) > PAGE_COUNT_REAP_THRESHOLD: + keepers = _pick_keepers(alive) + for target in alive: + if target["id"] in keepers: + continue + url = target.get("url", "") + if not _is_junk(url): + continue + if _close_target(http_endpoint, target["id"]): + report.junk_closed.append(url) + else: + report.errors.append(f"close_failed junk {url[:100]}") + if report.junk_closed: + emit(f"cdp_health CLOSED {len(report.junk_closed)} leaked junk tab(s)") + + emit(report.summary()) + return report + + +def sweep_cdp_endpoint_sync( + http_endpoint: str, + log: Callable[[str], None] | None = None, + reap_junk: bool = True, +) -> CdpHealthReport: + """Blocking wrapper for callers that are not already inside an event loop.""" + return asyncio.run(sweep_cdp_endpoint(http_endpoint, log=log, reap_junk=reap_junk)) + + +if __name__ == "__main__": # pragma: no cover - operational entry point + import sys + + endpoint = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:9223" + result = sweep_cdp_endpoint_sync(endpoint, log=print) + print(json.dumps({ + "reachable": result.reachable, + "page_targets": result.page_targets, + "hung_closed": result.hung_closed, + "junk_closed": result.junk_closed, + "errors": result.errors, + "skipped_reason": result.skipped_reason, + }, indent=2)) + sys.exit(0 if result.reachable else 1) diff --git a/council-automation/council_browser.py b/council-automation/council_browser.py index b30d0d7..3cfeffa 100644 --- a/council-automation/council_browser.py +++ b/council-automation/council_browser.py @@ -168,6 +168,38 @@ def _count_active(self) -> int: """Count active slot files (after cleanup).""" return len(list(self.sessions_dir.glob("slot-*.lock"))) + def _describe_holders(self) -> str: + """Describe every held slot as 'slot=N pid=P age=Ts alive=yes|no'. + + Purely diagnostic. This is what a BROWSER_BUSY message carries, so a + reader can tell real contention from a wedged holder without having + to reverse-engineer it from the source. + """ + parts: list[str] = [] + for slot in range(self.max_sessions): + slot_file = self.sessions_dir / f"slot-{slot}.lock" + if not slot_file.exists(): + continue + try: + fields = slot_file.read_text(encoding="utf-8").split() + holder_pid = int(fields[0]) + claimed_at = float(fields[1]) + except (OSError, ValueError, IndexError): + parts.append(f"slot={slot} pid=? age=? alive=?") + continue + # Same liveness probe _cleanup_stale uses; Windows os.kill can + # raise SystemError as well as OSError. + try: + os.kill(holder_pid, 0) + alive = "yes" + except (OSError, SystemError): + alive = "no" + age = max(0.0, time.time() - claimed_at) + parts.append( + f"slot={slot} pid={holder_pid} age={age:.0f}s alive={alive}" + ) + return ", ".join(parts) if parts else "none recorded" + def acquire(self, wait_timeout: float = SEMAPHORE_WAIT_TIMEOUT) -> int: """Acquire a named session slot. Waits up to wait_timeout seconds. @@ -199,7 +231,8 @@ def acquire(self, wait_timeout: float = SEMAPHORE_WAIT_TIMEOUT) -> int: if elapsed >= wait_timeout: raise BrowserBusyError( f"All {self.max_sessions} browser session slots are in use. " - f"Waited {wait_timeout}s. Wait for a session to finish or use --mode api." + f"Waited {wait_timeout}s. Holders: {self._describe_holders()}. " + f"Wait for a session to finish or use --mode api." ) time.sleep(1) @@ -293,6 +326,32 @@ def __exit__(self, *args): # ~/.claude/mcp-servers/browser-bridge/server.js. Don't grep-and-hunt. PERPLEXITY_COMMIT_KEY = "Space" +# Ceiling on chromium.connect_over_cdp(). Playwright's own default is 180s, and on +# 2026-08-22 a single hung renderer in the keeper Chrome made every attach pay that +# full 180s -- twice per run once the fallback fired the keeper too -- turning a +# 40s research run into a 5.5-minute failure and hiding the real cause. The +# pre-attach sweep in cdp_health.py removes the usual cause; this constant bounds +# the cost of any cause we have not seen yet. Fail fast, fall back, stay honest. +CDP_CONNECT_TIMEOUT_MS = 45_000 + + +# Optional per-run log file. Set by _init_artifact_dir() so the structured +# `activate_mode verify=... indicator=...` lines -- the single most useful +# diagnostic this runner emits -- survive after the process exits. Before +# 2026-08-22 they went to stderr only, which the MCP server discards on the +# success path and truncates to 300 chars on the error path, so a failed run +# left a run directory containing nothing at all. +_RUN_LOG: "object | None" = None + + +def _set_run_log(path) -> None: + """Open (append) the per-run log file. Best-effort; never raises.""" + global _RUN_LOG + try: + _RUN_LOG = open(path, "a", encoding="utf-8", errors="replace") + except Exception: + _RUN_LOG = None + def _log(msg: str) -> None: """Log to stderr (stdout reserved for JSON result). @@ -304,6 +363,12 @@ def _log(msg: str) -> None: hang pattern under concurrent /research-perplexity load. """ print(f" [browser] {msg}", file=sys.stderr, flush=True) + if _RUN_LOG is not None: + try: + _RUN_LOG.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} {msg}\n") + _RUN_LOG.flush() + except Exception: + pass def _load_selectors() -> dict: @@ -526,10 +591,20 @@ def _init_artifact_dir(self, query: str) -> None: self._artifact_dir = Path("~/.claude/council-logs/runs").expanduser() / run_id self._artifact_dir.mkdir(parents=True, exist_ok=True) self._artifact_count = 0 - - async def _save_artifact(self, page, label: str) -> None: - """Capture screenshot + HTML as forensic artifacts. Non-fatal, capped at 10.""" - if not self.save_artifacts or not self._artifact_dir: + # Persist this run's log next to its artifacts (see _set_run_log). + _set_run_log(self._artifact_dir / "run.log") + _log(f"run log: {self._artifact_dir / 'run.log'}") + + async def _save_artifact(self, page, label: str, force: bool = False) -> None: + """Capture screenshot + HTML as forensic artifacts. Non-fatal, capped at 10. + + `force=True` captures even when --save-artifacts was not passed. Every + FAILURE path sets it: queue runs never pass the flag, so on 2026-08-22 the + run directories existed but were empty and the next lane had nothing to + read but the error string. A screenshot of the failing page is the single + artifact that would have shown the sign-in wall immediately. + """ + if not (self.save_artifacts or force) or not self._artifact_dir: return if self._artifact_count >= 10: return @@ -928,8 +1003,30 @@ async def _start_via_cdp(self) -> bool: return False try: + # 2026-08-22: connect_over_cdp initialises EVERY attached page target, so + # one hung renderer blocks the whole attach for its full timeout. Reap + # unresponsive/leaked page targets over the browser-process HTTP API + # (which keeps answering when a renderer does not) before attaching. + try: + from cdp_health import sweep_cdp_endpoint + + health = await sweep_cdp_endpoint(endpoint, log=_log) + if health.closed_any: + _log( + "MONITOR-SIGNAL cdp_wedge_reaped " + f"hung={len(health.hung_closed)} junk={len(health.junk_closed)}" + ) + except Exception as sweep_error: + # A failed health check must never be the reason a run dies. + _log( + "cdp_health sweep skipped " + f"({type(sweep_error).__name__}: {sweep_error})" + ) + _log(f"Attaching to session_keeper via CDP at {endpoint} ...") - self._browser = await self.playwright.chromium.connect_over_cdp(endpoint) + self._browser = await self.playwright.chromium.connect_over_cdp( + endpoint, timeout=CDP_CONNECT_TIMEOUT_MS + ) contexts = self._browser.contexts if not contexts: _log("CDP attach: no contexts available (keeper not ready); fall back") @@ -1081,6 +1178,20 @@ async def _ensure_fresh_session(self, reason: str = "") -> None: succeeds. Because we CDP-attach to the same keeper Chrome that the keeper refreshes in-place, the live attached session picks up the new cookies. """ + # 2026-08-22: an interactive login (--save-session, the sole setter of + # use_persistent) must never be gated on session freshness. This guard + # aborts when critical cookies are expired -- which is precisely when a + # human needs to sign in -- so the one command that repairs a dead + # session refused to run *because* the session was dead, while every + # other failure path told the user to run it. The bail lives here rather + # than at the two call sites so a third caller cannot reintroduce it. + if getattr(self, "use_persistent", False): + _log( + f"Interactive login ({reason}): skipping the session-freshness " + f"guard. Stale cookies are the reason this run exists." + ) + return + freshness = self._check_session_freshness(self.session_path) if hasattr(self, "_query_inst"): self._query_inst["cookies_stale_critical"] = freshness.get("stale_critical", []) @@ -1341,6 +1452,48 @@ def _parse_cookie_string(cookie_str: str) -> list[dict]: }) return cookies + @staticmethod + async def _is_signed_in(page) -> bool | None: + """Whether this page's context is signed in to a Perplexity account. + + Returns True (signed in), False (definitively signed out), or None when it + could not be determined -- callers must not treat None as either answer. + + Two independent checks, cheapest-authoritative first: + 1. ``/api/auth/session`` -- next-auth returns ``{}`` for no session. This + is the same endpoint the site itself uses, so it cannot drift with a + UI redesign the way a selector can. + 2. The sign-in wall's provider buttons ("Continue with Google" etc.), + which only render when signed out. + """ + try: + payload = await page.evaluate( + """async () => { + const r = await fetch('/api/auth/session', {credentials: 'include'}); + if (!r.ok) return null; + const t = (await r.text()).trim(); + if (!t) return {}; + try { return JSON.parse(t); } catch (_) { return null; } + }""" + ) + if isinstance(payload, dict): + return bool(payload) + except Exception as auth_error: + _log(f"auth probe error={auth_error!r}") + + try: + wall = await page.evaluate( + """() => Array.from(document.querySelectorAll('button, a')) + .some(el => /continue with (google|apple|email)|single sign-on/i + .test((el.textContent || '')))""" + ) + if wall: + return False + except Exception as wall_error: + _log(f"sign-in-wall probe error={wall_error!r}") + + return None + async def validate_session(self) -> bool: """Check if we're logged in to Perplexity.""" page = await self.context.new_page() @@ -1354,12 +1507,32 @@ async def validate_session(self) -> bool: textarea = self.selectors.get("textarea", "#ask-input") try: await page.wait_for_selector(textarea, timeout=10000) - _log("Session valid: found input element") - return True except Exception: _log("Session invalid: input element not found (not logged in?)") - await self._save_artifact(page, "validate_failure") + await self._save_artifact(page, "validate_failure", force=True) return False + + # 2026-08-22: the composer renders for SIGNED-OUT visitors too, so the + # presence of #ask-input proves the page loaded, not that we are logged + # in. A signed-out session then fails several steps later as the wholly + # misleading "Failed to activate research mode", because slash commands + # only exist for an account. Ask Perplexity's own auth endpoint, and + # fall back to the sign-in wall's buttons if that call cannot be made. + signed_in = await self._is_signed_in(page) + if signed_in is False: + _log( + "Session invalid: SIGNED OUT (Perplexity auth endpoint reports no " + "session). Cookies are present but dead server-side -- a human " + "must sign in again; refreshing cookies cannot fix this." + ) + _log("MONITOR-SIGNAL perplexity_signed_out") + await self._save_artifact(page, "validate_signed_out", force=True) + return False + if signed_in is None: + _log("Session check inconclusive (auth probe unavailable); proceeding") + else: + _log("Session valid: signed in") + return True finally: await page.close() @@ -1517,6 +1690,24 @@ async def _verify_council_activation(self, page) -> bool: Tier 2: text-scan for 'Model council' (tolerates DOM drift). Both miss → SELECTOR_DRIFT_DETECTED, return False (was: optimistic True). """ + # Tier 0 (2026-07-22): aria-label + aria-pressed on the icon-only mode + # button (same Perplexity redesign that broke the research verifier). + try: + aria_found = await page.evaluate("""() => { + const els = document.querySelectorAll('[aria-label]'); + for (const el of els) { + const label = (el.getAttribute('aria-label') || '').trim().toLowerCase(); + if ((label === 'model council' || label === 'council') + && el.getAttribute('aria-pressed') === 'true') return true; + } + return false; + }""") + if aria_found: + _log("activate_mode verify=OK mode=council indicator=tier0_aria_pressed") + return True + except Exception as _e0: + _log(f"activate_mode verify=tier0_ERROR mode=council exception={_e0!r}") + # Tier 1: stable aria-label selector try: three_models = self.selectors.get("threeModelsDropdown", "button[aria-label='3 models']") @@ -1545,12 +1736,35 @@ async def _verify_council_activation(self, page) -> bool: async def _verify_research_activation(self, page) -> bool: """Verify Research mode activated via 2-tier selector cascade. + Tier 0 (2026-07-22): aria-label + aria-pressed on the toolbar mode + button. Perplexity moved the indicator from a TEXT pill to an + ICON-ONLY button (aria-label="Deep research", aria-pressed="true") + with EMPTY textContent, so the tier1/tier2 text scans below can no + longer see it and falsely report SELECTOR_DRIFT. Verified via live DOM + probe. aria-pressed is the reliable active-state discriminator. Tier 1: exact-text match for the activated mode pill ("Deep research" or "Research" exactly). Catches the canonical activated state. Tier 2: case-insensitive contains scan for 'deep research' or exact 'research'. Tolerates minor Perplexity UI tweaks. - Both miss → SELECTOR_DRIFT_DETECTED, return False (was: optimistic True). + All miss → SELECTOR_DRIFT_DETECTED, return False (was: optimistic True). """ + # Tier 0: aria-label + aria-pressed on the icon-only mode button + try: + aria_found = await page.evaluate("""() => { + const els = document.querySelectorAll('[aria-label]'); + for (const el of els) { + const label = (el.getAttribute('aria-label') || '').trim().toLowerCase(); + if ((label === 'deep research' || label === 'research') + && el.getAttribute('aria-pressed') === 'true') return true; + } + return false; + }""") + if aria_found: + _log("activate_mode verify=OK mode=research indicator=tier0_aria_pressed") + return True + except Exception as e: + _log(f"activate_mode verify=tier0_ERROR mode=research exception={e!r}") + # Tier 1: exact-text match on the toolbar mode pill try: primary_found = await page.evaluate("""() => { @@ -1591,10 +1805,28 @@ async def _verify_research_activation(self, page) -> bool: async def _verify_labs_activation(self, page) -> bool: """Verify Labs mode activated via 2-tier selector cascade. + Tier 0 (2026-07-22): aria-label + aria-pressed on the icon-only mode + button (same Perplexity redesign that broke the research verifier). Tier 1: exact-text 'Labs' on a toolbar pill. Tier 2: case-insensitive contains 'labs' (looser fallback). - Both miss → SELECTOR_DRIFT_DETECTED, return False (was: optimistic True). + All miss → SELECTOR_DRIFT_DETECTED, return False (was: optimistic True). """ + # Tier 0: aria-label + aria-pressed on the icon-only mode button + try: + aria_found = await page.evaluate("""() => { + const els = document.querySelectorAll('[aria-label]'); + for (const el of els) { + const label = (el.getAttribute('aria-label') || '').trim().toLowerCase(); + if (label === 'labs' && el.getAttribute('aria-pressed') === 'true') return true; + } + return false; + }""") + if aria_found: + _log("activate_mode verify=OK mode=labs indicator=tier0_aria_pressed") + return True + except Exception as e: + _log(f"activate_mode verify=tier0_ERROR mode=labs exception={e!r}") + # Tier 1: exact-text match on the toolbar mode pill try: primary_found = await page.evaluate("""() => { @@ -3096,11 +3328,13 @@ async def _run_impl(self, query: str) -> dict: if not await self.validate_session(): return { "error": "Session expired or not logged in. Run: python council_browser.py --save-session", + "code": "SESSION_SIGNED_OUT", "step": "validate", } else: return { "error": "Session expired or not logged in. Run: python council_browser.py --save-session", + "code": "SESSION_SIGNED_OUT", "step": "validate", } @@ -3118,6 +3352,26 @@ async def _run_impl(self, query: str) -> dict: ) await page.wait_for_timeout(2000) + # Force a desktop viewport on THIS page's own CDP session. + # The keeper Chrome (:9222) is shared with the /takeover phone + # bridge, which applies a mobile device-metrics override (~400px) + # for its screencast. At mobile width Perplexity collapses the + # composer's mode selector to an icon with no "Research"/"Deep + # research" text, so activate_mode's text verify fails with + # SELECTOR_DRIFT_DETECTED ("Failed to activate mode"). + # setDeviceMetricsOverride is per-session and last-writer-wins on + # the renderer, so setting it here makes the runner immune to + # whatever emulation the takeover left behind. (Diagnosed 2026-07-16.) + try: + _vp_cdp = await self.context.new_cdp_session(page) + await _vp_cdp.send("Emulation.setDeviceMetricsOverride", { + "width": 1440, "height": 900, + "deviceScaleFactor": 1, "mobile": False, + }) + _log("Forced desktop viewport 1440x900 (guard vs takeover mobile emulation)") + except Exception as _vp_e: + _log(f"Desktop viewport override failed (non-fatal): {_vp_e!r}") + # submit_lock was acquired BEFORE self.start() (above) so # Chrome launches ARE inside the lock — this prevents the # ProcessSingleton race between concurrent Claude sessions @@ -3132,8 +3386,27 @@ async def _run_impl(self, query: str) -> dict: # released only in the outer `finally`. _log(f"Activating {self.perplexity_mode} mode...") if not await self.activate_mode(page): - await self._save_artifact(page, "activate_failure") - return {"error": f"Failed to activate {self.perplexity_mode} mode", "step": "activate"} + await self._save_artifact(page, "activate_failure", force=True) + # Activation is the LAST step in a long chain, so it is where + # unrelated upstream faults surface. Say which browser we were + # on, whether we are actually signed in, and where the evidence + # is -- 2026-08-22 was six hours of chasing a selector that was + # never broken. The leading phrase is unchanged so anything + # matching on it keeps working. + signed_in = await self._is_signed_in(page) + diagnosis = ( + "perplexity_signed_out" if signed_in is False + else "signed_in_selector_drift" if signed_in is True + else "auth_state_unknown" + ) + return { + "error": f"Failed to activate {self.perplexity_mode} mode", + "step": "activate", + "diagnosis": diagnosis, + "attached": "cdp_keeper" if self._cdp_attached else "local_profile", + "url": page.url, + "artifact_dir": str(self._artifact_dir) if self._artifact_dir else None, + } _log(f"Submitting query: {query[:80]}...") await self.submit_query(page, query) @@ -3167,7 +3440,7 @@ async def _run_impl(self, query: str) -> dict: completed = await self.wait_for_completion(page, self.timeout) if not completed: _log("WARNING: Timed out waiting for completion, extracting partial results") - await self._save_artifact(page, "timeout") + await self._save_artifact(page, "timeout", force=True) _log("Extracting results...") results = await self.extract_results(page) @@ -3200,7 +3473,7 @@ async def _run_impl(self, query: str) -> dict: try: pages = self.context.pages if pages: - await self._save_artifact(pages[-1], "unhandled_exception") + await self._save_artifact(pages[-1], "unhandled_exception", force=True) except Exception: pass return { diff --git a/council-automation/council_query.py b/council-automation/council_query.py index 3d0941e..5640478 100644 --- a/council-automation/council_query.py +++ b/council-automation/council_query.py @@ -72,7 +72,7 @@ def _diag_asyncio_exception_handler(loop, context): sys.stderr.flush() # Apply the handler when the event loop is created. -_original_run = _asyncio_diag.run +_original_run = getattr(_asyncio_diag.run, "_council_original_run", _asyncio_diag.run) def _diag_asyncio_run(coro, *args, **kwargs): """Wrap asyncio.run to install our exception handler and surface failures.""" try: @@ -88,6 +88,7 @@ async def _runner(): print(f"[DIAG-ASYNCIO-RUN-RAISED] {type(e).__name__}: {e}", file=sys.stderr, flush=True) raise +_diag_asyncio_run._council_original_run = _original_run _asyncio_diag.run = _diag_asyncio_run import anthropic @@ -934,13 +935,28 @@ def format_synthesis_output(results: dict) -> str: error_msg = results["error"] code = results.get("code", "UNKNOWN") step = results.get("step", "unknown") + # 2026-08-22: this block used to spell out the literal token that the + # MCP bridge searches stdout for when deciding whether a run was + # "busy". Every failure of every kind -- including a signed-out + # account -- was therefore re-authored as "another browser session is + # active", and lanes deleted lock files for hours against a fault that + # had no lock in it. The notes below are written with slashes so they + # can never be confused with the structured Code field above, which is + # the only thing anything should match on. return ( f"# Research/Council Query FAILED\n\n" f"**Error:** {error_msg}\n" f"**Code:** {code}\n" f"**Step:** {step}\n\n" - f"If BROWSER_BUSY: another session is using Playwright. Wait ~2 min.\n" - f"If session expired: run `python council_browser.py --save-session`\n" + f"Read the Code field above; the notes below are keyed to it.\n" + f"- BROWSER/BUSY: another session is using Playwright. Wait ~2 min.\n" + f"- SESSION/STALE: Perplexity cookies are expired and auto-refresh " + f"could not renew them. Run `/cache-perplexity-session`, then retry. " + f"This aborts BEFORE submitting, so no queue slot was wasted — do not " + f"treat it as a transient failure to retry blindly.\n" + f"- SESSION/SIGNED/OUT: the account is signed out server-side. " + f"Refreshing cookies cannot fix it; a human must sign in " + f"interactively via `python council_browser.py --save-session`.\n" ) synthesis = results.get("synthesis", {}) diff --git a/council-automation/extended_research_runner.py b/council-automation/extended_research_runner.py index 19220d4..c93a7ef 100644 --- a/council-automation/extended_research_runner.py +++ b/council-automation/extended_research_runner.py @@ -2617,17 +2617,39 @@ def main() -> int: # detection, parser bug, malformed responses) abort the run instead of # burning the full pass budget. The caller can salvage raw responses from # salvaged-responses.md. - recent_statuses = [p.get("status") for p in ledger["pass_log"][-PARSE_FAIL_STREAK_THRESHOLD:]] + recent_passes = ledger["pass_log"][-PARSE_FAIL_STREAK_THRESHOLD:] + recent_statuses = [p.get("status") for p in recent_passes] if ( len(recent_statuses) >= PARSE_FAIL_STREAK_THRESHOLD and all(s == "PARSE-FAILED" for s in recent_statuses) ): + # 2026-08-22: this message used to GUESS ("expired session / Cloudflare / + # parser regression") while the runner already held the real reason in + # each pass record's own `error` field. On 2026-07-22 and again today + # that guess sent people to /cache-perplexity-session for a fault that + # had nothing to do with cookies, costing hours both times. Report what + # the runner was actually told, and only then offer the speculation. + underlying = [] + for record in recent_passes: + err = str(record.get("error") or "").strip() + if err and err not in underlying: + underlying.append(err[:200]) + observed = ( + " Underlying errors reported by the passes themselves: " + + "; ".join(underlying) + + "." + if underlying + else " The passes recorded no underlying error, which itself points at " + "a runner-side parser problem rather than an upstream failure." + ) termination_reason = ( f"PARSE-FAILED-STREAK ({PARSE_FAIL_STREAK_THRESHOLD} consecutive passes returned " - f"unparseable responses). Likely causes: expired Perplexity session — run " - f"`/cache-perplexity-session`; bot detection / Cloudflare lockout; or " - f"runner-side parser regression. Raw responses preserved in salvaged-responses.md " - f"for manual review." + f"unparseable responses)." + observed + + f" If those errors do not name a cause, the usual suspects are an expired " + f"Perplexity session (run `/cache-perplexity-session`), bot detection / " + f"Cloudflare lockout, or a runner-side parser regression — but check the " + f"per-run run.log under ~/.claude/council-logs/runs/ before assuming any of " + f"them. Raw responses preserved in salvaged-responses.md for manual review." ) log(termination_reason, "ERROR") break diff --git a/council-automation/refresh_session.py b/council-automation/refresh_session.py index 6ef7d1f..c0ba95c 100644 --- a/council-automation/refresh_session.py +++ b/council-automation/refresh_session.py @@ -42,9 +42,75 @@ def _log(msg: str) -> None: print(f"[refresh_session] {msg}", flush=True) +import datetime as _dt + +AUTH_SESSION_URL = "https://www.perplexity.ai/api/auth/session" + + +async def _ask_perplexity_whether_signed_in(page) -> bool | None: + """Ask Perplexity's own auth endpoint whether this session is signed in. + + next-auth returns an empty object for an anonymous visitor and a populated + user object for a real session, so this is authoritative and immune to UI + drift -- unlike looking for a composer element, which is rendered to signed + out visitors too. + + Returns: + True or False when the endpoint answers, None when it cannot be reached + or returns something unrecognised (caller decides what to do). + """ + try: + payload = await page.evaluate( + """async (url) => { + // Vary the URL, not just the cache mode: `cache: 'no-store'` + // only governs the browser's own HTTP cache, while a CDN edge + // cache or a service worker's Cache Storage keys on the URL and + // would happily replay a stale 200 for either. + const bust = url + (url.includes('?') ? '&' : '?') + '_cb=' + Date.now(); + const response = await fetch(bust, { + credentials: 'include', + cache: 'no-store', + }); + if (!response.ok) return null; + return await response.json(); + }""", + AUTH_SESSION_URL, + ) + except Exception as exc: # network error, navigation mid-flight, CSP, ... + _log(f"Auth endpoint unreachable ({type(exc).__name__}: {exc})") + return None + if not isinstance(payload, dict): + return None + if not payload: + return False + + # A non-empty body is not proof of a live session. next-auth computes + # `expires` from the JWT at issuance, so it can already be in the past. + expires = payload.get("expires") + if isinstance(expires, str): + try: + deadline = _dt.datetime.fromisoformat(expires.replace("Z", "+00:00")) + except ValueError: + _log(f"Auth endpoint returned an unparseable expires ({expires!r}); " + f"treating the session as UNVERIFIED rather than guessing.") + return None + if deadline <= _dt.datetime.now(_dt.timezone.utc): + _log(f"Auth endpoint returned a session that expired at {expires}.") + return False + return True + + async def _navigate_and_check_auth(context) -> tuple: """Navigate to Perplexity and check for logged-in state. + 2026-08-22: this used to return True as soon as it found '#ask-input'. That + element is present on the SIGNED OUT homepage too, so the check reported + "Auth confirmed" on a completely dead session, saved the dead cookies, and + every run afterwards failed with "session expired or not logged in". A + refresh that reports success while nothing works is worse than one that + fails, so the authoritative auth endpoint is asked first and the selector is + only a fallback for when that endpoint cannot be reached. + Returns (page, logged_in) tuple. """ page = await context.new_page() @@ -54,10 +120,29 @@ async def _navigate_and_check_auth(context) -> tuple: _log("Waiting for auth hydration...") await page.wait_for_timeout(3000) + signed_in = await _ask_perplexity_whether_signed_in(page) + if signed_in is True: + _log("Auth confirmed: Perplexity's auth endpoint reports an active session") + return page, True + if signed_in is False: + _log( + "Auth check FAILED: SIGNED OUT. Perplexity's auth endpoint reports no " + "session, so the stored cookies are dead server-side. Saving them again " + "cannot help -- a human must sign in interactively." + ) + _log("MONITOR-SIGNAL perplexity_signed_out") + return page, False + + # Endpoint indeterminate. Fall back to the old selector check, but say plainly + # that it is weak evidence rather than logging "Auth confirmed". for selector in ["#ask-input", "textarea[placeholder]", "[data-testid='ask-input']"]: try: await page.wait_for_selector(selector, timeout=10000) - _log(f"Auth confirmed: found '{selector}'") + _log( + f"Auth UNVERIFIED: auth endpoint unreachable; found '{selector}', which " + f"is also present when signed out. Proceeding, but do not treat this as " + f"proof the session is alive." + ) return page, True except Exception: continue diff --git a/council-automation/session_keeper.py b/council-automation/session_keeper.py index a42952e..9fb2801 100644 --- a/council-automation/session_keeper.py +++ b/council-automation/session_keeper.py @@ -320,6 +320,62 @@ async def _save_cookies_and_storage(context, page) -> int: return len(filtered) +# next-auth's session endpoint is the authoritative answer to "is this browser +# signed in": it returns the session object when there is one and a bare {} when +# there is not. Unlike a DOM selector it cannot drift when Perplexity restyles +# the page, and it cannot be fooled by UI that renders for signed-out visitors. +import datetime as _dt + +AUTH_SESSION_URL = "https://www.perplexity.ai/api/auth/session" + + +async def _ask_perplexity_whether_signed_in(page) -> bool | None: + """True/False if the auth endpoint answered, None if it could not be reached. + + None is deliberately distinct from False: "I could not tell" must never be + reported as "signed out", and must never be reported as "signed in" either. + """ + try: + payload = await page.evaluate( + """async (url) => { + // Vary the URL, not just the cache mode: `cache: 'no-store'` + // only governs the browser's own HTTP cache, while a CDN edge + // cache or a service worker's Cache Storage keys on the URL and + // would happily replay a stale 200 for either. + const bust = url + (url.includes('?') ? '&' : '?') + '_cb=' + Date.now(); + const response = await fetch(bust, { + credentials: 'include', + cache: 'no-store', + }); + if (!response.ok) return null; + return await response.json(); + }""", + AUTH_SESSION_URL, + ) + except Exception as exc: + _log(f"Auth endpoint unreachable ({type(exc).__name__}: {exc})") + return None + if not isinstance(payload, dict): + return None + if not payload: + return False + + # A non-empty body is not proof of a live session. next-auth computes + # `expires` from the JWT at issuance, so it can already be in the past. + expires = payload.get("expires") + if isinstance(expires, str): + try: + deadline = _dt.datetime.fromisoformat(expires.replace("Z", "+00:00")) + except ValueError: + _log(f"Auth endpoint returned an unparseable expires ({expires!r}); " + f"treating the session as UNVERIFIED rather than guessing.") + return None + if deadline <= _dt.datetime.now(_dt.timezone.utc): + _log(f"Auth endpoint returned a session that expired at {expires}.") + return False + return True + + async def _navigate_and_warm(page) -> bool: """Navigate to perplexity.ai, wait for auth hydration, return logged_in.""" try: @@ -328,9 +384,28 @@ async def _navigate_and_warm(page) -> bool: _log(f"Navigation failed: {e}") return False await page.wait_for_timeout(3000) + + # 2026-08-22: ask the server, not the page. This used to return True on the + # mere presence of `#ask-input`, which Perplexity renders to signed-out + # visitors -- so a dead session was certified healthy every cycle and its + # expired cookies were written over the saved jar. See the patch notes. + signed_in = await _ask_perplexity_whether_signed_in(page) + if signed_in is True: + return True + if signed_in is False: + _log("Auth check FAILED: SIGNED OUT server-side (/api/auth/session " + "returned an empty session). Not persisting cookies -- a human must " + "sign in; refreshing cannot revive a session the server has dropped.") + _log("MONITOR-SIGNAL perplexity_signed_out") + return False + + # Endpoint unreachable: fall back to the old selector probe, but say plainly + # that this is unverified rather than reporting a confirmed login. for selector in ("#ask-input", "textarea[placeholder]", "[data-testid='ask-input']"): try: await page.wait_for_selector(selector, timeout=5000) + _log(f"Auth UNVERIFIED: could not reach the auth endpoint; proceeding " + f"on the presence of {selector}, which also renders when signed out.") return True except Exception: continue @@ -484,7 +559,24 @@ async def main_loop(interval_s: int, cdp_port: int = DEFAULT_CDP_PORT) -> None: browser = None context = None try: - browser = await pw.chromium.connect_over_cdp(f"http://127.0.0.1:{cdp_port}") + # 2026-08-22: one hung renderer in this very Chrome made connect_over_cdp + # time out 54 times in a row over six hours, silently taking the whole + # research pipeline down -- and the queue monitor's remediation is to run + # THIS script, so without the sweep the auto-fix could never work either. + endpoint = f"http://127.0.0.1:{cdp_port}" + try: + from cdp_health import sweep_cdp_endpoint + + health = await sweep_cdp_endpoint(endpoint, log=_log) + if health.closed_any: + _log( + "reaped wedged/leaked targets before attach: " + f"hung={len(health.hung_closed)} junk={len(health.junk_closed)}" + ) + except Exception as sweep_error: + _log(f"cdp_health sweep skipped ({type(sweep_error).__name__}: {sweep_error})") + + browser = await pw.chromium.connect_over_cdp(endpoint, timeout=45_000) contexts = browser.contexts if not contexts: _log("ERROR: connect_over_cdp returned 0 contexts") @@ -492,19 +584,33 @@ async def main_loop(interval_s: int, cdp_port: int = DEFAULT_CDP_PORT) -> None: context = contexts[0] _log(f"Connected via CDP. {len(contexts)} context(s), {len(context.pages)} page(s)") - # Inject cookies from playwright-session.json into the default context. - await context.add_cookies(old_cookies) - _log(f"Injected {len(old_cookies)} cookies into CDP context") - # Use or open a page in the keeper context. if context.pages: page = context.pages[0] else: page = await context.new_page() - # Initial warm + cookie write. + # 2026-08-22: probe the LIVE context BEFORE injecting anything. + # This used to inject the saved jar first, unconditionally. When that jar + # was stale -- precisely the case this daemon exists to catch -- it + # overwrote a healthy live session with dead cookies, saw the browser go + # signed-out as a direct result, and persisted that. The keeper was the + # thing killing the session, in a loop, while logging success. + # Injection is a REPAIR, so it only makes sense once the live context has + # been shown to be broken. _log("Navigating to perplexity.ai (initial warm)...") logged_in = await _navigate_and_warm(page) + + if logged_in: + _log("Live context is already signed in — NOT injecting the saved jar; " + "it cannot be fresher than what the browser is already holding.") + else: + _log(f"Live context is not signed in — injecting {len(old_cookies)} saved " + f"cookies as a repair attempt, then re-checking.") + await context.add_cookies(old_cookies) + logged_in = await _navigate_and_warm(page) + _log("Repair from saved cookies " + ("succeeded" if logged_in else "failed")) + if not logged_in: _log("ERROR: not logged in — re-run `python council_browser.py --save-session`") sys.exit(1) diff --git a/council-automation/test_browser_busy_attribution.py b/council-automation/test_browser_busy_attribution.py new file mode 100644 index 0000000..cbe9de4 --- /dev/null +++ b/council-automation/test_browser_busy_attribution.py @@ -0,0 +1,182 @@ +"""Regression tests for the 2026-08-22 BROWSER_BUSY mis-attribution. + +Every research failure was reported to callers as BROWSER_BUSY, because the MCP +bridge searched the child's whole stdout for that literal token and +``format_synthesis_output`` printed it in an unconditional troubleshooting +block. The visible symptom -- "another browser council/research session is +active" -- reads as ordinary contention, so lanes deleted lock files for hours +against a fault that had no lock in it. + +These tests pin both halves: the runner must not emit the bare token for a +non-busy failure, and the bridge's own regex (read out of server.js, not +re-typed here) must key on the structured Code field. +""" +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +import pytest + +from council_browser import SessionSemaphore +from council_query import format_synthesis_output + +SERVER_JS = ( + pathlib.Path.home() / ".claude" / "mcp-servers" / "browser-bridge" / "server.js" +) + +# A PID that cannot be running. Windows PIDs are multiples of 4 and 0 is the +# System Idle Process, so this is never a live user process. +DEAD_PID = 999_999_999 + + +# --------------------------------------------------------------------------- +# The runner's side: do not print a token the bridge greps for +# --------------------------------------------------------------------------- +def test_signed_out_failure_does_not_emit_the_busy_token(): + """The exact failure that was misreported all of 2026-08-22.""" + out = format_synthesis_output( + { + "error": "Session expired or not logged in. Run: python council_browser.py --save-session", + "code": "SESSION_SIGNED_OUT", + "step": "validate", + } + ) + assert "BROWSER_BUSY" not in out, ( + "a signed-out failure must not contain the busy token anywhere -- the " + "bridge greps stdout for it" + ) + assert "**Code:** SESSION_SIGNED_OUT" in out + + +def test_every_non_busy_failure_is_free_of_the_busy_token(): + """Not just the signed-out case -- the guidance block is unconditional.""" + for code in ("SESSION_STALE", "UNKNOWN", "SELECTOR_DRIFT", None): + out = format_synthesis_output( + {"error": "something failed", "code": code, "step": "submit"} + ) + assert "BROWSER_BUSY" not in out, f"leaked the busy token for code={code}" + + +def test_a_real_busy_failure_still_announces_itself_in_the_code_field(): + out = format_synthesis_output( + { + "error": "All 8 browser session slots are in use.", + "code": "BROWSER_BUSY", + "step": "lock", + } + ) + assert "**Code:** BROWSER_BUSY" in out + + +# --------------------------------------------------------------------------- +# The bridge's side: exercise the regex that actually ships in server.js +# --------------------------------------------------------------------------- +def _bridge_says_busy(stdout: str) -> bool: + """Run server.js's own busy-detection regex against `stdout`. + + The pattern is extracted from server.js rather than re-typed, so this test + fails if someone loosens it back to a substring search. + """ + source = SERVER_JS.read_text(encoding="utf-8") + marker = "].test(result)) {" + line = next( + (ln for ln in source.splitlines() if ".test(result)) {" in ln), None + ) + assert line is not None, ( + "server.js no longer tests a regex against the child's stdout -- if it " + "went back to result.includes(...), that is the bug this file exists for" + ) + pattern = line.strip()[len("if ("):line.strip().rindex(".test(result)")] + script = ( + "const re = " + pattern + ";" + "const input = JSON.parse(process.argv[1]);" + "process.stdout.write(re.test(input) ? 'yes' : 'no');" + ) + result = subprocess.run( + ["node", "--eval", script, json.dumps(stdout)], + capture_output=True, + text=True, + check=True, + ) + return result.stdout.strip() == "yes" + + +@pytest.mark.skipif(not SERVER_JS.exists(), reason="browser-bridge not installed") +def test_bridge_does_not_call_a_signed_out_run_busy(): + stdout = format_synthesis_output( + { + "error": "Session expired or not logged in.", + "code": "SESSION_SIGNED_OUT", + "step": "validate", + } + ) + assert not _bridge_says_busy(stdout), ( + "the bridge reported a signed-out account as BROWSER_BUSY -- this is the " + "exact 2026-08-22 outage" + ) + + +@pytest.mark.skipif(not SERVER_JS.exists(), reason="browser-bridge not installed") +def test_bridge_still_recognises_a_genuine_busy_run(): + stdout = format_synthesis_output( + {"error": "All 8 slots in use.", "code": "BROWSER_BUSY", "step": "lock"} + ) + assert _bridge_says_busy(stdout), "real contention must still be reported as busy" + + +# --------------------------------------------------------------------------- +# Orphaned-lock reclaim, which the filing asked to have pinned +# --------------------------------------------------------------------------- +def test_slot_held_by_a_dead_pid_is_reclaimed(tmp_path: pathlib.Path): + """A crashed holder must not block the fleet forever.""" + semaphore = SessionSemaphore(max_sessions=1, sessions_dir=tmp_path) + (tmp_path / "slot-0.lock").write_text(f"{DEAD_PID} {time.time():.0f}\n", encoding="utf-8") + + slot = semaphore.acquire(wait_timeout=5) + + assert slot == 0 + holder = (tmp_path / "slot-0.lock").read_text(encoding="utf-8").split() + assert int(holder[0]) == os.getpid(), "the live process should now hold the slot" + semaphore.release() + + +def test_a_live_holder_is_never_evicted(tmp_path: pathlib.Path): + """The reclaim must not be so eager that it steals an in-flight run.""" + semaphore = SessionSemaphore(max_sessions=1, sessions_dir=tmp_path) + (tmp_path / "slot-0.lock").write_text( + f"{os.getpid()} {time.time():.0f}\n", encoding="utf-8" + ) + + with pytest.raises(Exception) as excinfo: + semaphore.acquire(wait_timeout=1) + + assert "in use" in str(excinfo.value) + + +def test_busy_message_names_the_holder_and_whether_it_is_alive(tmp_path: pathlib.Path): + """BROWSER_BUSY used to carry no diagnostic at all.""" + semaphore = SessionSemaphore(max_sessions=1, sessions_dir=tmp_path) + (tmp_path / "slot-0.lock").write_text( + f"{os.getpid()} {time.time():.0f}\n", encoding="utf-8" + ) + + with pytest.raises(Exception) as excinfo: + semaphore.acquire(wait_timeout=1) + + message = str(excinfo.value) + assert f"pid={os.getpid()}" in message + assert "alive=yes" in message + assert "age=" in message + + +def test_corrupt_slot_file_does_not_crash_the_holder_description(tmp_path: pathlib.Path): + semaphore = SessionSemaphore(max_sessions=2, sessions_dir=tmp_path) + (tmp_path / "slot-0.lock").write_text("not-a-pid\n", encoding="utf-8") + + described = semaphore._describe_holders() + + assert "slot=0" in described diff --git a/council-automation/test_cdp_health.py b/council-automation/test_cdp_health.py new file mode 100644 index 0000000..ac53c3c --- /dev/null +++ b/council-automation/test_cdp_health.py @@ -0,0 +1,193 @@ +"""Tests for cdp_health — the pre-attach CDP wedge reaper (2026-08-22). + +The property that matters most is NEGATIVE: this module runs on the hot path of +every research run, so it must never raise and must never close a page it has not +proved unresponsive. A bug here does not degrade research, it deletes work. +""" +from __future__ import annotations + +import asyncio + +import cdp_health + + +# -------------------------------------------------------------------------- +# URL classification +# -------------------------------------------------------------------------- +def test_junk_markers_match_the_tabs_that_actually_leaked(): + # These are the exact URLs reaped from the keeper Chrome on 2026-08-22. + assert cdp_health._is_junk("https://www.perplexity.ai/discover") + assert cdp_health._is_junk( + "https://www.perplexity.ai/discover/you/un-security-council-holds-seco-SuSl" + ) + assert cdp_health._is_junk("about:blank") + assert cdp_health._is_junk("") + + +def test_query_results_and_the_home_tab_are_never_junk(): + # Closing either of these would destroy a user's actual work. + assert not cdp_health._is_junk("https://www.perplexity.ai/search/abc-123") + assert not cdp_health._is_junk("https://www.perplexity.ai/") + + +def test_search_pages_are_protected_from_a_single_missed_probe(): + """A streaming deep-research page pins its JS thread; one miss is not a wedge.""" + assert cdp_health._is_protected("https://www.perplexity.ai/search/abc-123") + assert not cdp_health._is_protected("https://www.perplexity.ai/discover") + + +def test_pick_keepers_always_spares_one_perplexity_home_tab(): + alive = [ + {"id": "t1", "url": "https://www.perplexity.ai/discover"}, + {"id": "t2", "url": "https://www.perplexity.ai/"}, + {"id": "t3", "url": "https://www.perplexity.ai/"}, + ] + keepers = cdp_health._pick_keepers(alive) + assert keepers == {"t2"}, "must keep exactly the first non-junk Perplexity page" + + +def test_pick_keepers_with_no_perplexity_page_returns_empty(): + alive = [{"id": "t1", "url": "https://example.com/"}] + assert cdp_health._pick_keepers(alive) == set() + + +# -------------------------------------------------------------------------- +# Failure containment +# -------------------------------------------------------------------------- +def test_unreachable_endpoint_degrades_instead_of_raising(): + """If the health check itself fails, the run must proceed as it always did.""" + report = cdp_health.sweep_cdp_endpoint_sync("http://127.0.0.1:1") + assert report.reachable is False + assert report.closed_any is False + assert report.skipped_reason + assert "unreachable" in report.summary() + + +def test_malformed_json_list_is_not_treated_as_targets(monkeypatch): + monkeypatch.setattr(cdp_health, "_http_get_json", lambda *a, **k: {"not": "a list"}) + report = cdp_health.sweep_cdp_endpoint_sync("http://127.0.0.1:9223") + assert report.reachable is True + assert report.skipped_reason == "malformed_json_list" + assert report.closed_any is False + + +def test_healthy_endpoint_closes_nothing(monkeypatch): + targets = [ + {"type": "page", "id": "t1", "url": "https://www.perplexity.ai/", + "webSocketDebuggerUrl": "ws://x/1"}, + {"type": "page", "id": "t2", "url": "https://www.perplexity.ai/search/a", + "webSocketDebuggerUrl": "ws://x/2"}, + ] + closed: list[str] = [] + monkeypatch.setattr(cdp_health, "_http_get_json", lambda *a, **k: targets) + monkeypatch.setattr(cdp_health, "_close_target", + lambda ep, tid: closed.append(tid) or True) + + async def always_alive(ws_url, timeout=5.0): + return True + + monkeypatch.setattr(cdp_health, "_target_answers", always_alive) + report = cdp_health.sweep_cdp_endpoint_sync("http://127.0.0.1:9223") + assert report.page_targets == 2 + assert closed == [], "a healthy endpoint must not lose a single tab" + + +def test_hung_page_is_closed_and_named(monkeypatch): + targets = [ + {"type": "page", "id": "good", "url": "https://www.perplexity.ai/", + "webSocketDebuggerUrl": "ws://x/good"}, + {"type": "page", "id": "wedged", "url": "https://www.perplexity.ai/", + "webSocketDebuggerUrl": "ws://x/wedged"}, + ] + closed: list[str] = [] + monkeypatch.setattr(cdp_health, "_http_get_json", lambda *a, **k: targets) + monkeypatch.setattr(cdp_health, "_close_target", + lambda ep, tid: closed.append(tid) or True) + + async def one_wedge(ws_url, timeout=5.0): + return "wedged" not in ws_url + + monkeypatch.setattr(cdp_health, "_target_answers", one_wedge) + report = cdp_health.sweep_cdp_endpoint_sync("http://127.0.0.1:9223") + assert closed == ["wedged"] + assert report.hung_closed == ["https://www.perplexity.ai/"] + assert report.closed_any is True + + +def test_a_busy_search_page_that_recovers_is_not_closed(monkeypatch): + """The regression this guards: killing an in-flight deep-research query.""" + targets = [ + {"type": "page", "id": "busy", "url": "https://www.perplexity.ai/search/abc", + "webSocketDebuggerUrl": "ws://x/busy"}, + ] + closed: list[str] = [] + calls = {"n": 0} + monkeypatch.setattr(cdp_health, "_http_get_json", lambda *a, **k: targets) + monkeypatch.setattr(cdp_health, "_close_target", + lambda ep, tid: closed.append(tid) or True) + monkeypatch.setattr(cdp_health, "PROTECTED_REPROBE_DELAY_S", 0.01) + + async def slow_then_alive(ws_url, timeout=5.0): + calls["n"] += 1 + return calls["n"] > 1 # misses the first probe, answers the second + + monkeypatch.setattr(cdp_health, "_target_answers", slow_then_alive) + report = cdp_health.sweep_cdp_endpoint_sync("http://127.0.0.1:9223") + assert calls["n"] == 2, "a protected page must get a second probe" + assert closed == [], "a busy-but-alive query page must survive" + assert report.hung_closed == [] + + +def test_junk_is_only_reaped_above_the_threshold(monkeypatch): + """Below the threshold a few leaked tabs are harmless; do not touch them.""" + targets = [ + {"type": "page", "id": f"t{i}", "url": "https://www.perplexity.ai/discover", + "webSocketDebuggerUrl": f"ws://x/{i}"} + for i in range(3) + ] + closed: list[str] = [] + monkeypatch.setattr(cdp_health, "_http_get_json", lambda *a, **k: targets) + monkeypatch.setattr(cdp_health, "_close_target", + lambda ep, tid: closed.append(tid) or True) + + async def always_alive(ws_url, timeout=5.0): + return True + + monkeypatch.setattr(cdp_health, "_target_answers", always_alive) + report = cdp_health.sweep_cdp_endpoint_sync("http://127.0.0.1:9223") + assert closed == [] + assert report.junk_closed == [] + + +def test_target_answers_returns_false_when_the_renderer_never_replies(monkeypatch): + """The core discriminator: silence within the budget means wedged.""" + + class NeverReplies: + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + async def send(self, _payload): + return None + + async def recv(self): + await asyncio.sleep(10) # outlives the timeout + + import websockets + + monkeypatch.setattr(websockets, "connect", lambda *a, **k: NeverReplies()) + result = asyncio.run(cdp_health._target_answers("ws://x/1", timeout=0.05)) + assert result is False + + +def test_target_answers_is_forgiving_when_the_socket_itself_refuses(monkeypatch): + """A target that cannot take a websocket is not evidence of a wedge.""" + import websockets + + def boom(*a, **k): + raise OSError("connection refused") + + monkeypatch.setattr(websockets, "connect", boom) + assert asyncio.run(cdp_health._target_answers("ws://x/1", timeout=0.05)) is True diff --git a/council-automation/test_session_auth_probe.py b/council-automation/test_session_auth_probe.py new file mode 100644 index 0000000..e63f14b --- /dev/null +++ b/council-automation/test_session_auth_probe.py @@ -0,0 +1,176 @@ +"""Regression tests for the 2026-08-22 Perplexity session-keeper faults. + +Two distinct bugs are pinned here. + +1. The keeper injected the saved cookie jar into the live browser context BEFORE + checking whether that context was signed in. With a dead jar on disk that + overwrote a healthy session with expired cookies, observed the resulting + signed-out state, and saved it -- a loop that destroyed the session while + logging "Refresh complete" every cycle. Ordering is the whole fix, so the + ordering is what these tests assert. + +2. Auth was inferred from the presence of `#ask-input`, which Perplexity renders + to signed-out visitors. The probe now asks next-auth's session endpoint, and + must treat an empty body, a past `expires`, and an unreachable endpoint as + three different answers. +""" +from __future__ import annotations + +import asyncio +import datetime as dt +import pathlib +import re +import sys + +import pytest + +HERE = pathlib.Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) + +import session_keeper # noqa: E402 + + +# -------------------------------------------------------------------------- +# 1. Ordering: probe the live context before injecting anything into it. +# -------------------------------------------------------------------------- + +def _keeper_source() -> str: + return (HERE / "session_keeper.py").read_text(encoding="utf-8") + + +def test_saved_cookies_are_not_injected_before_the_auth_probe(): + """add_cookies must never run before the first _navigate_and_warm call. + + This is the exact fault: injecting a stale jar into a live context is + destructive, and it can only be judged safe after the context has been shown + to be signed out. + """ + source = _keeper_source() + first_warm = source.index("await _navigate_and_warm(page)") + injection = source.index("await context.add_cookies(old_cookies)") + assert injection > first_warm, ( + "session_keeper injects the saved cookie jar before probing the live " + "context. That overwrites a healthy session with whatever is on disk -- " + "the 2026-08-22 outage. Probe first; inject only to repair a context " + "already proven signed out." + ) + + +def test_injection_is_guarded_by_a_signed_out_branch(): + """The injection must sit on the failure branch, not run unconditionally.""" + source = _keeper_source() + injection = source.index("await context.add_cookies(old_cookies)") + window = source[max(0, injection - 600):injection] + assert re.search(r"if logged_in:", window), ( + "The cookie injection is no longer guarded by a signed-in check." + ) + + +def test_cookies_are_only_persisted_when_logged_in(): + """A failed auth check must abort before _save_cookies_and_storage runs.""" + source = _keeper_source() + abort = source.index("if not logged_in:") + save = source.index("await _save_cookies_and_storage(context, page)") + assert abort < save, ( + "The keeper can reach the cookie-save with logged_in False, which is how " + "a dead jar overwrote a good one." + ) + + +# -------------------------------------------------------------------------- +# 2. The auth probe itself. +# -------------------------------------------------------------------------- + +class FakePage: + """Minimal stand-in for a Playwright page whose evaluate() is scripted.""" + + def __init__(self, result=None, raises: Exception | None = None): + self._result = result + self._raises = raises + self.urls: list[str] = [] + + async def evaluate(self, _script, arg): + self.urls.append(arg) + if self._raises is not None: + raise self._raises + return self._result + + +def _probe(page): + return asyncio.run(session_keeper._ask_perplexity_whether_signed_in(page)) + + +def _iso(delta: dt.timedelta) -> str: + return (dt.datetime.now(dt.timezone.utc) + delta).isoformat().replace("+00:00", "Z") + + +def test_empty_session_object_is_signed_out(): + """next-auth answers {} for an anonymous visitor. That is a hard False.""" + assert _probe(FakePage({})) is False + + +def test_populated_unexpired_session_is_signed_in(): + assert _probe(FakePage({"user": {"id": "x"}, "expires": _iso(dt.timedelta(days=7))})) is True + + +def test_expired_session_is_signed_out_even_though_the_body_is_populated(): + """A JWT session carries an `expires` computed at issuance and can be past. + + Testing truthiness alone would call this signed in. + """ + assert _probe(FakePage({"user": {"id": "x"}, "expires": _iso(dt.timedelta(days=-1))})) is False + + +def test_unreachable_endpoint_is_unverified_not_signed_out(): + """None and False must stay distinct: 'I could not tell' is not 'signed out'. + + Conflating them would either discard a good cookie jar or certify a dead one. + """ + assert _probe(FakePage(raises=RuntimeError("net::ERR_CONNECTION_RESET"))) is None + + +def test_non_ok_response_is_unverified(): + assert _probe(FakePage(None)) is None + + +def test_unparseable_expires_is_unverified_rather_than_assumed_good(): + assert _probe(FakePage({"user": {"id": "x"}, "expires": "not-a-date"})) is None + + +def test_probe_url_is_cache_busted(): + """A CDN or service worker keys on URL, so the URL itself must vary. + + `cache: 'no-store'` alone only governs the browser's own HTTP cache. + """ + page = FakePage({}) + _probe(page) + source = _keeper_source() + assert "_cb=" in source and "Date.now()" in source, ( + "The auth probe no longer varies its URL, so a cached 200 could replay a " + "session that has since been revoked." + ) + assert "cache: 'no-store'" in source + + +def test_probe_targets_the_auth_endpoint_not_a_dom_selector(): + page = FakePage({}) + _probe(page) + assert page.urls == [session_keeper.AUTH_SESSION_URL] + + +def test_selector_probe_is_only_a_fallback(): + """The selector loop may remain, but only below the endpoint check. + + Those selectors render for signed-out visitors, which is what made them + useless as the primary signal. Anchor on the `for selector in (` loop rather + than on a bare selector string -- the selector names also appear in comments + explaining this very fault, and matching those would test nothing. + """ + source = _keeper_source() + endpoint_check = source.index("signed_in = await _ask_perplexity_whether_signed_in(page)") + selector_loop = source.index("for selector in (") + assert selector_loop > endpoint_check, ( + "The DOM selector loop is consulted before the authoritative endpoint " + "check, so a signed-out session can be certified healthy again." + ) + assert "#ask-input" in source[selector_loop:selector_loop + 200] diff --git a/mcp-servers/browser-bridge/lib/browser-discovery.js b/mcp-servers/browser-bridge/lib/browser-discovery.js new file mode 100644 index 0000000..8022c3c --- /dev/null +++ b/mcp-servers/browser-bridge/lib/browser-discovery.js @@ -0,0 +1,462 @@ +/** + * browser-discovery.js — best-effort census of Chromium-family browser instances + * running on this machine, so the bridge can DECLARE what it cannot see. + * + * Why this exists (2026-09-10): + * `browser_get_tabs` returned five tabs from the single Chrome the extension is + * installed in, with nothing in the response saying that a second Chrome was + * running. An agent read that silence as absence, told Austin his fantasy + * football league "must be on your phone", and started a phone-takeover relay. + * The tab was open the whole time in a second Chrome on --remote-debugging-port=9223. + * See STALE-TABS-AND-SINGLE-BROWSER-BLINDNESS-EVIDENCE-2026-09-10.md. + * + * Design constraints, in priority order: + * 1. DETECT AND DECLARE ONLY. This module never attaches to, drives, or reads + * pages from another browser. It answers "does another browser exist?" so the + * tool response can say so. Controlling another instance is a separate, + * explicitly opt-in decision that has NOT been made. + * 2. Never block the MCP stdio transport. Everything is async with hard + * deadlines; every failure degrades to "unknown", never to a throw. + * 3. Cheap on the hot path. One PowerShell spawn per TTL window, not per call. + * + * Windows notes: + * - `wmic.exe` is REMOVED in Windows 11 24H2 and later and is deliberately not + * used here. `Get-CimInstance Win32_Process` is the supported replacement. + * - Chrome is multi-process. A chrome.exe PID is NOT a browser instance; only + * the process whose command line has no `--type=` switch is a browser root. + */ + +import { spawn } from 'node:child_process'; +import http from 'node:http'; +import { readFile } from 'node:fs/promises'; +import { join } from 'node:path'; + +/** Chromium-family executables worth counting as "a browser instance". */ +const BROWSER_EXES = new Set(['chrome.exe', 'msedge.exe', 'chromium.exe', 'brave.exe']); + +/** Small fallback sweep for CDP endpoints whose port we could not read from a command line. */ +const FALLBACK_PORT_LO = 9222; +const FALLBACK_PORT_HI = 9235; + +export const DISCOVERY_TTL_MS = 15_000; + +const POWERSHELL = process.env.SystemRoot + ? `${process.env.SystemRoot}\\System32\\WindowsPowerShell\\v1.0\\powershell.exe` + : 'powershell.exe'; + +const CIM_SCRIPT = ` +$ErrorActionPreference = 'Stop' +Get-CimInstance -ClassName Win32_Process -Filter "Name='chrome.exe' OR Name='msedge.exe' OR Name='chromium.exe' OR Name='brave.exe'" | + Select-Object ProcessId, Name, CommandLine | + ConvertTo-Json -Compress -Depth 3 +`; + +let _cache = { expiresAt: 0, value: null }; +let _inFlight = null; + +/** + * Extract a Chromium switch value from a Windows command line. + * Handles `--k=v`, `--k v`, and quoted values containing spaces. Never split a + * Windows command line on whitespace — quoted profile paths break that. + * @param {string} commandLine + * @param {string} name switch name without leading dashes + * @returns {string|null} + */ +export function getSwitchValue(commandLine, name) { + if (!commandLine) return null; + const escaped = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + const re = new RegExp(`(?:^|\\s)--${escaped}(?:=|\\s+)(?:"((?:\\\\.|[^"])*)"|([^\\s]+))`, 'i'); + const m = commandLine.match(re); + if (!m) return null; + const raw = m[1] ?? m[2]; + return raw == null ? null : raw.replace(/\\"/g, '"'); +} + +/** + * True when a Chromium process is a BROWSER ROOT rather than a renderer/GPU/utility + * child. Chromium passes `--type=` to every child; the root has no `--type`. + * Heuristic, not a Chromium API contract. + * @param {{CommandLine?: string}} proc + */ +export function isBrowserRoot(proc) { + const cmd = proc?.CommandLine || ''; + if (!cmd) return false; // no command line readable → cannot classify, don't count it as a root + return !/\s--type(?:=|\s)/i.test(cmd); +} + +/** + * Enumerate Chromium-family processes with their command lines. + * Resolves to [] on any failure — this is best-effort telemetry, never a hard error. + * @param {{timeoutMs?: number}} [opts] + * @returns {Promise>} + */ +export function listChromiumProcesses({ timeoutMs = 2000 } = {}) { + if (process.platform !== 'win32') return Promise.resolve([]); + + return new Promise((resolve) => { + let stdout = ''; + let settled = false; + const finish = (v) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(v); + }; + + let child; + try { + child = spawn( + POWERSHELL, + ['-NoLogo', '-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Bypass', '-Command', CIM_SCRIPT], + { windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] }, + ); + } catch { + resolve([]); + return; + } + + const timer = setTimeout(() => { + try { child.kill(); } catch { /* already gone */ } + finish([]); + }, timeoutMs); + + child.stdout.setEncoding('utf8'); + child.stdout.on('data', (c) => { + stdout += c; + if (stdout.length > 4_000_000) { try { child.kill(); } catch { /* noop */ } } + }); + child.stderr.on('data', () => { /* discard */ }); + child.on('error', () => finish([])); + child.on('close', () => { + if (!stdout.trim()) return finish([]); + try { + const parsed = JSON.parse(stdout); + finish(Array.isArray(parsed) ? parsed : [parsed]); + } catch { + finish([]); + } + }); + }); +} + +/** + * Read a Chromium profile's DevToolsActivePort file. Line 1 is the live port, + * line 2 a browser-target GUID. The file can be STALE after a crash, so a port + * read from here is a hint that must still be probed before being believed. + * @param {string} userDataDir + * @returns {Promise} + */ +export async function readDevToolsActivePort(userDataDir) { + try { + const text = await readFile(join(userDataDir, 'DevToolsActivePort'), 'utf8'); + const first = text.replace(/^/, '').split(/\r?\n/)[0]; + const port = Number((first || '').trim()); + return Number.isInteger(port) && port >= 1 && port <= 65535 ? port : null; + } catch { + return null; + } +} + +/** + * Probe one loopback port for a Chromium DevTools HTTP endpoint. + * Host is pinned to 127.0.0.1 and the Host header set explicitly — Chrome's + * DNS-rebinding protection rejects unexpected Host headers on /json. + * @param {number} port + * @param {{deadlineMs?: number}} [opts] + * @returns {Promise<{port:number, browser:string}|null>} + */ +export function probeCdpPort(port, { deadlineMs = 300 } = {}) { + return new Promise((resolve) => { + let settled = false; + const finish = (v) => { if (!settled) { settled = true; clearTimeout(deadline); resolve(v); } }; + + const req = http.request( + { + host: '127.0.0.1', + port, + path: '/json/version', + method: 'GET', + headers: { Host: `127.0.0.1:${port}`, Accept: 'application/json' }, + agent: false, + timeout: deadlineMs, + }, + (res) => { + let body = ''; + res.setEncoding('utf8'); + res.on('data', (c) => { + body += c; + if (body.length > 128 * 1024) req.destroy(); + }); + res.on('end', () => { + if (res.statusCode !== 200) return finish(null); + try { + const v = JSON.parse(body); + const ws = v.webSocketDebuggerUrl; + const looksLikeCdp = typeof ws === 'string' + && /^ws:\/\/(?:127\.0\.0\.1|localhost):\d+\/devtools\/browser\//i.test(ws) + && typeof v.Browser === 'string' + && /(Chrome|Chromium|Edg|Brave)/i.test(v.Browser); + finish(looksLikeCdp ? { port, browser: v.Browser } : null); + } catch { + finish(null); + } + }); + }, + ); + + const deadline = setTimeout(() => req.destroy(), deadlineMs); + req.on('timeout', () => req.destroy()); + req.on('error', () => finish(null)); + req.end(); + }); +} + +/** Probe many ports with bounded concurrency. */ +async function probePorts(ports, concurrency = 5) { + const list = [...new Set(ports)].filter((p) => Number.isInteger(p) && p >= 1 && p <= 65535); + const found = []; + let cursor = 0; + const worker = async () => { + while (cursor < list.length) { + const port = list[cursor++]; + const hit = await probeCdpPort(port); + if (hit) found.push(hit); + } + }; + await Promise.all(Array.from({ length: Math.min(concurrency, list.length) }, worker)); + return found.sort((a, b) => a.port - b.port); +} + +/** + * Best-effort census of browser instances on this machine. + * + * Returns `{ known, browserRoots, cdpEndpoints, asOf }`. `known:false` means + * discovery failed or is unsupported here — callers MUST then report coverage as + * unknown rather than claiming completeness. + * + * @param {{force?: boolean, budgetMs?: number}} [opts] + */ +export async function getBrowserCensus({ force = false, budgetMs = 2500 } = {}) { + const now = Date.now(); + if (!force && _cache.value && now < _cache.expiresAt) return _cache.value; + if (_inFlight) return _inFlight; + + _inFlight = (async () => { + const census = { known: false, browserRoots: [], cdpEndpoints: [], asOf: new Date().toISOString(), note: null }; + try { + const procs = await listChromiumProcesses({ timeoutMs: Math.min(budgetMs, 2000) }); + if (procs.length === 0) { + census.note = process.platform === 'win32' + ? 'Process enumeration returned nothing (PowerShell/CIM unavailable, timed out, or blocked).' + : `Process enumeration is only implemented for win32; this host is ${process.platform}.`; + return census; + } + + census.known = true; + const candidatePorts = []; + + for (const p of procs) { + if (!BROWSER_EXES.has(String(p.Name || '').toLowerCase())) continue; + if (!isBrowserRoot(p)) continue; + const cmd = p.CommandLine || ''; + const userDataDir = getSwitchValue(cmd, 'user-data-dir'); + const rawPort = getSwitchValue(cmd, 'remote-debugging-port'); + const port = rawPort && /^\d+$/.test(rawPort) ? Number(rawPort) : null; + // --remote-debugging-port=0 means "pick an ephemeral port": the command + // line does not carry the real one, DevToolsActivePort does. + if (port) candidatePorts.push(port); + census.browserRoots.push({ + pid: Number(p.ProcessId), + exe: p.Name, + userDataDir: userDataDir || null, + declaredDebuggingPort: port, + usesDebuggingPipe: /\s--remote-debugging-pipe(?:\s|$)/i.test(cmd), + }); + } + + const dirPorts = await Promise.all( + census.browserRoots.filter((r) => r.userDataDir).map((r) => readDevToolsActivePort(r.userDataDir)), + ); + for (const p of dirPorts) if (p) candidatePorts.push(p); + + for (let p = FALLBACK_PORT_LO; p <= FALLBACK_PORT_HI; p += 1) candidatePorts.push(p); + + census.cdpEndpoints = await probePorts(candidatePorts); + + // Correlate a live endpoint back to the root that declared it. + for (const ep of census.cdpEndpoints) { + const root = census.browserRoots.find((r) => r.declaredDebuggingPort === ep.port); + if (root) { + ep.pid = root.pid; + ep.userDataDir = root.userDataDir; + } + } + } catch (err) { + census.known = false; + census.note = `Discovery failed: ${err.message}`; + } + return census; + })(); + + try { + const value = await _inFlight; + _cache = { expiresAt: Date.now() + DISCOVERY_TTL_MS, value }; + return value; + } finally { + _inFlight = null; + } +} + +/** + * How many browser extension clients are actually connected to the bridge. + * + * Almost every server.js process on this machine runs in RELAY mode — the first + * process to bind ws:8765 owns the browser clients and the rest forward to it. So + * a relay's own `bridge.browserClients` is EMPTY and using it would report + * "0 of 2 browsers queried" while happily returning that browser's tabs. Ask the + * primary's health endpoint instead, and fall back conservatively. + * + * @param {{localCount:number, tabsReturned:boolean, healthPort:number, deadlineMs?:number}} input + * @returns {Promise} + */ +export function getConnectedBrowserCount({ localCount, tabsReturned, healthPort, deadlineMs = 400 }) { + if (localCount > 0) return Promise.resolve(localCount); + + return new Promise((resolve) => { + let settled = false; + // A tab listing came back, so at least one browser client served it. + const fallback = () => resolve(tabsReturned ? 1 : 0); + const finish = (v) => { if (!settled) { settled = true; clearTimeout(deadline); resolve(v); } }; + + const req = http.request( + { host: '127.0.0.1', port: healthPort, path: '/health', method: 'GET', agent: false, timeout: deadlineMs }, + (res) => { + let body = ''; + res.setEncoding('utf8'); + res.on('data', (c) => { body += c; if (body.length > 256 * 1024) req.destroy(); }); + res.on('end', () => { + try { + const n = JSON.parse(body)?.bridge?.browserCount; + if (Number.isInteger(n) && n >= 0) return finish(n); + } catch { /* fall through */ } + if (!settled) { settled = true; clearTimeout(deadline); fallback(); } + }); + }, + ); + const deadline = setTimeout(() => req.destroy(), deadlineMs); + req.on('timeout', () => req.destroy()); + req.on('error', () => { if (!settled) { settled = true; clearTimeout(deadline); fallback(); } }); + req.end(); + }); +} + +/** Reset the TTL cache. Tests only. */ +export function _resetCensusCache() { + _cache = { expiresAt: 0, value: null }; + _inFlight = null; +} + +/** + * Build the coverage + freshness envelope for a tab listing. + * + * The shape and the wording here are deliberate, and follow the research pass run + * on 2026-09-10 (precedents: Elasticsearch `_shards`/`timed_out` partial results, + * GraphQL `data` + `errors`, the DNS TC bit, HTTP 206 range declaration): + * - `status` is an ENUM, not a boolean. COMPLETE / PARTIAL / SCOPED / UNAVAILABLE + * are four materially different states that `complete: false` conflates. + * - `negativeEvidence` states the INFERENCE the caller is permitted to draw, + * because that — not "results may be incomplete" — is what the incident got + * wrong. It names the invalid inference explicitly. + * - It is ALWAYS present, including in the ordinary single-browser case, so an + * agent can distinguish "the tool knows it saw everything" from "the tool + * simply did not mention its limits". + * - It is a SUCCESS result, never isError: the tabs that were observed are real + * and useful. isError is reserved for producing no observation at all. + * + * @param {{tabCount:number, connectedBrowserClients:number, census:Awaited>}} input + */ +export function buildCoverage({ tabCount, connectedBrowserClients, census }) { + const asOf = new Date().toISOString(); + const observedCount = connectedBrowserClients; + + if (!census || !census.known) { + return { + status: 'SCOPED', + negativeEvidence: 'UNKNOWN', + scope: 'the single browser client currently connected to this bridge', + summary: + 'This list covers only the one browser the bridge extension is connected to. ' + + 'This bridge could not determine whether other browsers are running on this machine, ' + + 'so a tab missing from this list is NOT evidence that the tab is not open somewhere else.', + observedCount, + detectedCount: null, + observedTabCount: tabCount, + unobserved: [], + discoveryNote: census?.note || 'Browser discovery unavailable.', + asOf, + }; + } + + const detectedCount = census.browserRoots.length; + const unobserved = []; + + // Every detected browser root beyond the ones actually connected is unobserved. + // We cannot map a specific root to the connected client, so this is expressed as + // a count plus per-instance detail, not as a claim about which one is which. + if (detectedCount > observedCount) { + for (const root of census.browserRoots) { + const endpoint = census.cdpEndpoints.find((e) => e.pid === root.pid); + unobserved.push({ + reasonCode: endpoint ? 'REACHABLE_VIA_CDP_NOT_CONNECTED' : 'NO_BRIDGE_EXTENSION_CONNECTION', + reason: endpoint + ? 'A separate browser instance with a reachable local DevTools endpoint. This bridge does not drive it.' + : 'A separate browser instance with no bridge extension connection and no reachable DevTools endpoint found.', + exe: root.exe, + pid: root.pid, + userDataDir: root.userDataDir, + cdpEndpoint: endpoint ? `http://127.0.0.1:${endpoint.port}` : null, + tabCount: null, + }); + } + } + + const partial = detectedCount > observedCount; + + return { + status: partial ? 'PARTIAL' : 'COMPLETE', + negativeEvidence: partial ? 'UNSAFE' : 'SAFE_WITHIN_DECLARED_SCOPE', + scope: partial + ? `${observedCount} of ${detectedCount} Chromium-family browser instances detected on this machine` + : 'all Chromium-family browser instances detected on this machine', + summary: partial + ? `Only ${observedCount} of ${detectedCount} browser instances running on this machine were queried. ` + + 'A tab missing from this list is NOT evidence that the tab is not open — it may be open in an ' + + 'instance this bridge cannot see. Check the cdpEndpoint values under coverage.unobserved ' + + '(GET /json/list) before concluding that a page is not open.' + : 'Every detected browser instance on this machine was queried. A tab missing from this list is ' + + 'genuinely not open in any browser this bridge could detect.', + observedCount, + detectedCount, + observedTabCount: tabCount, + unobserved, + ...(partial && { + unobservedNote: + 'The bridge cannot map its connected extension client back to a specific process, so exactly ' + + `${observedCount} of the ${unobserved.length} instances listed here IS the browser these tabs came from. ` + + 'The observedCount/detectedCount ratio is the reliable figure; treat the per-instance list as candidates.', + }), + asOf, + }; +} + +/** + * One-line, model-facing banner emitted ABOVE the JSON when a negative inference + * would be unsafe. Placement is the point: a caveat below a long array is read + * after the model has already anchored on the list. + * @param {ReturnType} coverage + * @returns {string|null} + */ +export function coverageBanner(coverage) { + if (!coverage || coverage.negativeEvidence === 'SAFE_WITHIN_DECLARED_SCOPE') return null; + return `RESULT STATUS: ${coverage.status} — NEGATIVE EVIDENCE ${coverage.negativeEvidence}.\n${coverage.summary}`; +} diff --git a/mcp-servers/browser-bridge/lib/config.js b/mcp-servers/browser-bridge/lib/config.js index 460e179..7d2919b 100644 --- a/mcp-servers/browser-bridge/lib/config.js +++ b/mcp-servers/browser-bridge/lib/config.js @@ -38,13 +38,32 @@ export const CONFIG = { perplexityAuto: 600_000, // automate_perplexity_task (10 min max) }, - // Relay-specific timeouts - relayReconnectDelay: 3_000, + // Relay-specific timeouts. + // relayReconnectDelay is the BASE of a full-jitter exponential backoff, not a + // fixed sleep. It was a constant 3_000 until 2026-08-22, which made every + // relay client retry in the same millisecond — ~20 processes logging the + // identical ECONNREFUSED 10,415 times in 24h. Each of those drops removed and + // re-added this server's tools, and a tool-list change invalidates the entire + // prompt cache, so the herd was costing ~$400/day. Small base = fast recovery + // once an owner exists; the cap bounds the retry rate while none does. + relayReconnectDelay: 250, + relayReconnectCapMs: 8_000, ppidPollInterval: 10_000, // WS bridge zombie/reconnect detection appMsgTimeout: 45_000, // zombie detection: 2 missed 20s keepalives waitForBrowserTimeout: 5_000, // max wait for browser client reconnect + + // Grace period before a disconnected relay's tabs are closed. + // A relay disconnect used to close that session's tabs IMMEDIATELY. Measured + // 2026-09-10: the intellegix-relay lane's server process exited at 12:43:06 and + // a replacement connected 2.3s later, so the cleanup destroyed a tab the caller + // was still working with. The caller then read the (accurate) tab list, did not + // find its tab, and concluded the page was never open — which is the incident in + // STALE-TABS-AND-SINGLE-BROWSER-BLINDNESS-EVIDENCE-2026-09-10.md. Deferring the + // cleanup lets a restart reclaim its own tabs. A genuinely-gone session just has + // its tabs closed this many ms later. + sessionCleanupGrace: 45_000, }; // --------------------------------------------------------------------------- diff --git a/mcp-servers/browser-bridge/lib/websocket-bridge.js b/mcp-servers/browser-bridge/lib/websocket-bridge.js index 17bf2a1..2147be2 100644 --- a/mcp-servers/browser-bridge/lib/websocket-bridge.js +++ b/mcp-servers/browser-bridge/lib/websocket-bridge.js @@ -17,6 +17,42 @@ export class WebSocketBridge extends EventEmitter { this.pendingRequests = new Map(); // requestId -> { resolve, reject, timer } this.heartbeatTimer = null; this.cachedPageContext = null; + // projectPath -> { timer, sessionId } — session cleanups deferred so a + // restarting lane can reclaim its own tabs. See CONFIG.sessionCleanupGrace. + this.pendingSessionCleanups = new Map(); + } + + /** + * Close a relay session's tabs after a grace period, so an MCP server process + * that is merely restarting does not have its own tabs destroyed underneath it. + * Cancelled by _cancelPendingCleanup when a relay for the same project reconnects. + */ + _scheduleSessionCleanup(sessionId, projectPath) { + const key = projectPath || `session:${sessionId}`; + const existing = this.pendingSessionCleanups.get(key); + if (existing) clearTimeout(existing.timer); + + const timer = setTimeout(() => { + this.pendingSessionCleanups.delete(key); + const cleanup = { type: 'session_cleanup', payload: { sessionId } }; + for (const [clientWs] of this.browserClients) this._send(clientWs, cleanup); + log.info('session_cleanup_sent', { sessionId: sessionId.slice(0, 8), projectPath, deferredMs: CONFIG.sessionCleanupGrace }); + }, CONFIG.sessionCleanupGrace); + if (typeof timer.unref === 'function') timer.unref(); + + this.pendingSessionCleanups.set(key, { timer, sessionId }); + log.info('session_cleanup_deferred', { sessionId: sessionId.slice(0, 8), projectPath, graceMs: CONFIG.sessionCleanupGrace }); + } + + /** A relay for this project reconnected in time — keep its tabs. */ + _cancelPendingCleanup(projectPath) { + if (!projectPath) return false; + const pending = this.pendingSessionCleanups.get(projectPath); + if (!pending) return false; + clearTimeout(pending.timer); + this.pendingSessionCleanups.delete(projectPath); + log.info('session_cleanup_cancelled', { sessionId: pending.sessionId.slice(0, 8), projectPath, reason: 'relay reconnected within grace period' }); + return true; } start() { @@ -77,11 +113,8 @@ export class WebSocketBridge extends EventEmitter { // When a relay disconnects, emit event for recovery handling and tell browser clients to close its session tabs if (closingInfo && closingInfo.role === 'stdio-relay' && closingInfo.sessionId) { this.emit('relayDisconnected', { sessionId: closingInfo.sessionId, pid: closingInfo.pid }); - const cleanup = { type: 'session_cleanup', payload: { sessionId: closingInfo.sessionId } }; - for (const [clientWs] of this.browserClients) { - this._send(clientWs, cleanup); - } - console.error(`[WebSocketBridge] Sent session_cleanup for relay session ${closingInfo.sessionId.slice(0, 8)}`); + this._scheduleSessionCleanup(closingInfo.sessionId, closingInfo.projectPath); + console.error(`[WebSocketBridge] Deferred session_cleanup for relay session ${closingInfo.sessionId.slice(0, 8)} (${CONFIG.sessionCleanupGrace}ms grace)`); } }); @@ -124,8 +157,11 @@ export class WebSocketBridge extends EventEmitter { browserInfo.sessionId = msg.payload?.sessionId; browserInfo.lastActivity = Date.now(); browserInfo.pid = msg.payload?.pid; + browserInfo.projectPath = msg.payload?.projectPath; this.relayClients.set(ws, browserInfo); } + // A replacement relay for the same project: keep the outgoing session's tabs. + this._cancelPendingCleanup(msg.payload?.projectPath); log.info('relay_connect', { clientId: browserInfo?.id, pid: msg.payload?.pid, sessionId: msg.payload?.sessionId?.slice(0, 8), totalRelays: this.relayClients.size }); this.emit('relayConnected', { sessionId: msg.payload?.sessionId, pid: msg.payload?.pid, projectPath: msg.payload?.projectPath, projectLabel: msg.payload?.projectLabel }); return; @@ -376,6 +412,8 @@ export class WebSocketBridge extends EventEmitter { stop() { clearInterval(this.heartbeatTimer); + for (const [, pending] of this.pendingSessionCleanups) clearTimeout(pending.timer); + this.pendingSessionCleanups.clear(); for (const [ws] of this.browserClients) ws.close(1000, 'Server shutting down'); for (const [ws] of this.relayClients) ws.close(1000, 'Server shutting down'); for (const [, pending] of this.pendingRequests) { diff --git a/mcp-servers/browser-bridge/server.js b/mcp-servers/browser-bridge/server.js index 50cfe70..0771144 100644 --- a/mcp-servers/browser-bridge/server.js +++ b/mcp-servers/browser-bridge/server.js @@ -25,7 +25,7 @@ import { WebSocket } from 'ws'; import { writeFileSync, mkdirSync, existsSync, unlinkSync, readFileSync } from 'node:fs'; import { basename, dirname, join, resolve as pathResolve } from 'node:path'; import { homedir } from 'node:os'; -import { execFileSync, execFile } from 'node:child_process'; +import { execFile } from 'node:child_process'; import { randomUUID } from 'node:crypto'; import { CONFIG, _debugLog } from './lib/config.js'; @@ -36,6 +36,7 @@ import { ContextManager } from './lib/context-manager.js'; import { WebSocketBridge } from './lib/websocket-bridge.js'; import { startHealthServer } from './lib/health-server.js'; import { MetricsCollector } from './lib/metrics.js'; +import { getBrowserCensus, buildCoverage, coverageBanner, getConnectedBrowserCount } from './lib/browser-discovery.js'; _debugLog(`imports OK — cwd=${process.cwd()} argv=${process.argv.join(' ')} ppid=${process.ppid}`); _debugLog('[council-mcp] build=2026-04-21T10'); @@ -565,6 +566,19 @@ class BrowserBridgeServer { return { content }; } + // When a negative inference from this result would be unsafe, lead with a + // plain-language banner ABOVE the JSON rather than relying on a field the + // model may skim past. Redundant with result.coverage on purpose. + const banner = coverageBanner(result?.result?.coverage); + if (banner) { + return { + content: [ + { type: 'text', text: banner }, + { type: 'text', text: JSON.stringify(result, null, 2) }, + ], + }; + } + return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }], }; @@ -597,10 +611,28 @@ class BrowserBridgeServer { case 'browser_navigate': { const url = Validator.url(args.url); const tabId = Validator.tabId(args.tabId); - return this.bridge.broadcast({ + const res = await this.bridge.broadcast({ type: 'navigate', payload: this._withSession({ url, tabId }), }); + // The extension silently opens a NEW tab when the requested tabId is not + // in this MCP session's tab group — which is the normal case after this + // server process restarts, because sessionId is regenerated per process. + // Reporting success with a different tabId and no explanation left callers + // operating on a tab they never asked for. Say so instead. + if (res && typeof res === 'object' && tabId && res.tabId && res.tabId !== tabId) { + return { + ...res, + requestedTabId: tabId, + retargeted: true, + retargetReason: + `Tab ${tabId} is not owned by this MCP session, so a new tab (${res.tabId}) was opened and ` + + 'navigated instead. The requested tab was NOT navigated and still holds its previous page. ' + + `Use tabId ${res.tabId} for follow-up calls.`, + }; + } + if (res && typeof res === 'object' && tabId) return { ...res, requestedTabId: tabId, retargeted: false }; + return res; } case 'browser_load_dynamic': { @@ -730,8 +762,38 @@ class BrowserBridgeServer { }); } - case 'browser_get_tabs': - return this.bridge.broadcast({ type: 'get_tabs', payload: this._withSession({}) }); + case 'browser_get_tabs': { + // The tab list itself is live (chrome.tabs.query at call time), but it + // covers ONE browser: the extension client this bridge elected. Austin + // lost a morning on 2026-09-10 because that scope was never stated and an + // agent read silence as absence. Declare the scope every time. + const started = Date.now(); + const [listing, census] = await Promise.all([ + this.bridge.broadcast({ type: 'get_tabs', payload: this._withSession({}) }), + getBrowserCensus().catch(() => null), + ]); + const tabs = Array.isArray(listing?.tabs) ? listing.tabs : []; + const connectedBrowserClients = await getConnectedBrowserCount({ + localCount: this.bridge.browserClients?.size ?? 0, + tabsReturned: Array.isArray(listing?.tabs), + healthPort: CONFIG.healthPort, + }); + const coverage = buildCoverage({ tabCount: tabs.length, connectedBrowserClients, census }); + // `result` is serialized first, deliberately: a caveat placed after a long + // array is read after the model has already anchored on the list. + return { + result: { + coverage, + freshness: { + status: 'FRESH', + asOf: coverage.asOf, + ageMs: Date.now() - started, + note: 'Enumerated live at call time from the connected browser; not a cached snapshot.', + }, + }, + tabs, + }; + } case 'browser_switch_tab': { const tabId = Validator.tabId(args.tabId); @@ -951,7 +1013,7 @@ class BrowserBridgeServer { const ctxFile = join(cacheDir, `session_context_${invocationId}.md`); if (includeContext) { try { - const ctxOut = execFileSync('python', [join(scriptDir, 'session_context.py'), process.cwd()], { + const { stdout: ctxOut } = await execFileAsync('python', [join(scriptDir, 'session_context.py'), process.cwd()], { timeout: CONFIG.timeouts.councilExec, encoding: 'utf-8', env: pythonEnv, @@ -989,8 +1051,15 @@ class BrowserBridgeServer { cwd: scriptDir, }); let result = raw.stdout; - // Check for browser busy error (concurrent session holding the profile lock) - if (result.includes('BROWSER_BUSY')) { + // Check for browser busy error (concurrent session holding the profile lock). + // 2026-08-22: this used to be `result.includes('BROWSER_BUSY')`, a bare + // substring search over the child's whole stdout. council_query.py prints an + // unconditional troubleshooting block on EVERY failure, and that block + // contained the literal token — so a signed-out account, a selector drift and + // a real lock contention all came back to the caller as "another browser + // session is active". Lanes deleted lock files for hours against a fault that + // had no lock in it. Match only the structured Code field the runner emits. + if (/^\s*\*\*Code:\*\*\s*BROWSER_BUSY\s*$/m.test(result)) { log.warn('query_browser_busy', { invocationId, queryType, elapsedMs: Date.now() - startMs }); return { error: 'Another browser council/research session is active. Wait ~2 min or use --mode api.', @@ -1031,7 +1100,7 @@ class BrowserBridgeServer { case 'council_metrics': { const scriptDir = join(homedir(), '.claude', 'council-automation'); const pyEnv = { ...process.env, PYTHONIOENCODING: 'utf-8' }; - const result = execFileSync('python', [ + const { stdout: result } = await execFileAsync('python', [ join(scriptDir, 'council_metrics.py'), '--json', ], { timeout: CONFIG.timeouts.councilExec, @@ -1047,7 +1116,7 @@ class BrowserBridgeServer { const scriptDir = join(homedir(), '.claude', 'council-automation'); const pyEnv = { ...process.env, PYTHONIOENCODING: 'utf-8' }; - const result = execFileSync('python', [ + const { stdout: result } = await execFileAsync('python', [ join(scriptDir, 'council_query.py'), level === 'full' ? '--read-full' : level === 'synthesis' ? '--read' : '--read-model', ...(level !== 'full' && level !== 'synthesis' ? [level] : []), @@ -1370,6 +1439,70 @@ class BrowserBridgeServer { // Relay mode — connect to existing WS server as client // ----------------------------------------------------------------------- + /** + * Owner re-election. Called when a relay client sees ECONNREFUSED, which + * means the process that owned the WebSocket port has exited. + * + * The election primitive is the bind itself: on Windows an exclusive bind to + * a specific loopback address is an OS-backed mutex, so exactly one racing + * process wins and the rest get EADDRINUSE. That is only true while we bind + * the SAME explicit address every time and never opt into SO_REUSEADDR / + * reusePort / exclusive:false — those make Windows accept a second binder and + * dispatch connections nondeterministically, which would give us two owners. + * + * Losing is the normal case and is NOT an error: the winner is now serving, + * so the loser just goes back to relaying. A promoted owner starts a fresh + * browser-session epoch — in-memory state does not transfer with the port. + */ + async _tryPromoteToOwner(scheduleReconnect) { + if (this._electing || this._relayConnected) return; + this._electing = true; + try { + await this.bridge.start(); + this.healthServer = await startHealthServer(this.bridge, rateLimiter, this.metrics); + this._relayConnected = false; + this._relayMode = false; + if (this._relayReconnectTimer) { + clearTimeout(this._relayReconnectTimer); + this._relayReconnectTimer = null; + } + // BUG FIX (2026-08-29): _connectAsRelay() shadows this.bridge.broadcast and + // this.bridge.getStatus with instance-level relay overrides that forward + // through this._relayWs. Promotion rebinds the real WS+health servers but + // NEVER removed those overrides, so a promoted (formerly-relay) process kept + // running the stale relay-mode broadcast()/getStatus() forever: + // - getStatus() always reported {mode:'relay', connected:false, + // clientCount:0} on /health regardless of real browser-client state — + // a false negative that made the extension look disconnected when it + // might not have been. + // - broadcast() tried to send over this._relayWs, which is null/dead + // post-promotion (the ECONNREFUSED that triggered promotion means the + // old primary — and this relay connection to it — is gone), so any + // code path calling this.bridge.broadcast() directly on the promoted + // instance (rather than through the real WebSocketBridge#_onMessage + // relay_forward path) would wrongly reject with "Relay not connected + // to primary server" instead of using the newly-live real bridge. + // Deleting the own-properties restores WebSocketBridge.prototype's real + // broadcast()/getStatus() now that this instance genuinely owns the ports. + delete this.bridge.broadcast; + delete this.bridge.getStatus; + _debugLog('_tryPromoteToOwner() WON election — now primary'); + console.error('[BrowserBridge] Previous owner exited — promoted to primary'); + } catch (err) { + if (err.code === 'EADDRINUSE') { + // Expected: another client won the race, or the old listener has not + // released yet. Not an error, and deliberately not logged as one. + _debugLog('_tryPromoteToOwner() lost election — staying relay'); + } else { + console.error('[BrowserBridge] Promotion failed:', err.message); + } + this._electing = false; + scheduleReconnect(); + return; + } + this._electing = false; + } + _connectAsRelay() { _debugLog(`_connectAsRelay() entered — target ws://${CONFIG.wsHost}:${CONFIG.wsPort}`); return new Promise((resolve, reject) => { @@ -1378,6 +1511,30 @@ class BrowserBridgeServer { this._relayPending = new Map(); this._relayReconnectTimer = null; this._relayConnected = false; + this._relayAttempt = 0; // full-jitter backoff exponent + this._electing = false; // single-flight guard for promotion + + // Full jitter (AWS "Exponential Backoff and Jitter"). A constant delay + // made ~20 relay clients retry in the SAME millisecond; measured + // 2026-08-22, that herd produced 10,415 ECONNREFUSED in 24h against + // only 104 process starts. Randomising the whole interval disperses it. + const backoffDelay = () => { + const capped = Math.min( + CONFIG.relayReconnectCapMs, + CONFIG.relayReconnectDelay * 2 ** Math.min(this._relayAttempt, 6), + ); + this._relayAttempt += 1; + return Math.floor(Math.random() * capped); + }; + + const scheduleReconnect = () => { + if (this._relayReconnectTimer || this._electing) return; // single-flight + const delay = backoffDelay(); + this._relayReconnectTimer = setTimeout(() => { + this._relayReconnectTimer = null; + connect(false); + }, delay); + }; const connect = (isInitial = false) => { _debugLog(`_connectAsRelay() connect() isInitial=${isInitial}`); @@ -1386,6 +1543,7 @@ class BrowserBridgeServer { ws.on('open', () => { this._relayWs = ws; this._relayConnected = true; + this._relayAttempt = 0; // reset backoff only on a real connection _debugLog('_connectAsRelay() WS open — sending relay_init'); console.error('[BrowserBridge] Relay connected to primary WS server'); @@ -1448,15 +1606,27 @@ class BrowserBridgeServer { pending.reject(new Error('Relay connection lost')); } this._relayPending.clear(); - this._relayReconnectTimer = setTimeout(() => connect(false), CONFIG.relayReconnectDelay); + scheduleReconnect(); }); ws.on('error', (err) => { - _debugLog(`_connectAsRelay() WS error: ${err.code || err.message} isInitial=${isInitial} connected=${this._relayConnected}`); - console.error('[BrowserBridge] Relay WS error:', err.message); + const code = err.code || err.message; + _debugLog(`_connectAsRelay() WS error: ${code} isInitial=${isInitial} connected=${this._relayConnected}`); if (isInitial && !this._relayConnected) { reject(err); + return; + } + // ECONNREFUSED means the OWNER PROCESS IS GONE — its session ended and + // nobody re-bound the port. Retrying a dead port forever is what caused + // the outage: the tools stay missing, and every drop/re-add of them + // invalidates the whole prompt cache (tools sit at prefix position 0). + // So try to BECOME the owner instead of waiting for one to reappear. + if (code === 'ECONNREFUSED') { + this._tryPromoteToOwner(scheduleReconnect); + return; } + console.error('[BrowserBridge] Relay WS error:', err.message); + scheduleReconnect(); }); }; diff --git a/mcp-servers/browser-bridge/test-coverage-honesty.js b/mcp-servers/browser-bridge/test-coverage-honesty.js new file mode 100644 index 0000000..2bc29e4 --- /dev/null +++ b/mcp-servers/browser-bridge/test-coverage-honesty.js @@ -0,0 +1,240 @@ +/** + * test-coverage-honesty.js — regression tests for the 2026-09-10 incident. + * + * The bug: browser_get_tabs returned a bare {tabs:[...]} from the ONE browser the + * bridge extension is connected to, with nothing saying other browsers existed. + * An agent read that silence as absence and reported a confident false negative. + * + * These tests fail against the pre-fix code (which had no coverage envelope, no + * retarget disclosure, and closed a relay's tabs the instant it disconnected) and + * pass against the fix. + * + * Run with: node --test test-coverage-honesty.js + */ + +import { describe, it, mock } from 'node:test'; +import assert from 'node:assert/strict'; + +import { + buildCoverage, + coverageBanner, + getSwitchValue, + isBrowserRoot, +} from './lib/browser-discovery.js'; +import { WebSocketBridge } from './lib/websocket-bridge.js'; +import { CONFIG } from './lib/config.js'; + +// --------------------------------------------------------------------------- +// The incident, reproduced as a fixture +// --------------------------------------------------------------------------- + +/** Two Chromes running: the bridge's Default profile, and the session keeper on 9223. */ +const TWO_CHROME_CENSUS = { + known: true, + asOf: '2026-09-10T12:39:00.000Z', + browserRoots: [ + { + pid: 1001, + exe: 'chrome.exe', + userDataDir: 'C:\\Users\\example\\AppData\\Local\\Google\\Chrome\\User Data', + declaredDebuggingPort: null, + usesDebuggingPipe: false, + }, + { + pid: 2002, + exe: 'chrome.exe', + userDataDir: 'C:\\Users\\example\\.claude\\config\\session_keeper_profile', + declaredDebuggingPort: 9223, + usesDebuggingPipe: false, + }, + ], + cdpEndpoints: [{ port: 9223, browser: 'Chrome/153.0.8010.36', pid: 2002 }], +}; + +const ONE_CHROME_CENSUS = { + known: true, + asOf: '2026-09-10T12:39:00.000Z', + browserRoots: [{ pid: 1001, exe: 'chrome.exe', userDataDir: 'C:\\...\\User Data', declaredDebuggingPort: null, usesDebuggingPipe: false }], + cdpEndpoints: [], +}; + +describe('coverage envelope — the tab list must declare its scope', () => { + it('REGRESSION: two browsers, one connected → PARTIAL and negative evidence UNSAFE', () => { + const coverage = buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: TWO_CHROME_CENSUS }); + + // This is the assertion the pre-fix response could not satisfy at all: the old + // shape was {tabs:[...]} with no coverage key of any kind. + assert.equal(coverage.status, 'PARTIAL'); + assert.equal(coverage.negativeEvidence, 'UNSAFE'); + assert.equal(coverage.observedCount, 1); + assert.equal(coverage.detectedCount, 2); + }); + + it('names the invalid inference explicitly, not just "may be incomplete"', () => { + const coverage = buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: TWO_CHROME_CENSUS }); + assert.match(coverage.summary, /NOT evidence that the tab is not open/); + }); + + it('tells the caller where to look — the unseen browser CDP endpoint is actionable', () => { + const coverage = buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: TWO_CHROME_CENSUS }); + const keeper = coverage.unobserved.find((u) => u.cdpEndpoint === 'http://127.0.0.1:9223'); + assert.ok(keeper, 'the session-keeper Chrome must appear under unobserved with its endpoint'); + assert.equal(keeper.reasonCode, 'REACHABLE_VIA_CDP_NOT_CONNECTED'); + assert.equal(keeper.tabCount, null, 'must not invent a tab count for a browser it did not query'); + }); + + it('single browser → COMPLETE, and a negative conclusion is allowed', () => { + const coverage = buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: ONE_CHROME_CENSUS }); + assert.equal(coverage.status, 'COMPLETE'); + assert.equal(coverage.negativeEvidence, 'SAFE_WITHIN_DECLARED_SCOPE'); + assert.deepEqual(coverage.unobserved, []); + }); + + it('coverage is ALWAYS present — the common case must not be silent', () => { + for (const census of [ONE_CHROME_CENSUS, TWO_CHROME_CENSUS, null]) { + const coverage = buildCoverage({ tabCount: 1, connectedBrowserClients: 1, census }); + assert.ok(coverage.status, 'status must always be set'); + assert.ok(coverage.negativeEvidence, 'negativeEvidence must always be set'); + assert.ok(coverage.scope, 'scope must always name the universe'); + } + }); + + it('discovery failure must NOT be reported as complete coverage', () => { + const coverage = buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: { known: false, note: 'PowerShell unavailable' } }); + assert.equal(coverage.status, 'SCOPED'); + assert.equal(coverage.negativeEvidence, 'UNKNOWN'); + assert.notEqual(coverage.status, 'COMPLETE'); + }); + + it('banner leads with the status and only appears when a negative inference is unsafe', () => { + const partial = coverageBanner(buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: TWO_CHROME_CENSUS })); + assert.match(partial, /^RESULT STATUS: PARTIAL — NEGATIVE EVIDENCE UNSAFE\./); + + const complete = coverageBanner(buildCoverage({ tabCount: 5, connectedBrowserClients: 1, census: ONE_CHROME_CENSUS })); + assert.equal(complete, null, 'the ordinary case must not add banner noise to every call'); + }); +}); + +// --------------------------------------------------------------------------- +// Windows command-line parsing +// --------------------------------------------------------------------------- + +describe('chromium command-line parsing', () => { + it('reads a quoted user-data-dir containing spaces', () => { + const cmd = '"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe" --user-data-dir="C:\\Users\\A B\\.claude\\config\\session_keeper_profile" --remote-debugging-port=9223'; + assert.equal(getSwitchValue(cmd, 'user-data-dir'), 'C:\\Users\\A B\\.claude\\config\\session_keeper_profile'); + assert.equal(getSwitchValue(cmd, 'remote-debugging-port'), '9223'); + }); + + it('handles space-separated switch values', () => { + assert.equal(getSwitchValue('chrome.exe --remote-debugging-port 9222', 'remote-debugging-port'), '9222'); + }); + + it('returns null for an absent switch', () => { + assert.equal(getSwitchValue('chrome.exe --headless', 'remote-debugging-port'), null); + }); + + it('counts only browser roots — renderer/GPU children are not instances', () => { + assert.equal(isBrowserRoot({ CommandLine: '"chrome.exe" --user-data-dir="C:\\x"' }), true); + assert.equal(isBrowserRoot({ CommandLine: '"chrome.exe" --type=renderer --lang=en-US' }), false); + assert.equal(isBrowserRoot({ CommandLine: '"chrome.exe" --type=gpu-process' }), false); + assert.equal(isBrowserRoot({ CommandLine: '' }), false, 'unreadable command line must not be counted as a root'); + }); +}); + +// --------------------------------------------------------------------------- +// Session cleanup grace — the mechanism that destroyed the caller's tab +// --------------------------------------------------------------------------- + +describe('relay disconnect must not destroy a restarting lane\'s tabs', () => { + it('REGRESSION: cleanup is deferred, not sent on the disconnect tick', () => { + const bridge = new WebSocketBridge(); + const sent = []; + const fakeBrowser = { readyState: 1, send: (d) => sent.push(JSON.parse(d)), close: () => {} }; + bridge.browserClients.set(fakeBrowser, { id: 'b1', connectedAt: Date.now() }); + + bridge._scheduleSessionCleanup('04cd2eca-4ca4-4532-857a-b496d6d73ed5', 'C:\\dev\\intellegix-relay'); + + // Pre-fix behaviour sent session_cleanup synchronously here. It must not. + assert.deepEqual(sent, [], 'no session_cleanup may be sent during the grace period'); + assert.equal(bridge.pendingSessionCleanups.size, 1); + bridge.stop(); + }); + + it('a relay reconnecting for the same project cancels the pending cleanup', () => { + const bridge = new WebSocketBridge(); + bridge._scheduleSessionCleanup('04cd2eca-4ca4-4532-857a-b496d6d73ed5', 'C:\\dev\\intellegix-relay'); + // 2.3 seconds later a replacement process connected for the same directory — + // that is exactly what happened at 12:43:06 → 12:43:08 on 2026-09-10. + const cancelled = bridge._cancelPendingCleanup('C:\\dev\\intellegix-relay'); + assert.equal(cancelled, true); + assert.equal(bridge.pendingSessionCleanups.size, 0); + bridge.stop(); + }); + + it('a different project does not cancel someone else\'s pending cleanup', () => { + const bridge = new WebSocketBridge(); + bridge._scheduleSessionCleanup('aaaaaaaa-0000-0000-0000-000000000000', 'C:\\dev\\project-a'); + assert.equal(bridge._cancelPendingCleanup('C:\\dev\\project-b'), false); + assert.equal(bridge.pendingSessionCleanups.size, 1); + bridge.stop(); + }); + + it('the cleanup does eventually fire for a session that really ended', async () => { + const original = CONFIG.sessionCleanupGrace; + CONFIG.sessionCleanupGrace = 20; + try { + const bridge = new WebSocketBridge(); + const sent = []; + bridge.browserClients.set({ readyState: 1, send: (d) => sent.push(JSON.parse(d)), close: () => {} }, { id: 'b1', connectedAt: Date.now() }); + bridge._scheduleSessionCleanup('deadbeef-0000-0000-0000-000000000000', 'C:\\dev\\gone'); + await new Promise((r) => setTimeout(r, 80)); + assert.equal(sent.length, 1); + assert.equal(sent[0].type, 'session_cleanup'); + assert.equal(sent[0].payload.sessionId, 'deadbeef-0000-0000-0000-000000000000'); + bridge.stop(); + } finally { + CONFIG.sessionCleanupGrace = original; + } + }); +}); + +// --------------------------------------------------------------------------- +// Navigate retarget disclosure +// --------------------------------------------------------------------------- + +/** Mirrors the browser_navigate branch in server.js _handleToolCall. */ +function decorateNavigateResult(res, tabId) { + if (res && typeof res === 'object' && tabId && res.tabId && res.tabId !== tabId) { + return { + ...res, + requestedTabId: tabId, + retargeted: true, + retargetReason: `Tab ${tabId} is not owned by this MCP session, so a new tab (${res.tabId}) was opened and navigated instead.`, + }; + } + if (res && typeof res === 'object' && tabId) return { ...res, requestedTabId: tabId, retargeted: false }; + return res; +} + +describe('browser_navigate must disclose a retarget', () => { + it('REGRESSION: the exact 2026-09-10 call reports retargeted:true', () => { + // Requested 1435686256 (chrome://newtab), got back 1435686303, success:true. + const out = decorateNavigateResult({ success: true, url: 'https://fantasy.espn.com/football/', tabId: 1435686303 }, 1435686256); + assert.equal(out.retargeted, true); + assert.equal(out.requestedTabId, 1435686256); + assert.equal(out.tabId, 1435686303); + assert.match(out.retargetReason, /new tab \(1435686303\)/); + }); + + it('an honoured tabId reports retargeted:false', () => { + const out = decorateNavigateResult({ success: true, url: 'https://example.com/', tabId: 77 }, 77); + assert.equal(out.retargeted, false); + assert.equal(out.requestedTabId, 77); + }); + + it('no tabId requested → no retarget fields invented', () => { + const out = decorateNavigateResult({ success: true, url: 'https://example.com/', tabId: 77 }, undefined); + assert.equal(out.retargeted, undefined); + }); +}); diff --git a/patterns/AUTONOMOUS_TRIAGE_PATTERN.md b/patterns/AUTONOMOUS_TRIAGE_PATTERN.md new file mode 100644 index 0000000..4c07769 --- /dev/null +++ b/patterns/AUTONOMOUS_TRIAGE_PATTERN.md @@ -0,0 +1,258 @@ +# AUTONOMOUS_TRIAGE_PATTERN.md + +An always-on error + inefficiency triage loop for a small-team internal app. Detects real user friction in production, has an LLM propose a fix, applies static-rules safety checks over the proposal, drafts a patch, waits for a human to approve + merge, then notifies the affected users the moment their issue is fixed and live. + +Human-in-loop at the MERGE step only. Everything upstream (detection, classification, patch draft) is autonomous. + +**Reference implementation:** `reference-impl/autonomous-triage/` (Next.js 16 + Prisma 5 + Neon Postgres + Vercel Blueprint, shipped 2026-07-10, ~2200 LOC). + +**When to use this pattern:** an internal app with <20 users, a per-user audit stream, a cron scheduler, and an admin who can approve merges within a day. Cost at 8-user signal density: ~$0.32/month. + +**When NOT to use:** consumer-scale apps (this is not a distributed telemetry system), regulated environments where autonomous code drafting requires ISO 27001 attestation, or teams without an admin willing to be the merge gate. + +--- + +## The pipeline in one picture + +``` +Cron (*/15 min) Cron (*/5 min) Cron (nightly) + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌────────────────┐ ┌────────────────┐ +│ auto-triage │ │ close-loop │ │ triage-cleanup │ +│ Stage A: SQL │ │ Poll GH PRs │ │ 30d prune + │ +│ Stage B: gate │ │ Match │ │ orphan sweep │ +│ Stage C: patch│ │ triage/ │ │ │ +└───────────────┘ │ → notify users │ └────────────────┘ + │ └────────────────┘ + ▼ ▲ + triage_tickets Admin merges PR + │ ▲ + ▼ │ + /admin/triage — review, approve locally, apply patch, push, PR +``` + +The three crons + one admin page is the whole system. The rest is code that fills those slots. + +--- + +## Prerequisites in the host app + +The pattern's runtime depends on **7 things** the host app must expose. If any is missing, the setup command emits `TODO(prereq: X)` stubs and the receiving Claude fills them in. + +| # | Prereq | Ref-impl uses | Portable substitute | +|---|---|---|---| +| 1 | Per-user audit event stream | `audit_events` table with `event_type`, `route`, `element_text`, `fetch_url`, `fetch_method`, `fetch_status`, `error_msg`, `user_id`, `session_id`, `occurred_at` | Any per-user click/fetch/error log. If the host app has none, this pattern isn't yet applicable — first ship a client-side audit stream (see `patterns/API_PATTERNS.md` for shape) | +| 2 | Cron scheduler | `vercel.json` cron entries | Any per-schedule executor: Vercel Cron, GitHub Actions cron, Cloudflare Cron Triggers, systemd timers, or a `cron`-labelled endpoint hit by an external pinger | +| 3 | Object storage for patch artifacts | Vercel Blob (`@vercel/blob`, `access: 'private'`) | S3 + signed URLs, R2, or a local `/tmp/triage-patches/` dir for single-node deployments | +| 4 | Admin auth gate | NextAuth 4 + `isAdmin(user.role)` | Any admin check exposed to server code | +| 5 | GitHub PAT with `repo` scope | `GITHUB_PAT` env var | Same — the PAT is the least-portable piece and is required for repo access | +| 6 | Anthropic API key | `ANTHROPIC_API_KEY` env var | Same | +| 7 | Per-user notification channel | `user_notifications` bell row + Web Push via `notifyUser()` helper | Any per-user pipe: SES, Pushover with per-user keys, Slack DMs, Discord webhooks, or a bell-feed table | + +**Explicit non-prerequisite:** the host app does NOT need Prisma. The reference-impl uses Prisma because the host does; port the two Prisma models to whatever ORM/SQL the target uses. Same for MUI on the dashboard — the pattern is React-optional. + +--- + +## Component-by-component walkthrough + +Each subsection maps directly to a file under `reference-impl/autonomous-triage/`. Read the file and this section side-by-side. + +### Schema (`schema/triage_tables.prisma`) + +Two models. `triage_tickets` is the state store for one row per unique detected pattern. `system_state` is a key/value scratch space for cron watermarks. + +**Portability notes:** +- `id` uses Postgres `gen_random_uuid()`. Swap for SQLite `randomblob(16)` or app-side UUID. +- `String[]` and `Int[]` are Postgres native arrays. On MySQL/SQLite, use `Json`. +- `pplx_response Json?` — the column name is a historical artifact of an initial Perplexity design; kept for schema stability during the pivot to Sonnet. Rename freely. +- Indexes optimize for `WHERE status = 'open' AND pplx_response IS NULL` (Stage B pickup) and `WHERE status = 'in_progress' AND patch_blob_pathname IS NULL` (Stage C pickup). Keep both if you keep the pipeline. + +### Signal detection (`src/lib/triage/signals.ts`) + +Two exported functions, both raw SQL over `audit_events`: +- `detectErrorClusters(prisma)` — per-user URL+status bucketing (COUNT >= 3) UNION cross-user error_msg pattern matching (DISTINCT user_id >= 2). +- `detectInefficiency(prisma)` — rage-click detection via SQL window function. + +**Threshold calibration is empirical.** The `COUNT >= 3` per-user threshold was chosen after measuring 30 days of the reference-impl's actual signal (5 real active users, ~7 clusters/month). At different scales: +- 100 users → keep 3 or lift to 5 +- 1000 users → lift to 10 and add a cross-user weight +- Under 10 users → the pattern is signal-poor; consider a rules-only version without the LLM gate + +**Portability notes:** +- The SQL uses Postgres `INTERVAL` syntax and regex functions (`~*`, `regexp_replace(...)`). SQLite / MySQL need equivalents. +- "User reached goal" definition: any 2xx `POST`/`PATCH`/`PUT` fetch in the same `session_id` within the window. If the host app has a different mutation pattern, adjust. +- `element_text` for rage-click identification is the button label. Ensure the audit stream captures this and not just an anonymous DOM path. + +### Sonnet triage gate (`src/lib/triage/sonnet-gate.ts`) + +Sends a triage_ticket snapshot to Claude Sonnet 4.5 with structured JSON output (Zod-validated). Returns: + +```typescript +{ fixable, severity, approach, blast_radius, estimated_users_affected, file_hints[], skip_reason? } +``` + +**Never throws.** Any failure — timeout, non-2xx, empty response, parse failure, schema mismatch — returns `{ fixable: false, skip_reason: 'sonnet_...' }` so the caller proceeds uniformly. + +**Why Sonnet and not Perplexity, GPT-4, or Gemini:** +- Perplexity's browser-driven API (via Playwright in the toolkit's `research_query`) can't be called from a server cron. +- Sonnet's structured-JSON adherence with `response_format` + Zod is more reliable than GPT-4's function-calling at this size. +- Cost per gate call is ~$0.005; at 7 tickets/month that's $0.04/month baseline. + +**Portability:** the SDK client is a raw `fetch` (not the Anthropic SDK) so this works in any JS runtime. Port to Python by lifting the prompt + Zod schema into pydantic. + +### Blast-radius classifier (`src/lib/triage/blast-radius.ts`) + +Pure static rule table. First-match-wins on file paths. Has **authoritative veto** over the Sonnet gate — can only make Sonnet's verdict MORE restrictive, never less. + +The rules ranked strictest-to-loosest: +- `HARD_BLOCK`: `prisma/schema.prisma`, `prisma/migrations/`, `lib/auth/`, `middleware.ts`, `.env*`, `**/credential*|secret|token|apikey*` +- `HARD_BLOCK` (with diff content): any `**/route.ts` that adds a `prisma.*.{create|update|delete|upsert|executeRaw}` call +- `HIGH`: `next.config.*`, `package.json`, `package-lock.json` +- `MEDIUM`: `**/api/**/route.ts` (no DB write), `**/components/**/*.tsx` with logic +- `LOW`: `**/*.tsx` (JSX text-only), `**/lib/**/*.ts` (Zod tightening, null-checks, prompt strings), test files +- Default: `MEDIUM` (conservative) + +**Called twice:** +1. After the Sonnet gate, over Sonnet's `file_hints[]` — cheap pre-filter before we pay for patch generation. +2. After patch generation, over the ACTUAL `+++ b/` lines from the diff PLUS the diff content — the belt-and-suspenders check. + +**Portability:** these rules are 100% host-agnostic in structure but Next.js/Prisma-shaped by content. Swap file paths for the target stack (`app/routes/` in Rails, `pkg/api/` in Go, etc.). + +### Patch generator (`src/lib/triage/patch-generator.ts`) + +For a ticket where Sonnet said `fixable=true` and the classifier didn't `HARD_BLOCK`: +1. Fetch current master SHA via `GET /repos/{owner}/{name}/git/refs/heads/master` with the `GITHUB_PAT`. +2. Fetch each `file_hint`'s contents at that SHA. Hallucinated hints (404) are silently dropped. If < 1 real file, skip. +3. Sonnet drafts a unified diff + summary using the two-section fence: + ``` + --- summary --- + ... + --- diff --- + ... + ``` +4. Post-generation guard: run the classifier over the ACTUAL `+++ b/` diff paths + diff content. If HARD_BLOCK, don't upload. +5. Upload `summary.md` + `suggested-fix.patch` to private blob storage. +6. Mark ticket `patch_blob_pathname`, `patch_summary`, `master_sha`, `status='in_progress'`. + +**Prompt budgets baked in:** `MAX_FILE_HINTS = 5`, `MAX_FILE_LINES = 400`, `MAX_TOTAL_FILE_CHARS = 40_000`, `PATCH_TIMEOUT_MS = 40_000`. + +**Cost:** ~$0.05-0.15 per patch call. At 2 patches/month, ~$0.20/month. + +**Portability:** +- GitHub owner + repo name are hardcoded at the top of the file. Extract to env vars for portability. +- Vercel Blob upload → S3 with a `contentType` header and signed download URL for the dashboard proxy. +- The `two-section fence` output format is important — the parser splits on it. Keep it if you swap models. + +### Admin dashboard (`src/app/(admin)/admin/triage/page.tsx` + `src/app/api/admin/triage/*`) + +MUI Table with row-expand, streams diff from private blob via admin-authed proxy, three actions: +- **Approve** — marks `approved_by_user_id` + `approved_at`; copies a full `git checkout && git checkout -b triage/ && curl ... && git apply ...` command to the clipboard. Ticket stays `in_progress`. +- **Skip** — opens a reason dialog; marks `status='skipped'`. +- **Archive** — hides from default view. + +**The admin never touches git through the dashboard.** The apply command runs on their laptop in their own Claude Code session. The dashboard is a review-and-approve UI, not an executor. + +**Portability:** +- The Table is MUI + React Query. Port to any UI kit. +- Auth gate is `getServerSession(authOptions)` + `isAdmin(user.role)`. Use the host app's admin check. +- Blob proxy: `head()` the private blob to get its signed URL, then `fetch()` and stream through. Any S3-equivalent works. + +### Close-loop cron (`src/app/api/cron/triage-close-loop/route.ts` + `src/lib/triage/close-loop.ts`) + +Every 5 minutes: +1. Read watermark from `system_state.triage_close_loop.value.last_processed_pr_number`. +2. `GET /repos/{owner}/{name}/pulls?state=closed&sort=updated&direction=desc&per_page=50&base=master` with the PAT. +3. Filter for `merged_at != null` AND `number > watermark`. +4. For each PR whose head branch matches `triage/`: + - Look up the ticket. + - If `status='in_progress' AND resolved_at IS NULL`, use `updateMany` with those conditions in the WHERE — count=0 means another cron pass already claimed it and we skip. + - For each `affected_user_id`, call `notifyUser({ title, body, kind, url })` which writes the bell row + sends Web Push + mirrors to admin. +5. Advance the watermark to the highest PR number seen (even for non-triage PRs). + +**Portability:** +- GH poll: same as patch-gen, hardcode → env var. +- The `notifyUser` helper is the host app's per-user notification abstraction — bell + push + email + whatever. Its interface is `notifyUser(userId, { title, body, url?, kind })`. Ship this ABSTRACTION expectation, not a specific implementation. + +### Cleanup cron (`src/app/api/cron/triage-cleanup/route.ts`) + +Nightly at 03:00 UTC: +1. `DELETE FROM triage_tickets WHERE status IN ('resolved','archived','skipped') AND updated_at < NOW() - INTERVAL '30 days'`. +2. Raw SQL: strip orphan user_ids from `affected_user_ids` arrays and recompute `estimated_users`. + +Tiny, uneventful. Not strictly required, but keeps the table lean. + +--- + +## Adaptation matrix — what to swap for a different stack + +The reference-impl's stack (Next.js 16 + Prisma 5 + Neon + Vercel Blueprint + NextAuth 4) is one point in the design space. Below is the matrix of substitutions for other host apps. + +| Component | Ref-impl | If host app uses… | Change | +|---|---|---|---| +| ORM | Prisma 5 | Drizzle | Regenerate schema; the raw SQL in `signals.ts` works unchanged | +| ORM | Prisma 5 | TypeORM / Sequelize | Rewrite `.upsert()`, `.updateMany()`, `.findUnique()` calls to that ORM's idioms | +| ORM | Prisma 5 | Python + SQLAlchemy | Full port; the SQL survives, the wrapper doesn't | +| DB | Neon Postgres | Postgres self-hosted | No change | +| DB | Neon Postgres | MySQL 8 | Replace `INTERVAL '15 minutes'` with `INTERVAL 15 MINUTE`, replace `~*` regex with `REGEXP`, replace `String[]` with JSON columns | +| DB | Neon Postgres | SQLite | Replace INTERVAL / window functions with app-side computation over recent rows; SQLite lacks arrays entirely | +| Cron | Vercel Cron | GitHub Actions | Each cron becomes a `.github/workflows/cron-XXX.yml` calling the API endpoint | +| Cron | Vercel Cron | AWS EventBridge → Lambda | Wrap each route in a Lambda handler | +| Cron | Vercel Cron | Cloudflare Workers Cron Triggers | Direct swap; adjust the `maxDuration` semantics | +| Blob | Vercel Blob (private) | S3 | `put()` → `s3.putObject()`; `head()` → `s3.getObject()` + signed URL | +| Blob | Vercel Blob (private) | Cloudflare R2 | Same as S3 with R2 endpoint | +| Blob | Vercel Blob (private) | Local disk | Write to `/tmp/triage-patches/{date}/{id}/`; scale limit is single-node | +| Auth | NextAuth 4 sessions | Clerk / Auth0 / Supabase Auth | Swap `getServerSession(authOptions)` for the host's session helper | +| Auth | `isAdmin(role)` | Anything | Swap the predicate | +| Notification | `user_notifications` + Web Push + mirror | Slack DMs | Replace `notifyUser` with a Slack webhook per user | +| Notification | `user_notifications` + Web Push + mirror | Email only | Replace with `sendEmail({to, subject, body})` per user | +| Notification | `user_notifications` + Web Push + mirror | Pushover per-user | Replace with `sendPushover({token, user_key, message})` per user | +| Frontend | MUI + React Query | Tailwind + SWR | Rewrite `page.tsx`; API endpoints unchanged | +| Frontend | MUI + React Query | Vue / Svelte / HTMX | Same — API endpoints are the contract | +| LLM gate | Sonnet 4.5 via fetch | GPT-4o with function calling | Replace `fetch` + JSON parsing with the OpenAI function-call adapter | +| LLM gate | Sonnet 4.5 via fetch | Anthropic SDK | Direct swap; keep the Zod validation | + +--- + +## Cost model (portable) + +Applies across stacks. Signal density is what varies. + +| Signal density | Tickets/mo | Sonnet gate | Sonnet patch | Total /mo | +|---|---|---|---|---| +| 5-10 real users, low friction | ~5-10 | ~$0.03 | ~$0.20 | **~$0.25** | +| 25 users | ~20-40 | ~$0.15 | ~$1.00 | **~$1.15** | +| 100 users | ~100-200 | ~$0.60 | ~$5.00 | **~$5.60** | +| 500+ users | Threshold-lifted; add cross-user weight | — | — | Reassess architecture — this pattern targets internal apps | + +**Threshold recommendation as user count scales:** lift the per-user COUNT threshold linearly with user count. At 100 users, `COUNT >= 10` per-user. Cross-user distinct-user threshold can stay at 2 or lift to 3. + +--- + +## Human-in-loop guarantees + +The pattern makes these hard promises: + +1. **Nothing auto-merges.** The dashboard's Approve action marks a ticket approved and copies an apply command; the admin runs it. No workflow bypasses this. +2. **HARD_BLOCK is absolute.** Once the classifier returns HARD_BLOCK, no path downstream can un-block. The rule table can only be edited in code. +3. **No autonomous DB writes.** The `route.ts` + `prisma.*.{create,update,delete,upsert}` combination is a HARD_BLOCK content rule. Sonnet cannot draft a fix that adds a new DB write in an API route. +4. **User-visible notifications are gated on ticket resolution.** The close-loop cron only fires `notifyUser` when a `triage/` PR is merged. Notifications never fire on a Sonnet skip, a classifier veto, or an admin manual skip. + +Break any of these and you're outside the pattern; document the deviation prominently. + +--- + +## Reading order for the receiving Claude + +1. This file end-to-end (~700 lines) — architecture + component walkthrough + adaptation matrix. +2. `commands/setup-autonomous-triage.md` — the interactive playbook. +3. `reference-impl/autonomous-triage/ADAPTATION.md` — quickstart cheatsheet. +4. `reference-impl/autonomous-triage/src/lib/triage/` files — the runtime logic, best read in order: signals → sonnet-gate → blast-radius → patch-generator → close-loop. +5. `reference-impl/autonomous-triage/src/app/api/cron/auto-triage/route.ts` — the orchestrator that ties them together. +6. The dashboard + tests as needed. + +--- + +## Provenance + +Extracted 2026-07-11 from the ASR PO System Enterprise implementation (`intellegix/ASR-PO-System-Enterprise`, PRs #49-54, shipped 2026-07-10). Reference-impl files are copied verbatim; do not edit them without also updating the source repo, or they'll drift. From fc8bb0d6ba3f69373a821355185c58ad6f4d9fb0 Mon Sep 17 00:00:00 2001 From: Austin Kidwell Date: Thu, 10 Sep 2026 06:29:07 -0700 Subject: [PATCH 2/3] chore(deps): bump github/codeql-action to v4.37.9 in one step, and group it Supersedes Dependabot #87, #88 and #90, none of which could pass CI on its own. github/codeql-action/{init,autobuild,analyze,upload-sarif} are one action and refuse to run at mixed versions. Dependabot, with no group configured, opened a separate PR per sub-action on 2026-08-31, so each PR left the workflow half bumped and CodeQL failed with "Loaded a configuration file for version '4.37.9', but running version '4.37.6'" (run 33384590794). #89 passed only because it touches scorecard.yml, whose upload-sarif step runs alone. All four references move to cdf488f595d80d6e07e03d4674febd5ab45fa938 together. That SHA was verified against the upstream repository, not the PR title: the annotated tag v4.37.9 in github/codeql-action resolves to it, and the SHA being replaced resolves to v4.37.6. The groups block stops the same split happening next release. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01F7yhQQARuYRqGTdJJZWvZH --- .github/dependabot.yml | 10 ++++++++++ .github/workflows/codeql.yml | 6 +++--- .github/workflows/scorecard.yml | 2 +- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 91e83cd..935fce9 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -11,3 +11,13 @@ updates: schedule: interval: weekly open-pull-requests-limit: 5 + groups: + # github/codeql-action/{init,autobuild,analyze,upload-sarif} are one action + # and refuse to run at mixed versions: "Loaded a configuration file for + # version '4.37.9', but running version '4.37.6'". Ungrouped, Dependabot + # opened one PR per sub-action (#87/#88/#90 on 2026-08-31) and every one of + # them failed CodeQL on its own, because merging any single one leaves the + # workflow mixed. Grouping makes it one PR that is mergeable. + codeql-action: + patterns: + - "github/codeql-action*" diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 52d0223..ed470e5 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -24,12 +24,12 @@ jobs: persist-credentials: false - name: Initialize CodeQL - uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 + uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 with: languages: python - name: Autobuild - uses: github/codeql-action/autobuild@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 + uses: github/codeql-action/autobuild@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 + uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 diff --git a/.github/workflows/scorecard.yml b/.github/workflows/scorecard.yml index c46e98e..ff12b5a 100644 --- a/.github/workflows/scorecard.yml +++ b/.github/workflows/scorecard.yml @@ -32,6 +32,6 @@ jobs: publish_results: true - name: Upload SARIF - uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 + uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 with: sarif_file: results.sarif From 6dda6fd1d625901352d465a93714da314c7a0d4a Mon Sep 17 00:00:00 2001 From: Austin Kidwell Date: Thu, 10 Sep 2026 06:37:00 -0700 Subject: [PATCH 3/3] docs(handoffs): browser-bridge honesty fix and toolkit reconciliation Findings, corrections to the brief's unverified hypotheses, and a claims table marking every material claim CONFIRMED or HYPOTHESIS with its evidence. The public-history PII finding is stated but its literal path is not: that would put a fresh pointer to the API-keys directory into a public repo. The path is in a private note at ~/.claude/SECURITY-FINDING-2026-09-10-public-history-pii.md. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01F7yhQQARuYRqGTdJJZWvZH --- ...6-09-10_0633-browser-bridge-and-toolkit.md | 245 ++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 handoffs/2026-09-10_0633-browser-bridge-and-toolkit.md diff --git a/handoffs/2026-09-10_0633-browser-bridge-and-toolkit.md b/handoffs/2026-09-10_0633-browser-bridge-and-toolkit.md new file mode 100644 index 0000000..30c1d91 --- /dev/null +++ b/handoffs/2026-09-10_0633-browser-bridge-and-toolkit.md @@ -0,0 +1,245 @@ +# Browser bridge honesty fix, and the toolkit repo reconciled + +Two commits already public in this repository's history carry Austin's name, role and three email addresses, and one of them also names the local folder where his API keys are kept, so that needs his attention before anything else in this document. No key values were exposed and nothing has been rewritten. Separately, the browser bridge now declares when it can only see one of two browsers instead of returning a bare tab list, and the eight unpushed local commits are reconciled and waiting on his review in PR #92. + +--- + +## 1. The thing to read first: what is already public + +**No credentials were found anywhere.** Not in the working tree, not in the ten stale +`.bak` copies, not in any of 256 commits. `MEMORY.md`, `API_KEY_REGISTRY.md` and +`settings.json` have never been committed here. + +What was found is PII and one signpost, both **already on `origin/master`**, both +verified as ancestors of it with `git merge-base --is-ancestor`: + +| Commit | Date | What it publishes | Public for | +|---|---|---|---| +| `3bd5b0b` | 2026-02-20 | Historical `CLAUDE.md`: full name, employer, job title, and **one line naming the local directory where API keys are stored** | ~6 months 3 weeks | +| `5622cca` | 2026-06-09 | Historical `CLAUDE.md`: three email addresses | ~3 months | + +The exact path is deliberately **not reproduced in this file**, because this file lives +in the public repository and restating it would create a fresh, prominent pointer to +where the keys are. It is written out in full, with the reasoning and the remediation +options, in a private note outside this repo: + +``` +~/.claude/SECURITY-FINDING-2026-09-10-public-history-pii.md +``` + +`CLAUDE.md` is no longer in the tree at the tip — commit `25eddb6` untracked it and the +repo now ships `CLAUDE.md.example` only. **That fixed the tip and does nothing to +history.** + +**Nothing was force-pushed, rewritten or deleted**, per the brief. There is also a +practical reason a rewrite is not remediation here: this repository has **57 stars and +14 forks**. Rewriting upstream history does not touch a fork, and fork-network objects +stay reachable by SHA. Anyone who cloned in the last six months already has it. + +**Two decisions are Austin's and were left to him.** Whether this repository should be +public at all — the brief says that has never actually been decided — and whether to +rotate on the basis that the keys folder's location has been public for six months. + +--- + +## 2. Job one — the browser bridge + +### What actually happened on 2026-09-10, evidenced + +The incident report's headline hypothesis was that the tab list was stale. **It was +not.** `handleGetTabs` in `extension/background.js` does a live `chrome.tabs.query({})` +on every call, and a live reproduction this session showed a newly created tab appear in +the very next `browser_get_tabs`. The list was accurate. The tab had been **destroyed**. + +Root cause, from `~/.claude/mcp-debug.log`: + +``` +12:39:22 PID 10776 sessionId=04cd2eca project=intellegix-relay +12:43:06 relay stdin closed — parent exited +12:43:06 relay session orphaned: 04cd2eca +12:43:08 PID 22680 sessionId=a643b291 +``` + +Each bridge process gets a fresh random `sessionId` (`server.js:80`), tabs are grouped by +it, and a relay disconnect made `websocket-bridge.js` broadcast `session_cleanup` +**immediately** — closing every tab in that group. The lane's process exited at 12:43:06 +and its replacement connected 2.3 seconds later under a new id, so the restart could not +reclaim its own tabs. That is 05:39 / 05:43 PT, exactly the incident window. It also +matches the "stuck tabs disappear between attempts" symptom in +`NAVIGATE-HANG-EVIDENCE-2026-08-29.md`. + +### Corrections to the brief + +The brief's hypotheses were explicitly marked unverified. Two do not survive: + +- **"Tab list is stale" — REFUTED.** Live query, verified by reproduction. The real + fault was destruction, not staleness. +- **"relay.mjs throws away a 7-vs-5 discrepancy" — REFUTED as described.** `relay.mjs:194` + filters on `^https?:` to build the takeover picker; `chrome://newtab` is not + controllable by design. No second-browser signal is being discarded there. +- **"Navigate silently retargets" — CONFIRMED** (`background.js:656-668`), reproduced. +- **"Blind to other browsers, reports partial state as complete" — CONFIRMED**, and this + is the design flaw the brief singled out. Fixed. + +### The fix + +Everything is server-side. **No tool was added, renamed or re-described, and no tool +description changed** — a tool-list change invalidates the prompt cache at position 0 for +every lane on this machine. No fleet restart was needed or performed; the new code takes +effect as each lane restarts its own bridge process naturally. The session-keeper Chrome +on port 9223 was never touched. + +- **`lib/browser-discovery.js`** (new) — enumerates Chromium browser *roots* on Windows + via `Get-CimInstance Win32_Process` (`wmic` is gone in 24H2), identifies them by the + absence of a `--type=` switch, reads `DevToolsActivePort`, and does a bounded loopback + CDP probe with an explicit `Host` header. **Detect and declare only — it never attaches + to another browser.** 15-second cache. +- **`browser_get_tabs`** now returns a `coverage` envelope before the data: `status` + (`COMPLETE` / `PARTIAL` / `SCOPED` / `UNAVAILABLE`), `negativeEvidence` + (`SAFE_WITHIN_DECLARED_SCOPE` / `UNSAFE` / `UNKNOWN`), `observedCount` vs + `detectedCount`, and `unobserved[]` naming the CDP endpoint to try. When a negative + conclusion would be invalid it prepends a banner: `RESULT STATUS: PARTIAL — NEGATIVE + EVIDENCE UNSAFE`, and the summary names the invalid inference in words rather than + hedging: *this is NOT evidence that the tab is not open*. The envelope is **always + present**, so the ordinary complete case is an explicit "complete", not silence. +- **`browser_navigate`** now returns `requestedTabId`, `retargeted` and `retargetReason` + instead of `success: true` on a tab the caller never asked for. +- **`websocket-bridge.js` + `config.js`** — `session_cleanup` is deferred by + `sessionCleanupGrace` (45s) and **cancelled** if a relay reconnects for the same project + path. A genuinely dead session still gets cleaned up, 45 seconds later. + +Design informed by two Perplexity passes, each authored through the `prompt-engineering` +skill with real code and real symptoms attached. The precedents it surfaced and the fix +follows: Elasticsearch `_shards` / `timed_out`, GraphQL returning `data` *and* `errors`, +the DNS truncation bit, and HTTP 206 declaring its range. All four say the same thing — +**a partial result is a success that states its own scope**, not an error, and the scope +goes before the data. + +### Verified + +- 18 new tests in `test-coverage-honesty.js`, all passing; they fail against the pre-fix + code. Proven by running the backed-up original: it sends `session_cleanup` + synchronously and has no `_scheduleSessionCleanup`; the new code sends none and cancels + on reconnect. +- 10/10 `npm test`; 60/60 across the existing handler, context-manager and reliability + suites. +- Live census: 382ms, correctly detected both Chromes and named + `http://127.0.0.1:9223` / `session_keeper_profile`. +- Full end-to-end over real MCP stdio: the `PARTIAL` / `UNSAFE` banner appears and the + retarget disclosure is correct. +- **Perplexity verified working through the patched server** (`PPLX_BLOCKS=1`, + `PPLX_ISERROR=false`), because the research pipeline for every lane runs through this + bridge. + +One bug in my own fix was caught by that end-to-end run and is worth recording: +`observedCount` was 0 for every lane, because relay-mode processes have an empty +`browserClients` map and **every lane runs in relay mode**. Fixed by querying the +primary's `/health` for `bridge.browserCount`. The unit tests all passed while this was +broken; only the end-to-end run found it. + +--- + +## 3. Job two — the repo + +`origin` was 25 commits ahead, local had 8 `origin` had never seen, and an untracked +`__pycache__` sat in the tree. + +**PR #92** — all checks green, labelled `needs-austin`. It carries the content of those +8 commits, the browser-bridge fix, and the dependency work. Two things were deliberately +left out: + +- **The CRLF flip.** Local commit `0ec3912` rewrote 211 of 218 tracked text files from LF + to CRLF without changing one line of content (`+61050/-61050`). All 218 text blobs on + `origin/master` were checked and every one is LF-only. The files in #92 are staged as + LF, so its diff is content only: 22 files, not 227. A new `.gitattributes` pins + `* text=auto` — a no-op against the current tree, there purely to stop the next + automated sweep repeating it. +- **Ten `.bak-2026-08-22-*` snapshots** (~14k lines of stale duplicates). Verified every + one still exists on disk under `~/.claude` before excluding them, so nothing is lost, + and git already holds the prior versions. + +### The five Dependabot PRs — the reason not to bulk-merge on title + +Three of the five were **failing CI and could not have passed**, which the titles do not +show. `github/codeql-action/{init,autobuild,analyze,upload-sarif}` is one action that +refuses to run at mixed versions. With no `groups` config, Dependabot opened a separate PR +per sub-action, so merging any single one leaves the workflow half-bumped: + +``` +Loaded a configuration file for version '4.37.9', but running version '4.37.6' +``` + +- **#87, #88, #90** — failing `Analyze (Python)` for that reason. Superseded by #92, which + moves all four references together. Commented, left open until #92 lands. +- **#89** — green only because `scorecard.yml`'s `upload-sarif` is the sole + `codeql-action` step in that file. Rolled into #92 anyway so the repo is never mixed. +- **#91** (pydantic `>=2.13.4` → `>=2.13.5`) — diff read line by line, and 2.13.5 confirmed + on PyPI: uploaded 2026-08-28, not yanked, currently latest. Green and ready. + +The SHA was verified **against the upstream repository, not the PR title**: the annotated +tag `v4.37.9` in `github/codeql-action` resolves to `cdf488f5…`, and the SHA being replaced +resolves to `v4.37.6`. `.github/dependabot.yml` now groups the family so the split cannot +recur. PR #92 passing `Analyze (Python)` is the proof the combined bump works where the +split ones could not. + +### Why nothing merged + +`master` requires **one approving review** (`required_approving_review_count: 1`, +read from the branch-protection API, not inferred from the refusal message). That is +deliberate on a public repo and I did not use `--admin` to go around it. All five +Dependabot PRs plus #92 are labelled `needs-austin` with the decision stated in plain +language in a comment. + +### Local checkout + +Working tree is clean; the `__pycache__` is gone and `.gitignore` gains a +`**/__pycache__/` catch-all in #92. Local `master` is intentionally **left as it was** +until #92 merges — resetting it now would discard content `origin` does not yet have. A +safety branch `backup/local-master-pre-sync-2026-09-10` was created first and holds all 8 +commits, including the three `.bak` blobs that exist only in git and not on disk. Once +#92 merges, the whole reconciliation is one command: + +``` +git checkout master && git reset --hard origin/master +``` + +--- + +## 4. Other findings, with confidence and severity + +| Finding | Confidence | Severity | +|---|---|---| +| **The Chrome extension exists in three copies and the live one matches neither source.** `background.js`: repo 1835 lines, `~/.claude` 2016, the Dropbox copy Chrome actually loads 1862. `content.js` runs the other way — the live copy is 572 against 593 in both sources. Chrome loads the Dropbox copy. Not touched: deploying an unreviewed merge could break the bridge for the whole fleet. | CONFIRMED | **High** — the code being audited is not the code running | +| `sessionId` is regenerated per process (`server.js:80`), so a restart can never reclaim its own tab group by identity. The 45s grace works around this; keying on project path would fix it properly. | CONFIRMED | Medium | +| CI runs `automated-loop`, `health-check` and `minecraft` only. The three new `council-automation` test modules and the 18 browser-bridge node tests are **not run by CI**. Not wired up here — they need a browser and would make CI flaky. | CONFIRMED | Medium | +| `.bak-*` files accumulate inside a public repo through an automated commit sweep with nobody reading them. Ten were about to be pushed. | CONFIRMED | Medium | +| The `browser_get_tabs` **tool description** was deliberately left unchanged, so a model that never reads the response body has no advance warning the envelope exists. Changing it would invalidate the prompt cache fleet-wide. Worth batching into the next unavoidable tool-list change. | CONFIRMED | Low, but a real gap | +| PR #52 is from an outside contributor (`mouse-value-add`), open since 2026-07-03. Not reviewed — out of scope, and it is a product call on a repo whose visibility is unresolved. | CONFIRMED | Informational | +| Benign scanner hits at the tip, checked individually and all fine: docker-compose example creds in `agents/devops.md:122`, `automated-loop/.env.example`, a deliberately fake truncated key in `automated-loop/tests/test_log_redactor.py:19`, a `nonexistent.invalid` URL in `health-check/tests/test_hc_db_pg.py:73`, and "BAD!" doc examples in `patterns/SECURITY_CHECKLIST.md:42` and `rules/python-scripts.md:109`. | CONFIRMED | None | + +--- + +## 5. Claims table + +| Claim | Evidence | Status | +|---|---|---| +| Tab list is live, not stale | `extension/background.js` `handleGetTabs` → `chrome.tabs.query({})`; live repro, new tab appeared in the next call | CONFIRMED | +| Tabs were destroyed by `session_cleanup` on relay disconnect | `~/.claude/mcp-debug.log` 12:39:22 / 12:43:06 / 12:43:08 sequence | CONFIRMED | +| `sessionId` is fresh per process | `server.js:80` `this.sessionId = randomUUID()` | CONFIRMED | +| Navigate silently retargets | `extension/background.js:656-668`; reproduced 1435686256 → 1435686303 with `success: true` | CONFIRMED | +| relay.mjs's "7 tabs, 5 controllable" is a URL-scheme filter, not a coverage signal | `relay.mjs:194` `/^https?:/i` filter, printed at line 200 | CONFIRMED — refutes the brief | +| Two Chrome instances were running; only one was reachable | live census, 382ms, found `session_keeper_profile` on `127.0.0.1:9223` | CONFIRMED | +| The fix does not change the tool list | no add/rename/description change in `server.js`; all edits are response fields and `lib/*` | CONFIRMED | +| Perplexity still works after the change | end-to-end through the patched server, `PPLX_BLOCKS=1`, `PPLX_ISERROR=false` | CONFIRMED | +| Nothing about to be pushed contains a credential | scan of the exact staged bytes of all 21 files, 14 credential patterns, **with a positive control proving the scanner was reading them** | CONFIRMED | +| An earlier version of that scan was worthless | it reported 0 hits while `grep` found matches in the same bytes; re-run with a positive control | CONFIRMED — my own error, caught before the push | +| `3bd5b0b` and `5622cca` are ancestors of `origin/master` | `git merge-base --is-ancestor`, both exit 0 | CONFIRMED | +| Repo is public with 57 stars and 14 forks | `gh repo view --json isPrivate,visibility,createdAt,stargazerCount,forkCount` | CONFIRMED | +| `origin/master` is entirely LF | all 218 text blobs read and counted; 0 contain CRLF | CONFIRMED | +| `0ec3912` is line-ending noise | 227 files differ, 22 differ under `--ignore-cr-at-eol` | CONFIRMED | +| Three codeql PRs cannot pass CI individually | run 33384590794 error line, quoted above; PR #92 with all four bumped passes the same check | CONFIRMED | +| `cdf488f5…` really is codeql-action v4.37.9 | `gh api repos/github/codeql-action/commits/v4.37.9` | CONFIRMED | +| pydantic 2.13.5 is real and not yanked | PyPI JSON API, uploaded 2026-08-28 | CONFIRMED | +| `master` requires 1 approving review | `gh api .../branches/master/protection` → `required_approving_review_count: 1` | CONFIRMED | +| Keying tab groups on project path instead of `sessionId` would remove the need for the grace period | reasoning from the code, not implemented or tested | HYPOTHESIS | +| The extension three-way drift is causing other unexplained bridge behaviour | drift is measured; a causal link to any specific bug is not | HYPOTHESIS |