Skip to content

CDP: leave a connected browser running, never hang connecting, retry a 500 - #10

Merged
jehrr merged 1 commit into
mainfrom
fix/cdp-lifecycle
Oct 22, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix/cdp-lifecycle

Conversation

@jehrr

@jehrr jehrr commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Problem

Reported by the manual test of v0.2.0 (#3 and #9) and reproduced on a local Chrome started with --remote-debugging-port before fixing:

engine before cause
pyppeteer browser DEAD after run 1; run 2 hangs until killed; Event loop is closed tracebacks teardown called Browser.close(), which for a connected browser sends CDP Browser.close; connect() has no way to fail
Playwright tabs 1 → 2 → 3 over two runs Browser.close() over CDP only disconnects; the page opened for the run stayed

The hang has a narrower cause than "no timeout": pyppeteer's connect() opens the WebSocket in a background task and waits for the answer to its first command. A refused port or an HTTP 500 handshake kills that task, and connect() waits forever. Measured against a local server answering 500 and against a closed port: both hung, and the real error showed only as a "Task exception was never retrieved" traceback at exit.

pyppeteer also took the remote browser's own first tab (pages()[0]) and navigated it away.

Behaviour after

  • pyppeteer: a connected browser gets its page closed and disconnect(); a launched one is closed. Pending connection tasks are drained before the loop closes. Over CDP it opens a new page. It connects through a small wrapper that waits on the first command and the background task, so a refusal raises the real reason (HTTP 500, Connection refused, or a handshake that never completed) within the timeout.
  • Playwright: over CDP it closes the page it opened, and the context if it created one, then disconnects.
  • Both: a handshake refused with HTTP 500/502/503 is retried twice (3 s, 6 s). A managed browser can refuse that way while a previous session on the same profile closes. The policy is data in page_flow; 401, a closed port and a timeout fail at once with exit 5.

Evidence

scenario Playwright pyppeteer
local Chrome, 2 runs exit 0 ×2, browser alive, tabs 1→1, 32 rows exit 0 ×2, browser alive, tabs 1→1, 32 rows, no tracebacks
local server answering 500 2 retries, exit 5 in 9 s, status in message 2 retries, exit 5 in 9 s, status in message
closed port exit 5 at once exit 5 at once
accepts TCP, never answers — exit 5 in 20 s (was: hang)

A password planted in the endpoint URL never reached the output. Non-CDP runs of both engines are unchanged.

Not verified live on the 2Captcha Scraping Browser API: every endpoint available here answered 401 (expired credentials), which both engines now report at once without retrying. The 500-retry path is exercised against a local server that answers 500.

Tests

  • The retry policy as data (Playwright's message form, websockets status_code and response.status_code, 401, refused, timeout, an unrelated "500").
  • Each driver's teardown on recorded fakes, connected and launched.
  • pyppeteer's connect against a real local 500 server and a closed port: raises promptly with the status.
  • Both retry loops: two 500s then success; a 401 once.
  • Six controls in the matching engine venvs, each caught by its named check. Two tests were changed to report a raised error by name after a control showed them crashing instead of failing by name.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P

…a 500

Two defects from a manual test of v0.2.0, both reproduced on a local Chrome
started with --remote-debugging-port before fixing.

pyppeteer shut down the browser it connected to. Its teardown called
`Browser.close()`, which for a connected browser sends CDP `Browser.close`:
the browser on the other end died, `Event loop is closed` tracebacks were
printed, and the next run hung. Reproduced: browser DEAD after run 1, run 2
killed by a 60 s timeout. A connected browser now has its page closed and
the connection dropped (`disconnect()`); only a browser the engine launched
is closed. The connection's tasks are drained before the event loop is
closed. It also took the remote browser's own first tab (`pages()[0]`) and
navigated it away; it now opens a new page.

The hang itself was pyppeteer's `connect()`: it opens the WebSocket in a
background task and waits for the answer to its first command, so a
refused port or an HTTP 500 handshake died in that task and `connect()`
waited forever. Measured against a local server answering 500 and against a
closed port: both hung. The engine now connects through a small wrapper
that waits on the command AND the background task and raises the real
reason — `HTTP 500`, `Connection refused`, or a handshake that never
completed — within the timeout.

Playwright left a tab per run in a browser it connected to:
`Browser.close()` over CDP only disconnects. It now closes the page it
opened, and the context if it created one. Reproduced: 1 -> 2 -> 3 tabs over
two runs; after the fix 1 -> 1 -> 1.

Both engines now retry a CDP handshake refused with HTTP 500, 502 or 503,
twice, after 3 s and 6 s — a managed browser can refuse while a previous
session on the same profile closes. The policy is data in `page_flow`;
401, a closed port and a timeout fail at once.

Verified:
- local Chrome, two runs per engine: browser ALIVE, tabs 1 -> 1, 32 rows
  each, no tracebacks.
- local server answering 500: both engines retry twice and exit 5 in 9 s
  with the status in the message; closed port: exit 5 at once; an endpoint
  that accepts TCP and never answers: pyppeteer exits 5 in 20 s instead of
  hanging. A password planted in the endpoint never reached the output.
- NOT verified live on the Scraping Browser API: every endpoint in the
  workspace answered 401 (expired credentials), which both engines now
  report at once without retrying.
- non-CDP runs of both engines unchanged.
- tests: the retry policy as data; each driver's teardown on recorded fakes;
  pyppeteer's connect against a real local 500 server and a closed port;
  the retry loops. Six controls in the engine venvs, each caught by its
  named check; two tests were changed to report a raised error by name after
  a control showed them crashing instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P
@jehrr
jehrr merged commit 3080e1e into main Oct 22, 2026
7 checks passed
@jehrr
jehrr deleted the fix/cdp-lifecycle branch October 22, 2026 22:10
@jehrr jehrr mentioned this pull request Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant