Repository navigation
CDP: leave a connected browser running, never hang connecting, retry a 500 - #10
Merged
Merged
Conversation
…a 500 Two defects from a manual test of v0.2.0, both reproduced on a local Chrome started with --remote-debugging-port before fixing. pyppeteer shut down the browser it connected to. Its teardown called `Browser.close()`, which for a connected browser sends CDP `Browser.close`: the browser on the other end died, `Event loop is closed` tracebacks were printed, and the next run hung. Reproduced: browser DEAD after run 1, run 2 killed by a 60 s timeout. A connected browser now has its page closed and the connection dropped (`disconnect()`); only a browser the engine launched is closed. The connection's tasks are drained before the event loop is closed. It also took the remote browser's own first tab (`pages()[0]`) and navigated it away; it now opens a new page. The hang itself was pyppeteer's `connect()`: it opens the WebSocket in a background task and waits for the answer to its first command, so a refused port or an HTTP 500 handshake died in that task and `connect()` waited forever. Measured against a local server answering 500 and against a closed port: both hung. The engine now connects through a small wrapper that waits on the command AND the background task and raises the real reason — `HTTP 500`, `Connection refused`, or a handshake that never completed — within the timeout. Playwright left a tab per run in a browser it connected to: `Browser.close()` over CDP only disconnects. It now closes the page it opened, and the context if it created one. Reproduced: 1 -> 2 -> 3 tabs over two runs; after the fix 1 -> 1 -> 1. Both engines now retry a CDP handshake refused with HTTP 500, 502 or 503, twice, after 3 s and 6 s — a managed browser can refuse while a previous session on the same profile closes. The policy is data in `page_flow`; 401, a closed port and a timeout fail at once. Verified: - local Chrome, two runs per engine: browser ALIVE, tabs 1 -> 1, 32 rows each, no tracebacks. - local server answering 500: both engines retry twice and exit 5 in 9 s with the status in the message; closed port: exit 5 at once; an endpoint that accepts TCP and never answers: pyppeteer exits 5 in 20 s instead of hanging. A password planted in the endpoint never reached the output. - NOT verified live on the Scraping Browser API: every endpoint in the workspace answered 401 (expired credentials), which both engines now report at once without retrying. - non-CDP runs of both engines unchanged. - tests: the retry policy as data; each driver's teardown on recorded fakes; pyppeteer's connect against a real local 500 server and a closed port; the retry loops. Six controls in the engine venvs, each caught by its named check; two tests were changed to report a raised error by name after a control showed them crashing instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Reported by the manual test of v0.2.0 (#3 and #9) and reproduced on a local Chrome started with
--remote-debugging-portbefore fixing:Event loop is closedtracebacksBrowser.close(), which for a connected browser sends CDPBrowser.close;connect()has no way to failBrowser.close()over CDP only disconnects; the page opened for the run stayedThe hang has a narrower cause than "no timeout": pyppeteer's
connect()opens the WebSocket in a background task and waits for the answer to its first command. A refused port or an HTTP 500 handshake kills that task, andconnect()waits forever. Measured against a local server answering 500 and against a closed port: both hung, and the real error showed only as a "Task exception was never retrieved" traceback at exit.pyppeteer also took the remote browser's own first tab (
pages()[0]) and navigated it away.Behaviour after
disconnect(); a launched one is closed. Pending connection tasks are drained before the loop closes. Over CDP it opens a new page. It connects through a small wrapper that waits on the first command and the background task, so a refusal raises the real reason (HTTP 500,Connection refused, or a handshake that never completed) within the timeout.page_flow; 401, a closed port and a timeout fail at once with exit 5.Evidence
A password planted in the endpoint URL never reached the output. Non-CDP runs of both engines are unchanged.
Not verified live on the 2Captcha Scraping Browser API: every endpoint available here answered 401 (expired credentials), which both engines now report at once without retrying. The 500-retry path is exercised against a local server that answers 500.
Tests
websocketsstatus_codeandresponse.status_code, 401, refused, timeout, an unrelated "500").🤖 Generated with Claude Code
https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P