fix(newsletter): record walks headed, skip the root load, and dismiss signup modals - #146
Merged
Merged
Conversation
… signup modals Two of five articles in issue 403 could not be recorded at all, and the third recorded the wrong thing. All three failures were in how the walk reaches and holds the page. capture-gifs now passes --headed and --no-root to the engine. Headless Chromium is fingerprinted and blocked outright by Cloudflare-fronted sites: the same Medium article returns the real page headed and "Sorry, you have been blocked" headless. Separately, the engine loads the site root before handing over, and on those sites the root-then-deep-link hop is itself what trips the bot check, while going straight to the article is served normally. The root load was also wasted footage the runner only trimmed back off. Both flags are new in site-capture and need that engine. The flow now presses Escape and clicks a close control at each beat. Medium interrupts a scroll partway down with a signup modal that then covers the body for the rest of the walk, which is what the first successful Netflix recording captured. Dismissing costs nothing on a page with no modal, so it runs at every beat rather than only on arrival. Verified by extracting frames rather than trusting the beat log: the Netflix walk now shows article body throughout. openai.com still fails, serving its own "This page couldn't load" error to the automation browser regardless of these flags, so that article needs an image or a hand-recorded clip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two of five articles in issue 403 could not be recorded at all, and the one that did record captured the wrong thing. All three failures were in how the walk reaches and holds the page.
Depends on EthicalML/agent-skills-marketplace#16, which adds
--no-rootand the tolerant root load to the site-capture engine. Merge and update the plugin before this lands, or the flags are ignored.Headless was blocked
Same URL, same navigation, differing only in mode:
"Attention Required! | Cloudflare"The root hop was also blocked
The engine loads the site root before handing over to the flow. On a Cloudflare-fronted site that hop is itself what trips the bot check:
"Attention Required! | Cloudflare"--no-rootskips it, which also removes footage the runner only trimmed back off.Medium covers the article mid-scroll
The first recording that got through captured the article and then a Medium signup modal, which appears partway down a scroll and sits over the body for the rest of the walk. The flow now presses Escape and clicks a close control at each beat rather than only on arrival, since the modal appears on scroll rather than on load. It costs nothing on a page that has none.
Verification
Frames extracted from the recording rather than trusting the beat log, which only proves the script ran. The Netflix walk now shows article body throughout, with no modal.
Still failing
openai.comserves its own "This page couldn't load" error to the automation browser regardless of these flags. That article needs a still image or a hand-recorded clip; nothing here fixes it.