feat(leads): step through paginated directories - #145
Merged
Merged
Conversation
A listing page is a fraction of the directory behind it, and a fraction reported as a total is indistinguishable from a small directory. Seeds now walk the pager before opening anything. Following the next page by URL is tried before anything else, so the fetch-first path still applies and an ordinary paginated directory costs one cheap request per page rather than one browser render per page. The three ways of finding that URL are ordered by how much the page is actually asserting. rel="next" is the site stating it outright. Link text is a convention, and is matched whole — a link reading "next" is a pager, one reading "next steps" is prose — while an aria-label is matched on the word, because labels are written to be read aloud and say "Next page". Incrementing a page parameter is a guess, so it comes last and only when the URL already carries one; inventing ?page=2 asks every site on the internet for a page that may not exist. Disabled controls are refused, since a greyed-out arrow means this is the last page and following it loops. The guess needed a second guard that only a live run revealed. A site that ignores its page parameter answers every guess with the same page, and the walk dutifully fetched twenty-one identical pages before stopping at the cap. A page that contributes no new entries is now the end of the list whatever its pager claims, which took the same run from twenty-one page loads to two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vu1nz Security Review0 finding(s) in PR #? No security issues found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A listing page is a fraction of the directory behind it, and a fraction reported as a total is indistinguishable from a small directory. Seeds now walk the pager before opening anything.
URL-following first, so fetch-first survives
Following the next page by URL is tried before anything else, so an ordinary paginated directory costs one cheap request per page rather than one browser render per page. Clicking a JS-only control is the expensive fallback.
Ordered by how much the page is asserting
rel="next"aria-label?page=NLink text is matched whole — a link reading "next" is a pager, one reading "next steps" is prose.
aria-labelis matched on the word, because labels are written to be read aloud ("Next page").The page-parameter guess only fires when the URL already carries one. Inventing
?page=2asks every site for a page that may not exist. Disabled controls are refused — a greyed-out arrow means last page, and following it loops.The guard a live run made necessary
The guess needed a second stop condition that no unit test would have surfaced. A site that ignores its page parameter answers every guess with the same page — so the walk fetched 21 identical pages before hitting the cap.
A page contributing no new entries is now the end of the list, whatever its pager claims:
Checks
tsc --noEmitclean🤖 Generated with Claude Code