Repository navigation
Scraper API on the shared output contract; --url keeps its market; docs and packaging - #11
Merged
Merged
Conversation
…cs and packaging From a manual test of v0.2.0 (#14, #15, #16), plus one defect found while verifying #14. A market in --url was overwritten. `--gl`/`--hl` defaulted to US/en and were written over the URL, so `--url ".../search?q=vpn&hl=ja&gl=JP"` fetched the US store — USD, English titles, rows labelled US — silently. Found by comparing the Scraper API with Playwright on that URL: 23 of 30 apps shared and 58 field differences, with the Scraper API the one showing JPY. The URL's own gl/hl now outrank the defaults; a typed flag replaces them and says so. After the fix the two engines returned the same 30 apps with no field different. Scraper API (#14): it wrote rows only — no `.meta.json`, so `diff_runs.py` could not use it; `page`/`position` empty; `--format both` where the engines default to json; and "not run against this site" text. It now reads the page state through the engines' `_handle_state` and writes through `finish_run`, fills page and position, defaults to json, normalises `--out` the same way, and pins hl/gl in the request — without them Play chose the market from the API's exit while rows were labelled with a default. Run live 2026-10-22: a category (96 rows), a JP search (30), an app (1, position 1), a developer (10), all exit 0 with a sidecar; a `gl` Play does not serve exits 2; $0.0006 for the five requests. An app row from `parse_products` also gets position 1, as the engines' own app run gives it. diff_runs (#15): a refusal now ends with advice for its actual reason (the same --gl, the same --mode, re-run the incomplete side, compare reviews on review_id) instead of "re-run the incomplete side" for every reason. Docs (#15, #16 and the pyppeteer note): README explains why `original_price` and `discount_pct` are null (0 of 411 capture rows, 0 of 877 live rows); the Scraper API is in the engine table; pyppeteer on Apple Silicon is documented — its Chromium download has no arm64 entry. `.dockerignore` leaves out `.venv-*/`, `venv-*/` and `captures/`: a build with a planted 20 MB venv sent 288 kB of context, the image built, ran `--help` and holds no `.env` and no captures. Tests: the Scraper API path through its own `main()` with the network replaced by captures; market precedence for `--url` with and without typed flags, and a fake-driver run of a JP URL; diff_runs' advice line for two complete markets and for an incomplete run. Six controls, each caught by its named check — one of them showed the diff test matching "same --gl" in the problem text rather than the advice, and the test now reads the advice line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P
Merged
jehrr
added a commit
that referenced
this pull request
Oct 22, 2026
Version and changelog for the five fixes merged as #7-#11, from a manual test of v0.2.0. The section opens with the behaviour changes an existing caller will see — reviews order and paging, exit 2 for a --gl Play does not serve, --url keeping its market, --out suffixes, the no_new_products stop_reason, and the Scraper API's json default and sidecar — followed by the correction of the README's earlier claim about the store's review orderings. The auditor's own reproduction steps were re-run against the merged code before this commit: 13 of 13 runnable items now give the expected result. Claude-Session: https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P Co-authored-by: Petr <petr@2captcha.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
From the manual test of v0.2.0 (#14, #15, #16), plus one defect found while verifying #14.
--urllost its market (new).--gl/--hldefaulted toUS/enand were written over the URL, so--url ".../search?q=vpn&hl=ja&gl=JP"fetched the US store without saying so: USD prices, English titles, rows labelled US. Comparing the Scraper API with Playwright on that URL showed 23 of 30 apps shared and 58 differing fields, and the Scraper API was the one returning JPY.Scraper API (#14): wrote rows only — no
.meta.json(sodiff_runs.pycould not use the output),page/positionempty,--format bothwhere the engines default tojson, and docstring/help text saying it had not been run on this site. It also sent a URL withouthl/glas-is, so Play chose the market from the API's exit while the rows were labelled with a default.#15:
diff_runs.pyended every refusal with "re-run the incomplete side", including when both runs were complete;original_price/discount_pctwere always null with no explanation; the rotation log in reviews (fixed in #7).#16:
.dockerignoresent per-engine virtualenvs andcaptures/to the build, about 240 MB.Behaviour after
gl/hloutrank the defaults. A typed--gl/--hlreplaces them and logs--gl DE replaces gl=JP in --url._handle_state; output throughfinish_run(sidecar, the same exit codes);page/positionfilled;--format json;--outnormalised;hl/glpinned in the request. An app row fromparse_productsgets position 1, as the engines' app run gives it.diff_runs.pygives advice for the actual reason: the same--gl, the same--mode, re-run the incomplete side, or compare reviews onreview_id.original_price/discount_pctare null, the Scraper API row in the engine table, and pyppeteer on Apple Silicon (its Chromium download has no arm64 entry)..dockerignore:.venv-*/,venv-*/,captures/.Evidence (2026-10-22)
page1,position1,screenshot_count9), developer 10, all exit 0 with a sidecar;gl=xx→ exit 2. $0.0006 for the five requests.original_price/discount_pctnull on 0 of 411 rows parsed from the captures and 0 of 877 rows from today's live runs..venv-playwright→ 288 kB of context; the image builds, runs--help, and contains no.envand no captures.main()with captures in place of the network; market precedence (resolution and a fake-driver JP run);diff_runsadvice for two complete markets and for an incomplete run. Six controls, each caught by its named check. One of them showed the diff test matching "same --gl" in the problem text rather than the advice, and the test now reads the advice line.🤖 Generated with Claude Code
https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P