feat: add subdivisionCode to proxy configuration#592
Conversation
github-actions
Bot
added
t-unblocking
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
Thank you @daniil-poletaev !
Regarding the new separate option, I have another question. Are we planning to support passing multiple countries at some point? (e.g., "I want to scrape xyz.com either through Belgium or the Netherlands to make the proxy pool larger").
In such a case, separate country and subdivision code options would require us to change the API again, as we couldn't match the country and the subdivision code
countryCode: ['BE', 'NL'],
subdivisionCode: ['BRU', 'AMS'] // which country do these match?Compare to the alternative countryCode: ["BE-BRU", "NL-AMS"], which disambiguates by default.
But, if we're positive that we don't want to go this way, it's a rather theoretical problem :)
| const CHECK_ACCESS_REQUEST_TIMEOUT_MILLIS = 4_000; | ||
| const CHECK_ACCESS_MAX_ATTEMPTS = 2; | ||
| const COUNTRY_CODE_REGEX = /^[A-Z]{2}$/; | ||
| const SUBDIVISION_CODE_REGEX = /^[A-Z0-9]{2}$/; |
There was a problem hiding this comment.
Note that, e.g., United Kingdom or Belgium do three-lettered codes
There was a problem hiding this comment.
Good catch 👍
Fixed it
|
|
||
| if (subdivisionCodeToUse && !countryCodeToUse) { | ||
| throw new Error( | ||
| '"subdivisionCode" requires "countryCode" to be set.', |
There was a problem hiding this comment.
Perhaps we should make this even more explicit (e.g., mention ProxyConfiguration in the error message), so the user knows immediately where this comes from
Maybe, but tbh it still would be a breaking change and we can change format later if it would be needed, so I would keep the format as-is for now 🤔 |
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
Alright, I don't have any further objections then :) (see small typo below)
Thank you @daniil-poletaev !
| * Same option as `subdivisionCode` which can be used to | ||
| * configurate the proxy by UI input schema. You should use the `subdivisionCode` option in your crawler code. |
There was a problem hiding this comment.
| * Same option as `subdivisionCode` which can be used to | |
| * configurate the proxy by UI input schema. You should use the `subdivisionCode` option in your crawler code. | |
| * Same option as `subdivisionCode` which can be used to | |
| * configure the proxy by UI input schema. You should use the `subdivisionCode` option in your crawler code. |
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
Sorry for the last-minute note - can you also please make changes to https://docs.apify.com/sdk/js/docs/concepts/proxy-management , just like with the Python PR? Thanks!
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
no further comments, thank you @daniil-poletaev !
| const CHECK_ACCESS_REQUEST_TIMEOUT_MILLIS = 4_000; | ||
| const CHECK_ACCESS_MAX_ATTEMPTS = 2; | ||
| const COUNTRY_CODE_REGEX = /^[A-Z]{2}$/; | ||
| const SUBDIVISION_CODE_REGEX = /^[A-Z0-9]{1,3}$/; |
There was a problem hiding this comment.
I would highly recommend add there a comment that explains that 1-3 range for an ISO standard that expects only 2 chars 😄
This was something that confused me.
This PR adds new variable `subdivision_code` to Apify proxy configuration. This variable is currently needed for US state targeting. More context: https://apify.slack.com/archives/C010Q0FBYG3/p1776704073278519 Same PR in JS: apify/apify-sdk-js#592
bliuchak
added
the
validated
4 participants
This PR adds new variable
subdivisionCodeandapifyProxySubdivisionto Apify proxy configuration. This variable is currently needed for US state targeting.More context: https://apify.slack.com/archives/C010Q0FBYG3/p1776704073278519
Same PR in python: apify/apify-sdk-python#878