Charge the crawler that does not announce itself - #363
Merged
Merged
Conversation
…ror"
A misconfigured Infura project took down every ETH and POL operation. It had
"require API key secret" switched on while ETHEREUM_RPC_URL carried only the
project id, so every call came back
403 private key only is enabled in Project ID settings
Nothing failed over, because the endpoint was a single string. Balance reads
logged "checkEVMBalance: lookup failed" and sends from the wallet extension
died outright.
What made it expensive was the reporting, not the outage. estimateFees() was
called OUTSIDE the try in prepareTransaction, so the throw escaped to the
route's outer catch, which answers serverError() with no argument: the payer
saw the bare string "Internal server error", with no chain, no status and no
cause. A dead provider was indistinguishable from a bug in our own code.
So:
- evm-rpc.ts resolves an endpoint LIST per chain (configured first, then
keyless public fallbacks) and walks it. Only transport failures fail over,
a network error or a non-2xx status, which mean the provider is broken. A
JSON-RPC error inside a 200 is the chain talking and is returned as-is:
every provider would repeat it, and for a send retrying it would mean
resubmitting.
- Errors name each host and status, never the URL, which carries the API key.
- estimateFees() moved inside the try, so a fee failure is PREPARE_FAILED
plus the real message instead of "Internal server error".
- USDC_BASE resolves to Base, not Ethereum. These are substring matches and
the wrong one reads a nonce from the wrong chain without failing loudly.
- "already known" from eth_sendRawTransaction is now success, not failure. It
means the node already holds these exact signed bytes, so the money moved;
marking the row failed was the worst of both records. The hash is keccak256
of bytes we already hold, so it needs no provider to report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extracted from launchpadder-web's ip-protection-service, which had been doing
VPN/proxy/Tor detection against ip-api for a while. Not yet wired to anything:
landing it on its own so the extraction is reviewable separately from any
policy change it might later inform.
Three defects were fixed on the way out, each real rather than stylistic:
- `fetch(url, { timeout: 5000 })` does nothing. `timeout` is not a fetch
option in any runtime, so the call had NO deadline. Now an AbortSignal.
- The reputation table scored google/amazon/microsoft/cloudflare as TRUSTED.
For scraper detection that is exactly inverted: those are the providers
scraping runs from. Hosting is now a positive signal of automation.
- A failed lookup returned `proxy: false, vpn: false`, indistinguishable from
a clean result, so a dead upstream silently read as "everyone is innocent".
Failures are marked `unknown` and score nothing.
Header signals (./headers.js) carry the cheap half and need no upstream: the
load-bearing one is that every Chromium since 76 sends Sec-Fetch-Mode and
cannot suppress it, so a request claiming Chrome/148 without it is an HTTP
client wearing a copied string. Self-declared agents are exempt from that
check — Googlebot claims Chrome and sends no Sec-Fetch-Mode, and convicting
it would cut off indexing.
Scoring is three-valued (human / unclear / automated) because two is not
enough to act well, and hosting alone never convicts: corporate VPNs,
university egress and privacy relays all resolve to infrastructure while
carrying real readers.
21 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The crawl gateway judges by user agent, and the traffic that actually costs us
does not lie in one. A headless Chromium in a Singapore cloud region was 95.8%
of this site's reported HUMAN traffic — 152,131 hits in a month — and passed
every check we have, because it is not pretending to be a browser. It is one.
It renders the page and fires the analytics beacon, which is exactly why it
was counted as a reader.
The tell was geography, and it needed the countries/cities panels added to
CrawlProof's stats API to see at all: 152,131 hits from Singapore, of which
267 resolve to the city of Singapore. Country resolves, city does not, which
is what a datacenter allocation looks like in a geo database.
So judge the one thing it cannot dress up. AWS, GCP and DigitalOcean each
publish their address space as a machine-readable file, so the ranges are
knowable exactly — no IP reputation service, no per-request lookup, no quota,
and without our storing anybody's address to work it out. Live against the
real files: 12,619 prefixes merge to 1,194 ranges, and 200k lookups take
177ms.
It ASKS FOR MONEY rather than refusing. A datacenter address is not
misconduct: agents, integrations and corporate egress live there too, and an
agent that wants the pages can buy the same pass a declared crawler buys.
402 is a question a caller can answer; 403 is not.
Never charged, each for a reason that would otherwise have hurt:
/api/* Stripe, Column and Plaid call us from cloud addresses by
nature, as do the x402 endpoints themselves.
no client IP Railway runs ON a cloud and healthchecks `/`. Charging that
marks the deploy unhealthy and breaks every future deploy.
signed in They already pay us.
search bots Googlebot and the retrieval half send readers back; charging
them de-indexes the site.
Ships INERT. `CLOUD_CHARGE=pages` turns it on. This can answer a real customer
with a payment demand if a rule is wrong, on the live payment platform, so it
is switched on deliberately with the logs watched rather than by the act of
merging it. Before the first range download lands the list is empty and
matches nobody, so a slow or failed fetch degrades to today's behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runtime test builds a throwaway Next app from an explicit list of src/lib files and runs the real HTTP pipeline against it. A module the interception files import but the list omits does not fail to compile — the fixture cannot resolve it, the proxy module fails to load, and EVERY request answers 500. That is what four of these tests were reporting. Adding cloud-gate.ts, and a comment saying why the list has to track the imports, since the failure it produces points nowhere near the cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan331 finding(s) HIGH/CRITICAL: 33 | MEDIUM: 40 | LOW: 258
…and 281 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The crawl gateway judges by user agent, and the traffic that actually costs us does not lie in one.
A headless Chromium in a Singapore cloud region was 95.8% of this site's reported HUMAN traffic — 152,131 hits in a month — and passed every check we have, because it is not pretending to be a browser. It is one. It renders the page and fires the analytics beacon, which is exactly why it was counted as a reader.
Finding it
It needed the countries/cities panels added to CrawlProof's stats API (crawlproof.com#279) to see at all:
Singapore never appears in the bot panel. And of those 152,131 hits, 267 resolve to the city of Singapore — 0.18%. Country resolves, city does not, which is what a datacenter allocation looks like in a geo database.
The approach
Judge the one thing it cannot dress up. AWS, GCP and DigitalOcean each publish their address space as a machine-readable file, so the ranges are knowable exactly — no IP reputation service, no per-request lookup, no quota, and without our storing anybody's address to work it out.
Verified against the live files: 12,619 prefixes merge to 1,194 ranges; 200k lookups take 177ms (~0.9µs each).
It asks for money rather than refusing. A datacenter address is not misconduct — agents, integrations and corporate egress live there too, and an agent that wants the pages can buy the same pass a declared crawler buys. 402 is a question a caller can answer; 403 is not.
Never charged
/api/*/. Charging that marks the deploy unhealthy and breaks every future deployShips inert
CLOUD_CHARGE=pagesturns it on. This can answer a real customer with a payment demand if a rule is wrong, on the live payment platform, so it is switched on deliberately with the logs watched rather than by the act of merging it. Before the first range download lands the list is empty and matches nobody, so a slow or failed fetch degrades to today's behaviour.Also here
@profullstack/footprint— extracted from launchpadder'sip-protection-service. Three real defects fixed on the way out:fetch(url, {timeout: 5000})was a no-op (not a fetch option, so the call had no deadline);google/amazon/microsoftwere scored trusted, exactly inverted for scraper detection; and a failed lookup returnedproxy: false, indistinguishable from clean, so an outage read as "everyone is innocent".proxy.runtime.test.ts— the fixture-app file list now includescloud-gate.ts. A missing entry there does not fail to compile; the fixture cannot resolve the module and every request answers 500, which is how that test earned its keep here.Testing
packages/footprint, 12 incloud-gate(every exclusion pinned individually), 5 inproxy.runtime.src/lib/banking/service.test.ts, verified identical on cleanorigin/master— pre-existing and unrelated.tsc --noEmitclean.🤖 Generated with Claude Code