Never charge an uptime monitor, and add Oracle's ranges - #365
Merged
Merged
Conversation
Two things found by watching the gate run in production rather than by reading it. **Uptime monitors would have paged us.** A monitor counts 200-399 as up — CrawlProof's own checkHttp does exactly that — so answering one 402 does not bill it, it reports THE SITE AS DOWN. Ours runs on Railway, which is to say from a cloud address, so the range check alone would have caught it and we would have built ourselves a false alarm generator and then been woken by it. Exempted by user agent, ours and the common third parties. **Oracle was missing.** It publishes ap-singapore-1 and ap-singapore-2 in a clean official file, which is exactly the shape of host this gate exists for. With it the list is 13,724 prefixes merging to 1,671 ranges. Verified against the live files, including the control that HTTP probes could not give us — every request from this machine arrives from a DigitalOcean address, so no forged X-Forwarded-For can simulate a residential visitor: 3.0.0.1 AWS matched 140.238.1.1 Oracle Singapore matched 67.205.189.229 our own DO box matched 86.1.2.3 UK residential NOT matched 24.60.1.2 US Comcast NOT matched Azure and Alibaba are still uncovered and now say so in the source. Azure's file URL carries a date and rotates weekly; Alibaba publishes nothing and would need its ASNs resolved through BGP data. Both matter for Singapore, where Alibaba is a common host for this exact traffic — so a miss here does NOT establish that an address is residential, and the comment says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan331 finding(s) HIGH/CRITICAL: 33 | MEDIUM: 40 | LOW: 258
…and 281 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things found by watching the gate run in production, not by reading it.
Uptime monitors would have paged us
A monitor counts 200-399 as up — CrawlProof's own
checkHttpdoes exactly that — so answering one 402 does not bill it, it reports the site as down. Ours runs on Railway, i.e. from a cloud address, so the range check alone would have caught it. We would have built ourselves a false-alarm generator and then been woken by it.Exempted by user agent: ours plus the common third parties (UptimeRobot, Pingdom, StatusCake, Better Uptime, Checkly, updown.io, …).
Oracle was missing
It publishes
ap-singapore-1andap-singapore-2in a clean official file — exactly the shape of host this gate exists for. With it: 13,724 prefixes merging to 1,671 ranges.The control HTTP probes could not give us
Every request from the dev box arrives from a DigitalOcean address, so no forged
X-Forwarded-Forcan simulate a residential visitor — which is why my earlier prod probes all showed 402 and looked alarming. Verified at the matcher against the live files instead:Real users are not matched. That was the open question after #364 went live.
Still uncovered, and the source now says so
Azure's range file URL carries a date and rotates weekly; Alibaba publishes nothing and would need its ASNs resolved through BGP data. Both matter for Singapore, where Alibaba is a common host for this exact traffic — so a miss here does not establish that an address is residential, and
UNCOVERED_PROVIDERSdocuments it rather than leaving it to be rediscovered.Testing
61 tests across
cloud-gate,footprintandproxy.runtime;tsc --noEmitclean.🤖 Generated with Claude Code