feat(ads): share the visitor id with /ad.js, salt + rotate IP hashes - #191
Merged
Merged
Conversation
Three changes to make ad metering identify visitors it previously couldn't, and to stop the fallback identifier from being reversible. 1. Shared visitor id. /ad.js only ever *read* localStorage['crawlproof.visitor']; nothing but /stats.js wrote it, so a publisher running the ad tag without the analytics tag sent an empty visitor on every impression -- ~69% of production impressions carried no visitor id. The minting logic moves to lib/tracker/visitorSnippet.ts and is now inlined verbatim by both routes, so the ad tag mints the id when it is the first CrawlProof script on the page. Still localStorage, still no cookie. 2. Salted, rotating IP hashes. lib/ads/serve.ts hashed IPs as bare sha256(ip) truncated to 32 hex chars. IPv4 is 2^32 addresses, so that is a reversible encoding of the IP, not a pseudonym -- and ip_hash is the only identifier the ad network has for terminal traffic, which has no localStorage at all. Both implementations (here and lib/rateLimit.ts, which used a hardcoded constant prefix) collapse into lib/ipHash.ts. Two exported variants over one implementation, because the callers need opposite properties: abuse caps look back 24h+ and must not reset at the rotation boundary, so they keep a stable salt; ad metering only needs to recognise an IP for hours, so it rotates daily and the ability to correlate expires on its own. Click dedupe queries every salt window inside its 6h lookback, so the check doesn't silently miss after each rotation. IP_HASH_SALT unset reproduces the legacy digest byte for byte, so deploying this without the env var set changes nothing. 3. Documented ?v= for terminal publishers. A terminal has no cookies and no localStorage, so unlike the web tag we cannot mint an id for the caller -- without it every fetch looks like a new person, which is why a scheduled curl loop reads as a spike of unique visitors. The slot manager now ships a snippet that generates one opaque id per machine at install time. Also replaces the raw control bytes embedded in lib/rateLimit.ts's CONTROL_OR_WS regex with \u escapes. They made the file read as binary to grep and to git (which diffed it as Bin), so searches over it silently returned nothing. Behaviour is identical and now covered by tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both tags are string-templated into third-party pages, so a stray brace or a bad interpolation would ship a script that throws on every publisher site. Containment assertions cannot catch that; parsing can. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan52 finding(s) HIGH/CRITICAL: 4 | MEDIUM: 45 | LOW: 3
…and 2 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the investigation into the periodic ad-impression spikes on
/ads?range=1h.1.
/ad.jsnow mints the visitor id/ad.jsonly ever readlocalStorage['crawlproof.visitor']— nothing but/stats.jswrote it. A publisher running the ad tag without the analytics tag therefore sent an empty visitor on every impression. In production that was ~69% of impressions with no visitor id (5,979 of 19,171 over 7 days), which leftip_hashas the only signal available for click dedupe and frequency capping.The minting logic moves to
lib/tracker/visitorSnippet.tsand is inlined verbatim by both routes. StilllocalStorage, still no cookie.Scope note, deliberately documented in the module: the tag runs in the publisher's page, so this storage is partitioned to the publisher's origin. It is a stable per-site visitor id and cannot follow anyone across publishers — that would need third-party storage, which browsers no longer grant. Per-site frequency capping and returning-visitor counts work; cross-publisher reach does not, by platform design.
2. Salted, daily-rotating IP hashes
lib/ads/serve.tshashed IPs as baresha256(ip)truncated to 32 hex chars. IPv4 is 2^32 addresses, so that is a reversible encoding of the IP rather than a pseudonym — andip_hashis the only identifier the ad network has for terminal traffic, which has no localStorage at all.lib/rateLimit.tsused a hardcoded constant prefix, which is a pepper in name only since it lives in the source.Both collapse into
lib/ipHash.ts. It exports two variants over one implementation, because the callers need opposite properties:hashIp()checkAnonymousLimitlooks back 24h+; a rotating salt would refill every anonymous quota at the boundary — a quota bypass, not a privacy winhashIpRotating()rotatingIpHashCandidates()returns every salt window inside a lookback, so click dedupe doesn't silently miss for the first hours after each rotation.Deploying this changes nothing until
IP_HASH_SALTis set — unset reproduces the legacy digest byte for byte, so no stored hash is invalidated and no rate-limited visitor gets a free quota. Setting it in prod resets abuse counters once and breaks in-flight click dedupe for up to 6h; both self-heal.3.
?v=documented for terminal publishersA terminal has no cookies and no localStorage, so unlike the web tag we cannot mint an id for the caller — without it every fetch looks like a new person, which is exactly why a scheduled
curlloop reads as a spike of unique visitors. The slot manager now ships a snippet that generates one opaque random id per machine at install time, with an explicit warning not to use a hostname, username, or IP.Incidental:
lib/rateLimit.tswas binaryIts
CONTROL_OR_WSregex embedded raw control bytes (including a NUL) directly in the literal instead of escapes. That made the file read as binary togrep— searches over it silently returned nothing, with no error — and git diffed it asBin. Now spelled with\uescapes; behaviour is identical and pinned by new tests. This was found by accident when greps forhashIpin that file returned nothing.Testing
npm run typecheck— cleannpm test— 1,417 passed, 1 failedtests/contract/tracker-geo.test.ts, which needs the GeoLite2 mmdb. Pre-existing and environmental — verified by stashing this branch's changes and re-running it on the baseline, where it fails identically. CI installs the dependency vianpm ci, so it passes there.ip-hash(12),ad-visitor-id(6), andurl-control-chars(10). Thead-visitor-idset includes a parse check on both generated tags, since containment assertions can't catch a broken interpolation.Not included
Impression-side dedupe (mirroring the 6h click window) is the change that would actually flatten the spikes. It is a clean follow-up now that the hash is properly salted, but it is a metering behaviour change and deserves its own PR. The immediate mitigation is a one-line change to
/usr/local/bin/profullstack-sponsor-adon the dev box, outside this repo.🤖 Generated with Claude Code