tracker: count people, not beacons (visitor rollup + scripted cap) - #263
Merged
Merged
Conversation
ralyodio
marked this pull request as ready for review
September 21, 2026 18:54
"Human visits" was every beacon from a non-crawler user agent: the page view plus four scroll depths, every click and every form submit, three to four per page view. On four properties it read 51,531 in a week while the raw event table held 420 distinct visitor ids in a day, which is also what datafa.st reported for the same sites. Nothing outside the 24h raw table kept a visitor id, so weekly uniques could not be stated at all. - tracker_visitor_daily_stats: one row per (project, UTC day, visitor id) with event and pageview counts and which side of the human/bot line the visitor ended the day on. tracker_touch_visitor upserts it per beacon in one round trip and applies the scripted cap: more than 500 events or 200 page views from one visitor in a day flips it to bot for the day, and the ingest route counts every later beacon from it under bot:scripted. A stock-UA headless browser was the largest "human" source and nothing in the classifier could see it. - tracker_visitor_totals / tracker_visitor_daily_series: exact distinct visitors over a window (and the window before) and per day. - Stats page, /dashboard cards and /dashboard/analytics lead with "Human visitors" and "Page views" from the rollup; the old figure stays as "Human events" with a definition that says what it is. A failed rollup read leaves the tile out rather than showing 0. - /api/tracker/v1/stats totals: visitors and pageviews come from the rollup (null when unreadable, never 0), events is the old number. The CLI line prints all three. totalsFromSeries no longer returns beacons as visitors, so `crawlproof dashboard` cost-per-visitor is finally per visitor. Migration applied to production 2026-09-21 via the Supabase MCP; rows begin that day and the UI captions any window that reaches further back. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ralyodio
force-pushed
the
worktree-tracker-visitor-rollup
branch
from
September 21, 2026 18:57
478aeb5 to
5005f4a
Compare
ThreatCrush Security Scan48 finding(s) HIGH/CRITICAL: 2 | MEDIUM: 31 | LOW: 15
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
datafa.st and crawlproof disagreed by ~100x on four sites: crawlproof said 50k humans, datafa.st said 400. datafa.st was right.
"Human visits" was
sum(count) where bucket not like 'bot:%'overtracker_daily_stats, which/api/trackbumps for every beacon stats.js auto-fires (pageview, scroll_25/50/75/100, button_click, form_submit, internal_click…), 3–4 per page view, from any user agent that does not announce itself as a bot. Same four sites, from the tracker's own tables on 2026-09-21:That 420 is datafa.st's 400. And
tracker_eventsis pruned at 24h with no rollup keeping visitor ids, so weekly uniques could not be stated at all.What
tracker_visitor_daily_stats— one row per (project, UTC day, visitor id) with event/pageview counts and the visitor's side of the human/bot line.tracker_touch_visitorupserts it per beacon in one round trip.botfor the day and the route counts every later beacon underbot:scripted. The largest "human" source on bittorrented today is a stock-UA driven browser (398 events on 4 page views, CN, yopmail/jubt*.xyz referrers) that the UA classifier cannot see. Fails open: no visitor id or a failed RPC leaves the UA verdict standing.tracker_visitor_totals/tracker_visitor_daily_series— exact distinct visitors over a window (+ the window before) and per day./dashboardcards and/dashboard/analyticslead with Human visitors and Page views; the old number stays as Human events with a definition that says what it is. A failed rollup read drops the tile instead of showing 0. Windows reaching before 2026-09-21 carry a caption./api/tracker/v1/statstotalsis now{ visitors, pageviews, events }withvisitors/pageviewsfrom the rollup (null when unreadable, never 0).crawlproof statsprints all three.totalsFromSeriesno longer returns beacons as visitors, socrawlproof dashboardcost-per-visitor is per visitor.Migration
20260921120000_tracker_visitor_rollup.sql— applied to production via the Supabase MCP on 2026-09-21 and smoke-tested (touch increments, cap flips at N+1 and is sticky, totals exclude the demoted visitor, test rows deleted). Rows begin today; nothing to backfill from.Verification
tsc --noEmitclean;vitest run2,258 passed (19 new intests/tracker-visitor-rollup.test.ts).npm run lintis broken on master independently of this PR (next lintno longer exists in this Next version).Not in this PR
🤖 Generated with Claude Code