Lead analytics with human visits; show bot crawls apart - #230
Merged
Merged
Conversation
Every headline number on /dashboard, /dashboard/analytics and the
per-project stats page was bot-inclusive. On one property 99% of ~257k
weekly hits were a single AI training crawler, and the card read as
"80k pageviews a day" for a site with a few hundred real readers.
Definition, used in SQL and in lib/tracker/humans.ts: a hit is a human
when its bucket does NOT start with `bot:` (AI referrals, search, social,
referral, direct), a bot when it does. humans + bots = events, exactly.
Migration 20260905120000_tracker_human_split.sql (NOT applied yet):
- new dashboard_project_traffic(p_project_ids, p_since) ->
(project_id, day, humans, bots) from tracker_daily_stats
- tracker_project_totals gains humans / prev_humans
- tracker_daily_series and tracker_daily_series_multi gain humans
- tracker_project_daily_series gains humans and bots
- tracker_recent_series gains humans
All are drop + create (return types change), same argument signatures,
columns only appended, security invoker, grants re-issued. The three
*_multi functions carrying `set work_mem` are deliberately untouched.
UI:
- /dashboard cards: "{n} human visits" headline, "{n} bot hits · Past 7
days" beneath, sparkline of humans, read from dashboard_project_traffic
- /dashboard/analytics: tiles are Human visits, AI referrals, Bot crawls,
All events; verdict, ranking, per-property trend and sparklines are all
human-based; the By property table has Humans / Previous / Change /
Trend / AI / Bots; the portfolio chart stacks human visits per property
with bot crawls as one dashed unstacked overlay
- project stats: tiles are Human visits, AI referrals, Bot crawls;
Traffic pulse stacks humans + bots (the bot-inclusive pageview band is
gone), with AI referrals and interactions as overlays
- every labelled figure carries the definition as hover text and a
caption
Before the migration is applied the missing `humans` column is derived
as events - bots, which is exact for every bucket-based leg, so the
pages are correct either way. `humans` is never backfilled from the
bot-inclusive event table on pre-bucket days.
Tests: lib/tracker/totals.ts and humans.ts are covered in
tests/tracker-human-split.test.ts; portfolio-analytics.test.ts covers
the new series field and the fallback.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC
ThreatCrush Security Scan39 finding(s) HIGH/CRITICAL: 2 | MEDIUM: 28 | LOW: 9
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every headline number on
/dashboard,/dashboard/analyticsand the per-project stats page was bot-inclusive. On genrewatch.com 99% of ~257k weekly hits are Meta's training crawler, and the card read as "80k pageviews a day" for a site with a few hundred real readers.Definition (SQL and TS agree)
bot:(AI referrals, search, social, referral, direct). A person arriving from ChatGPT is a person.bot:(named AI crawlers +bot:other).lib/tracker/humans.tsand the migration header.Migration: NOT APPLIED
supabase/migrations/20260905120000_tracker_human_split.sqlmust be applied by hand via the Supabase MCP (projectywcizjsgrcmhgyplldac) before this merges. It:dashboard_project_traffic(p_project_ids uuid[], p_since date) -> (project_id, day, humans, bots)fromtracker_daily_stats(grant authenticated, service_role);dashboard_project_pageviewsis left in placetracker_project_totals: +humans,prev_humanstracker_daily_series,tracker_daily_series_multi: +humanstracker_project_daily_series: +humans,botstracker_recent_series: +humansReturn types change, so each is
drop function if exists+create function. Argument signatures are identical and columns are only appended, so existing callers keep working. All staysecurity invoker; grants are re-issued after the drops.tracker_top_pages_multi/tracker_top_actions_multi/tracker_top_exit_pages_multi(which carryset work_memfrom an applied migration) are deliberately untouched.Until the migration lands, the code derives the missing
humanscolumn asevents - bots(exact for every bucket-based leg), so nothing renders as "0 human visits" in the gap. The one thing that needs the migration is the/dashboardcards, which call the newdashboard_project_trafficRPC; before it exists they show the existing "unavailable / query failed, not zero" state rather than a wrong number.UI
/dashboardcards:{n} human visitsheadline,{n} bot hits · Past 7 daysbeneath, sparkline of humans./dashboard/analytics: tiles are Human visits (accent), AI referrals, Bot crawls, All events (muted). Verdict, property ranking, per-property trend and sparklines are all human-based. "By property" table: Humans / Previous / Change / Trend / AI / Bots. Portfolio chart stacks human visits per property with bot crawls (all properties) as one dashed, unstacked overlay.SeriesPointgainshumans, so the SSR page and/api/projects/:id/tracker-statsagree.humansis never backfilled from the bot-inclusive event table on pre-bucket-rollup days; those days honestly show 0 humans.Tests
tests/tracker-human-split.test.ts(new):humansFrom/toCount,toProjectTotals,sumTotals,totalsTrends(humans trend down while bot-inflated events trend up),buildBucketAxishumans.tests/portfolio-analytics.test.ts:humanson series rows, no legacy backfill of humans, pre-migration fallback.tsc --noEmit: clean.vitest run: 153 files passed, 1 skipped; 2008 tests passed, 7 skipped.next lint(eslint is not present in the node_modules available to the worktree).🤖 Generated with Claude Code
https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC