Charge AI training crawlers for access with @profullstack/x402-gateway - #205
Merged
Conversation
GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider, Applebot-Extended and friends now get 402 Payment Required with an x402 offer ($1 buys a day, USDC settled by CoinPay) or the sales page at /crawl; a paid pass in x-crawl-pass lets them through. People, Googlebot and retrieval crawlers are untouched: the gate runs first in src/proxy.ts and returns null for them before the existing rate-limit, bot and profile logic. robots.txt is now generated from the same lists in src/app/robots.txt/route.ts, replacing the two conflicting producers (public/robots.txt, which won, and the shadowed src/app/robots.ts) with the union of their disallow lists. ai.txt flips Training to disallow and points at /crawl. Needs COINPAY_X402_KEY and CRAWL_PAY_TO in the deployment env; until both are set training crawlers get 402 with an empty offer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
ThreatCrush Security Scan99 finding(s) HIGH/CRITICAL: 11 | MEDIUM: 29 | LOW: 59
…and 49 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
The root middleware.ts (Supabase token refresh + referral cookie) was never loaded: with the app under src/, Next 16 only scans src/ for a proxy or middleware file, and would refuse to build if it saw both. So sessions were not being refreshed and ?ref= was not being tracked. src/proxy.ts is now the one live file and runs, in order: the x402 crawl gate, the Supabase session refresh, the existing rate-limit / bot / profile logic, then the referral cookie. A refused training crawler gets its 402 before anything touches Supabase. The refreshed session cookie is written onto whichever response goes out (pass- through, select-profile redirect, 429 or 403), because Supabase rotates refresh tokens and a dropped refresh would sign the user out on the next request. The matcher takes the union of both files' exclusions. Tests cover the refresh (fetch target, tokens written back, 7-day httpOnly cookie), fresh sessions left alone, 401 clearing vs 503 keeping the cookie, the redirect carrying the cookie, a crawler never reaching Supabase, and the referral cookie for valid and malformed codes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC
ralyodio
marked this pull request as ready for review
September 5, 2026 16:41
This was referenced Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes, for whom
Integrates
@profullstack/x402-gateway0.1.0 so AI training crawlers pay for access instead of reading bittorrented.com for free./api/search/*,/api/dht/*)402 Payment Requiredon every route with an x402 offer (JSON), or the HTML sales page if they ask for HTML;/robots.txt,/crawl,/security.txt,/.well-known/stay readable/crawlcoinpay x402 pay https://bittorrented.com/crawl)x-crawl-passMatching is the crawler's self-declared user-agent token, so this is a toll for crawlers that identify themselves, not a defence against ones that lie.
Files
src/lib/crawl-gateway.ts(new): onecreateGatewayinstance,siteUrlfromNEXT_PUBLIC_APP_URL(the env the app already uses for canonicals), nothing Node-only imported.src/proxy.ts: the gateway runs first (const answer = await gateway.handle(request); if (answer) return answer;), then the Supabase session refresh, then the existing rate-limit / bot / profile logic, then the referral cookie (see the next section).proxyis nowasyncand always returns aResponse(NextResponse.next()for pass-through). Composed into the live file rather than a new one.src/app/robots.txt/route.ts(new):robotsRoute(gateway, { disallow: ['/api/', '/login', '/settings', '/dashboard/'] }), so robots.txt and the gate are generated from one set of lists. Every named group repeats the private-path rules (a crawler that finds its own group ignoresUser-agent: *).public/robots.txtandsrc/app/robots.tsdeleted. They disagreed with each other (/login+/settingsvs/dashboard/), the static file was the one actually served, and the metadata route cannot express per-agent groups in a reliable order. The new route carries the union of both disallow lists.public/ai.txt:Training: allow->disallow(Retention and Commercial-Use too), with a comment pointing at/crawlfor paid access..env.example: documents the two new vars.src/app/robots.txt/route.test.ts(new) asserts GPTBot and meta-externalagent getDisallow: /(+Allow: /crawl), OAI-SearchBot getsAllow: /, andDisallow: /api/survives in every group.src/proxy.test.tsnow awaits the async proxy and adds a gateway block: meta-externalagent on/browsegets 402 with an x402 v2 body, a Chrome UA on/browseand/api/search/torrentsfalls through unchanged, Googlebot and OAI-SearchBot pass, a training crawler may read/robots.txt. One existing expectation changed: GPTBot on/api/dht/browseis now 402 (gateway) instead of 403 (bot block), which is the intended ordering.The dead root
middleware.ts, folded in (second commit)The root
middleware.ts(Supabase token refresh + referral cookie) was dead code on this Next 16 layout: with the app undersrc/app, Next only scanssrc/forproxy/middlewarefiles (build/index.js,rootDir = join(appDir, '..')), and would throw E900 if it saw both. So the session refresh had not been running, and neither had?ref=tracking. Fix in this PR:middleware.tsdeleted.src/proxy.tsis the one live file.proxy()in this order: x402 gate, then session refresh, then rate-limit / bot / profile, then referral cookie. A refused training crawler gets its 402 before anything touches Supabase (tested)./select-profileredirect, a 429 or a 403. The old file could only ever attach it to a pass-through; since Supabase rotates refresh tokens, a refresh whose result is dropped would have signed the user out on the next request. The JWT expiry decode usesatob+TextDecoderinstead ofBuffer, so the file has no Node-only API./^[a-zA-Z0-9_-]{1,64}$/) thentrackReferralCodefrom@profullstack/stack/referrals, which sets the non-httpOnlyreferral_codecookie for 30 days.[GET] /api/... — ip) kept.logo.svg, fonts,css,js,map), so the proxy now uses the union:_next/static|_next/image|favicon.ico|logo.svg|*.(png|jpg|jpeg|gif|webp|svg|ico|woff|woff2|ttf|eot|css|js|map).src/proxy.test.ts, +10): expiring session for a browser is refreshed againstSUPABASE_URL/auth/v1/token?grant_type=refresh_tokenand the new tokens land insb-auth-token(httpOnly, 7 days); fresh session untouched, no fetch; no session, no cookies at all; refreshed cookie survives the select-profile 307; 401 clears the cookie, 503 keeps the stale one;meta-externalagentwith an expiring session gets 402 and Supabase is never called; valid?ref=setsreferral_code, a malformed one does not; both cookies on one browser request. Pass-through is now asserted asNextResponse.next()(x-middleware-next: 1) instead ofundefined.Deployment: two env vars
COINPAY_X402_KEYcp_live_..., from the business's API Keys tab) withpayments:create. The legacy business key is refused by CoinPay's x402 routes. Also the pass-signing secret.CRAWL_PAY_TOUntil both are set, training crawlers still get 402, but with an empty offer (
accepts: []) and the sales page says payments are switched off. Nothing is sold, nothing is given away, humans are unaffected either way.Where this deploys and how the vars get there
.github/workflows/deploy-droplet.ymlruns after the "CI Pipeline" workflow succeeds onmaster(or by manual dispatch). It rsyncs the checkout to a DigitalOcean droplet at${{ secrets.DROPLET_HOST }}(userDROPLET_USER, portDROPLET_PORT) into/home/ubuntu/www/bittorrented.com/media-streamer, thenpnpm install --frozen-lockfile && pnpm buildon the droplet and restarts thebittorrentedsystemd unit (pnpm start, i.e.next start, plus the iptv and podcast workers).Env vars reach the app through the
ENV_FILEGitHub Actions secret: the workflow writes its contents verbatim to.envon the runner and rsyncs that file to the droplet on every deploy. So add these two lines to theENV_FILErepository secret (not by hand on the droplet, where the next deploy would overwrite them):Then re-run the deploy (or merge the next change).
Checks (after the second commit)
pnpm typecheck: cleanpnpm lint: 0 errors (248 pre-existing warnings, none in touched files)pnpm test: 2731 passed, 9 skipped, 0 failed (206 files). In two earlier runssrc/lib/spotify/librespot.test.ts"caps the played list" hit the 5stestTimeoutwhilenext buildwas competing for CPU; it is a 60-iteration synchronous loop that runs ~4.9s alone, untouched by this PR, and passed on a quiet box.pnpm build: passes;ƒ Proxy (Middleware)compiled fromsrc/proxy.tsalone,/robots.txtprerendered static, 134/134 pages. The pre-commit hook was bypassed for both commits because its full-suite run flaked on timing-bound tests (librespot"caps the played list", RSS jsdom e2e) on a box at load average 13; both pass alone and the full suite passed clean before the commit.Not merged; review before merge.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC