0.4.0: a free allowance that answers with a price, not a 429

README.md

@@ -102,6 +102,61 @@ export const GET = robotsRoute(gateway, { disallow: ['/login', '/api/'] });
 
 Without `coinpay.apiKey` and `payTo` the gateway still answers training crawlers with 402 and the page says payments are off. Nothing is sold, but nothing is given away either.
 
+## A free allowance that sells instead of refusing
+
+Named crawlers are one problem. The other is an ordinary client hammering the
+site far past what a reader would. The usual answer is `429 Too Many Requests`,
+which tells them to go away. That is the right answer when you have nothing to
+sell. Here you do.
+
+```js
+createGateway({
+  siteUrl: 'https://your-site.com',
+  coinpay: { apiKey: process.env.COINPAY_X402_KEY },
+  payTo: process.env.CRAWL_PAY_TO,
+
+  freeQuota: 100,                    // 100 requests a minute, free, no account
+  benefits: [
+    'No rate limit',
+    'Bulk export endpoint',
+    'Every field, not just the summary',
+  ],
+});
+```
+
+Past the allowance the gateway answers `402` with the offer, the RateLimit
+headers, and a body that says what ran out, when it comes back, and what a pass
+costs. The moment a caller runs out of free requests is the best sales pitch the
+site will ever get: it has just demonstrated it wants more than the free tier
+and is still holding the request.
+
+`freeQuota` takes `{ requests, windowSeconds, paths, identify, store }`. A pass
+is checked before the allowance, so a paying caller is never metered. The sales
+page, `robots.txt` and `.well-known/` stay reachable when the allowance is gone,
+because being unable to reach the page that sells the fix would be the worst
+possible failure of a throttle that exists to sell something.
+
+### On rotating addresses to get around it
+
+The default identity is the caller's address, and a proxy rotation defeats it.
+That is not a hole to be patched, because the arithmetic already argues for
+paying:
+
+- Residential proxy bandwidth is sold **by the gigabyte**, and a crawl big
+  enough to be worth rotating for passes a dollar on the first day.
+- A rotation still fetches every page one at a time. It buys no speed.
+- A pass is a flat price with nothing to maintain and nothing to keep working.
+
+So the throttled page makes that case in as many words rather than pretending to
+be undefeatable. Detection is a race you re-run every time someone changes
+tactics. Price is not: the better your free tier and the clearer your paid one,
+the less anyone bothers. Set `benefits` to the things a rotation genuinely
+cannot get, a bulk endpoint above all, and evasion stops being worth the effort
+rather than being blocked.
+
+Where you *can* identify a caller properly, do: pass `identify` and key on an API
+key or an account, and the allowance becomes exact.
+
 ## Crawlers that do not say who they are
 
 The lists catch crawlers that name themselves. Two do not: a VPS fleet wearing a browser string, and a residential-proxy rotation cycling a few Chrome strings across hundreds of addresses. `denyCidrs` handles the first (`['51.38.0.0/16', '54.38.0.0/16', …]` for one provider's ranges); `chargeSpoofedBrowsers` handles both by asking a question only a browser can answer. A request that answers it is left alone. One that cannot gets the same 402 as GPTBot, which costs the site a hash instead of a render.

index.d.ts

@@ -74,9 +74,51 @@ export interface GatewayOptions {
   contact?: string;
   /** Awaited before the buyer's receipt is sent, and its errors are swallowed: a sale is recorded, and a recording failure never costs the buyer the pass. */
   onSale?: (sale: Sale) => void | Promise<void>;
+  /**
+   * A free allowance for ordinary callers, after which the gateway answers 402
+   * with the offer instead of letting the request through. A bare number means
+   * that many requests per minute. A valid pass is never metered.
+   */
+  freeQuota?: number | FreeQuotaOptions;
+  /** What a pass unlocks, listed on the sales page and in the 402 body. */
+  benefits?: string[];
   fetch?: typeof fetch;
 }
 
+export interface QuotaHit {
+  /** Requests in the current window, including this one. */
+  count: number;
+  /** Seconds until the window rolls over. */
+  resetSeconds: number;
+}
+
+/** Somewhere to count requests. Supply one to share an allowance across a fleet. */
+export interface QuotaStore {
+  hit(key: string, windowSeconds: number): QuotaHit | Promise<QuotaHit>;
+}
+
+export interface FreeQuotaOptions {
+  /** Free requests per window. */
+  requests: number;
+  /** Window length. Default 60. */
+  windowSeconds?: number;
+  /**
+   * What to count against. Defaults to the caller's address, which a rotation
+   * defeats; that is answered with price rather than detection, see the README.
+   */
+  identify?: (request: Request) => string | null;
+  /** Default: an in-process counter, so each instance grants its own allowance. */
+  store?: QuotaStore;
+  /** Only meter these paths or prefixes. Default: everything the gate sees. */
+  paths?: string[];
+}
+
+/** An in-process fixed-window counter. */
+export function memoryQuotaStore(options?: {
+  now?: () => number;
+  sweepEvery?: number;
+}): QuotaStore & { sweep(): void; readonly size: number };
+
 export interface AcceptEntry {
   scheme: 'exact';
   network: string;

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@profullstack/x402-gateway",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "type": "module",
   "description": "Sell crawl access to AI training crawlers by the day over x402, settled by CoinPay. One middleware: 402 with an offer, a sales page with CLI instructions, signed passes, and a robots.txt that keeps search crawlers welcome.",
   "keywords": [

src/index.js

@@ -1,5 +1,6 @@
 import { isTrainingAgent, RETRIEVAL_AGENTS, TRAINING_AGENTS } from './agents.js';
 import { clientIp, compileCidrs, inCidrs, isSpoofedBrowser } from './edge.js';
+import { memoryQuotaStore, meters, normaliseQuota, quotaHeaders, spend } from './quota.js';
 import { renderPage } from './page.js';
 import { mintPass, readPass } from './pass.js';
 import { robotsTxt } from './robots.js';
@@ -20,6 +21,7 @@ export { renderPage } from './page.js';
 export { mintPass, readPass } from './pass.js';
 export { robotsTxt } from './robots.js';
 export { buildOffer, decodePayment, expectedFor, METHODS, verifyAndSettle } from './x402.js';
+export { memoryQuotaStore, normaliseQuota, quotaHeaders, spend } from './quota.js';
 
 /**
  * A gateway that sells crawl access to training crawlers, by the day, over x402.
@@ -119,10 +121,23 @@ export function createGateway(options = {}) {
       status,
       headers: { 'content-type': 'application/json; charset=utf-8', ...noStore, ...headers },
     });
-  const html = (body, status) =>
-    new Response(body, { status, headers: { 'content-type': 'text/html; charset=utf-8', ...noStore } });
+  const html = (body, status, headers = {}) =>
+    new Response(body, {
+      status,
+      headers: { 'content-type': 'text/html; charset=utf-8', ...noStore, ...headers },
+    });
 
-  const pageCtx = (days = 1) => ({
+  const pageCtx = (days = 1, usage = null) => ({
+    quota: o.freeQuota
+      ? {
+          requests: o.freeQuota.requests,
+          windowSeconds: o.freeQuota.windowSeconds,
+          used: usage?.count ?? null,
+          resetSeconds: usage?.resetSeconds ?? null,
+          exceeded: Boolean(usage?.overLimit),
+        }
+      : null,
+    benefits: o.benefits,
     days,
     total: money(o.priceCents * days),
     siteName: o.siteName,
@@ -155,10 +170,14 @@ export function createGateway(options = {}) {
    * body and not the headers, and a crawler that wanted the page can fetch it
    * again a moment later with the pass.
    */
-  async function sell(request) {
+  async function sell(request, context = {}) {
     const ua = request.headers.get('user-agent') ?? '';
     const proofHeader = request.headers.get('x-payment');
     const asked = daysFrom(request);
+    // Present when the free allowance is what stopped this request, rather than
+    // the crawler lists. It changes what the 402 says, not what it costs.
+    const usage = context.usage ?? null;
+    const rateHeaders = usage ? quotaHeaders(o.freeQuota, usage) : {};
 
     if (proofHeader) {
       if (!enabled) return json(receipt(asked, { error: 'Payments are not switched on here.' }), 402);
@@ -252,7 +271,33 @@ export function createGateway(options = {}) {
       );
     }
 
-    if (wantsHtml(request.headers.get('accept'))) return html(o.page(pageCtx(asked)), 402);
+    if (wantsHtml(request.headers.get('accept'))) {
+      return html(o.page(pageCtx(asked, usage)), 402, rateHeaders);
+    }
+
+    if (usage) {
+      // Say what ran out, when it comes back, and what a pass costs, in that
+      // order. A caller reading this is deciding between waiting, rotating
+      // addresses, and paying, and the numbers are the argument.
+      return json(
+        receipt(asked, {
+          error:
+            `Free allowance used: ${o.freeQuota.requests} requests per ` +
+            `${o.freeQuota.windowSeconds}s. It resets in ${usage.resetSeconds}s. ` +
+            `A pass removes the limit for ${price} a day.`,
+          quota: {
+            requests: o.freeQuota.requests,
+            windowSeconds: o.freeQuota.windowSeconds,
+            used: usage.count,
+            resetSeconds: usage.resetSeconds,
+          },
+          ...(o.benefits ? { unlocks: o.benefits } : {}),
+        }),
+        402,
+        rateHeaders,
+      );
+    }
+
     return json(receipt(asked, { error: `Payment required for training crawlers. Read ${buyUrl} for how.` }), 402);
   }
 
@@ -280,15 +325,41 @@ export function createGateway(options = {}) {
     const path = new URL(request.url).pathname;
     if (path === o.path) return sell(request);
     if (o.exempt && o.exempt(request)) return null;
+
+    /*
+     * A valid pass is checked before anything else that could refuse, because
+     * a pass is the thing being sold: whoever holds one is neither charged as
+     * a crawler nor metered against the free allowance. It used to be read
+     * only after the crawler lists matched, which was fine while the lists
+     * were the only reason to refuse and is not now that a quota exists.
+     */
+    const token = passFrom(request);
+    const paid = Boolean(token && (await readPass(token, { secret })));
+    if (paid) return null;
+
     const pays =
       o.isPaidAgent(request.headers.get('user-agent') ?? '') ||
       (o.chargeSpoofedBrowsers && isSpoofedBrowser(request));
-    if (!pays) return null;
+    if (pays && !isOpen(path)) return sell(request);
     if (isOpen(path)) return null;
 
-    const token = passFrom(request);
-    if (token && (await readPass(token, { secret }))) return null;
-    return sell(request);
+    /*
+     * Everyone else gets the free allowance. Running out is answered with a
+     * price rather than a 429: the caller has just shown it wants more than
+     * the free tier and is still holding the request, which is the best moment
+     * this site will ever get to sell it a pass.
+     */
+    if (o.freeQuota && meters(o.freeQuota, path)) {
+      const key = o.freeQuota.identify ? o.freeQuota.identify(request) : clientIp(request);
+      const usage = await spend(o.freeQuota, key);
+      // Under the limit the request carries on untouched. `handle` answers with
+      // a Response or nothing at all, and quietly growing that contract to
+      // smuggle headers out would break every adapter that checks it for truth.
+      // The allowance is advertised on the 402, which is where it is read.
+      if (usage?.overLimit) return sell(request, { usage });
+    }
+
+    return null;
   }
 
   return {
@@ -372,6 +443,8 @@ function normalise(options) {
     page: options.page ?? renderPage,
     contact: options.contact ?? '',
     onSale: options.onSale ?? null,
+    freeQuota: normaliseQuota(options.freeQuota),
+    benefits: Array.isArray(options.benefits) ? options.benefits : null,
     fetch: options.fetch ?? globalThis.fetch,
   };
 }

src/page.js

@@ -58,7 +58,43 @@ export function renderPage(ctx) {
     days = 1,
     total = price,
     maxDays = 30,
+    quota = null,
+    benefits = null,
   } = ctx;
+
+  /*
+   * Two audiences reach this page and they need different first sentences. A
+   * training crawler is here because it is on a list. A heavy reader is here
+   * because it ran out of the free allowance, and telling that one it is a
+   * training crawler is both wrong and insulting. The price is the same; the
+   * argument is not.
+   */
+  const throttled = Boolean(quota?.exceeded);
+  const headline = throttled
+    ? 'You have used up the free allowance.'
+    : 'Training crawlers pay for access here.';
+  const opening = throttled
+    ? `<p class="mut">${esc(String(quota.requests))} requests every ${esc(String(quota.windowSeconds))} seconds are free, no key and no account, and that is not changing. ` +
+      `You have gone past it${quota.resetSeconds ? `, and it resets in ${esc(String(quota.resetSeconds))} seconds` : ''}. ` +
+      `A pass lifts the limit rather than waiting it out.</p>`
+    : `<p class="mut">People read <a href="${esc(siteUrl)}">${esc(siteName)}</a> free. So do search engines and the retrieval crawlers behind AI answers, because they send readers back. A crawler that copies pages into a training corpus sends nobody back, so it pays for the time it spends.</p>`;
+
+  /*
+   * Said plainly because it is the honest argument and the one that actually
+   * lands. Rotating addresses to dodge a free-tier limit is not free: the
+   * bandwidth is metered and billed by the gigabyte, and a crawl big enough to
+   * be worth rotating for costs more in proxies than the pass does. Anyone
+   * weighing the two should be able to see that from here.
+   */
+  const arithmetic = throttled
+    ? `<h2>Before you reach for a proxy pool</h2>
+<p class="mut">Spreading the same crawl over rotating addresses works, and it is the expensive way to do this. Residential bandwidth is sold by the gigabyte, you still fetch every page one at a time, and the bill starts on the first day. A pass is ${esc(price)} a day, flat, with no rotation to maintain and nothing to keep working. We would rather sell you access than play that game, which is why the limit answers with a price instead of a refusal.</p>`
+    : '';
+
+  const unlocks =
+    benefits && benefits.length
+      ? `<h2>What a pass gets you</h2>\n<ul>\n${benefits.map((b) => `  <li>${esc(b)}</li>`).join('\n')}\n</ul>`
+      : '';
   const window =
     minutes === 1440
       ? 'one day'
@@ -82,8 +118,8 @@ export function renderPage(ctx) {
 </head>
 <body>
 <main>
-<h1>Training crawlers pay for access here.</h1>
-<p class="mut">People read <a href="${esc(siteUrl)}">${esc(siteName)}</a> free. So do search engines and the retrieval crawlers behind AI answers, because they send readers back. A crawler that copies pages into a training corpus sends nobody back, so it pays for the time it spends.</p>
+<h1>${esc(headline)}</h1>
+${opening}
 
 <div class="price">${esc(days > 1 ? total : price)} <span class="mut" style="font-size:1rem;font-weight:400">for ${esc(days > 1 ? `${days} × ${window}` : window)} of requests</span></div>
 ${
@@ -97,6 +133,9 @@ ${
     : '<p><strong>Payments are not switched on here yet.</strong> The offer below is empty until the operator configures a payout address, so for now this crawler is simply refused.</p>'
 }
 
+${unlocks}
+${arithmetic}
+
 <h2>How it works</h2>
 <ol>
   <li>Any page you fetch answers <code>402 Payment Required</code>. This page, fetched with <code>Accept: application/json</code>, returns the x402 offer: USDC, <code>exact</code> scheme, on ${esc(networks || 'Base, Polygon or Ethereum')}.</li>