Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Sell crawl access to AI training crawlers, by the day, over [x402](https://x402.

People read your site free. So do search engines and the retrieval crawlers behind AI answers, because they send readers back. A crawler that copies pages into a training corpus sends nobody back, so it pays: every page answers `402 Payment Required` with an x402 offer, paying the offer returns a signed pass, and the pass opens the site for a day.

A crawler that wants longer buys more days in one payment. `?days=7` on the sales page quotes seven days at the daily price, and the days a proof buys are read off the value it authorizes, so paying seven times the price — however it was asked for — returns a pass that expires seven days out. `maxDays` caps how many one proof can buy.

One middleware. No database. Runs in Node, Bun and at the edge.

```
Expand Down Expand Up @@ -82,7 +84,8 @@ export const GET = robotsRoute(gateway, { disallow: ['/login', '/api/'] });
| `coinpay.apiKey` | | a **scoped** CoinPay key (`cp_live_…`, from the business's API Keys tab) with `payments:create`. The legacy business key is refused by CoinPay's x402 routes. |
| `payTo` | | EVM address that receives the USDC, on Base, Polygon and Ethereum alike |
| `priceCents` | `100` | |
| `passMinutes` | `1440` | a day |
| `passMinutes` | `1440` | a day: the term one price buys |
| `maxDays` | `30` | the most terms one proof may buy at once |
| `header` | `x-crawl-pass` | where the pass goes; `Authorization: Bearer` works too |
| `path` | `/crawl` | the sales page |
| `openPaths` | `[]` | extra paths a refused crawler may read (`robots.txt`, the sales page, `security.txt` and `.well-known/` always are) |
Expand Down
19 changes: 18 additions & 1 deletion index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,27 @@ export interface Sale {
token: string;
expiresAt: string;
userAgent: string;
/** Per term (`passMinutes`). */
priceCents: number;
/** Terms this proof bought. */
days: number;
/** `priceCents * days`. */
totalCents: number;
currency: string;
}

export interface PageContext {
siteName: string;
siteUrl: string;
buyUrl: string;
/** Per day, e.g. "1.00 USD". */
price: string;
minutes: number;
/** Days this page's offer quotes (`?days=`), 1 by default. */
days: number;
/** `price` times `days`. */
total: string;
maxDays: number;
header: string;
enabled: boolean;
offer: Offer;
Expand All @@ -37,8 +48,10 @@ export interface GatewayOptions {
/** Default 100 ($1). */
priceCents?: number;
currency?: string;
/** What a payment buys. Default 1440 (a day). */
/** What one price buys. Default 1440 (a day). */
passMinutes?: number;
/** The most terms one proof may buy at once (`?days=` and paid multiples are clamped to it). Default 30. */
maxDays?: number;
/** Request header the pass is presented in. Default 'x-crawl-pass'. */
header?: string;
/** The sales page. Default '/crawl'. */
Expand Down Expand Up @@ -133,6 +146,10 @@ export const X402_METHODS: typeof METHODS;
export function buildOffer(args: { payTo: string; priceCents: number; resource: string; description?: string; maxTimeoutSeconds?: number; methods?: typeof METHODS }): Offer;
export function decodePayment(header: string | null | undefined): Record<string, unknown> | null;
export function expectedFor(payment: unknown, offer: Offer): { amount: string; resource: string; payTo: string; asset: string } | null;
/** The value a proof authorizes, in the token's smallest unit, or null. */
export function paidValueOf(payment: unknown): bigint | null;
/** How many terms `value` buys at `unit` per term: a whole number in [1, maxDays], or 0. */
export function daysPaid(value: bigint | null, unit: string | number | bigint, maxDays: number): number;
export function verifyAndSettle(
payment: unknown,
expected: { amount: string; resource: string; payTo: string; asset: string },
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@profullstack/x402-gateway",
"version": "0.2.2",
"version": "0.3.0",
"type": "module",
"description": "Sell crawl access to AI training crawlers by the day over x402, settled by CoinPay. One middleware: 402 with an offer, a sales page with CLI instructions, signed passes, and a robots.txt that keeps search crawlers welcome.",
"keywords": [
Expand Down
131 changes: 111 additions & 20 deletions src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ export { buildOffer, decodePayment, expectedFor, METHODS, verifyAndSettle } from
* or as an HTML sales page if it asked for HTML -- on every path but the few it
* needs to read to comply. Paying the offer, at the sales page or on any 402'd
* URL, returns a signed pass good for `passMinutes`, presented in `header` on
* every request after that.
* every request after that. A crawler that wants longer buys more days at
* once: `?days=N` on the sales page quotes N terms, and a proof for N times
* the price — however it was asked for — buys a pass that lasts N terms.
*
* Framework-agnostic: `handle(request)` takes a Fetch `Request` and resolves to
* a `Response` to send, or null to let the request through. The adapters in
Expand All @@ -42,7 +44,8 @@ export { buildOffer, decodePayment, expectedFor, METHODS, verifyAndSettle } from
* @param {string} [options.payTo] EVM address that receives the USDC
* @param {number} [options.priceCents=100]
* @param {string} [options.currency='USD']
* @param {number} [options.passMinutes=1440] a day
* @param {number} [options.passMinutes=1440] a day: the term one payment buys
* @param {number} [options.maxDays=30] the most terms one proof may buy at once
* @param {string} [options.header='x-crawl-pass']
* @param {string} [options.path='/crawl'] the sales page
* @param {string[]} [options.openPaths] extra paths a refused crawler may read
Expand All @@ -65,22 +68,45 @@ export function createGateway(options = {}) {
const denied = compileCidrs(o.denyCidrs);
const isOpen = (path) => openPaths.some((p) => (p.endsWith('/') ? path.startsWith(p) : path === p));

const price = `${(o.priceCents / 100).toFixed(2)} ${o.currency}`;
const money = (cents) => `${(cents / 100).toFixed(2)} ${o.currency}`;
const price = money(o.priceCents);
const buyUrl = `${o.siteUrl}${o.path}`;

const offer = () =>
/**
* How many terms a request is asking to buy: `?days=N`, clamped to
* [1, maxDays]. Anything unparseable is one day, which is what the offer
* always meant before there was a way to ask for more.
*/
const daysFrom = (request) => {
const raw = new URL(request.url).searchParams.get('days');
const n = Number.parseInt(raw ?? '', 10);
if (!Number.isFinite(n) || n < 1) return 1;
return Math.min(n, o.maxDays);
};

/** The offer for `days` terms: the same entries, `days` times the price. */
const offer = (days = 1) =>
enabled
? buildOffer({
payTo: o.payTo,
priceCents: o.priceCents,
priceCents: o.priceCents * days,
resource: buyUrl,
description: `${o.passMinutes} minutes of crawl access to ${o.siteUrl}`,
description: `${days * o.passMinutes} minutes of crawl access to ${o.siteUrl}${days > 1 ? ` (${days} × ${o.passMinutes})` : ''}`,
})
: { x402Version: 2, accepts: [] };

const receipt = (extra = {}) => ({
...offer(),
pass: { price, minutes: o.passMinutes, header: o.header, buy: buyUrl },
const receipt = (days = 1, extra = {}) => ({
...offer(days),
pass: {
price,
minutes: o.passMinutes,
days,
total: money(o.priceCents * days),
maxDays: o.maxDays,
header: o.header,
buy: days > 1 ? `${buyUrl}?days=${days}` : buyUrl,
buyDays: `${buyUrl}?days=<n>`,
},
...extra,
});

Expand All @@ -96,12 +122,15 @@ export function createGateway(options = {}) {
const html = (body, status) =>
new Response(body, { status, headers: { 'content-type': 'text/html; charset=utf-8', ...noStore } });

const pageCtx = () => ({
const pageCtx = (days = 1) => ({
days,
total: money(o.priceCents * days),
siteName: o.siteName,
siteUrl: o.siteUrl,
buyUrl,
price,
minutes: o.passMinutes,
maxDays: o.maxDays,
header: o.header,
enabled,
offer: offer(),
Expand Down Expand Up @@ -129,14 +158,35 @@ export function createGateway(options = {}) {
async function sell(request) {
const ua = request.headers.get('user-agent') ?? '';
const proofHeader = request.headers.get('x-payment');
const asked = daysFrom(request);

if (proofHeader) {
if (!enabled) return json(receipt({ error: 'Payments are not switched on here.' }), 402);
if (!enabled) return json(receipt(asked, { error: 'Payments are not switched on here.' }), 402);
const payment = decodePayment(proofHeader);
if (!payment) return json(receipt({ error: 'X-PAYMENT is not base64 JSON.' }), 402);
const current = offer();
const expected = expectedFor(payment, current);
if (!expected) return json(receipt({ error: 'Proof does not match an offered network.' }), 402);
if (!payment) return json(receipt(asked, { error: 'X-PAYMENT is not base64 JSON.' }), 402);
const unit = expectedFor(payment, offer(1));
if (!unit) return json(receipt(asked, { error: 'Proof does not match an offered network.' }), 402);

/*
* The money decides the term, not the URL. A proof is an authorization
* for an exact value, and the value the buyer signed is what CoinPay
* will move -- so the days it buys are read off the proof: a whole
* number of day-prices, at most maxDays. `?days=` shaped the offer the
* buyer read; if they then signed for a different multiple, they get
* what they paid for, and if they signed for something that is not a
* multiple they get nothing, before anyone is charged.
*/
const days = daysPaid(paidValueOf(payment), unit.amount, o.maxDays);
if (!days) {
return json(
receipt(asked, {
error: `Pay a whole number of days: ${unit.amount} per day in the token's smallest unit, up to ${o.maxDays} days. Add ?days=<n> to ${buyUrl} for the offer.`,
}),
402,
);
}
const expected = expectedFor(payment, offer(days));
const term = days * o.passMinutes * 60;

const now = Math.floor(Date.now() / 1000);
const coinpay = { apiKey: o.coinpay.apiKey, baseUrl: o.coinpay.baseUrl, fetch: o.fetch };
Expand All @@ -145,7 +195,7 @@ export function createGateway(options = {}) {
let expiresAt = null;
let replayed = false;
if (result.ok) {
expiresAt = now + o.passMinutes * 60;
expiresAt = now + term;
} else if (result.replay) {
/*
* Paid once, lost the answer, asked again with the same proof. Answered
Expand All @@ -158,12 +208,12 @@ export function createGateway(options = {}) {
const paid = await settleAgain(payment, coinpay);
const validBefore = validBeforeOf(payment);
if (paid && validBefore) {
expiresAt = Math.min(now + o.passMinutes * 60, validBefore + o.passMinutes * 60);
expiresAt = Math.min(now + term, validBefore + term);
replayed = true;
}
}
if (!expiresAt || expiresAt <= now) {
return json(receipt({ error: result.reason ?? 'Payment could not be settled.' }), 402);
return json(receipt(days, { error: result.reason ?? 'Payment could not be settled.' }), 402);
}

const ref = nonceOf(payment) ?? result.ref ?? null;
Expand All @@ -178,6 +228,8 @@ export function createGateway(options = {}) {
expiresAt: expires,
userAgent: ua,
priceCents: o.priceCents,
days,
totalCents: o.priceCents * days,
currency: o.currency,
});
} catch {
Expand All @@ -189,6 +241,8 @@ export function createGateway(options = {}) {
ok: true,
pass: pass.token,
expires_at: expires,
days,
minutes: days * o.passMinutes,
header: o.header,
replayed,
use: `curl -H "${o.header}: ${pass.token}" ${o.siteUrl}/`,
Expand All @@ -198,8 +252,8 @@ export function createGateway(options = {}) {
);
}

if (wantsHtml(request.headers.get('accept'))) return html(o.page(pageCtx()), 402);
return json(receipt({ error: `Payment required for training crawlers. Read ${buyUrl} for how.` }), 402);
if (wantsHtml(request.headers.get('accept'))) return html(o.page(pageCtx(asked)), 402);
return json(receipt(asked, { error: `Payment required for training crawlers. Read ${buyUrl} for how.` }), 402);
}

/**
Expand Down Expand Up @@ -253,6 +307,42 @@ export function createGateway(options = {}) {
/** Whether the caller would rather read a page than a JSON offer. */
export const wantsHtml = (accept = '') => String(accept ?? '').toLowerCase().includes('text/html');

/** The value a proof authorizes, in the token's smallest unit, or null. */
export function paidValueOf(payment) {
const raw = payment?.payload?.authorization?.value;
if (raw === undefined || raw === null || raw === '') return null;
try {
const value = BigInt(raw);
return value > 0n ? value : null;
} catch {
return null;
}
}

/**
* How many terms a paid value buys at `unit` per term: a whole number in
* [1, maxDays], or 0 when it is not one. Integer arithmetic on the smallest
* unit, so a price that is not a round number of cents still divides exactly.
*
* @param {bigint|null} value
* @param {string|number|bigint} unit
* @param {number} maxDays
* @returns {number}
*/
export function daysPaid(value, unit, maxDays) {
if (value === null) return 0;
let per;
try {
per = BigInt(unit);
} catch {
return 0;
}
if (per <= 0n || value % per !== 0n) return 0;
const days = value / per;
if (days < 1n || days > BigInt(maxDays)) return 0;
return Number(days);
}

function normalise(options) {
const siteUrl = String(options.siteUrl ?? '').replace(/\/+$/, '');
if (!siteUrl) throw new Error('createGateway needs siteUrl');
Expand All @@ -268,6 +358,7 @@ function normalise(options) {
priceCents: Number.isFinite(options.priceCents) ? options.priceCents : 100,
currency: options.currency ?? 'USD',
passMinutes: Number.isFinite(options.passMinutes) && options.passMinutes > 0 ? options.passMinutes : 1440,
maxDays: Number.isInteger(options.maxDays) && options.maxDays >= 1 ? options.maxDays : 30,
header: String(options.header ?? 'x-crawl-pass').toLowerCase(),
path: options.path ?? '/crawl',
openPaths: options.openPaths ?? [],
Expand Down
19 changes: 15 additions & 4 deletions src/page.js
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ export function renderPage(ctx) {
training = [],
retrieval = [],
contact,
days = 1,
total = price,
maxDays = 30,
} = ctx;
const window =
minutes === 1440
Expand Down Expand Up @@ -82,7 +85,12 @@ export function renderPage(ctx) {
<h1>Training crawlers pay for access here.</h1>
<p class="mut">People read <a href="${esc(siteUrl)}">${esc(siteName)}</a> free. So do search engines and the retrieval crawlers behind AI answers, because they send readers back. A crawler that copies pages into a training corpus sends nobody back, so it pays for the time it spends.</p>

<div class="price">${esc(price)} <span class="mut" style="font-size:1rem;font-weight:400">for ${esc(window)} of requests</span></div>
<div class="price">${esc(days > 1 ? total : price)} <span class="mut" style="font-size:1rem;font-weight:400">for ${esc(days > 1 ? `${days} × ${window}` : window)} of requests</span></div>
${
days > 1
? `<p class="mut">This offer is for ${days} days at ${esc(price)} a day. The plain page at <code>${esc(buyUrl)}</code> quotes one.</p>`
: `<p class="mut">Want longer? Add <code>?days=&lt;n&gt;</code> to this URL for an offer of up to ${maxDays} days at ${esc(price)} a day, or simply pay a whole multiple of the price: the pass lasts as many days as you paid for.</p>`
}
${
enabled
? ''
Expand All @@ -93,13 +101,15 @@ ${
<ol>
<li>Any page you fetch answers <code>402 Payment Required</code>. This page, fetched with <code>Accept: application/json</code>, returns the x402 offer: USDC, <code>exact</code> scheme, on ${esc(networks || 'Base, Polygon or Ethereum')}.</li>
<li>Sign the payment and retry with the proof in an <code>X-PAYMENT</code> header. The response is a JSON receipt carrying a pass.</li>
<li>Send the pass in <code>${esc(header)}</code> on every request for the next ${esc(window)}. When it expires, buy another. The sale is the pass, not the page: fetch the page again with the pass.</li>
<li>Send the pass in <code>${esc(header)}</code> on every request until it expires — ${esc(window)} per day paid, so a proof for three times the price buys three. When it expires, buy another. The sale is the pass, not the page: fetch the page again with the pass.</li>
</ol>

<h2>Pay with the CoinPay CLI</h2>
<p>Settlement is by CoinPay: the buyer's USDC goes straight to the site's wallet and CoinPay's relayer pays the gas, so you need USDC and nothing else.</p>
<pre><code>npm install -g @profullstack/coinpay
coinpay x402 pay ${esc(buyUrl)} --output pass.json</code></pre>
coinpay x402 pay ${esc(buyUrl)} --output pass.json
# or a week at once:
coinpay x402 pay "${esc(buyUrl)}?days=7" --output pass.json</code></pre>
<p>The command fetches this page, reads the offer, opens a browser tab to approve the payment with the CoinPay Wallet extension or any EIP-6963 wallet (MetaMask, Rabby, Coinbase Wallet), and writes the receipt to <code>pass.json</code>. Then:</p>
<pre><code>PASS=$(node -p "require('./pass.json').pass")
curl -H "${esc(header)}: $PASS" ${esc(siteUrl)}/</code></pre>
Expand All @@ -109,7 +119,8 @@ curl -H "${esc(header)}: $PASS" ${esc(siteUrl)}/</code></pre>
# 402 with { "x402Version": 2, "accepts": [ ... ] }
# sign an EIP-3009 transferWithAuthorization for one entry, then:
curl -sS -H "X-PAYMENT: &lt;base64 proof&gt;" ${esc(buyUrl)}
# 200 with { "ok": true, "pass": "cp_...", "expires_at": "...", "header": "${esc(header)}" }</code></pre>
# 200 with { "ok": true, "pass": "cp_...", "expires_at": "...", "days": 1, "header": "${esc(header)}" }</code></pre>
<p class="mut">The days a proof buys are read off the value it authorizes: a whole multiple of the one-day amount, up to ${maxDays}. <code>?days=&lt;n&gt;</code> only changes what the offer quotes, so a standard client that pays exactly what is asked gets <em>n</em> days.</p>
<p class="mut">The proof is x402 v2 in CoinPay's dialect: <code>{ x402Version: 2, scheme: "exact", network: "&lt;CAIP-2&gt;", payload: { signature, authorization } }</code>, base64-encoded. A proof is single-use; retrying with the same one returns the same pass, not a second charge.</p>

<h2>Who pays and who does not</h2>
Expand Down
Loading
Loading