Skip to content

Scraping

github-actions[bot] edited this page Nov 18, 2026 · 24 revisions

Scraping

A scrape is a job, not a ninth primitive. scrape() returns a JobHandle, so it inherits .enqueue(), the retry policy, the worker's cancellation, the dead-letter path and x jobs show — the same factory shape backfill(), llm() and agent() use (The eight primitives).

As of 2026-10-01: 53 source files, 32 owned X_SCRAPE_* codes, 2 borrowed. Zero runtime dependencies outside @ultimat3/* — puppeteer is passed in, never imported (Drivers).

What ships

Capability Mechanism
Browser automation ScrapePage, driver-blind — the same body runs on a browser, a recording, or a string of HTML
The second transport http on the same session: the browser's cookies, headers, proxy, host list, rate limit and cancellation
Actionability waits for visible, enabled, unobstructed and still before acting — not for the selector alone
Host allow list intercept.ts, one decision, asked by every driver and the HTTP leg, before a byte leaves
robots.txt obeyed by default, on both legs, deadlined and capped
Rate limiting navigations per second, across both legs. There is no unpaced mode
Secrets declared by name, resolved in the worker, and a typed secret refuses later pixel captures
Sessions capture, persist sealed, restore, validate, burn
One session per connection concurrency: { key, limit: 1, whenBusy: 'fail' } — the job's own keyed cap
An exit per run egress: (input, ctx) => proxyUrl, resolved in the worker — never carried in the queue payload — and dialled by both legs and the robots read
A rented browser remoteBrowser({ cdpUrl: resolver }) — acquired per session, released exactly once
A prompt answered mid-run eventPrompt({ timeout }) + answerPrompt(), over the stored job event bus, with the browser open
Usage on every result ScrapeReport.usage — a measurement, never a quota
The silent-green alarm expect: { minRows, maxDrop } — a scrape that returns nothing fails instead of succeeding
Offline testing two drivers that never touch the network, pinned against the real one by a parity suite
Wedge watchdog ends a run whose browser has gone silent: kills the process and closes the browser at the fire, an attached one too

The declaration

import { t } from '@ultimat3/action';
import { scrape } from '@ultimat3/scraping';

export const dailyOrders = scrape({
  name: 'orders.daily',
  input: t.object({ orgId: t.uuid, page: t.number.int() }),
  extract: t.object({ id: t.string, title: t.string }),
  idempotencyKey: (input) => `orders:${input.orgId}:${input.page}`,
  tenant: (input) => input.orgId,
  allowHosts: ['shop.example'],
  expect: { minRows: 1 },
  async run({ page, http, step }) {
    await step.run('list', () => page.goto('https://shop.example/orders'));
    return (await page.values('.row')).map((row) => ({ id: row.attrs['data-id'], title: row.text }));
  },
});
Field Required What it decides
name yes the durable queue key, never an export name
input yes the job's input schema
extract yes every row, parsed. A row the schema rejects is X_SCRAPE_OUTPUT_INVALID, never a stored partial
idempotencyKey yes derived from input alone, like every job's
tenant yes or 'none' — a job runs with no request behind it, so an unscoped read would be unscoped for real
allowHosts yes ['*'] is legal and is a decision spelled out; [] is refused at declaration
run yes the body: { input, page, http, step, ctx, secrets }
block — resource types to drop (image, font, …)
rate — navigations per second. Default 1; 0 or negative is refused at declaration
robots — 'obey' (default) or { ignore: '<reason>' } — a reason is required
expect / history — the yield alarm, below
secrets — names, never values
auth / prompt — session lifecycle, and where an out-of-band code comes from — see A session a service is built on
egress — (input, ctx) => string | undefined, sync or async: the proxy URL THIS run leaves through, looked up in the worker. Wins over the driver's proxy
onSettled — how the run ended, told once: the ScrapeReport on completed, the code and the last attempt's usage otherwise — see Usage
recover — a function — see Recovery
artifacts — where a failed run's HTML is written: { storage: () => disk('artifacts') } — a thunk, read per write
driver — which browser. Absent uses the process's — scrapeDriver in apps/<app>/runtime.ts
watchdog — { idleMs, graceMs }, defaults 120000 / 5000
retry / timeout / pageTimeout / queue / concurrency — the job knobs, unchanged — concurrency takes the job's keyed form too (Jobs and workflows)
clock — pins this definition's clock. Absent uses the process's — @ultimat3/core's ambient clock, which is what a test replaces

The CLI has no scrape command and no scrape generator As of 2026-08-20 — g ships thirteen positionals and scrape is not one of them. A scrape is reached through the job CLI:

x jobs list --name orders.daily --json
x jobs show <id> --json
x jobs retry <id> --from-step list --json

The page vocabulary

One interface, three drivers, no driver type in it.

Call Notes
goto(url, { timeout }) refused before a byte leaves on a blocked host or a disallowed path
waitFor(selector, { state, timeout }) attached → visible → enabled → actionable; each implies the ones above
click / type / fill / select every one waits for actionable first
values / text / html / count reads; text() with no argument is the whole document
evaluate(expression) returns unknown — parse it
frame(nameOrSelector) re-resolves on every call; nothing hands out a stale handle
screenshot { fullPage } or { clip }, never both
pdf the print layout: format (a4, letter, …) or widthMm + heightMm, marginMm, printBackground, preferCssPageSize, headerHtml / footerHtml. Two sizes, half a size or a size beside preferCssPageSize is X_SCRAPE_PDF_OPTIONS_INVALID (422), before any driver
download({ timeout }) the file the last click, key or navigation produced — the newest since it, older ones cleared — or X_SCRAPE_DOWNLOAD_TIMEOUT. A file from an earlier trigger, or one that landed after its own download() timed out, never answers a later one. Off the wire on the browser driver, never a download directory, capped at maxDownloadBytes — see The port is complete
cookies / session the handoff to the HTTP leg, as a value you can inspect
query(selector) every match as a snapshot — tag, attributes, text, and visible. The one definition of "visible" in the framework; nothing else may compute it
offline(enabled) the browser's own offline mode. Not @ultimat3/testing's sealed network, which patches fetch in the test process and a browser's requests never traverse — a test built on that would pass against a fully online app
focus(selector, { timeout }) waits for actionable, then moves focus — the setup for a press()
press(chord) 'Meta+K', 'Escape', 'Shift+Tab' on whatever holds focus. Modifiers are Meta, Control, Alt, Shift in the browser's own spelling; 'Ctrl+K' is X_SCRAPE_KEY_INVALID on every driver, offline included — the parse is the one thing an offline driver can be wrong about, so it runs there too, then stops
accessibility(selector, { max }) what a screen reader is told about each match — role and name after ARIA and label association, at most max (default 25) nodes, from this handle's document: on a frame(…) handle, the frame's. The browser driver answers what Chrome computed; the offline drivers answer the HTML-AAM mapping of the markup, never role= read alone: a <div onclick> maps to no role on every driver, and that is the finding
console() / network() bounded rings, 200 entries each
pageErrors() / pageErrorsDropped() uncaught exceptions, a third ring — not console lines
activity() tells the wedge watchdog the run is working on a channel this page cannot see — your own CDP session, a long warm-up. Every verb above already reports; this touches the idle clock only and counts no usage. On every driver

The wedge watchdog ends the run at the fire, on every driver (As of 2026-11). Past watchdog.idleMs with no activity it kills the OS process, starts the graceful close under graceMs (a second kill past it), then aborts with X_SCRAPE_WEDGED. On remoteBrowser() there is no process to kill, and a call blocked on the socket — evaluate takes no signal — never unwound to close(): the run kept driving the site for minutes. Closing the browser is what rejects that call, and it rejects as X_SCRAPE_WEDGED, not X_SCRAPE_BROWSER_UNREACHABLE. A run that drives the browser through its own channel calls page.activity() between steps rather than routing keep-alives through page.evaluate.

A frame verb acts on the frame, and this had to be fixed. frame() returns a target that overrides every verb addressing a document — click, type, select, clear, query. Until 2026-08-25 clear was missing from that list, so frame(…).fill() cleared the parent's same-id field and then appended to the frame's: a remembered username submitted as oldUserNEWUSER while a parent field was silently emptied. On the offline drivers one shared overlay made it worse — page.values('#password') read back what was typed into the frame.

driver-parity-frames.test.ts pins every frame verb across all three drivers, because the parity suite that exists to catch a CDP/offline divergence had no frame coverage at all.

screenshot/pdf take no timeout — CaptureRequest.timeout and the port's CaptureOptions.timeoutMs were deleted in 4.0.0 (Upgrading). No driver had ever honoured them, and a deadline enforced above the driver would have had to race the clock's sleep, which under a test's frozenClock() resolves on the first microtask — so every capture in every test would have timed out. The driver's own default is the honest bound.

pageErrors() is a separate stream from console(), and that is not tidiness. An uncaught exception calls no console method, so a page whose script died can answer console(): [] — gate on console alone and it reads as clean. Nothing in this package observed pageerror before 2026-08-21: console and the renderer-crash error event were subscribed, and a throw fell between them. The two events are also not interchangeable — error latches the crash flag, after which every call answers X_SCRAPE_PAGE_CRASHED, which is classified terminal; routing a pageerror there would dead-letter a scrape of a page that still renders perfectly.

A PageError carries message, an optional stack and at. Keep the stack — it names the module and line that threw (at Cart (/app/islands/cart.tsx:31:18)), which is the difference between a report and a lead; message alone says what went wrong and never where. It is truncated at MAX_PAGE_ERROR_CHARS (4,000) and marked when it is, because one blown stack is thousands of frames. An absent stack means the payload carried none — a throw 'string' — never that the throw had no origin.

pageErrorsDropped() makes the count a floor: the ring is bounded, so pageErrors().length is what survived eviction and not what happened.

click takes no index. It clicks the first match, on every driver.

Two transports, one session

Drive the browser through login and navigation, then pull the bulk off the site's own JSON endpoints. http.request() carries the browser's cookies (scoped per RFC 6265 §5.1.3/§5.1.4), its headers, its proxy, the same allowHosts, the same robots gate, the same pacing and the same cancellation. A redirect is followed one hop at a time under the same gates — each Location is screened against allowHosts and robots, recorded in page.network() under the URL actually requested, and re-scoped for cookies — so a scraped endpoint cannot 302 the worker onto a host the allow list never named. A hop to another origin carries no credential you set (authorization, proxy-authorization, a hand-written cookie), and it stays dropped for the rest of the chain; a same-origin hop keeps them. A 301/302 re-asks a POST as a GET and leaves every other method — a PUT or a DELETE — intact with its body, 303 re-asks everything but GET/HEAD, and 307/308 carry both. res.url is the final URL, a chain past MAX_REDIRECT_HOPS (10) is X_SCRAPE_REDIRECT_LOOP, and a javascript: URL is refused outright rather than read as a hostless scheme. Header names are case-insensitive, so a header you declare replaces the session's in any spelling — { 'User-Agent': 'Mine', Cookie: '…' } wins over the browser's agent and the jar instead of being joined to them.

Two hundred paginated pages clicked through is minutes and two hundred chances to break; the same data off the endpoint behind them is seconds, and a JSON endpoint changes far less often than a DOM.

Response bodies are counted as they arrive, capped at DEFAULT_HTTP_MAX_BYTES (32 MiB) — .text() on a hostile stream is a heap the worker never gets back.

Drivers

Driver Use Network
localBrowser({ launcher, executablePath }) a browser in this container real
remoteBrowser({ launcher, cdpUrl }) an attached CDP endpoint — a fixed URL, or a CdpResolver that rents one per session real
fixtureBrowser(dir) committed page recordings none
fakeBrowser(pages) inline pages, for a unit test none

The launcher is passed in, never imported. CdpLauncherLike is two methods — launch and connect — so puppeteer, puppeteer-core or anything with the same shape satisfies it, and this package declares no browser dependency at all. A launcher with no launch is X_SCRAPE_REMOTE_REQUIRED, not a crash. The browser it hands back must expose target().createCDPSession() — puppeteer's does — because allowHosts is enforced at the browser too, on every tab a scraped page opens; its pages and the browser must carry every verb the page forwards to, checked when the session opens (X_SCRAPE_LAUNCHER_INCOMPLETE, see The port is complete).

driver-parity.test.ts runs one suite against all three and pins the single honest divergence: the offline drivers have no layout engine, so ElementSnapshot.box and hitTarget are absent rather than faked.

The port is complete

Every page verb answers on every driver; none is X_NOT_IMPLEMENTED (As of 2026-11-15). Thirteen refusals came out of cdp-target.ts and html-target.ts: a port method one driver cannot honour is a second path, so each one was implemented there — or, for a launcher missing a method, moved to a check when the session opens. port-complete.test.ts calls every ScrapeTarget member on the offline and the browser driver, on the page and on a frame, keyed by the port's own members — a member added without a row does not compile.

Verb Browser driver Offline drivers
download() the file's bytes off the wire: the browser-level Fetch pause at the Response stage streams the body (Fetch.takeResponseBodyAsStream + IO.read) under a byte cap — localBrowser({ maxDownloadBytes }), default DEFAULT_MAX_DOWNLOAD_BYTES (25 MiB) — and fulfils the response with the bytes read; Browser.downloadWillBegin names it, and Chrome's own copy is denied. Over the cap nothing is buffered: a declared Content-Length over it is left to Chrome (download() times out), an undeclared attachment is abandoned at the cap and failed, an undeclared non-attachment is never read. Works on remoteBrowser(), whose disk is not yours. A script-made blob: download is read through the page, its size checked first the recorded downloads entry the LAST click armed — a click that downloads nothing, or a key, disarms it
offline(enabled) the browser's own network condition every navigation — goto, a link's click — fails X_SCRAPE_BROWSER_UNREACHABLE while set, the code a browser's failed navigation carries. No JS runs, so an assertion on what the app's offline code renders fails rather than passing
accessibility() what the browser computed — on a frame handle, scoped to the frame's document, found through the page's own session (Page.getFrameTree → DOM.getFrameOwner → its contentDocument). A cross-site frame under site isolation renders in another process the page session cannot read: X_SCRAPE_FRAME_OUT_OF_PROCESS, fixed by localBrowser({ launcher, options: { args: ['--disable-site-isolation-trials'] } }) (measured on Chrome 157) the HTML-AAM mapping of the markup — the frame's own markup on a frame handle: explicit role, else the implicit one; the name from aria-labelledby, aria-label, <label for>, alt, content, title, placeholder. CSS and script are not seen. A <div onclick> answers no role and no name
press / focus / colorScheme / prepare / cookies / session CdpPageLike and CdpBrowserLike keep keyboard, focus, createCDPSession, setOfflineMode, emulateMediaFeatures, evaluateOnNewDocument, cookies, setCookie optional in the type, so a launcher written against 31.x still compiles — and checks every one when the session opens: a missing one is X_SCRAPE_LAUNCHER_INCOMPLETE naming each, and the launched browser is closed. Never a refusal on the first run that reached the verb. CompleteLauncher is the same shape with each required, which the built-in launchers are declared as unchanged

A frame handle reads its frame. page.frame(…).accessibility() answers from the frame's document on every driver; it was X_NOT_IMPLEMENTED on all three until 2026-11-15.

What a browser dependency beside scraping still buys. A direct puppeteer import for a download, an offline toggle, a keyboard chord, an accessibility read, an init script, a theme, cookies or a frame's form is now a page verb, on every driver, with allowHosts, robots, pacing, redaction and the wedge watchdog applied. What the port does not cover, and a direct import still does: request/response bodies of subresources, a download Chrome makes from an HTML response with no Content-Disposition, file uploads, and multiple tabs.

The silent-green alarm

A scraper that returns zero rows because the site changed its markup is the failure that does not look like one.

expect: { minRows: 1, maxDrop: 0.5, window: 7 },
history: yourYieldHistory,
Rule Needs history Fires when
minRows no the run returned fewer rows than the floor
maxDrop yes the run fell more than this fraction below the trailing median

minRows is an absolute floor, checked before the history gate, so it is legal on its own. maxDrop is a fraction in [0, 1) — 50 or 1 would make the alarm unable to fire, and a negative one fires on a run that matched its baseline, so each is refused (X_INVARIANT) where the scrape is declared. It is a fraction of a trailing median and only a history: store can supply one — declaring it alone is refused at declaration with X_SCRAPE_YIELD_HISTORY_MISSING, because with no store the baseline is [] forever and the alarm could never fire.

A collapsed run is not recorded. Three broken runs at 2 rows would make the median 2, and the fourth broken run would be within maxDrop of it — a scraper that re-baselines onto its own failure has silenced the alarm exactly when it was working. maxDrop needs MIN_BASELINE_RUNS (3) runs after it is declared before it can fire: a delay, not a hole.

Robots, hosts, rate

Gate Default Refusal
allowHosts required, no default X_SCRAPE_HOST_BLOCKED, before a byte leaves — on the page, on every other tab the page opens, on the HTTP leg and on the robots read
robots 'obey' X_SCRAPE_ROBOTS_DISALLOWED
rate 1 navigation/second none — it paces, across both legs

A wildcard admits names, then checks where they lead. It never admits localhost or *.localhost — loopback by name, and a name Chrome would send around an egress proxy; allowHosts: ['*', 'localhost'] is the opt-in. A browser launched with an exit gets --proxy-bypass-list=<-loopback>, so loopback goes through the proxy too. On the two legs this package dials itself — the HTTP leg and the robots read — a host admitted only by '*' or '*.example.com' is resolved, and an answer that is loopback, private, link-local or the metadata address is X_SCRAPE_HOST_BLOCKED; the request is then pinned to the approved address. An exact rule ('intranet.example') is the decision to reach that host and is not resolved. Through a proxy the proxy resolves, so nothing is checked locally. The browser leg screens the same way before it releases a request — the page's own and a popup's: an inward or unresolvable answer fails the request and lands in page.network() as refused: 'host'. It cannot pin, because the browser resolves again for itself, so a name that rebinds between the two is open, and a WebSocket opened by a page is not intercepted at all → Known gaps.

Ignoring robots takes a written reason: robots: { ignore: 'contract with the operator, ticket OPS-441' }. A bare false is a decision with no author.

The /robots.txt read is deadlined (10s), capped (500 KiB) and dialled through the session's own exit — the run's egress, else the driver's proxy — asked per read, because the exit is resolved inside driver.open() while the gate is an argument to it. An offline driver reports the run's egress too, so its one real request leaves the same way. A redirect is followed one hop at a time, at most MAX_ROBOTS_REDIRECTS (5, RFC 9309's floor), each hop asked of allowHosts first — a hop off the list is never requested. (robotsFetcher / robotsGate called with no allowHosts follow only a hop on the starting hostname — http: → https:, a moved path.) What the read answers decides the run, after the "access results" rules of RFC 9309 (As of 2026-10):

The read Means The gate
2xx the file its rules, cached for the run
2xx with a body past the cap (500 KiB) the file's head the whole lines inside the cap are parsed (RFC 9309 §2.5), cached for the run; a head with no line end is unreachable
4xx (not 429), a redirect off the list or past the hop cap unavailable no restrictions, cached for the run
5xx, 429, a refused connection, the deadline, the run cancelled, a fetchText that rejects unreachable complete disallow — every path on the origin is X_SCRAPE_ROBOTS_DISALLOWED — and not cached: the next navigation asks again, so the run recovers when the file answers

429 is the one row that is our policy, not the RFC's: RFC 9309 calls every 4xx unavailable, 429 included. Ultimate refuses on it because a server rate-limiting the crawler has not said "no rules" — stricter than the RFC requires, never looser.

A hand-written fetchText answers the text, undefined (no file) or { unreachable: '<why>' }.

Robots patterns are walked, not compiled — a wildcard-dense rule from a scraped site is linear rather than catastrophic backtracking on the worker's only thread.

Secrets

Declared by name, resolved in the worker, never in the definition:

secrets: ['SHOP_PASSWORD'],
run: async ({ page, secrets }) => {
  await page.type('#password', secrets.get('SHOP_PASSWORD'));
}

Redaction is by VALUE, on four surfaces: page.html(), page.console(), page.network() URLs and page.pageErrors() — plus the HTTP leg's response body, which reaches an error cause and from there a dead-letter row that x jobs show prints. Three of those were unredacted until 2026-08-25 while this package's own header promised all of them. Every spelling of a value is redacted: raw, percent-encoded (hex escapes in either case), form-encoded (+ for a space) and HTML-escaped as text, as an attribute or by a template (&#39;) — a password in a query string is never the raw string.

A credential-bearing URL — a proxy exit, a provider's connect URL — hands the redaction set the whole URL, its password, its query, and only those query values and path segments that are credential-shaped (a key whose words — split on _, -, . and camelCase — include token, key, secret, auth, sig, session and the like, so apiKey counts and keyboard does not; a long letters-and-digits run; or the id after /devtools/browser/ or /devtools/page/). stealth=true&proxy=residential does not blank every true and residential in the artifact.

One stated limit: a secret shorter than 4 characters is not redacted. A 3-character token would match too much ordinary page text to be safe to blank; declare a longer one.

Typing a Secret taints the page. A later screenshot() or pdf() is refused with X_SCRAPE_SECRET_EXPOSED — a screenshot of a filled login form is the password, in pixels, in object storage, forever. Refused rather than masked: a mask over pixels is a guess about layout, and page.html() already gives a redacted artifact that is exact.

Sessions

Reuse is both the fast path and the safe path — logging in on every run is slow and is itself the signal anti-bot systems look for.

Call What it does
restorableSession(plan) the stored session this run may restore, or undefined
ensureAuthenticated(plan) log in when there is nothing to restore
burnSession(plan, seen) delete it — a flagged profile stays flagged, so a retry that reloads it re-trips the block. seen is the version of the record this run found — restorable or not — or saved (recordVersion(restored); a record from before versions shipped reads its savedAt). A record another run on the same key saved since is left in place; the refusal tombstone a failed login writes follows the same rule. The compare narrows the race and does not close it: neither the store nor StorageDriver has a conditional write
memorySessionStore() / storageSessionStore(() => disk('sessions')) where it lives. The disk is a THUNK, read per call: a scrape is declared when its module loads, and the app's disks exist only after boot ran defineStorage()

A stored session is sealed. storageSessionStore writes { "sealed": "x1.…" } — core's seal() under the app's master key (x secrets init), purpose scrape-session. No cookie value, localStorage token or origin is readable from the bucket.

Found on load() Answer
this store's sealed record, saved under this key the session
nothing undefined — the run logs in
an unsealed object, a corrupt one, one whose tag fails under a key this process holds, one copied from another key's path burned, then undefined — a session is a cache, so nothing is migrated
one sealed under a key this process does not hold yet (X_SEAL_KEY_UNKNOWN, mid-rotation) undefined, the object kept: a pod that already holds the new key still restores it
no master key at all X_SEAL_KEY_MISSING, before the browser opens — run x secrets init

Upgrading into sealed sessions discards what was stored before, refusal markers included. A record written before sealing shipped (23.0.0) is unsealed, so its first load() burns it: the run logs in again, and a credential the site had already refused is presented once more instead of being refused before the browser opens. On a site that locks an account after repeated failures, correct that credential before the first run after upgrading (Upgrading, 22.x → 23.0.0).

The key hashes each segment, and this changed on 2026-08-25. A session key is <tenant>/s/<discriminator>, and both segments used to be sanitised by collapsing every run of non-[a-zA-Z0-9._-] to a single -. So alice@corp.com and alice-corp.com produced one key, as did acct/1 and acct-1, and tenants acme corp and acme-corp. The browser then loaded account A's cookies, auth.validate() answered true — the session is valid, for the wrong account — and A's rows were stored under B's tenant. Each segment now carries a hash of its raw value, so two spellings can no longer collide.

Migration: none to run. Every stored key changes spelling, a miss reads as "no session", so the run logs in again and writes the new key. One extra login per stored session, no error, and the old objects are orphaned until the bucket's lifecycle rule collects them.

A session record survives a refusal rather than being deleted with it: refusedAt is read before driver.open(), so a replay, a manual x jobs retry or a second enqueue refuses without spending a browser, a CDP attach or one more wrong password at a site that locks accounts after three.

parseSessionState completes a stored cookie rather than asserting it. A jar entry carrying only name and value gets domain: '', path: '/', httpOnly: false, secure: false. An empty domain matches no host — inferring one from whichever URL is asking is exactly how a bank.test session cookie reaches evilbank.test. Before 4.0.0 the parser claimed such an entry was a whole ScrapeCookie and cookieHeaderFor then threw a bare TypeError on domain.trim().

Session material is credential material: it is tenant-scoped, it never reaches a log line, an event field, an artifact or a screenshot, and sessionDigest() summarises it as counts and an origin.

A session a service is built on

One declaration: a caller opens a session, events stream out, input goes in mid-run, one result comes back. Every piece is a field on scrape() or on its driver.

import { t } from '@ultimat3/action';
import {
  type CdpLauncherLike,
  type CdpResolver,
  eventPrompt,
  remoteBrowser,
  scrape,
  storageSessionStore,
} from '@ultimat3/scraping';
import { disk } from '@ultimat3/storage';

// The app's own: the browser vendor's client, its row store, its login and its scrape body.
declare const puppeteer: CdpLauncherLike;
declare const provider: {
  create(input: { tag: string | undefined; proxy: string | undefined; signal: AbortSignal | undefined }): Promise<{
    readonly id: string;
    readonly connectUrl: string;
    readonly priceCents: number;
  }>;
  end(id: string): Promise<void>;
};
declare const connections: {
  byId(id: string): Promise<{ readonly exit: string } | null>;
};
declare const runs: { record(runId: string, usage: unknown): Promise<void> };
declare function login(): Promise<void>;
declare function run(): Promise<readonly unknown[]>;

const rent: CdpResolver = async ({ runId, egress, signal }) => {
  const rented = await provider.create({ tag: runId, proxy: egress, signal });
  return {
    cdpUrl: rented.connectUrl,
    release: () => provider.end(rented.id),
    cost: { minor: rented.priceCents, currency: 'USD' },
  };
};

export const syncAccounts = scrape({
  name: 'accounts.sync',
  // Ids only: the input is the queue row, and a proxy URL carries its account.
  input: t.object({ connectionId: t.uuid, orgId: t.uuid }),
  extract: t.object({ id: t.string }),
  idempotencyKey: ({ connectionId }) => `accounts:${connectionId}`,
  tenant: ({ orgId }) => orgId,
  allowHosts: ['bank.example'],
  driver: remoteBrowser({ launcher: puppeteer, cdpUrl: rent }),
  // Resolved in the worker, under the job's tenant — a `.sealed()` column on the connection row.
  egress: async ({ connectionId }) => (await connections.byId(connectionId))?.exit,
  concurrency: { key: ({ connectionId }) => connectionId, limit: 1, whenBusy: 'fail' },
  auth: {
    store: storageSessionStore(() => disk('sessions')),
    key: ({ connectionId }) => connectionId,
    login,
  },
  prompt: eventPrompt({ timeout: 300_000 }),
  run,
  // How the run ended: the report with its usage, or the code it failed with.
  async onSettled(settled) {
    if (settled.input === undefined) return;
    await runs.record(settled.runId, settled.outcome === 'completed' ? settled.result.usage : settled.usage);
  },
});
Field Rule
egress the run's exit, resolved in the WORKER: (input, ctx), sync or async, under the job's tenant. The input names a row; the row holds the exit. An exit whose password is also in the input rode the queue payload — x_jobs holds it in the clear, and redactInput hides a key by name for display only — so it is refused before the browser opens: X_SCRAPE_EGRESS_IN_PAYLOAD, terminal. Both legs dial it and the robots read leaves through it. localBrowser({ options: { args } }) keeps the caller's args and appends --proxy-server after them; a proxy switch of the caller's own (--proxy-server, --proxy-bypass-list, --proxy-pac-url, --proxy-auto-detect, --no-proxy-server) or args that are not a list of strings are X_SCRAPE_LAUNCH_ARGS_INVALID, terminal, before anything launches. Credentials in the URL never reach the process arguments — a launched browser answers the proxy through page.authenticate() — and are redacted by value everywhere
cdpUrl: resolver called once per open() with { scrape, runId, egress, signal }. release() runs exactly once: in the session's close() after the browser is quit — on success, failure and cancellation — and straight away when the attach itself fails. The framework ships no vendor
cdpUrl: 'ws://…' + egress X_SCRAPE_EGRESS_UNSUPPORTED when the exits differ: a fixed URL is a browser already bound to one
concurrency: { key, limit, whenBusy } the job's field. whenBusy: 'fail' settles a second run of one connection failed with X_JOB_KEY_BUSY, body never run
auth.store the declaration that persists a session. Absent means every run logs in. auth.login and auth.validate are handed runId beside input
prompt: eventPrompt({ timeout }) below

A prompt answered while the browser stays open

eventPrompt({ timeout, pollMs? }) polls the job event bus in process. It is not step.waitForEvent: that suspends the run and closes the browser the site is waiting in.

The bus must be the stored one (EventBus.stored): the answer is published by a web process and read by the worker. Outside development and test, an in-memory bus is refused when the prompt is asked — X_DRIVER_UNAVAILABLE, fix eventBus: postgresEventBus({ executor }) in apps/<app>/runtime.ts — rather than timing out minutes later as X_SCRAPE_PROMPT_UNANSWERED.

Fact Detail
event name derived, never chosen: promptEventName(runId, index) → scrape-prompt:<runId>:<n>, n from 1 per attempt. PromptRequest carries both, and the run's input — what a handler ties the prompt to in the app's own tables
answering answerPrompt({ runId, index, answer }) from the app's own action. The name is not an authorization — the action's policy is
on the bus the answer is sealed for that one event name and expires in 10 minutes (ttl). A payload answerPrompt() did not write is refused, never typed
stale answers only an answer published after the prompt was asked is consumed — the run id is the same on every attempt
while it waits the worker's heartbeat keeps the claim; one browser round trip per poll keeps the wedge watchdog and a rented browser alive
it ends on the answer · timeout → X_SCRAPE_PROMPT_UNANSWERED (terminal) · the run's signal · a browser that died
the bus must be the stored one (postgresEventBus) wherever the worker and the answering process differ. Every x boot installs it; to swap it, eventBus: in apps/<app>/runtime.ts (Adapters)

Saying a prompt is pending is the app's — wrap the handler, it is a function:

const waitForAnswer = eventPrompt({ timeout: 300_000 });
// on the definition:
prompt: async (request) => {
  await runEvents.insert({ runId: request.runId, index: request.index, label: request.label });
  return waitForAnswer(request);
},

Usage

ScrapeReport.usage, and the same counts on the scrape.ok / scrape.failed log lines.

Field Counts
browserMs from before driver.open() to the report, on the run's clock
navigations page.goto() calls that reached the driver — a refused host is not one
httpRequests requests the HTTP leg put on the wire, one per redirect hop
bytesIn response-body bytes the HTTP leg read. The browser leg's traffic is not measured
promptsAnswered prompts a handler answered
browserCost the resolver's cost, a Money ({ minor, currency, scale? }) — a float is X_VALIDATION_FAILED. Absent when nothing was rented

No quota, plan or ledger reads these numbers. What a number is worth is the app's.

Reaching the app. A worker hands a job's return value to one place: the job's onSettled (Jobs and workflows). A scrape's adds what is scrape-specific.

settled.outcome Carries
completed result — the whole ScrapeReport: rows, artifacts, usage
dead-lettered / dropped error, code, and usage: what the last attempt used. A failed run has no report and was billed all the same
refused code: 'X_JOB_KEY_BUSY'; usage is undefined — the body never ran

After the row is settled, under the job's tenant. A completed run carries its report, a value, so it is told at most once across a crash; any other ending is at least once — re-driven with redriven: true and usage: undefined when its worker died first. A hook that throws is X_JOB_ON_SETTLED_FAILED and changes nothing. run({ progress }) is the job's own progress(done, total, note?), passed through.

A test's clock

import { afterEach, beforeEach } from 'bun:test';
import { noWaitClock } from '@ultimat3/core';
import { installClock } from '@ultimat3/core/adapter';

let release = (): void => {};
beforeEach(() => {
  release = installClock(noWaitClock);
});
afterEach(() => release());
Clock (@ultimat3/core, since 31.0.0) Sleeping For
noWaitClock one turn of the event loop; now() and deadlines are real a run that waits on something the test does — an answer it publishes, a second run it starts
frozenClock() advances virtual time a test OF a timeout

A run waits on the process's clock unless its definition pins one, so the declaration under test carries no clock: written for the test's sake.

What never reaches a log, an error or an artifact

A prompt answer, the password of an egress URL and every credential-bearing part of a CDP URL are added to the run's redaction set — secrets.conceal(value), which a run body may call too. X_SCRAPE_CDP_ATTACH_FAILED names scheme and host only, for a fixed URL as for a resolved one: a provider's connect URL is its access token.

Recovery

recover: async ({ page, failure, attempt }) => { /* return true to re-run the body once */ }

A recover hook is a function, and only a function: recover: 'agent' was removed in 26.0.0, and anything but a function is refused at scrape(). A recovery that asks a model to re-derive the selector is your own hook calling your own llm() action with page.html(). A hook that answers anything other than a boolean is X_SCRAPE_RECOVER_REFUSED.

Error codes

Every owned code is split by whether the same request can succeed unchanged. x errors explain <CODE> --json prints the cause, a runnable fix and the docs URL for any of them (Error codes).

Retryable Why
X_SCRAPE_CDP_ATTACH_FAILED, X_SCRAPE_BROWSER_UNREACHABLE the browser, not the page
X_SCRAPE_TIMEOUT, X_SCRAPE_WEDGED, X_SCRAPE_DOWNLOAD_TIMEOUT a moment, not a property of the site
X_SCRAPE_HTTP_FAILED 408, 409, 425, 429, any 5xx, a deploy — transient far more often than not. Which 4xx are permanent is @ultimat3/core's isRetryableStatus, not this package's, As of 2026-08-23: a private copy here called 408 and 425 terminal while the rest of the framework called them retryable, and a terminal classification dead-letters the run on the attempt that failed rather than spending its declared attempts
X_SCRAPE_BLOCKED retryable and it burns the session first: retrying a block on the same flagged cookies re-trips it every time

Everything else is terminal, including X_SCRAPE_SELECTOR_MISSING (the markup changed), X_SCRAPE_OUTPUT_INVALID (the rows are the wrong shape), X_SCRAPE_YIELD_COLLAPSED, X_SCRAPE_SECRET_EXPOSED, X_SCRAPE_KEY_INVALID (the chord is the caller's own literal), X_SCRAPE_PROMPT_UNANSWERED, X_SCRAPE_BROWSER_MISSING (there is no browser binary to launch), X_SCRAPE_EGRESS_UNSUPPORTED (the exit is the run's input and the driver is the deploy's constant) X_SCRAPE_LAUNCH_ARGS_INVALID (the args are the deploy's literal), X_SCRAPE_LAUNCHER_INCOMPLETE (the launcher is the deploy's library) and X_SCRAPE_FRAME_OUT_OF_PROCESS (site isolation is the browser's configuration).

Testing offline

fakeBrowser and fixtureBrowser never touch the network. An unrecorded page or request throws rather than falling through, so a green suite cannot be secretly live.

One request escapes that, and it is the robots read. Under the default robots: 'obey', an offline driver still fires a real fetch at https://<host>/robots.txt before the first navigation — the gate is built in scrape-run.ts and knows nothing about which driver open() will return. Measured again As of 2026-08-20: one egress per origin per run. Under bun test the sealed network refuses it, and a refused read is unreachable, so the run fails X_SCRAPE_ROBOTS_DISALLOWED on its first navigation — the leak is now loud rather than green. Declare robots: { ignore: 'offline fixture' } on an offline scrape until it is closed → Known gaps.

What it does not do

Not shipped Detail
Browser-leg bytes in usage.bytesIn the CDP port subscribes to requests, not to their sizes. bytesIn is the HTTP leg alone
Usage on a FAILED run's result a failed run has no report. Its counts are on the scrape.failed log line
A vendor no browser provider, proxy inventory, captcha or OTP relay. CdpResolver, egress and PromptHandler are the seams
A capture deadline deleted in 4.0.0; the driver's own default is the bound
CAPTCHA solving, fingerprint evasion, proxy rotation none of it. X_SCRAPE_BLOCKED burns the session and re-raises
A scaffold g has no scrape positional; write the declaration by hand

Pages as files: PDF and cards

page.pdf(layout) is the invoice path, through the browser the app already brings. Who may reach the page is the app's policy: print from a route a signed token opens, never from a page a secret was typed into (X_SCRAPE_SECRET_EXPOSED keeps its rule). The offline drivers answer different deterministic bytes per layout, so a dropped margin fails a test. As of 2026-11.

import { t } from '@ultimat3/jobs';
import type { CdpLauncherLike } from '@ultimat3/scraping';
import { localBrowser, scrape } from '@ultimat3/scraping';
import { disk } from '@ultimat3/storage';

declare const puppeteer: CdpLauncherLike; // the app's own `puppeteer-core`, passed in

export const printInvoice = scrape({
  name: 'invoices.print',
  input: t.object({ orgId: t.uuid, invoiceId: t.uuid }),
  extract: t.object({ key: t.string }),
  idempotencyKey: ({ invoiceId }) => `invoices.print:${invoiceId}`,
  tenant: ({ orgId }) => orgId,
  allowHosts: ['app.example'],
  driver: localBrowser({ launcher: puppeteer }),
  async run({ input, page }) {
    // The app's own print route, behind a token it signs and verifies — its policy, not this one.
    await page.goto(`https://app.example/invoices/${input.invoiceId}/print`);
    const pdf = await page.pdf({ format: 'a4', marginMm: { top: 15, bottom: 15 } });
    const key = `org/${input.orgId}/invoices/${input.invoiceId}.pdf`;
    await disk().put(key, pdf, { contentType: 'application/pdf' });
    return [{ key }];
  },
});

A route printed to PDF — pdf()

pdf() is a factory over action (@ultimat3/scraping, a PRIMITIVE_FACTORIES row). It renders one of the app's OWN routes through Chrome over raw CDP (Page.printToPDF), writes the bytes to a disk and answers { key, bytes, contentType: 'application/pdf' }. Schema, policy, rate limit, meter and audit are the action's; HTTP, OpenAPI, MCP, .job() and .as() come with it. As of 2026-11-09.

import { can } from '@ultimat3/policy';
import { t } from '@ultimat3/schema';
import { pdf } from '@ultimat3/scraping';
import { disk } from '@ultimat3/storage';

declare function printToken(orgId: string | undefined): string; // the app's short-lived token

export const invoicePdf = pdf({
  input: t.object({ id: t.uuid }),
  policy: can('invoices:read'),
  route: ({ input }) => `/invoices/${input.id}/print`, // a PATH on the app's origin, never a URL
  key: ({ input, ctx }) => `invoices/${ctx.actor?.orgId ?? 'none'}/${input.id}.pdf`,
  storage: () => disk('documents'),
  layout: { format: 'a4', marginMm: { top: 12, bottom: 12 }, printBackground: true },
  headers: ({ ctx }) => ({ authorization: `Bearer ${printToken(ctx.actor?.orgId)}` }),
});
Rule Refusal
the origin is APP_URL, read per call, or origin: () => … none → X_SCRAPE_PDF_ORIGIN_MISSING (500)
the route is a path on that origin else X_SCRAPE_PDF_ROUTE_INVALID (500)
the page's own status, read after load and document.fonts.ready 4xx/5xx → X_SCRAPE_PDF_ROUTE_FAILED (502; 4xx terminal, 5xx and 429 retryable)
the layout, checked at declaration X_SCRAPE_PDF_OPTIONS_INVALID
Bound Detail
browsers at once one Chrome per render, closed in a finally, at most DEFAULT_PDF_MAX_CONCURRENT (2) alive per process — chromePdfRenderer({ maxConcurrent })
a render with no slot waits up to queueTimeoutMs (default: the render's own timeout), then X_SCRAPE_PDF_BUSY (503, retryable)
what the page may load every request is paused on the page's session; one to the route's ORIGIN (scheme, host, port, compared whole) continues with headers added, every other is failed before it leaves — so the print token never reaches an <img src> a post carried, and a page cannot dial an internal host. A redirect off-origin is failed the same way
Chrome's sandbox ON: a print renders pages that carry user content. A container that cannot grant it fails the launch with X_CDP_LAUNCH_FAILED quoting Chrome's "No usable sandbox"; opt out in one visible line, chromePdfRenderer({ sandbox: false })

A test passes renderer: (any PdfRenderer); chromePdfRenderer({ launch }) takes any { connection, close }. The printing browser has no session of its own: a private route needs the app's short-lived token through headers (Known gaps).

The CDP launcher moved. findChrome, launchChrome, launchFoundChrome, cdpConnect, cdpConnectOver, the Cdp*Error classes and the CHROME_* / LAUNCH_* constants live in @ultimat3/scraping/cdp since 31.0.0 (its own entry, importing core only), and X_CDP_* are registered by scraping. It is their one import path: @ultimat3/testing no longer re-exports them, and E2E_BROWSER_CLOSE_MS is LAUNCHED_CLOSE_MS here (Upgrading #53).

A test launches through launchReady (As of 2026-11-15), from the same entry: launchChrome, then ready once the page it was handed EXISTS — heard as Target.targetCreated, never polled for, so a cold start (a Windows runner) is waited for rather than read early. The test takes launchTestMs(launchTimeoutMs) as its deadline — LAUNCH_TEST_MS at the default — so the launch's own X_CDP_LAUNCH_FAILED is what a slow browser reports, never Bun's timeout:

import { test } from 'bun:test';
import { findChrome, LAUNCH_TEST_MS, launchReady } from '@ultimat3/scraping/cdp';

const executable = await findChrome(process.env);

test.skipIf(executable === undefined)('the page the browser was handed', async () => {
  const browser = await launchReady({ executable: executable ?? '', timeoutMs: 20_000 });
  try {
    await browser.connection.send('Target.getTargets');
  } finally {
    await browser.close();
  }
}, LAUNCH_TEST_MS);

bun run scripts/launch-deadline.ts refuses a test that awaits a launch under a deadline of its own (X_TEST_LAUNCH_DEADLINE).

A social card for a row published at runtime is the same shape: page.screenshot() of the row's 1200 x 630 card page, disk().put(), then og: { image } — SEO covers the cards a static build takes itself. examples/dummy's runs/jobs/print-statement.ts is the worked example over a recording.

Clone this wiki locally