refactor!: Continued public surface minimization

CONTRIBUTING.md

@@ -63,6 +63,19 @@ yay -S libffi7 icu66 libwebp052 flite-unpatched
 sudo ln -s /usr/lib/libpcre.so /usr/lib/libpcre.so.3
 ```
 
+## Public API reports
+
+`docs/public-api/*.api.md` map what each package promises not to break (see [its README](docs/public-api/README.md)). After changing any package's public surface, regenerate and commit them:
+
+```sh
+pnpm build        # reports are generated from dist/
+pnpm api:extract
+```
+
+CI runs `pnpm api:check`, which fails when a committed report is out of date. Either the surface change is intended (commit the updated report so reviewers see the diff) or it is accidental (fix it).
+
+It also fails when an untagged signature references an `@internal` type, since the report would then use a type it never declares. Regenerating won't fix that: either drop the tag from the referenced type or keep it out of the public signature.
+
 ## Testing in Crawlee with vitest
 
 There are a few small differences between how testing in jest and vitest works. Mostly, they relate to what to do, and not do anymore.

docs/examples/puppeteer_capture_screenshot.mdx

@@ -36,7 +36,7 @@ Using `page.screenshot()`:
 
 <TabItem value="crawlerutilsscreenshot" label="Crawler Utils Screenshot" default>
 
-Using `utils.puppeteer.saveSnapshot()`:
+Using <ApiLink to="puppeteer-crawler/namespace/puppeteerUtils#saveSnapshot">`puppeteerUtils.saveSnapshot()`</ApiLink>:
 
 <RunnableCodeBlock className="language-js" type="puppeteer">
 	{PuppeteerCrawlerUtilsSnapshotSource}

docs/examples/puppeteer_crawler_utils_snapshot.ts

@@ -1,4 +1,4 @@
-import { launchPuppeteer, utils } from 'crawlee';
+import { launchPuppeteer, puppeteerUtils } from 'crawlee';
 
 const url = 'http://www.example.com/';
 // Start a browser
@@ -11,7 +11,7 @@ const page = await browser.newPage();
 await page.goto(url);
 
 // Capture the screenshot
-await utils.puppeteer.saveSnapshot(page, { key: 'my-key', saveHtml: false });
+await puppeteerUtils.saveSnapshot(page, { key: 'my-key', saveHtml: false });
 
 // Close Puppeteer
 await browser.close();

docs/guides/configuration.mdx

@@ -105,8 +105,8 @@ are launched in headful mode, i.e. with windows.
 
 Specifies the minimum log level, which can be one of the following values (in order of severity):
 `DEBUG`, `INFO`, `WARNING`, `ERROR` and `OFF`. By default, the log level is set to `INFO`,
-which means that `DEBUG` messages are not printed to console. See the <ApiLink to="core/class/Log">`utils.log`</ApiLink>
-namespace for logging utilities.
+which means that `DEBUG` messages are not printed to console. See the <ApiLink to="core/class/Log">`log`</ApiLink>
+instance for logging utilities.
 
 #### `CRAWLEE_VERBOSE_LOG`
 

docs/guides/public_api.mdx

@@ -0,0 +1,58 @@
+---
+id: public-api
+title: Public API
+description: What Crawlee promises not to break, and what it doesn't
+---
+
+Crawlee ships its type definitions, and those definitions contain more than its supported API.
+Some of what you can import is a deliberate promise; some of it is machinery that happens to be
+reachable. This page explains how to tell, so you can decide what to depend on.
+
+## Supported by default
+
+Anything Crawlee exports is supported unless it says otherwise. If you can import it and its
+documentation does not mark it as internal, it is covered by backwards compatibility: it will
+not change shape or disappear outside a major release, and if it ever does, the change is
+recorded in the upgrading guide.
+
+## Marked internal
+
+Some members carry an `@internal` tag in their documentation comment. Your editor shows it when
+you hover the symbol, and it means exactly one thing: **we do not promise anything about it.**
+It can change signature, behaviour or disappear entirely in any release, including a patch, and
+it will not appear in the upgrading guide when it does.
+
+It is still exported, still typed, and auto-completion still works. That is on purpose. We
+would rather leave you a way to unblock yourself — knowingly — than take it away and have you
+patch the package or give up. So the tag is a statement about support, not about access:
+
+```ts
+// Fine. Supported, and it will keep working.
+import { CheerioCrawler, Dataset } from 'crawlee';
+
+// Allowed, but you are on your own. Pin your Crawlee version
+// and expect to revisit this on every upgrade.
+import { someInternalHelper } from 'crawlee';
+```
+
+If you find yourself reaching for an internal member to get something done, that is worth
+telling us about, so you should [open an issue](https://github.com/apify/crawlee/issues). Those reports are
+what we use to decide which extension points deserve a real, supported API.
+
+## Things that are not types
+
+Not every promise is expressible in TypeScript, and a few things are contracts even though
+nothing checks them:
+
+- **Persisted state.** The layout of what Crawlee writes into a key-value store or request
+  queue is an implementation detail. Read it for debugging, do not build on it.
+- **Log message text.** Messages change freely; never match on them.
+- **Subclass hook ordering.** When you override a documented extension point, call `super`
+  where the base class expects it. Skipping it usually compiles and then misbehaves at runtime.
+
+## Where to check
+
+For anything beyond the obvious, the per-package surface maps under
+[`docs/public-api/`](https://github.com/apify/crawlee/tree/master/docs/public-api) in the
+repository are the authoritative inventory of what we promise. If a symbol is in there, it is
+supported; if not, it is not.

docs/guides/session_management.mdx

@@ -249,8 +249,11 @@ The `await using` syntax needs Node.js 24 or later. On Node.js 22 call <ApiLink
 
 A crawler accepts any object implementing the <ApiLink to="types/interface/ISessionPool">`ISessionPool`</ApiLink> interface as its `sessionPool` option, not just the built-in `SessionPool`. The contract is intentionally tiny — a single `getSession()` / `getSession(id)` method that hands out an <ApiLink to="types/interface/ISession">`ISession`</ApiLink> for a request. This lets you plug in a remote, shared, or database-backed session strategy without subclassing `SessionPool` or copying its internals.
 
+`ISessionPool` and `ISession` are owned by `@crawlee/types`, which the `crawlee` meta-package does not re-export. Add it to your dependencies to import them.
+
 ```ts
-import { BasicCrawler, Session, type ISessionPool } from 'crawlee';
+import { BasicCrawler, Session } from 'crawlee';
+import type { ISessionPool } from '@crawlee/types';
 
 class MySessionPool implements ISessionPool {
     private readonly sessions = new Map<string, Session>();

docs/public-api/README.md

@@ -1,83 +1,15 @@
 # Public API surface maps
 
-Each `*.api.md` file in this folder is a generated **map of the public, type-level
-interface** of one publishable `@crawlee/*` package — every exported class, method,
-property, function, and type, with full signatures. These reports define **where we
-promise backwards compatibility**.
+Each `*.api.md` file here is a generated map of the public, type-level interface of one `@crawlee/*` package. Being in a report is the backwards-compatibility promise; being absent means no promise, not that the symbol is unreachable. See the [Public API guide](../guides/public_api.mdx) for what that means for users.
 
-They are produced by [API Extractor](https://api-extractor.com/) from the built
-`dist/index.d.ts` of each package.
+- **Untagged** members are promised. The codebase does not use explicit `@public` tags.
+- **`@internal`** (and the legacy `@ignore`) members are trimmed from the report but stay exported and present in the `.d.ts`. Use `#private` / `private` when something should be unreachable, `@internal` when it should be reachable but unsupported.
+- **`@private` is inert**: it does not remove anything from the report.
 
-## Workflow
+The reports are generated by [`apify/api-extractor-report`](https://github.com/apify/api-extractor-report); its [README](https://github.com/apify/api-extractor-report#readme) explains how they are produced and what it fixes up in API Extractor's output. Regenerating and CI checks are covered in [CONTRIBUTING.md](https://github.com/apify/crawlee/blob/master/CONTRIBUTING.md#public-api-reports).
 
-- After changing any package's public surface, regenerate the reports and commit them:
+## Reading a report
 
-  ```sh
-  pnpm build        # the reports are generated from dist/
-  pnpm api:extract
-  ```
-
-- CI runs `pnpm api:check`, which fails if a committed report is out of date. A failing
-  check means you changed the public API: either that change is intentional (commit the
-  updated report — reviewers will see the surface diff) or it was accidental (fix it).
-
-- `api:check` also fails if a report ends up referencing a symbol it never declares, which
-  leaves the committed map describing a type nothing in it defines. Regenerating cannot fix
-  that; it has to be fixed in the source. In practice it means a `@public` symbol's signature
-  references an `@internal`/`@ignore`-d one, so the referenced type is trimmed out from under
-  it. Either drop the referenced type's tag (it is reachable from the public API, so users can
-  already depend on it) or keep it out of the public signature. An untagged symbol is
-  implicitly public, which is the convention here — the codebase does not use explicit
-  `@public` tags.
-
-  A symbol that is merely missing from the package's exports does **not** need fixing: see the
-  note on forgotten exports below.
-
-## Notes
-
-- The reports are generated as API Extractor's **`public`** variant, so symbols tagged
-  `@internal` (`@alpha`/`@beta` too) are excluded — only `@public` surface is tracked.
-  The legacy `@ignore` tag counts as `@internal` here; the generator rewrites it before
-  extraction, so an `@ignore`-d symbol is excluded too and cannot be referenced from a
-  `@public` signature.
-  The generator stages the variant as `<name>.public.api.md` under `temp/` and promotes it
-  onto the committed `<name>.api.md`, so the tracked filenames stay stable.
-- API Extractor builds the import list before it trims the non-`@public` declarations and
-  never revisits it, so a type reachable only from an `@internal` member would linger as a
-  bare import and read as public surface. There is no config option for this, so the
-  generator post-processes each report: it parses the fenced TypeScript and drops imports
-  whose binding is referenced by no declaration that survived the trim.
-- **Forgotten exports** — types the public API references but the entry point never exports —
-  are included in the report via `includeForgottenExports` and carry an explicit banner:
-
-  ```ts
-  // Not exported by the entry point; reachable only as a referenced type.
-  // @public (undocumented)
-  interface SitemapUrlData {
-  ```
-
-  Their *shape* is part of the surface we promise not to break, but their *name* is not
-  importable, so they are emitted without `export`. API Extractor labels them `@public
-  (undocumented)` like anything else, which is indistinguishable from a real export at a
-  glance, hence the added banner. The alternative was exporting every such type from its
-  package — ~38 new public exports, committing us to names we never meant to publish. If you
-  *want* one importable, export it deliberately and the report will show it with `export`.
-- Because API Extractor decides both of the above before the `@public` trim, it also offers
-  declarations for symbols reachable only from members that never reach the report. The
-  generator drops those the same way it drops dead imports, so the report carries nothing it
-  does not refer to. Only symbols flagged `ae-forgotten-export` are eligible, which is what
-  keeps genuinely reachable declarations (e.g. the `social` namespace in `@crawlee/utils`,
-  whose members are exposed through a `declare namespace` block) from being pruned.
-- `docs/public-api/temp/` holds intermediate reports (including the staged `.public.api.md`
-  files) and is git-ignored.
-- `@crawlee/cli` and `@crawlee/templates` are deliberately excluded — they are tooling
-  (a CLI binary and project scaffolding), not an importable API where we promise BC. The
-  exclude list lives in `scripts/api-extractor/run.ts`.
-- The generator lives in `scripts/api-extractor/`. It temporarily strips the build's
-  injected `// @ts-ignore` comment lines from the `.d.ts` files (restoring them
-  afterwards) because API Extractor's AST walker trips over some of them; a small number
-  of packages additionally need a sanitized-mirror fallback. See the comments in
-  `scripts/api-extractor/run.ts` for details.
-- These reports now cover only the `@public` surface. Further shrinking them — genuinely
-  hiding class internals (untagged `protected`/`_`-prefixed members) rather than merely
-  tagging them — is the goal tracked in issue #3109.
+- **Forgotten exports** are types the public API references but the entry point does not export. They appear without `export` under a `// Not exported by the entry point` banner: their shape is promised, their name is not importable. Export one deliberately if it should be.
+- **`crawlee.api.md` is nearly empty on purpose.** The meta-package only has `export *` lines, and everything they re-export is inventoried in the constituent package's report.
+- `@crawlee/cli` and `@crawlee/templates` are excluded: they are tooling, not an importable API.