Skip to content

feat(renderer): add opt-in Camoufox stealth renderer tier#149

Open
us wants to merge 1 commit into
mainfrom
feat/camoufox-optin-renderer
Open

feat(renderer): add opt-in Camoufox stealth renderer tier#149
us wants to merge 1 commit into
mainfrom
feat/camoufox-optin-renderer

Conversation

@us

@us us commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Summary

Adds Camoufox as an opt-in stealth renderer tier for fingerprint/bot-challenge–blocked targets (e.g. Cloudflare 403) that the existing CDP tiers can't pass. It is reached only when explicitly requested — it does not join the default auto failover ladder unless you turn it on. Existing deployments are unaffected.

This is the "Option C" design from the earlier feasibility evaluation: supported + selectable + not-in-default-auto + breaks nothing.

How a user opts in

Three explicit paths (in increasing scope):

  1. Per-request pinrenderer: "camoufox" on a scrape/crawl request.
  2. Config pinmode = "camoufox" pins every request to it.
  3. Auto-chain opt-in[renderer.camoufox] include_in_auto = true adds it as the last-resort tier in the auto chain.
[renderer.camoufox]
base_url = "http://127.0.0.1:9377"
# api_key = ""               # optional bearer token
# include_in_auto = false    # default: stay OUT of the auto ladder
# camoufox_timeout_ms = 60000

A configured endpoint with the default include_in_auto = false is constructed (so a per-request pin can reach it) but stays out of the auto chain — so simply configuring an endpoint changes nothing for existing auto-mode traffic.

Breaks nothing

  • Default-off camoufox cargo feature threaded crw-core → crw-renderer → crw-server → crw-cli. The lean build is byte-identical; cargo build with no extra features pulls no new deps.
  • REST, not CDP. Camoufox is never counted in cdp_tier_count, never charged CDP_TIER_OVERHEAD_MS, and gets its own deadline-budget branch outside the cdp gate (so a camoufox-only deployment isn't deadline-starved).
  • RendererMode::Camoufox is excluded from the cfg(not(feature = "cdp")) rejection (it needs no CDP), and rejected cleanly when built without the camoufox feature.
  • The only changed existing assertions are two MCP-schema renderer-enum counts (4 → 5), correct because the schema advertises camoufox unconditionally (like playwright).

Implementation

  • Config (crw-core): RendererMode::Camoufox, CamoufoxEndpoint { base_url, api_key, include_in_auto }, camoufox_timeout(), and a single-source-of-truth camoufox_in_ladder() predicate driving the opt-in policy + deadline math.
  • Kinds/metrics (crw-core/crw-renderer): RendererKind::Camoufox + RequestedRenderer::Camoufox (both → "camoufox"), credit_for = 3, tier_timeouts, and a 5th BreakerRegistry entry.
  • Renderer (crw-renderer/src/camoufox.rs): a reqwest-based PageFetcher driving the camofox-browser REST contract — create tab → evaluate(document.documentElement.outerHTML)always destroy session (even on error). Per-request fresh session (no cookie carry-over), missing-tabId retry-once, bot-wall/empty-HTML → retryable error, deadline-clamped per call, health-probe is_available().
  • Wiring (crw-renderer): independent construction block (not nested in the cdp guard); auto-exclusion applied at request time in fetch_with_js while pins bypass it.
  • Surface: MCP tool schema advertises camoufox; validate_renderer_pin accepts it generically; documented in config.default.toml.

REST contract

Verified against the camofox-browser openapi.json:

Call Endpoint
create tab (navigates) POST /tabs {userId, sessionKey, url}{tabId}
get DOM POST /tabs/{tabId}/evaluate {userId, expression}{ok, result}
cleanup DELETE /sessions/{userId}
health GET /health

Tests & verification

  • New: 12 renderer unit tests (happy path + cleanup-on-error + missing-tabId retry + wall/empty/ok=false/non-string-result + deadline short-circuit + cleanup-timeout isolation + is_available), opt-in construction/ladder tests, config camoufox_in_ladder truth-table + camoufox-only deadline test, MCP schema + breaker tests.
  • Gate: lean workspace 1044 tests pass / 0 fail; camoufox, cdp,camoufox, and lean builds all compile and test green; clippy + fmt clean.

Operating note

Requires a build with --features camoufox and a running camofox-browser sidecar. The tier auto-skips (via is_available() health probe) when the sidecar is unreachable, so it degrades gracefully rather than erroring every request.

Add Camoufox as an optional stealth renderer tier (REST, not CDP),
reached only when explicitly opted into. It does NOT join the default
auto failover ladder unless requested, so existing deployments are
unaffected. Backed by a camofox-browser REST sidecar.

Three explicit opt-in paths (Option C):
- per-request `renderer: "camoufox"` pin
- config `mode = "camoufox"` (pins every request to it)
- `[renderer.camoufox] include_in_auto = true` (joins the auto chain)

A configured endpoint with the default `include_in_auto = false` is
constructed (so a per-request pin can reach it) but stays OUT of the
auto chain — zero behavior change for existing auto-mode deployments.

Gated behind a default-off `camoufox` cargo feature threaded through
crw-core -> crw-renderer -> crw-server -> crw-cli, so the lean build is
byte-identical. Camoufox is REST: never counted as a CDP tier, never
charged CDP overhead, and gets its own deadline-budget branch outside
the cdp gate.

The renderer drives the camofox-browser openapi contract: create tab ->
evaluate document.documentElement.outerHTML -> always destroy session
(even on error). Bot-wall/empty-HTML detection surfaces a retryable
error; fresh per-request session avoids cookie carry-over.

Adds RendererMode::Camoufox, RequestedRenderer::Camoufox,
RendererKind::Camoufox (cost 3), CamoufoxEndpoint config, MCP schema
entry, and full unit + opt-in-semantics test coverage.
@us us force-pushed the feat/camoufox-optin-renderer branch from a8f8f77 to e53519e Compare June 15, 2026 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant