feat(renderer): add opt-in Camoufox stealth renderer tier#149
Open
us wants to merge 1 commit into
Open
Conversation
Add Camoufox as an optional stealth renderer tier (REST, not CDP), reached only when explicitly opted into. It does NOT join the default auto failover ladder unless requested, so existing deployments are unaffected. Backed by a camofox-browser REST sidecar. Three explicit opt-in paths (Option C): - per-request `renderer: "camoufox"` pin - config `mode = "camoufox"` (pins every request to it) - `[renderer.camoufox] include_in_auto = true` (joins the auto chain) A configured endpoint with the default `include_in_auto = false` is constructed (so a per-request pin can reach it) but stays OUT of the auto chain — zero behavior change for existing auto-mode deployments. Gated behind a default-off `camoufox` cargo feature threaded through crw-core -> crw-renderer -> crw-server -> crw-cli, so the lean build is byte-identical. Camoufox is REST: never counted as a CDP tier, never charged CDP overhead, and gets its own deadline-budget branch outside the cdp gate. The renderer drives the camofox-browser openapi contract: create tab -> evaluate document.documentElement.outerHTML -> always destroy session (even on error). Bot-wall/empty-HTML detection surfaces a retryable error; fresh per-request session avoids cookie carry-over. Adds RendererMode::Camoufox, RequestedRenderer::Camoufox, RendererKind::Camoufox (cost 3), CamoufoxEndpoint config, MCP schema entry, and full unit + opt-in-semantics test coverage.
a8f8f77 to
e53519e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Camoufox as an opt-in stealth renderer tier for fingerprint/bot-challenge–blocked targets (e.g. Cloudflare 403) that the existing CDP tiers can't pass. It is reached only when explicitly requested — it does not join the default
autofailover ladder unless you turn it on. Existing deployments are unaffected.This is the "Option C" design from the earlier feasibility evaluation: supported + selectable + not-in-default-auto + breaks nothing.
How a user opts in
Three explicit paths (in increasing scope):
renderer: "camoufox"on a scrape/crawl request.mode = "camoufox"pins every request to it.[renderer.camoufox] include_in_auto = trueadds it as the last-resort tier in theautochain.A configured endpoint with the default
include_in_auto = falseis constructed (so a per-request pin can reach it) but stays out of the auto chain — so simply configuring an endpoint changes nothing for existingauto-mode traffic.Breaks nothing
camoufoxcargo feature threadedcrw-core → crw-renderer → crw-server → crw-cli. The lean build is byte-identical;cargo buildwith no extra features pulls no new deps.cdp_tier_count, never chargedCDP_TIER_OVERHEAD_MS, and gets its own deadline-budget branch outside the cdp gate (so a camoufox-only deployment isn't deadline-starved).RendererMode::Camoufoxis excluded from thecfg(not(feature = "cdp"))rejection (it needs no CDP), and rejected cleanly when built without thecamoufoxfeature.camoufoxunconditionally (likeplaywright).Implementation
crw-core):RendererMode::Camoufox,CamoufoxEndpoint { base_url, api_key, include_in_auto },camoufox_timeout(), and a single-source-of-truthcamoufox_in_ladder()predicate driving the opt-in policy + deadline math.crw-core/crw-renderer):RendererKind::Camoufox+RequestedRenderer::Camoufox(both →"camoufox"),credit_for = 3,tier_timeouts, and a 5thBreakerRegistryentry.crw-renderer/src/camoufox.rs): areqwest-basedPageFetcherdriving the camofox-browser REST contract — create tab →evaluate(document.documentElement.outerHTML)→ always destroy session (even on error). Per-request fresh session (no cookie carry-over), missing-tabIdretry-once, bot-wall/empty-HTML → retryable error, deadline-clamped per call, health-probeis_available().crw-renderer): independent construction block (not nested in the cdp guard); auto-exclusion applied at request time infetch_with_jswhile pins bypass it.camoufox;validate_renderer_pinaccepts it generically; documented inconfig.default.toml.REST contract
Verified against the camofox-browser
openapi.json:POST /tabs{userId, sessionKey, url}→{tabId}POST /tabs/{tabId}/evaluate{userId, expression}→{ok, result}DELETE /sessions/{userId}GET /healthTests & verification
is_available), opt-in construction/ladder tests, configcamoufox_in_laddertruth-table + camoufox-only deadline test, MCP schema + breaker tests.camoufox,cdp,camoufox, and lean builds all compile and test green;clippy+fmtclean.Operating note
Requires a build with
--features camoufoxand a running camofox-browser sidecar. The tier auto-skips (viais_available()health probe) when the sidecar is unreachable, so it degrades gracefully rather than erroring every request.