Repository navigation
[pull] main from firecrawl:main - #395
Merged
Merged
Conversation
…aging (#4417) * feat(pdf): prefer fire-pdf's live remaining estimate in timeout messaging Phase 2 consumer side: 202 polls may now carry estimated_remaining_ms (computed by fire-pdf from measured lane throughput + real queue depth); the poll observer records the latest one with its observation time, and composeTimeoutProcessing prefers it — aged since observation, floored at a minute — over the static per-page formula, which remains the fallback for older fire-pdf builds and lanes without measured throughput. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pdf): atomic server-estimate snapshot; harden the wire field Local cubic round 1: estimated_remaining_ms is now finite+nonnegative with .catch(undefined) so a malformed value degrades to no-estimate instead of failing the poll; and the estimate travels as one typed {remainingMs, observedAtMs} snapshot across Meta, the poll observer, and the timeout composer — no half-populated states. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… functions (#4418) * fix(pdf): enforce the per-team cap inside both by-reference placement functions The rewrite (server-side GCS copy of a fire-engine handoff) and the streaming upload trusted their callers to have applied largePdfLimitBytes; only the router's gates enforced it. Both now refuse over-cap inputs themselves and return null, so an oversized file follows the pre-by-reference oversized path regardless of which caller asked. * test(pdf): pin cap configs explicitly and clean up the upload temp file Addresses cubic review: the cap-enforcement block now sets both tier caps itself instead of relying on schema defaults (and derives the test sizes from them), and the upload-accept test unlinks its temp file.
* feat(sdk): auto-resume large-document scrapes that keep processing server-side
Phase 3 of the large-PDF retry story, JS + Python SDKs (sync and
async). When /v2/scrape times out with
details.state=processing_continues (the document outlives the request
window by design and keeps processing), the SDK sleeps the advertised
retryAfterSeconds (payload first, Retry-After header fallback, clamped
5s-10min) and re-issues the request — which attaches to the in-flight
job via content adoption and returns the finished result. From the
caller's code, a large document becomes one slow successful call.
Strictly bounded and conservative: resumes only on the explicit
processing_continues signal (plain timeouts are untouched), at most 5
resumes and 20 minutes of total waiting, opt-out via
autoResume/auto_resume=false. The option is SDK-only and stripped from
the wire payload.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(sdk): scrape-call-specific autoResume; shared python resume policy
Local cubic round 1: ScrapeOptions is reused inside batch/crawl/search/
extract payloads, so the SDK-only option moves off the shared type —
JS gets ScrapeCallOptions (intersection, scrape-only), Python takes
auto_resume as a keyword argument on scrape()/client.scrape() and never
touches the pydantic wire model. The resume policy (bounds + signal
parsing) moves to a neutral utils/auto_resume module shared by the sync
and async transports.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(sdk): shared ResumeTracker; explicit async auto_resume parameter
Local cubic round 2: the full stateful resume decision (signal, bounds,
accounting, opt-out) lives once in utils/auto_resume.ResumeTracker and
both transports consume it; the async client exposes auto_resume as a
typed keyword parameter instead of popping it from **kwargs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(sdk): harden retryAfterSeconds parsing; client timeout outlasts the server wall
Cloud cubic round 1: Python rejects boolean/non-finite retryAfterSeconds
(bool is an int subclass) and falls back header→60s; the JS client-side
timeout now always exceeds the server's own wall (options.timeout or the
API's 5-min default, +30s) so axios can't abort in a photo-finish and
lose the processing_continues signal; the three pytest.raises assertions
pin FirecrawlError instead of Exception.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(sdk): resume-gated timeout cushion respects client config; overflow guard; wait-bound test
Cloud cubic round 2: autoResume:false restores the exact pre-existing
request timing (explicit timeout +5s, else the client default); with
resume on, the cushion floors at the configured client timeout (new
HttpClient.getTimeoutMs) so FirecrawlClient({timeoutMs}) is never
overridden downward. Python catches OverflowError on absurd JSON
integers, and the 20-minute total-wait bound gets its own test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(js-sdk): esbuild audit override for GHSA-g7r4-m6w7-qqqr in the firecrawl subpackage
The advisory published mid-review (audit passed at 20:30Z, failed at
20:48Z with no dependency changes) and blocks any js-sdk-touching PR.
Mirrors the override apps/js-sdk/package.json already carries; audit-ci
verified passing locally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(sdk): bump js-sdk to 4.37.0, python-sdk to 4.40.0
Minor bumps for the auto-resume feature so the merge publishes both
packages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Revert "fix(js-sdk): esbuild audit override for GHSA-g7r4-m6w7-qqqr in the firecrawl subpackage"
This reverts commit dc0b427.
* fix(js-sdk): complete the audit-override revert
The subpackage is its own pnpm workspace and pnpm 10 stores overrides in
pnpm-workspace.yaml — the earlier revert only cleaned package.json, so
the esbuild pin still shipped via the workspace file and lockfile. Both
now match main; the pin lives solely in the standalone audit PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(js-sdk): drop stray npm lockfile; dep files byte-identical to main
package-lock.json was an npm-install artifact swept in by a broad git
add (the package is pnpm-managed); pnpm-workspace.yaml and
pnpm-lock.yaml are restored to exactly main's state — this branch
changes no dependencies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ci: use trusted publishing for JS SDK * ci: keep pnpm for JS SDK publishing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )