Update scrapy requirement from >=2.17.0 to >=2.18.0

[dependabot]

Updates the requirements on scrapy to permit the latest version.

Release notes

Sourced from scrapy's releases.

2.18.0

• HttpxDownloadHandler now uses httpx2
• The Twisted-based HTTP/2 download handler is no longer experimental
• brotli and Zstandard support are now always available, and optional extras cover the rest of the optional features
• Late Crawler attributes, such as stats, now raise RuntimeError instead of being None before the crawl starts
• Item exporters now export fields in declaration order
• New MetaCopyDetectionMiddleware
• New optimization page and built-in stats reference

Full Changelog

Changelog

Sourced from scrapy's changelog.

Scrapy 2.18.0 (2026-08-20)

Highlights:

• HttpxDownloadHandler now uses httpx2 <https://httpx2.pydantic.dev/>__

• The Twisted-based HTTP/2 download handler is no longer experimental

• brotli and Zstandard support are now always available, and :ref:optional extras <extras> cover the rest of the optional features

• Late :class:~scrapy.crawler.Crawler attributes, such as :attr:~scrapy.crawler.Crawler.stats, now raise :exc:RuntimeError instead of being None before the crawl starts

• Item exporters now export fields in declaration order

• New :class:~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware

• New :ref:optimization <optimize> page and :ref:built-in stats reference <topics-stats-reference>

Modified requirements

-   ``brotli`` (``brotlicffi`` on PyPy) and Zstandard support (the standard
    library :mod:`compression.zstd` module on Python 3.14 and higher, the
    ``backports.zstd`` package on earlier versions) are now required, so
    ``br`` and ``zstd`` are always included in the ``Accept-Encoding`` header
    of requests, and Brotli- and Zstandard-compressed responses are always
    decoded. Websites may now serve such responses to crawls that previously
    did not advertise support for them.
The minimum required versions are ``brotli`` 1.2.0, ``brotlicffi``
1.2.0.0 and ``backports.zstd`` 1.3.0.
(:gh:4698, :gh:6978, :gh:7083, :gh:7929, :gh:8009)




Increased the minimum versions of the following dependencies:


cryptography_: 37.0.0 → 41.0.5


pyOpenSSL_: 22.0.0 → 24.3.0


queuelib_: 1.4.2 → 1.6.1


service_identity_: 23.1.0 → 24.2.0


w3lib_: 1.17.0 → 2.1.1

</tr></table>

... (truncated)

Commits

• 2216063 Bump version: 2.17.0 → 2.18.0
• 713360c Release notes for Scrapy 2.18 (#7969)
• ac738e0 Let parsel determine the type of nested selectors (#8040)
• 9da1a32 CI: Fix concurrency group collision between merged-PR and master-push runs (#...
• 5938dde Warn that XMLFeedSpider's html iterator can mangle void-like tags (#8045)
• bbb5a26 Document where the project name appears in a Scrapy project (#8044)
• 4f0b5fa Add a beginner's guide to Scrapy's main concepts (#8025)
• ab27dfd Document deriving the job directory from the spider name (#8035)
• 09a5d81 CI: Cancel in-progress workflow runs when their pull request is closed (#8029)
• d2ec88f Warn against using keep_fragments for JS-rendered fragment content (#8033)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

---

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🔍 Scrapy 2.18 adds required transitive deps

Scrapy 2.18.0 makes brotli and Zstandard support mandatory and raises minimum versions of cryptography, pyOpenSSL, queuelib, service_identity, and w3lib. The bump changes transitive resolution, so a fresh install and the scraper run warrant a check before merge.

---

Was this helpful? React with 👍 or 👎 to provide feedback.