Skip to content

Decode CSS escapes in a single pass - #203

Merged
wRAR merged 1 commit into
scrapy:masterfrom
itecz26:css-escape-single-pass
Oct 16, 2026
Merged

wRAR merged 1 commit into
scrapy:masterfrom
itecz26:css-escape-single-pass

Conversation

@itecz26

@itecz26 itecz26 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

CSS escapes are decoded more than once

unescape_ident() and the string branch of tokenize() decode escapes in separate re.sub() passes (unicode escapes, then simple escapes, plus escaped newlines first for strings). Each pass rescans what the previous one produced, so escape boundaries are lost:

>>> GenericTranslator().css_to_xpath(r"*[x='\\31 ']")
"descendant-or-self::*[@x = '1']"        # expected '\31 '
>>> GenericTranslator().css_to_xpath(r"*[x='\5c 41']")
"descendant-or-self::*[@x = '41']"       # expected '\41'
>>> GenericTranslator().css_to_xpath(r"a\\31")
'descendant-or-self::a1'                 # expected name() = 'a\31'

CSS Syntax §4.3.7 consumes each escape exactly once, left to right, so \\ is a literal backslash and the result of \5c is never read as the start of another escape. Here a selector can end up matching a different value or element name than the one it spells, which is the usual double-decoding problem for anything that inspects or builds selectors before translating them. Decoding every escape with one combined pattern in a single pass fixes both the identifier and the string path; valid input without a backslash next to an escape is unaffected and the existing tests pass unchanged.

unescape_ident() and the string tokenizer decoded escapes with several
re.sub() passes, so each pass rescanned the output of the previous one:
an escaped backslash followed by hex digits (\\31) was decoded again
as a unicode escape, and a backslash produced by \5c started a new
simple escape. Decode every escape in one left-to-right pass instead.
@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.79%. Comparing base (91f39df) to head (c3c52ac).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #203   +/-   ##
=======================================
  Coverage   99.79%   99.79%           
=======================================
  Files           3        3           
  Lines         987      987           
  Branches      161      162    +1     
=======================================
  Hits          985      985           
  Misses          1        1           
  Partials        1        1           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wRAR
wRAR merged commit ec61e4b into scrapy:master Oct 16, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants