Conversation
wappalyzer's display names often carry a vendor prefix that the CPE dictionary's bare product name omits (e.g. 'Apache Tomcat' vs CPE product 'tomcat'), or the reverse. The exact-match lookup in lookupTechVersion can never bridge that gap, so the CPE version field stays '*' even when the version is known. Add a length-guarded substring fallback: if no exact key matches, check whether a CPE product's lookup key is a substring of (or contains) a detected technology's normalized name, requiring both sides to be at least 5 chars to avoid false positives on short generic keys. Fixes #2550
CodeRabbit flagged two real issues in the substring-fallback approach: 1. False positives - 'react' is a character-for-character substring of 'preact', so the two would be wrongly conflated. 2. Nondeterminism - iterating a Go map in the substring fallback meant an ambiguous match could resolve differently across runs. Replace substring containment with whole-word token matching: technologyTokens() splits names on non-alphanumeric boundaries instead of concatenating them, so 'react' and 'preact' never share a token. buildTechVersionTokenIndex() builds the token->version map once and drops any token that maps to conflicting versions, the same ambiguity-safe pattern buildTechVersionMap already uses for exact names - so results no longer depend on map iteration order. Also excludes ~25 generic words (framework, suite, commerce, etc.) that are too common across unrelated products to trust as a sole match, and raises the minimum token length from 4 to 5. Verified against the real wappalyzergo + awesome-search-queries datasets: still recovers 58 of 91 previously-broken shared-CPE pairs, with zero known false-positive collisions.
…tching fix(cpe): substring fallback for vendor-prefix mismatches (#2550)
* fix: retry over https when a plaintext probe is rejected for missing TLS Ports above 1024 are probed as plain HTTP first, and the scheme retry only fires on a transport error. A TLS listener answers a plaintext request with a perfectly valid HTTP 400 saying TLS is required, so err == nil, the retry never happens, and a TLS-only service is reported as plain http with a 400 and no title or technologies. Detect that response and reuse the existing scheme retry. Only the distinctive server phrasings count (nginx, Netty, Apache, HAProxy); a bare "400 Bad Request" is a legitimate HTTP answer and is left alone, so the extra request is limited to targets that already told us to use TLS. Downstream this mattered: consumers that persist the probed scheme were recording TLS-only ports as http assets, and every HTTP-based scan of those targets then ran over plaintext and matched nothing. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * refactor: fold the TLS retry test into runner_test.go Tests for runner.go belong in runner_test.go; a per-scenario file drifts away from the code it covers. Also cut the four-line preamble on the retry down to the one fact that is not already on the next line. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * fix(test): check the error from server.Serve errcheck flagged the unchecked Serve in the TLS-only test listener. It always returns io.EOF there, since the listener yields a single connection, so discard it explicitly. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * refactor: let the TLS handshake decide the scheme, not the error text Matching the server's rejection wording only worked for the phrasings we had seen; nginx, Netty, Apache and HAProxy each word it differently and a titleless 400 matched nothing. Trigger the upgrade on the shape of the exchange instead — a plaintext probe answered 400 — and let the TLS handshake settle it: if TLS works the port is https, and if it does not the existing scheme fallback recovers the original http result. This also stops the upgrade consuming the single retry budget, so a target whose https attempt fails is no longer left without a result. Drops respondsOnlyOverTLS and its signal list. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * docs: cut the TLS upgrade comment to the fact that is not in the code Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * fix(test): read the request before the listener answers TestPlainHTTPPortStaysHTTP wrote its 400 on accept, before the client had finished sending. Go discards a reply that arrives on a channel it has not spoken on ("unsolicited response"), so the result went missing and the assertion saw zero results. It passed locally on timing luck and failed on all three CI runners. Read the request head first, with a deadline so a silent client cannot park the goroutine. The TLS listener's plaintext branch had the same dependency and is fixed alongside. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * fix: do not attempt the TLS upgrade in unsafe mode Unsafe mode bypasses the scheme fallback, so the upgrade had nothing to fall back to: a plain HTTP service answering 400 lost its result entirely rather than being reported as http. The rfc-path integration tests cover exactly that shape and caught it. Verified locally: `-unsafe` against a plain HTTP 400 went from 0 results back to 1, and both integration tests pass. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * chore: restore ports_optimization_test.go to dev Removing the string-matching test left a trailing blank-line diff, which is noise in review. The helper it covered is gone with the phrase list, so both ports_optimization files are unchanged from dev now. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * fix: keep the plaintext response until the HTTPS attempt succeeds Review found that the upgrade discarded a successful HTTP 400 before trying HTTPS, and the error path then issued a fresh HTTP request rather than restoring it. A transient, one-shot or rate-limited service could answer once, fail the HTTPS attempt, fail the repeat request, and vanish from the output despite having been reachable. Hold the plaintext response and restore it on any HTTPS failure, so no second HTTP request is made and nothing is lost. An earlier revision gated the upgrade on a TLS handshake preflight. That is dropped: the HTTPS request opens with the same handshake, so the preflight only duplicated it on the success path, and dialling outside the client bypassed transport.Proxy and CONNECT while reimplementing CustomIP and TLS impersonation. Going through the normal client path inherits all of it. Tests: a cleartext service that answers 400 exactly once and refuses afterwards, and one whose TLS handshake succeeds before it closes without an HTTP response. Both must still be reported as http. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * fix: restore the URL alongside the retained plaintext response Review found that the restore left URL as the object built for the failed HTTPS attempt, so SupportHTTP2 received protocol "http" with an https URL, and the stored-response filename hashed the https form for a result reported as http. URL is cloned before the upgrade and restored with the response; the comment claiming resp, req and protocol were the whole of the downstream state was wrong and is corrected. Adds the handshake-success-then-close test that the previous message claimed was present and was not: TLS completes, the connection closes without an HTTP response, and the cleartext service answers only once. Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c * test: cover restored scheme in HTTP/2 probe * fix: address PR review comments
* chore: require a tlsx version that exists on GitHub
tlsx v1.3.1 and v1.3.2 are cached in the Go module proxy but their tags
were deleted from GitHub, so anyone whose GOPRIVATE or GONOSUMDB routes
projectdiscovery modules around the proxy cannot resolve them:
go: github.com/projectdiscovery/tlsx@v1.3.2: invalid version:
unknown revision v1.3.2
v1.3.0 is a live tag whose commit is a descendant of the one v1.3.2
pointed at, so this loses nothing - the two share an identical go.mod
hash.
Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH
Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c
* chore: drop the stale tlsx v1.3.2 go.sum entries
go.sum still carried hashes for the version that was replaced, which
left the module graph inconsistent and failed lint with "no go files to
analyze: running `go mod tidy` may solve the problem".
Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH
Claude-Local-Session: claude --resume 0fe5b680-5ba5-43e6-b354-43d4388aaf9c
* chore: require tlsx v1.4.0
v1.4.0 is the first tlsx release whose GitHub tag and module-proxy record
agree, so it resolves on both fetch paths. It supersedes v1.3.0 (tag moved
after publication) and v1.3.1/v1.3.2 (tags deleted), all of which left dev
unbuildable for anyone whose GOPRIVATE bypasses the proxy.
Claude-Session: https://claude.ai/code/session_015AKXsNzjd3HjGMq4B8JCNH
Claude-Local-Session: claude --resume 6d6d6d7c-ada6-47f9-8739-1b416e252b29
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
…12 updates (#2592) Bumps the projectdiscovery group with 10 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.50` | `1.2.51` | | [github.com/projectdiscovery/clistats](https://github.com/projectdiscovery/clistats) | `0.1.5` | `0.1.6` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.21` | `0.8.22` | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.18` | `0.5.19` | | [github.com/projectdiscovery/goflags](https://github.com/projectdiscovery/goflags) | `0.1.76` | `0.2.1` | | [github.com/projectdiscovery/rawhttp](https://github.com/projectdiscovery/rawhttp) | `0.1.91` | `0.1.92` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.24` | `1.3.25` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.108` | `0.0.109` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.11.2` | `0.11.3` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.95` | `0.2.96` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.50 to 1.2.51 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.50...v1.2.51) Updates `github.com/projectdiscovery/clistats` from 0.1.5 to 0.1.6 - [Release notes](https://github.com/projectdiscovery/clistats/releases) - [Commits](projectdiscovery/clistats@v0.1.5...v0.1.6) Updates `github.com/projectdiscovery/dsl` from 0.8.21 to 0.8.22 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](projectdiscovery/dsl@v0.8.21...v0.8.22) Updates `github.com/projectdiscovery/fastdialer` from 0.5.18 to 0.5.19 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](projectdiscovery/fastdialer@v0.5.18...v0.5.19) Updates `github.com/projectdiscovery/goflags` from 0.1.76 to 0.2.1 - [Release notes](https://github.com/projectdiscovery/goflags/releases) - [Commits](projectdiscovery/goflags@v0.1.76...v0.2.1) Updates `github.com/projectdiscovery/hmap` from 0.0.101 to 0.0.102 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](projectdiscovery/hmap@v0.0.101...v0.0.102) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.47 to 0.1.48 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](projectdiscovery/networkpolicy@v0.1.47...v0.1.48) Updates `github.com/projectdiscovery/rawhttp` from 0.1.91 to 0.1.92 - [Release notes](https://github.com/projectdiscovery/rawhttp/releases) - [Commits](projectdiscovery/rawhttp@v0.1.91...v0.1.92) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.24 to 1.3.25 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.3.24...v1.3.25) Updates `github.com/projectdiscovery/useragent` from 0.0.108 to 0.0.109 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](projectdiscovery/useragent@v0.0.108...v0.0.109) Updates `github.com/projectdiscovery/utils` from 0.11.2 to 0.11.3 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](projectdiscovery/utils@v0.11.2...v0.11.3) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.95 to 0.2.96 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.95...v0.2.96) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.51 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/clistats dependency-version: 0.1.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.22 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.5.19 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/goflags dependency-version: 0.2.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.102 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.48 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/rawhttp dependency-version: 0.1.92 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.3.25 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.11.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: projectdiscovery ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(ci): update CodeQL actions * ci: validate CodeQL workflow changes
…0 in the external group (#2576) chore(deps): bump github.com/PuerkitoBio/goquery in the external group Bumps the external group with 1 update: [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery). Updates `github.com/PuerkitoBio/goquery` from 1.12.0 to 1.13.0 - [Release notes](https://github.com/PuerkitoBio/goquery/releases) - [Commits](PuerkitoBio/goquery@v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: github.com/PuerkitoBio/goquery dependency-version: 1.13.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: external ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.