Add proxy support for fetch indexing

[mxyhi]

Summary

ctx_fetch_and_index currently generates a subprocess script that calls bare fetch(url). In environments where direct DNS/network access is blocked or polluted, this makes fetch indexing fail even when users have standard proxy variables configured.

This PR adds proxy-aware fetch support for both JavaScript runtimes used by context-mode:

• Bun: passes fetch(targetUrl, { proxy }).
• Node: passes fetch(targetUrl, { dispatcher: new ProxyAgent(proxy) }) through undici.

Details

• Added src/fetch-proxy.ts to select proxy URLs from HTTPS_PROXY, https_proxy, HTTP_PROXY, http_proxy, ALL_PROXY, and all_proxy.
• Added NO_PROXY / no_proxy bypass support for exact hosts, suffix hosts, wildcard, host:port, and IPv4 CIDR entries.
• Updated ctx_fetch_and_index generated fetch code to call fetchWithProxy(url) instead of bare fetch(url).
• Added undici@^6.25.0 as a runtime dependency so Node 18+ can use a stable ProxyAgent path.
• Updated bundled artifacts.

Validation

• bunx vitest run tests/core/fetch-proxy.test.ts
• npm run typecheck
• npm test -- tests/core/fetch-proxy.test.ts tests/core/server.test.ts tests/core/search.test.ts
• npm run build

Also ran npm test; unrelated existing ABI-cache assertions in tests/core/cli.test.ts fail locally under Node 25 because ensureNativeCompat() returns early when hasModernSqlite() is true:

• cache hit: copies cached ABI binary to active path
• cache hit does not trigger rebuild

[mksglu]

Review

The proxy logic in fetch-proxy.ts is well-structured — NO_PROXY parsing with CIDR/suffix/wildcard support, correct Bun vs Node runtime branching, and clean subprocess code serialization. Good work.

Before we can merge

1. Remove bundle files from the diff.
cli.bundle.mjs and server.bundle.mjs are 356 lines of unreviable minified code. These are built by CI — they should not be in the PR. Remove them and let the build pipeline regenerate.

2. Target the next branch, not main.
All feature PRs go through next first. Please retarget.

3. SOCKS5 validation.
The PR reads ALL_PROXY which commonly points to socks5:// URLs, but undici's ProxyAgent only supports HTTP/HTTPS proxies (nodejs/undici#2224). A socks5:// URL will silently fail. Please either:

• Validate the protocol and throw a clear error: "SOCKS proxies are not supported — only HTTP/HTTPS"
• Or skip ALL_PROXY entirely and only read HTTP_PROXY / HTTPS_PROXY

Question: real-world demand

I want to understand the actual need before committing to maintaining proxy infrastructure:

• Are you personally behind a corporate proxy? Did ctx_fetch_and_index fail for you?
• How many users have reported this? I haven't seen proxy-related issues filed.
• Would documenting HTTP_PROXY support in the README be sufficient, or do you need the full NO_PROXY/CIDR logic?

This adds ~200 lines of source + undici as a new dependency. I want to make sure we're solving a real problem for real users, not a theoretical one.

Thanks for the thorough implementation — the code quality is solid. Just need clarity on demand before we merge.

[mxyhi]

Thanks for the review. I pushed an update addressing the merge blockers:

• Removed cli.bundle.mjs and server.bundle.mjs from the PR diff. The PR now only changes bun.lock, package.json, src/fetch-proxy.ts, src/server.ts, and tests/core/fetch-proxy.test.ts.
• Retargeted the PR to next and rebased the branch onto upstream/next.
• Removed ALL_PROXY / all_proxy support to avoid SOCKS ambiguity. The code now only reads HTTP_PROXY / HTTPS_PROXY variants.
• Added explicit validation for selected proxy URLs. If someone sets HTTP_PROXY or HTTPS_PROXY to socks*://..., it throws SOCKS proxies are not supported — only HTTP/HTTPS instead of failing later inside undici.

On demand: this came from a real failure I hit while using ctx_fetch_and_index against https://swr.vercel.app/docs/api and /docs/mutation. example.com fetched successfully, but the SWR docs path hung because direct DNS/network resolution returned non-Vercel/suspicious IPs and the generated fetch subprocess used bare fetch(url). Manually using Bun fetch(url, { proxy: "http://127.0.0.1:7890" }) succeeded.

I do not know of additional user reports, so I would frame this as one concrete reproducible environment rather than broad demand. README documentation alone would not have fixed this case because the generated fetch script needed an explicit cross-runtime proxy path: Bun needs { proxy }, and Node needs an undici dispatcher.

I kept NO_PROXY support because once HTTP_PROXY/HTTPS_PROXY is honored, users need a way to avoid proxying local/private routes. The CIDR support matches common existing NO_PROXY configs such as 100.64.0.0/10; happy to simplify that further if you prefer a smaller first version.

Validation after the update:

• npm run typecheck
• npm test -- tests/core/fetch-proxy.test.ts tests/core/server.test.ts tests/core/search.test.ts
• npm run build

All passed locally.

[mksglu]

Thanks for the quick turnaround — all three items addressed cleanly.

I haven't seen this need come up before, so I'd like to wait for a bit of community signal before merging. If others need proxy support too, they can 👍 this PR. Once we see demand, we'll merge it in.

Appreciate the contribution and the detailed repro.

[mxyhi]

I think everyone in mainland China needs it; you should understand.

[ousamabenyounes]

After reading the diff carefully I want to flag three concrete points before merge — not blockers per se, but the kind of things that bite later.

1. Duplication of all NO_PROXY logic between TS and runtime template

src/fetch-proxy.ts defines the bypass logic twice:

• TS version at module scope (lines ~172–242 of the diff): shouldBypassProxy, splitNoProxyToken, ipv4ToNumber, matchesIpv4Cidr, getProxyUrlForRequest, validateProxyUrl, normalizeHost, defaultPort.
• Runtime version inside buildFetchProxyRuntimeCode(...) template literal (lines ~81–158): the same 8 functions, same bodies, just with TS annotations stripped.

Two sources of truth for identical logic, with no compile-time link between them. The unit tests exercise the TS version (CIDR / suffix / wildcard / port edge cases), but the string-injected runtime version is only exercised by a smoke test that checks "Bun gets { proxy }, Node gets { dispatcher }". A NO_PROXY regression in the runtime copy would not be caught.

Cleaner alternative: ship a vendored plain-JS file (e.g. src/fetch-proxy.runtime.js) that the subprocess requires at runtime, and have the TS module re-export the same functions. One file, one set of tests, no string templating, type-checked and lintable.

This is the change I'd push back on hardest before merging — once it lands as-is, the two copies will drift the first time someone fixes a NO_PROXY edge case.

2. NO_PROXY="*:port" ignores the port (logic-order bug)

In shouldBypassProxy:

if (token.host === "*") return true;
if (token.port && token.port !== port) continue;

splitNoProxyToken("*:8080") returns { host: "*", port: "8080" }, the wildcard branch fires first, and the port check is never reached. So NO_PROXY="*:8080" bypasses every request, including :443. Likely not the intent given the rest of the parser carefully tracks port. Swap the two checks (or treat * with a port as a separate case).

3. validateProxyUrl doesn't wrap new URL(...)

const protocol = new URL(proxyUrl).protocol;

If a user sets HTTPS_PROXY=corp-proxy.local:8080 (missing scheme — common mistake), this throws a generic TypeError: Invalid URL from the URL constructor instead of the descriptive errors the function is built to surface. Wrapping in try/catch and rethrowing with the proxy var name in the message would make misconfigurations easier to debug.

Test coverage gap

The "selects HTTPS proxy before HTTP proxy" test passes both vars and checks HTTPS wins, but there's no test for the documented fallback: HTTPS request URL with only HTTP_PROXY set (no HTTPS_PROXY). Given HTTPS_PROXY_KEYS = ["HTTPS_PROXY", "https_proxy", "HTTP_PROXY", "http_proxy"] lists HTTP_PROXY as a fallback, that path should have an explicit test.

---

On the broader merge question: I'd lean toward shipping this. Honoring HTTP_PROXY / HTTPS_PROXY is table-stakes UNIX behavior (curl, git, npm, pip all do it), and a silent fetch failure is the kind of bug that doesn't generate GitHub 👍 because affected users just stop using the tool. But the duplication above is the right reason to ask for one more revision rather than wait for a community signal that won't arrive.

@mxyhi if you want a hand on this, I would be happy to contribute directly on your PR — I can push a commit that consolidates the runtime into a vendored JS file, fixes the *:port ordering, wraps the URL parsing in try/catch, and adds the missing HTTPS-URL → HTTP_PROXY fallback test. Just say the word and I will open the changes against your branch.

Generated by Ora Studio (With Claude)
Reviewed by Ousama Ben Younes