feat(connectors): add url source connector

[paulkarayan]

Part of making "playground" (the Test / workbench flow) run on the mainstream processing path instead of its own bespoke code.

Today playground ingests uploaded/sample files through a hidden, one-off plugin (plugin-utic-playground) with its own indexer + downloader. This adds a real url source connector so ingest can instead be a normal connector — "fetch this list of {url, filename}" — the same way every other source works.

What it does

• UrlIndexer: yields FileData for a configured list of {url, filename}.
• UrlDownloader: SSRF-safe fetch — validates every resolved address (is_global allowlist), pins the httpx transport to the validated IP, revalidates each redirect hop (no DNS-rebind TOCTOU). Private-IP guard is connector config (allow_private_ips), not an env var.
• Registered in the source registry; httpx is a lazy url extra. 14 unit tests.

Once released, playground's uploaded-file ingest becomes a plain url source — no special plugin.

Linkages

• platform-libs pk/url-source-connector-type declares this as a SourceConnectorType (needed for it to be usable in workflows).
• platform-plugins pk/playground-notify handles the uploader side of the same goal.
• Downstream: platform-api rewrites the playground DAG to emit this url source (separate PR, once this is released and plugin manifests reference it).

🤖 Generated with Claude Code

[cubic-dev-ai]

5 issues found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured_ingest/processes/connectors/url.py">

<violation number="1" location="unstructured_ingest/processes/connectors/url.py:14">
P3: The module docstring now says the URL connector is not registered, but the registry imports `.url` and adds `url_source_entry`. Updating this note would avoid misleading future changes around connector registration.</violation>

<violation number="2" location="unstructured_ingest/processes/connectors/url.py:187">
P2: Large URL downloads can spike worker memory because `_ssrf_safe_get` loads the entire response into `resp.content` before `UrlDownloader` writes it. Consider streaming the validated response to the download path, while keeping the per-redirect SSRF checks.</violation>
</file>

<file name="scripts/url_connector_spike.py">

<violation number="1" location="scripts/url_connector_spike.py:6">
P3: The documented run command can fail from a fresh checkout before the spike starts because direct script execution puts `scripts/` on `sys.path` instead of the repo root. Documenting module execution avoids that import-path trap for repo-local runs.</violation>

<violation number="2" location="scripts/url_connector_spike.py:55">
P3: Each spike run leaves its served and downloaded files in a new temp directory because `mkdtemp()` is never removed. Consider wrapping the work tree in `tempfile.TemporaryDirectory()` or removing `work` in `finally`.</violation>

<violation number="3" location="scripts/url_connector_spike.py:95">
P2: The redirect check proves redirect following, but not per-hop SSRF revalidation: it uses the `allow_private_ips=True` downloader, so a private redirect hop would be allowed either way. Consider adding a redirect-to-private case with the default guard, or monkeypatching `_validate_and_pin` to assert both hop hosts are validated.

(Based on your team's feedback about proposing specific test cases.) [FEEDBACK_USED]</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The redirect check proves redirect following, but not per-hop SSRF revalidation: it uses the allow_private_ips=True downloader, so a private redirect hop would be allowed either way. Consider adding a redirect-to-private case with the default guard, or monkeypatching _validate_and_pin to assert both hop hosts are validated.

(Based on your team's feedback about proposing specific test cases.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/url_connector_spike.py, line 95:

<comment>The redirect check proves redirect following, but not per-hop SSRF revalidation: it uses the `allow_private_ips=True` downloader, so a private redirect hop would be allowed either way. Consider adding a redirect-to-private case with the default guard, or monkeypatching `_validate_and_pin` to assert both hop hosts are validated.

(Based on your team's feedback about proposing specific test cases.) </comment>

<file context>
@@ -0,0 +1,123 @@
+            assert resp["file_data"].local_download_path == str(path.resolve())
+        assert contents == {"a.txt": "hello alpha", "b.txt": "hello beta"}, contents
+
+        # redirect is followed (each hop revalidated)
+        redirect_fd = next(iter(indexer.run()))
+        redirect_fd.metadata.url = f"{base}/redirect"
</file context>

[cubic-dev-ai]

P2: Large URL downloads can spike worker memory because _ssrf_safe_get loads the entire response into resp.content before UrlDownloader writes it. Consider streaming the validated response to the download path, while keeping the per-redirect SSRF checks.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured_ingest/processes/connectors/url.py, line 187:

<comment>Large URL downloads can spike worker memory because `_ssrf_safe_get` loads the entire response into `resp.content` before `UrlDownloader` writes it. Consider streaming the validated response to the download path, while keeping the per-redirect SSRF checks.</comment>

<file context>
@@ -0,0 +1,226 @@
+            continue
+        if resp.status_code != 200:
+            raise RuntimeError(f"GET {current} -> {resp.status_code}")
+        return resp.content
+    raise IngestValueError(f"Too many redirects for url: {url}")
+
</file context>

[cubic-dev-ai]

P3: Each spike run leaves its served and downloaded files in a new temp directory because mkdtemp() is never removed. Consider wrapping the work tree in tempfile.TemporaryDirectory() or removing work in finally.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/url_connector_spike.py, line 55:

<comment>Each spike run leaves its served and downloaded files in a new temp directory because `mkdtemp()` is never removed. Consider wrapping the work tree in `tempfile.TemporaryDirectory()` or removing `work` in `finally`.</comment>

<file context>
@@ -0,0 +1,123 @@
+        except Exception as e:  # noqa: BLE001
+            assert "refusing" in str(e).lower(), (bad, e)
+
+    work = Path(tempfile.mkdtemp())
+    served = work / "served"
+    served.mkdir()
</file context>

[cubic-dev-ai]

P3: The documented run command can fail from a fresh checkout before the spike starts because direct script execution puts scripts/ on sys.path instead of the repo root. Documenting module execution avoids that import-path trap for repo-local runs.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At scripts/url_connector_spike.py, line 6:

<comment>The documented run command can fail from a fresh checkout before the spike starts because direct script execution puts `scripts/` on `sys.path` instead of the repo root. Documenting module execution avoids that import-path trap for repo-local runs.</comment>

<file context>
@@ -0,0 +1,123 @@
+the SSRF guard (validator on public/private, private blocked by default, redirect
+followed with per-hop revalidation).
+
+Run:  python scripts/url_connector_spike.py
+"""
+
</file context>

[cubic-dev-ai]

P3: The module docstring now says the URL connector is not registered, but the registry imports .url and adds url_source_entry. Updating this note would avoid misleading future changes around connector registration.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured_ingest/processes/connectors/url.py, line 14:

<comment>The module docstring now says the URL connector is not registered, but the registry imports `.url` and adds `url_source_entry`. Updating this note would avoid misleading future changes around connector registration.</comment>

<file context>
@@ -0,0 +1,226 @@
+(`UrlDownloaderConfig.allow_private_ips`), and the TOCTOU in that original guard
+is closed (see `_ssrf_safe_get`).
+
+NOT YET REGISTERED. `add_source_entry("url", url_source_entry)` in
+`processes/connectors/__init__.py` + the utic_types enum + plugin manifests are
+follow-ups. Importing this module has no effect on the shipped registry.
</file context>

[paulkarayan]

Addressed cubic:

• P1 (path traversal): fixed — _safe_filename reduces caller filenames to a basename and rejects pure-traversal names; tested.
• Duplicate filenames: fixed — precheck rejects duplicate basenames.
• Coverage (default guard not exercised through the pipeline): added test_download_blocks_private_by_default — runs UrlDownloader.run with allow_private_ips=False and asserts the fetch is blocked.
• Streaming/memory (P2): deferred — the only current caller is playground (20 MB cap), and streaming with per-redirect revalidation adds real complexity. Worth doing if this connector is used for large general downloads; filing as follow-up rather than in this PR.
• Spike redirect test (P2): the spike is a smoke script; the real per-hop revalidation coverage is test_redirect_target_is_revalidated (monkeypatches _validate_and_pin, allow_private=False, asserts the post-redirect host is validated).

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 5 unresolved issues from previous reviews.

Re-trigger cubic</sub>