Add --allow-remote-refs to disable HTTP fetching of $ref by default (#3051)

* Add --allow-remote-refs flag to gate HTTP fetching of $ref targets

Remote $ref fetching over HTTP/HTTPS is now disabled by default.
When a $ref resolves to an HTTP(S) URL and --allow-remote-refs is not set,
a clear error message is shown instead of silently fetching remote content.

file:// URLs are still allowed without the flag since they are local.
--url input implicitly enables --allow-remote-refs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Improve error message for missing local $ref files

When a $ref points to a local file that doesn't exist, the error now
clearly states "$ref file not found: <path>" instead of raising a raw
FileNotFoundError.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Validate HTTP responses before parsing as schema content

Check status codes and Content-Type headers when fetching remote $ref
targets over HTTP. Returns clear error messages for HTTP errors (4xx/5xx)
and unexpected HTML responses instead of cryptic YAML parse errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove redundant local `import json` in test functions

The module-level import already covers these usages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add @pytest.mark.cli_doc marker for --allow-remote-refs

CLI reference docs will be auto-generated by CI from this marker.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Set status_code and headers on httpx mock responses

Remove isinstance/hasattr guards from http.get_body() and instead
fix all test mocks to set status_code=200 and headers={}, matching
real httpx.Response behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Use SchemaFetchError instead of bare Exception in HTTP responses

Add SchemaFetchError(Error) for HTTP error status codes and unexpected
HTML responses. This ensures errors are caught by the existing
`except Error` handler in __main__.py and shown as clean messages
instead of unhandled tracebacks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Wrap transport errors and normalize Content-Type check

Catch httpx transport exceptions (DNS, timeout, connection errors) and
wrap them in SchemaFetchError. Normalize Content-Type to lowercase
before checking for text/html.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Harden test assertions for blocked and missing refs

Assert httpx.get is not called when remote refs are blocked, preventing
real HTTP leaks if the gate regresses. Assert the specific file path
in the missing local ref error, not just the generic prefix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Change --allow-remote-refs default to warn instead of block

Per maintainer feedback, keep backward compatibility by allowing remote
$ref fetching by default but emit a FutureWarning when it happens
without explicit --allow-remote-refs. The flag becomes a three-state:
True (explicit opt-in, no warning), False (blocks), None (default,
allows with deprecation warning). The default will flip to False in
a future major version.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Support --no-allow-remote-refs via BooleanOptionalAction

Use BooleanOptionalAction (like --use-annotated) so users can
explicitly opt out with --no-allow-remote-refs from the CLI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add --no-allow-remote-refs to CLI_OPTION_META

BooleanOptionalAction registers both --allow-remote-refs and
--no-allow-remote-refs in argparse; the sync test requires both
to be in CLI_OPTION_META.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: butvinm

src/datamodel_code_generator/__init__.py

@@ -371,6 +371,10 @@ def _format_message(self, message: str) -> str:
         return message
 
 
+class SchemaFetchError(Error):
+    """Raised when fetching a remote schema fails (HTTP error, unexpected content type)."""
+
+
 def get_first_file(path: Path) -> Path:  # pragma: no cover
     """Find and return the first file in a path (file or directory)."""
     if path.is_file():

src/datamodel_code_generator/__main__.py

@@ -491,6 +491,7 @@ def validate_class_name_affix_scope(cls, v: str | ClassNameAffixScope | None) ->
     use_closed_typed_dict: bool = True
     allof_merge_mode: AllOfMergeMode = AllOfMergeMode.Constraints
     allof_class_hierarchy: AllOfClassHierarchy = AllOfClassHierarchy.IfNoConflict
+    allow_remote_refs: Optional[bool] = None  # noqa: UP045
     http_headers: Optional[Sequence[tuple[str, str]]] = None  # noqa: UP045
     http_ignore_tls: bool = False
     http_timeout: Optional[float] = None  # noqa: UP045
@@ -934,6 +935,7 @@ def run_generate_from_config(  # noqa: PLR0913, PLR0917
         use_closed_typed_dict=config.use_closed_typed_dict,
         allof_merge_mode=config.allof_merge_mode,
         allof_class_hierarchy=config.allof_class_hierarchy,
+        allow_remote_refs=config.allow_remote_refs,
         http_headers=config.http_headers,
         http_ignore_tls=config.http_ignore_tls,
         http_timeout=config.http_timeout,
@@ -1080,6 +1082,10 @@ def main(args: Sequence[str] | None = None) -> Exit:  # noqa: PLR0911, PLR0912,
         )
         return Exit.ERROR
 
+    # --url implies --allow-remote-refs since the user is explicitly fetching a remote schema
+    if config.url:
+        config.allow_remote_refs = True
+
     if config.check and config.output is None:
         print(  # noqa: T201
             "Error: --check cannot be used with stdout output (no --output specified)",

src/datamodel_code_generator/_types/generate_config_dict.py

@@ -116,6 +116,7 @@ class GenerateConfigDict(TypedDict, closed=True):
     use_closed_typed_dict: NotRequired[bool]
     allof_merge_mode: NotRequired[AllOfMergeMode]
     allof_class_hierarchy: NotRequired[AllOfClassHierarchy]
+    allow_remote_refs: NotRequired[bool | None]
     http_headers: NotRequired[Sequence[tuple[str, str]] | None]
     http_ignore_tls: NotRequired[bool]
     http_timeout: NotRequired[float | None]

src/datamodel_code_generator/_types/parser_config_dicts.py

@@ -111,6 +111,7 @@ class ParserConfigDict(TypedDict):
     use_closed_typed_dict: NotRequired[bool]
     allof_merge_mode: NotRequired[AllOfMergeMode]
     allof_class_hierarchy: NotRequired[AllOfClassHierarchy]
+    allow_remote_refs: NotRequired[bool | None]
     http_headers: NotRequired[Sequence[tuple[str, str]] | None]
     http_ignore_tls: NotRequired[bool]
     http_timeout: NotRequired[float | None]

src/datamodel_code_generator/arguments.py

@@ -125,6 +125,16 @@ def start_section(self, heading: str | None) -> None:
 # ======================================================================================
 # Base options for input/output
 # ======================================================================================
+base_options.add_argument(
+    "--allow-remote-refs",
+    help="Allow fetching remote $ref references over HTTP/HTTPS. "
+    "Currently remote fetching is allowed by default but emits a deprecation warning. "
+    "Pass --allow-remote-refs to opt in without warning, "
+    "or --no-allow-remote-refs to block remote fetching. "
+    "In a future version, remote fetching will be disabled by default.",
+    action=BooleanOptionalAction,
+    default=None,
+)
 base_options.add_argument(
     "--http-headers",
     nargs="+",

src/datamodel_code_generator/cli_options.py

@@ -265,6 +265,8 @@ class CLIOptionMeta:
     # General Options
     # ==========================================================================
     "--check": CLIOptionMeta(name="--check", category=OptionCategory.GENERAL),
+    "--allow-remote-refs": CLIOptionMeta(name="--allow-remote-refs", category=OptionCategory.GENERAL),
+    "--no-allow-remote-refs": CLIOptionMeta(name="--no-allow-remote-refs", category=OptionCategory.GENERAL),
     "--http-headers": CLIOptionMeta(name="--http-headers", category=OptionCategory.GENERAL),
     "--http-ignore-tls": CLIOptionMeta(name="--http-ignore-tls", category=OptionCategory.GENERAL),
     "--http-query-parameters": CLIOptionMeta(name="--http-query-parameters", category=OptionCategory.GENERAL),

src/datamodel_code_generator/config.py

@@ -141,6 +141,7 @@ class GenerateConfig(BaseModel):
     use_closed_typed_dict: bool = True
     allof_merge_mode: AllOfMergeMode = AllOfMergeMode.Constraints
     allof_class_hierarchy: AllOfClassHierarchy = AllOfClassHierarchy.IfNoConflict
+    allow_remote_refs: bool | None = None
     http_headers: Sequence[tuple[str, str]] | None = None
     http_ignore_tls: bool = False
     http_timeout: float | None = None
@@ -273,6 +274,7 @@ class ParserConfig(BaseModel):
     use_closed_typed_dict: bool = True
     allof_merge_mode: AllOfMergeMode = AllOfMergeMode.Constraints
     allof_class_hierarchy: AllOfClassHierarchy = AllOfClassHierarchy.IfNoConflict
+    allow_remote_refs: bool | None = None
     http_headers: Sequence[tuple[str, str]] | None = None
     http_ignore_tls: bool = False
     http_timeout: float | None = None

src/datamodel_code_generator/http.py

@@ -9,6 +9,8 @@
 
 from typing import TYPE_CHECKING, Any
 
+from datamodel_code_generator import SchemaFetchError
+
 if TYPE_CHECKING:
     from collections.abc import Sequence
 
@@ -35,14 +37,28 @@ def get_body(
 ) -> str:
     """Fetch content from a URL with optional headers and query parameters."""
     httpx = _get_httpx()
-    return httpx.get(
-        url,
-        headers=headers,
-        verify=not ignore_tls,
-        follow_redirects=True,
-        params=query_parameters,  # ty: ignore
-        timeout=timeout,
-    ).text
+    try:
+        response = httpx.get(
+            url,
+            headers=headers,
+            verify=not ignore_tls,
+            follow_redirects=True,
+            params=query_parameters,  # ty: ignore
+            timeout=timeout,
+        )
+    except Exception as e:
+        msg = f"Failed to fetch {url}: {e}"
+        raise SchemaFetchError(msg) from e
+    if response.status_code >= 400:  # noqa: PLR2004
+        msg = f"HTTP {response.status_code} error fetching {url}"
+        raise SchemaFetchError(msg)
+    content_type = response.headers.get("content-type", "").lower()
+    if "text/html" in content_type:
+        msg = (
+            f"Unexpected HTML response from {url} (Content-Type: {content_type}). Expected JSON or YAML schema content."
+        )
+        raise SchemaFetchError(msg)
+    return response.text
 
 
 def join_url(url: str, ref: str = ".") -> str:  # noqa: PLR0912

src/datamodel_code_generator/parser/base.py

@@ -1101,6 +1101,7 @@ def __init__(  # noqa: PLR0912, PLR0915
         )
         self.class_name: str | None = config.class_name
         self.wrap_string_literal: bool | None = config.wrap_string_literal
+        self.allow_remote_refs: bool | None = config.allow_remote_refs
         self.http_headers: Sequence[tuple[str, str]] | None = config.http_headers
         self.http_query_parameters: Sequence[tuple[str, str]] | None = config.http_query_parameters
         self.http_ignore_tls: bool = config.http_ignore_tls

src/datamodel_code_generator/parser/jsonschema.py

@@ -30,6 +30,7 @@
 from datamodel_code_generator import (
     AllOfClassHierarchy,
     AllOfMergeMode,
+    Error,
     InvalidClassNameError,
     JsonSchemaVersion,
     ReadOnlyWriteOnlyModelType,
@@ -3817,6 +3818,23 @@ def create_enum(reference_: Reference) -> DataType:
     def _get_ref_body(self, resolved_ref: str) -> dict[str, YamlValue]:
         """Get the body of a reference from URL or remote file."""
         if is_url(resolved_ref):
+            if not resolved_ref.startswith("file://"):
+                if self.allow_remote_refs is False:
+                    msg = (
+                        f"Fetching remote $ref is disabled: {resolved_ref}\n"
+                        "Use --allow-remote-refs to enable HTTP fetching of remote references."
+                    )
+                    raise Error(msg)
+                if self.allow_remote_refs is None:
+                    import warnings  # noqa: PLC0415
+
+                    warnings.warn(
+                        f"Fetching remote $ref without --allow-remote-refs: {resolved_ref}\n"
+                        "In a future version, remote $ref fetching will be disabled by default. "
+                        "Pass --allow-remote-refs explicitly to silence this warning.",
+                        FutureWarning,
+                        stacklevel=2,
+                    )
             return self._get_ref_body_from_url(resolved_ref)
         return self._get_ref_body_from_remote(resolved_ref)
 
@@ -3844,10 +3862,14 @@ def _get_ref_body_from_remote(self, resolved_ref: str) -> dict[str, YamlValue]:
         """Get reference body from a remote file path."""
         full_path = self.base_path / resolved_ref
 
-        return self.remote_object_cache.get_or_put(
-            str(full_path),
-            default_factory=lambda _: load_data_from_path(full_path, self.encoding),
-        )
+        try:
+            return self.remote_object_cache.get_or_put(
+                str(full_path),
+                default_factory=lambda _: load_data_from_path(full_path, self.encoding),
+            )
+        except FileNotFoundError:
+            msg = f"$ref file not found: {full_path}"
+            raise Error(msg) from None
 
     def resolve_ref(self, object_ref: str) -> Reference:
         """Resolve a reference by loading and parsing the referenced schema."""