Add remote file support to SourceFile

[tomdurrant]

Problem

Currently, SourceFile only supports local file paths. Users cannot directly specify remote files (e.g., gs://, s3://, http://) as data sources, even though rompy already has a robust transfer system via entry points that handles these protocols.

Proposed Solution

Extend SourceFile to automatically detect and download remote files before opening them with xarray. This leverages the existing transfer system (TransferBase implementations) already used by DataBlob.

Implementation Approach

1. Detect URI scheme using parse_scheme() from rompy.transfer.utils
2. For local files (file:// or plain paths): Use current behavior
3. For remote URIs: Use get_transfer() to download to a cache directory before calling xr.open_dataset()

Key Features

• Automatic protocol detection: No need for separate SourceRemoteFile class
• Caching: Downloaded files cached to avoid repeated downloads
• Leverage existing infrastructure: Uses rompy.transfer entry points (http, https, gs, s3, az, oceanum)
• Backward compatible: Existing local file usage unchanged

Proposed API

# Local file (existing behavior)
source = SourceFile(uri="/data/local.nc")

# Remote GCS file (new capability)
source = SourceFile(uri="gs://bucket/path/data.nc")

# Remote HTTP file with custom cache location
source = SourceFile(
    uri="https://example.com/data.nc",
    cache_dir="/path/to/cache"
)

# All open with the same interface
ds = source.open()

New Parameters

• cache_dir: Optional[Path] = None - Cache directory for downloaded files (defaults to system temp dir)
• Private attribute _local_path to track downloaded file and avoid re-downloads

Implementation Details

Location: src/rompy/core/source.py, SourceFile class (line 142)

Add method:

def _ensure_local(self) -> Path:
    """Ensure URI is available as local file, downloading if needed."""
    # Check if already cached
    # Detect scheme
    # If local, return path
    # If remote, use get_transfer() to download
    # Return local path

Update _open() to call _ensure_local() before xr.open_dataset().

Alternative Considered

Creating a separate SourceRemoteFile class was considered but rejected because:

• Violates DRY (duplicates SourceFile logic)
• Worse UX (users must know if file is remote)
• Breaks abstraction ("a file is a file")
• Doesn't leverage the existing pattern in DataBlob

Related Code

• rompy/core/data.py: DataBlob.get() demonstrates the pattern (line 114-158)
• rompy/transfer/registry.py: Transfer system with scheme-based dispatch
• Entry points in pyproject.toml: [project.entry-points."rompy.transfer"]

Testing Considerations

• Unit tests for local files (no change in behavior)
• Unit tests for mock remote URIs (gs://, s3://, http://)
• Integration tests with actual remote data (marked as @pytest.mark.slow)
• Cache behavior tests (verify no re-download)
• Error handling (invalid URIs, network failures)