Skip to content

Add RemoteInput.open - #24

Merged
kou merged 13 commits into
red-data-tools:mainfrom
tikkss:remote-input-open-without-cache
Sep 26, 2026
Merged

kou merged 13 commits into
red-data-tools:mainfrom
tikkss:remote-input-open-without-cache

Conversation

@tikkss

@tikkss tikkss commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

GitHub: GH-5

This patch will introduce the new high level API for IO like object (input only object, read only object).

RemoteInput.open downloads the data lazily on the first read. The data is stored in a system cache directory and kept on close to avoid re-download for the same URL:

RemoteInput.open("https://example.com/sample/data.csv") do |input|
  input.read
end

The cache file path is computed from the URL so that the same URL uses the same cache:

  • URL: https://example.com/sample/data.csv
  • Cache file path: <system cache directory>/red-remote-input/example.com-sample/data.csv
    • example.com-sample is the cache ID: host + directory (/ is replaced with -)
    • data.csv is the base name of the URL
      (data is used when the URL doesn't have a base name)

RemoteInput#clear_cache removes the cache explicitly:

input = RemoteInput.open("https://example.com/sample/data.csv")
input.read
input.close       # The cache is kept here
input.clear_cache # Remove the cache explicitly

RemoteInput.open accepts encoding options like IO.new:

RemoteInput.open("https://example.com/sample/data.csv",
                 encoding: "Windows-31J:UTF-8") do |input|
  input.read # => UTF-8 string
end

Without a block, RemoteInput.open returns a RemoteInput object that must be closed explicitly:

input = RemoteInput.open("https://example.com/sample/data.csv")
input.read
input.close

GitHub: red-data-toolsGH-5

This patch will introduce the new high level API for IO like object
(input only object, read only object).

`RemoteInput.open` downloads the data lazily on the first read. The data
is stored in a system temporary directory and removed on close:

```ruby
RemoteInput.open("https://example.com/sample/data.csv") do |input|
  input.read
end
```

`RemoteInput::Input#local_path` returns a temporary file path:

```ruby
RemoteInput.open("https://example.com/sample/data.csv") do |input|
  input.local_path
  # => /system temporary directory/red-remote-input/1234-567/data
  # 1234 is the process ID
  # 567 is the object ID
  # data is the fixed file name
end
```

Without a block, `RemoteInput.open` returns a `RemoteInput::Input`
object that must be closed explicitly:

```ruby
input = RemoteInput.open("https://example.com/sample/data.csv")
input.read
input.close
```
@tikkss

tikkss commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

GitHub Actions was disabled when I opened this PR, so no workflows ran.
I've since enabled it, but since events aren't re-delivered, I'll close and reopen this PR to trigger CI.

@tikkss tikkss closed this Sep 1, 2026
@tikkss tikkss reopened this Sep 1, 2026
Comment thread lib/remote_input/input.rb Outdated
Because `RemoteInput::Input` is redundant.
@tikkss

tikkss commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

I've made RemoteInput a class and moved the RemoteInput::Input implementation to it: c547b94
I've also merged test/test-input.rb into test/test-remote-input.rb.
Could you review this again? Thanks!

Comment thread lib/remote_input.rb Outdated
Comment thread lib/remote_input.rb Outdated
Comment thread lib/remote_input.rb Outdated
tikkss and others added 6 commits September 23, 2026 20:23
Because we only accept read only mode.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because `Downloader.new` accepts fallback URLs as positional arguments.
Because `mode` was removed, we need another way to read not UTF-8
encoding such as Windows-31J.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because we want to avoid re-download for the same URL. So, we don't
remove cache file on close.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because we don't remove the cache file on close yet, we need a way to
remove it explicitly.
@tikkss
tikkss force-pushed the remote-input-open-without-cache branch from 0ab538b to 1a9ecd6 Compare September 23, 2026 11:24
@tikkss tikkss changed the title Add RemoteInput.open without cache Add RemoteInput.open Sep 23, 2026
@tikkss

tikkss commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

I've applied the suggestions:

  • Removed mode and added encoding:/internal_encoding:/external_encoding:
    options like IO.new
  • Used the same cache ID for the same URL: the cache ID is computed from
    the URL (host + directory, / is replaced with -), e.g.
    https://example.com/sample/data.csv → example.com-sample/data.csv
  • Kept the cache file on close to avoid re-download and added
    RemoteInput#clear_cache to remove it explicitly

I've also updated the PR title and description for the new design.
Could you review this again?

Comment thread lib/remote_input.rb Outdated
Comment thread lib/remote_input.rb Outdated
tikkss and others added 5 commits September 25, 2026 20:34
Because we don't want to implement encoding validation in our side.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because it returns nonexistent path when `read` isn't called yet.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because `IO#path` exists.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because `RemoteInput#path` already exists and we may confuse them.
@tikkss

tikkss commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor Author

I've applied all suggestions and updated the PR description. Could you review this again?

Note: making #path a public API (that downloads the data lazily) is out of scope of this PR.

Comment thread lib/remote_input.rb
@kou
kou merged commit e70b6cb into red-data-tools:main Sep 26, 2026
9 checks passed
@tikkss
tikkss deleted the remote-input-open-without-cache branch September 26, 2026 06:17
@tikkss

tikkss commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for your polite review always!

tikkss added a commit to tikkss/red-remote-input that referenced this pull request Sep 26, 2026
GitHub: follow-up red-data-toolsGH-24

Because if URL has any query parameters, the cache ID may be conflicted.

The cache ID replaces `/`, `:`, `?`, and `*` with `-`. They may appear
in a URL without percent-encoding:

  * Host: alphanumerics and `- . _ ~ ! $ & ' ( ) * + , ; =`
    (RFC 3986 3.2.2)
  * Path: the host characters and `: @` (`/` is the separator)
    (RFC 3986 3.3)
  * Query: the path characters and `/ ?`
    (RFC 3986 3.4)

However, they can't be used in file names:

  * POSIX: `/`
  * Windows: `< > : " / \ | ? *`
    ("Naming Files, Paths, and Namespaces" below)

See also:

  * RFC 3986 "Uniform Resource Identifier (URI): Generic Syntax":
    https://www.rfc-editor.org/rfc/rfc3986
  * "Naming Files, Paths, and Namespaces" (Microsoft Learn):
    https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
kou added a commit that referenced this pull request Sep 28, 2026
GitHub: follow-up GH-24

Because if URL has any query parameters, the cache ID may be conflicted.
The query is now included in the cache ID:

  * URL: https://example.com/file?a=b
    * => cache ID: `example.com+a=b`
  * URL: https://example.com/file?a=c
    * => cache ID: `example.com+a=c`

So `clear_cache` doesn't remove caches for other queries.

The cache ID keeps only the following characters and replaces all other
characters with `-`:

  * The URL unreserved characters `A-Z a-z 0-9 - . _ ~` (RFC 3986 2.3)
  * `=` for query readability

See also:

  * RFC 3986 "Uniform Resource Identifier (URI): Generic Syntax":
    https://www.rfc-editor.org/rfc/rfc3986
  * "Naming Files, Paths, and Namespaces" (Microsoft Learn):
    https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file

---------

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants