Add RemoteInput.open - #24
Merged
kou merged 13 commits intoSep 26, 2026
Merged
Conversation
GitHub: red-data-toolsGH-5 This patch will introduce the new high level API for IO like object (input only object, read only object). `RemoteInput.open` downloads the data lazily on the first read. The data is stored in a system temporary directory and removed on close: ```ruby RemoteInput.open("https://example.com/sample/data.csv") do |input| input.read end ``` `RemoteInput::Input#local_path` returns a temporary file path: ```ruby RemoteInput.open("https://example.com/sample/data.csv") do |input| input.local_path # => /system temporary directory/red-remote-input/1234-567/data # 1234 is the process ID # 567 is the object ID # data is the fixed file name end ``` Without a block, `RemoteInput.open` returns a `RemoteInput::Input` object that must be closed explicitly: ```ruby input = RemoteInput.open("https://example.com/sample/data.csv") input.read input.close ```
Contributor
Author
|
GitHub Actions was disabled when I opened this PR, so no workflows ran. |
kou
reviewed
Sep 2, 2026
Because `RemoteInput::Input` is redundant.
Contributor
Author
|
I've made |
kou
reviewed
Sep 20, 2026
Because we only accept read only mode. Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because `Downloader.new` accepts fallback URLs as positional arguments.
Because `mode` was removed, we need another way to read not UTF-8 encoding such as Windows-31J. Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because we want to avoid re-download for the same URL. So, we don't remove cache file on close. Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because we don't remove the cache file on close yet, we need a way to remove it explicitly.
tikkss
force-pushed
the
remote-input-open-without-cache
branch
from
September 23, 2026 11:24
0ab538b to
1a9ecd6
Compare
Contributor
Author
|
I've applied the suggestions:
I've also updated the PR title and description for the new design. |
kou
reviewed
Sep 23, 2026
Because we don't want to implement encoding validation in our side. Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because it returns nonexistent path when `read` isn't called yet. Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because `IO#path` exists. Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Because `RemoteInput#path` already exists and we may confuse them.
Contributor
Author
|
I've applied all suggestions and updated the PR description. Could you review this again? Note: making |
kou
reviewed
Sep 26, 2026
Contributor
Author
|
Thanks for your polite review always! |
tikkss
added a commit
to tikkss/red-remote-input
that referenced
this pull request
Sep 26, 2026
GitHub: follow-up red-data-toolsGH-24 Because if URL has any query parameters, the cache ID may be conflicted. The cache ID replaces `/`, `:`, `?`, and `*` with `-`. They may appear in a URL without percent-encoding: * Host: alphanumerics and `- . _ ~ ! $ & ' ( ) * + , ; =` (RFC 3986 3.2.2) * Path: the host characters and `: @` (`/` is the separator) (RFC 3986 3.3) * Query: the path characters and `/ ?` (RFC 3986 3.4) However, they can't be used in file names: * POSIX: `/` * Windows: `< > : " / \ | ? *` ("Naming Files, Paths, and Namespaces" below) See also: * RFC 3986 "Uniform Resource Identifier (URI): Generic Syntax": https://www.rfc-editor.org/rfc/rfc3986 * "Naming Files, Paths, and Namespaces" (Microsoft Learn): https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file Co-authored-by: Sutou Kouhei <kou@clear-code.com>
kou
added a commit
that referenced
this pull request
Sep 28, 2026
GitHub: follow-up GH-24 Because if URL has any query parameters, the cache ID may be conflicted. The query is now included in the cache ID: * URL: https://example.com/file?a=b * => cache ID: `example.com+a=b` * URL: https://example.com/file?a=c * => cache ID: `example.com+a=c` So `clear_cache` doesn't remove caches for other queries. The cache ID keeps only the following characters and replaces all other characters with `-`: * The URL unreserved characters `A-Z a-z 0-9 - . _ ~` (RFC 3986 2.3) * `=` for query readability See also: * RFC 3986 "Uniform Resource Identifier (URI): Generic Syntax": https://www.rfc-editor.org/rfc/rfc3986 * "Naming Files, Paths, and Namespaces" (Microsoft Learn): https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file --------- Co-authored-by: Sutou Kouhei <kou@clear-code.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub: GH-5
This patch will introduce the new high level API for IO like object (input only object, read only object).
RemoteInput.opendownloads the data lazily on the first read. The data is stored in a system cache directory and kept on close to avoid re-download for the same URL:The cache file path is computed from the URL so that the same URL uses the same cache:
<system cache directory>/red-remote-input/example.com-sample/data.csvexample.com-sampleis the cache ID: host + directory (/is replaced with-)data.csvis the base name of the URL(
datais used when the URL doesn't have a base name)RemoteInput#clear_cacheremoves the cache explicitly:RemoteInput.openaccepts encoding options likeIO.new:Without a block,
RemoteInput.openreturns aRemoteInputobject that must be closed explicitly: