Skip to content

Include the query in the cache ID - #25

Merged
kou merged 5 commits into
red-data-tools:mainfrom
tikkss:include-query-in-cache-id
Sep 28, 2026
Merged

kou merged 5 commits into
red-data-tools:mainfrom
tikkss:include-query-in-cache-id

Conversation

@tikkss

@tikkss tikkss commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GitHub: follow-up GH-24

Because if URL has any query parameters, the cache ID may be conflicted.
The query is now included in the cache ID:

So clear_cache doesn't remove caches for other queries.

The cache ID keeps only the following characters and replaces all other
characters with -:

  • The URL unreserved characters A-Z a-z 0-9 - . _ ~ (RFC 3986 2.3)
  • = for query readability

See also:

GitHub: follow-up red-data-toolsGH-24

Because if URL has any query parameters, the cache ID may be conflicted.

The cache ID replaces `/`, `:`, `?`, and `*` with `-`. They may appear
in a URL without percent-encoding:

  * Host: alphanumerics and `- . _ ~ ! $ & ' ( ) * + , ; =`
    (RFC 3986 3.2.2)
  * Path: the host characters and `: @` (`/` is the separator)
    (RFC 3986 3.3)
  * Query: the path characters and `/ ?`
    (RFC 3986 3.4)

However, they can't be used in file names:

  * POSIX: `/`
  * Windows: `< > : " / \ | ? *`
    ("Naming Files, Paths, and Namespaces" below)

See also:

  * RFC 3986 "Uniform Resource Identifier (URI): Generic Syntax":
    https://www.rfc-editor.org/rfc/rfc3986
  * "Naming Files, Paths, and Namespaces" (Microsoft Learn):
    https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
@kou

kou commented Sep 26, 2026

Copy link
Copy Markdown
Member

How about using Digest::SHA256.hexdigest instead of escaping special characters?
Or how about using allow list style instead of deny list style? I feel that deny list style is difficult to maintain...

Because deny list style is difficult to maintain.

The cache ID keeps only the URL unreserved characters (RFC 3986 2.3)
and `=` for query readability.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
@tikkss

tikkss commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for your suggestion!

I like the allow list style! I've implemented it at bf0f9ab.
I don't like digest style because we can't find the source URL from a cache ID when we debug.

I've updated the PR description. Could you review it again?

@kou

kou commented Sep 27, 2026

Copy link
Copy Markdown
Member

It seems that example.com/a/b?c=d and example.com/a/b-c=d become the same cache ID.

Can we avoid it? Or we don't need to care about the case...?

Because "+" isn't in the allow list and can be used in a file name.

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
@tikkss

tikkss commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for your finding it!

I've changed the query separator from - to + at 0fe7cc5.
+ isn't in the allow list so does not become the same cache ID:

  • example.com/a/b?c=d
    • example.com-a-b+c=d
  • example.com/a/b-c=d
    • example.com-a-b-c=d

I've also updated the PR description.

@kou
kou merged commit 94e1bc4 into red-data-tools:main Sep 28, 2026
9 checks passed
@kou

kou commented Sep 28, 2026

Copy link
Copy Markdown
Member

Thanks.

@tikkss
tikkss deleted the include-query-in-cache-id branch September 28, 2026 20:19
@tikkss

tikkss commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for your review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants