Skip to content

feat: support generic http(s) urls - #8

Merged
coroa merged 5 commits into
mainfrom
feat/support-generic-http-urls
Mar 18, 2026
Merged

coroa merged 5 commits into
mainfrom
feat/support-generic-http-urls

Conversation

@coroa

@coroa coroa commented Feb 26, 2026 •

Copy link
Copy Markdown
Member

Fixes #5 .

With this change we re-introduce general http(s) urls based on a http head request, so that cached-http becomes a complete replacement for the original http plugin.

Of note: if head requests are unsupported, the plugin assumes the requested url exists with an mtime of 0.

@coroa
coroa requested a review from lkstrp February 26, 2026 11:05
@coroa

coroa commented Feb 26, 2026

Copy link
Copy Markdown
Member Author

We need to be a bit careful about releasing a version here, since i removed the monkey patching of snakemake-storage-plugin-http, so this would give us, if both are installled:

WorkflowError in file "/home/coroa/work/pypsa-eur/rules/retrieve.smk", line 28:
Multiple suitable storage providers found for query https://data.pypsa.org/workflows/eur/eurostat_balances/2026-02/nrg_bal_c: http, cached-http. Explicitly specify the storage provider.

I guess i could leave the monkey patching in place for a grace period and warn about the future incompatibility?

@coroa coroa mentioned this pull request Feb 26, 2026
@lkstrp

lkstrp commented Feb 26, 2026

Copy link
Copy Markdown
Member

I mean this is currently used for PyPSA-Eur and forks only. We can just pin the current version in Eur, release here a new version, and remove the pin and remove the default storage provider

@coroa
coroa force-pushed the feat/support-generic-http-urls branch from ef5edfd to 4eade7f Compare February 26, 2026 12:57
@coroa

coroa commented Feb 26, 2026

Copy link
Copy Markdown
Member Author

Yes, but since the update cycles of the soft-forks are somewhat irregular, let's just take the safe route and keep the monkey-patching around a bit longer.

@coroa
coroa merged commit a7a76ad into main Mar 18, 2026
3 checks passed
@lkstrp

lkstrp commented Mar 20, 2026

Copy link
Copy Markdown
Member

Sorry for the missed review !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add general url support

2 participants