Add Google cloud storage support

[coroa]

Contrarily to the other two storage options, we do not assume immutability, so quite some changes to track correctly the different lifetime between storage and local paths were added.

There is now a StorageProvider.is_immutable method which encodes whether a url of a certain StorageProvider may change or not.

immutable storages are returned with an mtime of 0 always, mutable ones (GCS) report the mtime of the server and update the local cache item if it is older than upstream.

[coroa]

@lkstrp @daniel-rdt would one of you be able to review?

[lkstrp]

Looks fine for me, but I would rather see this as another argument for moving all the checksums to pypsa-eur itself. And then we reduce this storage plugin to it's caching functionality.

We are producing more and more overhead now, and we wanna expand this for any user to be able to set own storages (and even mirrors at some point). Currently we need to extend this plugin for every storage service. It's just much simpler when serving checksums in versions.csv and comes with all the other benefits

[coroa]

We are producing more and more overhead now, and we wanna expand this for any user to be able to set own storages (and even mirrors at some point). Currently we need to extend this plugin for every storage service. It's just much simpler when serving checksums in versions.csv and comes with all the other benefits

don't agree. i am actually surprised how extensible the structure already is. where is the overhead? you mean the metadata round trips?

sure, we'd want to refactor out these metadata storage options, ie. #6 . already have an idea how that should look. and i guess some config option to link URL prefixes with a metadata form works be good enough

[coroa]

Tests fail because zenodo is moody again, but otherwise this seems to work great now.

[coroa]

@daniel-rdt would you then please test once more. for me this is done

[daniel-rdt]

@daniel-rdt would you then please test once more. for me this is done

Will do, I am in the train now, so let's see if the internet is good enough. But it should be fine for testing the general implementation.

[daniel-rdt]

Thanks @coroa. I tested your implementation again on the new retrieves introduced in open-energy-transition/open-tyndp#451 and it works now without any errors or bugs. Good to go for me!

[coroa]

Thank you, great.