Add Hugging Face filesystem support to fsspec

[lhoestq]

Rationale for this change

Add support for the Hugging Face filesystem in fsspec, which uses hf:// paths.
This allows to import HF datasets.

Authentication is done using the "hf.token" property.

Are these changes tested?

I tried locally but haven't added tests in test_fsspec.py (lmk if it's a requirement)

Are there any user-facing changes?

No changes, it simply adds support for hf:// URLs

[Fokko]

Hey @lhoestq Thanks for raising this PR. I think this is super interesting!

I think the PR needs a couple more things:

• Add the dependency to pyproject.toml.
• Add some docs.

[lhoestq]

I updated pyproject.toml and added some docs :)

PS: I also added the "hf" extra in pyproject.toml, lmk if this is fine

[kevinjqliu]

LGTM! Thanks for adding this.

[kevinjqliu]

is there a way to allow PyArrowFileIO as well?

[lhoestq]

There is no HF filesystem implementation in arrow C++ yet unfortunately ! But hopefully soon

[kevinjqliu]

#24 34.52 E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/o/openjdk-11/openjdk-11-jdk-headless_11.0.26%2b4-1%7edeb11u1_amd64.deb Error reading from server - read (104: Connection reset by peer) [IP: 146.75.30.132 80]

just retriggered CI

[Fokko]

Also retriggered the CI 😄

[lhoestq]

all green ! thanks

[kevinjqliu]

Thanks @lhoestq for the contribution and @Fokko for the review :)