Conversation
|
This looks like fun - please let me know when you are ready. Are you affiliated with IA in any way? If not, should we approach someone for their opinion? |
|
@martindurant thank you for reaching out. I'm from the Common Crawl Foundation. |
|
I asked Mark Graham @ IA a week ago and he did not reply. (I used to work there.) We want to stress test this PR and also PR #2124 -- I have a 500 TByte dataset of ~10 GByte files that I want to read from IA that ought to be a good stress test. |
|
@martindurant I have asked jjjake to weigh in. |
|
@wumpus Thanks for the heads up on this. My only suggestion is to use IA-S3 keys via Authorization header for everything, and not use cookies at all. You'll just need to make sure that the Authorization header survives the redirect to the data node. Everything else looks good to me. I'll pass this along to the rest of our team in case anyone else has feedback, but the rest looks good to me! Thanks for submitting this, lfoppiano! |
…e data node aiohttp builds a new request from the session's request_class for every hop, so InternetArchiveFileSystem now passes a ClientRequest subclass that sets `Authorization: LOW <access>:<secret>` on any request to a host in `.archive.org`. That both re-attaches the header on the data-node hop and keeps it from leaving archive.org should a data node redirect elsewhere; it is no longer in the session's default headers. Tests: the stand-in archive.org is now two servers on localhost (a different port is a different origin), asserting the header on every hop, and no header when the node is off-domain.
|
The failures seem to be real and persistent - but I don't see what they have to do with changes here. Perhaps update the branch to see if these were already fixed? |
4 participants
Adds
InternetArchiveFileSystem(fsspec/implementations/ia.py, protocol ia):ia://<identifier>/<filename>for files in archive.org items, plus registry entry, API docs, changelog and tests.It remap to HTTP (HTTPFileSystem) with a path mapping and credentials, from https://archive.org/download//, which redirects to a data node that support HTTP Range queries. That is the same URL the
internetarchivepackage downloads from.S3-like API (s3.us.archive.org, which an S3FileSystem + endpoint_url would use): Internet Archive data nodes ignore Range and answer a ranged GET with 200 and the whole file, so every block read would download the entire object. Measured on a 1.6 GB public item (Range: bytes=0-99 on every request):
IA documents this: "HTTP 1.1 Range headers are ignored" (https://archive.org/developers/ias3.html#how-this-is-different-from-normal-s3). botocore also misreads the 307 as an AWS region redirect and recurses on HeadBucket until RecursionError.
Credentials. Public items need none. For restricted items the class reads the ia.ini written by ia configure, found the way the internetarchive package finds it ($IA_CONFIG_FILE, $XDG_CONFIG_HOME/internetarchive/ia.ini, ~/.config/ia.ini, ~/.ia), with IA_ACCESS_KEY_ID/IA_SECRET_ACCESS_KEY overriding the file.
Cookies go into the session's cookie jar scoped to .archive.org rather than a Cookie header, because every download is a cross-origin redirect and aiohttp drops Cookie/Authorization headers there; the jar re-attaches them to the data node. 401/403 raise PermissionError (otherwise HTTPFileSystem._info reports a 403 as FileNotFoundError).
Tested with
fsspec/implementations/tests/test_ia.py.