Skip to content

Support for ia:// (without need of ffspec dependency) - #22

Merged
lfoppiano merged 3 commits into
mainfrom
feat/ia-scheme
Sep 16, 2026
Merged

lfoppiano merged 3 commits into
mainfrom
feat/ia-scheme

Conversation

@lfoppiano

@lfoppiano lfoppiano commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

This PR implements the issue #16 (it ships additional changes that are supposed to go into ffspec).

The implementation does rely on the HTTP backend, and not on the S3 one. The reason is due to the limitation in the IA service that does not allow ranged queries.

The workaround (implemented) is to not use IA's S3 endpoint at all. ia://item/file is rewritten to https://archive.org/download/item/file and read as ordinary HTTP.

Requesting a resource via the s3 path, the service returns 200 OK with the full 1.6 GB body instead of 206:

Request Answer
curl -r 0-99 to the data node directly, http 200, Content-Length 1689855303
Same over https 200, full length
Explicit Range: bytes=0-99 header through the front end with -L 307, then, when following the redirect 200 with full length

The IA service provides Accept-Ranges: bytes, but IA's own documentation states "HTTP 1.1 Range headers are ignored" (see: https://archive.org/developers/ias3.html#how-this-is-different-from-normal-s3)

I could not find the sources of the IA service, but looks like there are other incompatibilities with S3 (307 mismatch with the S3 convention). Result: using botocore will result in having thrown an exception of unbounded recursion.

@lfoppiano
lfoppiano changed the base branch from main to feat/test-suite September 8, 2026 07:47
@lfoppiano
lfoppiano changed the base branch from feat/test-suite to feature/create-testing-warcs September 8, 2026 07:50
@lfoppiano
lfoppiano changed the base branch from feature/create-testing-warcs to feat/test-suite September 8, 2026 07:50
@lfoppiano lfoppiano changed the title Support for ia:// Support for ia:// (without need of ffspec dependency) Sep 9, 2026
@lfoppiano
lfoppiano marked this pull request as ready for review September 9, 2026 07:43
@wumpus
wumpus self-requested a review September 14, 2026 15:30
Base automatically changed from feat/test-suite to main September 14, 2026 15:46
@lfoppiano
lfoppiano force-pushed the feat/ia-scheme branch 2 times, most recently from 3e1c443 to 04f37aa Compare September 15, 2026 20:05
@lfoppiano
lfoppiano merged commit e0fac53 into main Sep 16, 2026
5 checks passed
@lfoppiano lfoppiano linked an issue Sep 16, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add ia:// support to warc2zip and eventually fsspec

2 participants