feat: throw error on partial downloads

[mfroach]

If we receive a partial download due to network cutoff or truncation, with all valid records, no warning is thrown about records skipped and the user has no way of knowing the data is incomplete. Added a check for matching reported size and received size of dataset.

[wumpus]

This is an improvement -- I had no idea fsspec would just return EOF -- but can we also set the retry strategy for fsspec while we're at it? Looks like we can install aiohttp-retry and then use it like this:

import fsspec
from aiohttp_retry import RetryClient, ExponentialRetry

def custom_retry_client(**kwargs):
    """
    Creates a retry client tailored to catch specific HTTP status codes.
    """
    retry_options = ExponentialRetry(
        attempts=5,                     # Number of retries
        start_timeout=1.0,              # Initial backoff wait in seconds
        max_timeout=30.0,               # Maximum backoff ceiling
        factor=2.0,                     # Backoff multipliers
        statuses={403, 429, 503}        # Target specific failure status codes
    )
    # aiohttp_retry.RetryClient wraps around an aiohttp.ClientSession
    return RetryClient(retry_options=retry_options, **kwargs)

# Instantiate the filesystem using your tailored retry rules
fs = fsspec.filesystem("https", get_client=custom_retry_client)

with fs.open("https://example.com", "rb") as f:
    data = f.read()

If instead you are using s3 then there's a similar boto3 config.

@lfoppiano can you please have a look? I think our recent warcio changes have a similar problem.

[wumpus]

Oh @mfroach I probably should have started by asking you what WARC you were reading -- https or s3, our bucket or someone elses, etc etc.

[mfroach]

Oh @mfroach I probably should have started by asking you what WARC you were reading -- https or s3, our bucket or someone elses, etc etc.

I only wrote this to catch HTTP failures. As far as I can tell, s3fs handles this case with automatic retry or throws an exception.
I like the idea of bringing in aiohttp-retry though.

[lfoppiano]

Thanks @mfroach, @wumpus.

I've added some changes which integrate Greg's suggestion, is it OK if I push them directly on your fork, @mfroach and you can double check it? (you should make sure that "maintainer can modify/write" is enabled for your PR).

(Suggestion) For the (hopefully) next PR, perhaps you should create a branch on your fork.

[mfroach]

@lfoppiano please go ahead and push, yes! I will take a look. Sorry I forgot to create a new branch!

[lfoppiano]

@mfroach pushed. Thank you!

[mfroach]

Awesome. This looks much more robust. The added test is particularly good. Thanks @lfoppiano!

[lfoppiano]

👍👍