[FIX] Parallelise per-document chunk listing in S3ChunkDownloader

[noel-improv]

Description

S3ChunkDownloader.download reads extracted documents back from S3. It lists each document's chunks with a separate list_objects_v2 call, and those calls ran one at a time on the main thread. This runs them on a thread pool instead, so the listing no longer serialises. Documents still yield in the same order.

Changes

• List each document's chunk keys concurrently, sized by extraction_num_threads_per_worker (the setting the chunk downloads already use). Documents yield in prefix order. Listing and downloading use separate pools so listings prefetch while the current document downloads.
• Bound the listing with a sliding window of at most extraction_num_threads_per_worker documents in flight, refilled one per document consumed, so listing prefetches ahead without reading the whole collection.
• Download each document's chunks lazily, when the consumer reaches that document, rather than dispatching them at listing time. At most one document's chunk data is resident at a time (the same bound as the original serial code), and abandoning the generator early dispatches no downloads for unconsumed documents.
• Reuse the collection-level paginator for the per-document listing calls instead of rebuilding it per prefix.
• Read chunk_page.get('Contents', []) instead of chunk_page['Contents'], so a prefix that lists empty returns no keys instead of raising KeyError.
• Tests: document and chunk order preservation, a Barrier-based test that the listing runs concurrently, a bounded-window test driven from a consumer thread, and a guard that no later document's chunks download while the consumer is stalled on the first document.

Problem

Reading extracted documents back from S3 was slow. On a 5,000-document collection (16,536 chunk objects) the readback took 578s against 3s from local disk. The per-document listing accounted for 255.7s of that, running as 5,000 sequential round-trips on the main thread while the downloads already used a thread pool.

Related issue (if any): #

Testing

• Unit tests added/updated
• Integration tests added (as appropriate)
• Existing tests pass (pytest)
• Tested manually (describe below)

169 tests in tests/unit/indexing/load/ pass, including the four added here. The concurrency test fails with BrokenBarrierError against serial listing, and the lazy-dispatch guard fails against eager download dispatch (later documents' chunks download while the consumer is stalled on document 0), so they guard the behaviour rather than passing vacuously.

Measured against the same collection, listing isolated from downloads:

threads	listing time
serial	255.7s
16	16.5s
32	8.3s
64	4.4s

Listing is I/O-bound, so raising extraction_num_threads_per_worker above its default of 4 increases the gain; at the default it is smaller. This covers the listing part of readback. The chunk downloads (~322s) are unchanged.

Checklist

• Code follows existing style and conventions
• License headers present on new files
• Documentation updated (if applicable)
• No breaking changes (or clearly documented)

Public behaviour is unchanged: the same SourceDocument objects yield in the same order. No new files.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.