feat: deduplicate shared URL downloads across test suites

[dabrain34]

Introduce a centralized DownloadManager that ensures each URL is downloaded at most once, eliminating duplicate downloads both across test suites and within a single test suite.

• Add DownloadManager class in utils.py with download-once caching and centralized archive cleanup
• Refactor TestSuite.download() to use pre-downloaded archives from the manager across all three download paths
• Use a thread pool to download concurrently and make DownloadManager thread-safe so duplicate URLs are still fetched only once.

This feature allows to fast up considerably the download of AV1-ARGON* which was downloading each time the 6GB archive for every test vector.

Fix #309

[dabrain34]

@ylatuya ping

[dabrain34]

@rsanchez87 can you have a look to this PR as well? The idea would be to fast up the build of docker images containing all the test suites

[ylatuya]

We use Pool from multiprocessing

[dabrain34]

The idea is to move all the logic for download pool inside the download manager which handled the dedup, the thread lock, the extraction and everything to download through ThreadPoolExecutor.

Do you have a preference for multiprocessing for a reason ?

[rsanchez87]

@rsanchez87 can you have a look to this PR as well? The idea would be to fast up the build of docker images containing all the test suites

@dabrain34, tested with python3 fluster.py download AV1-ARGON-PROFILE0-CORE-ANNEX-B AV1-ARGON-PROFILE1-CORE-ANNEX-B AV1-ARGON-PROFILE2-CORE-ANNEX-B
master: 49m 40s
PR: 16m 2s (~3x faster, ZIP downloaded once instead of 3 times) ✅

Also regression tests ✔️

I’ll test again once the requested changes by @ylatuya are implemented. Thanks!

[dabrain34]

thanks for the test, indeed this is even better on low speed lines as we dont redownload all the time the AV1 zip file.

I'm currently addressing comments from ylatuya. When this is ready I will come back to you

[1ace]

@dabrain34 @ylatuya was the question of using threads vs processes resolved in another channel perhaps?

Is there anything else blocking this from being merged?

[ylatuya]

@dabrain34 I think the current implementation is overly complicated for what it's trying to achieve, the download logic is now spread in the DownloadManager and the TestSuite itself.

My proposal in the previous review may not have been expressed clearly, but what I was proposing is to move all the existing download logic from TestSuite, which already supports parallel downloads and deduplication, into a new DownloadManager. The new DownloadManager.download should now accept a list of test suites to support deduplication and parallel downloads over multiple test suites. The DownloadManager should also manage a shared cache for downloaded sources.

Fluster.download_test_suites uses the DownloadManager directly, from this:

     for test_suite in download_test_suites:
            test_suite.download(
                jobs,
                self.resources_dir,
                verify=True,
                keep_file=keep_file,
                retries=retries,
            )

To this:

    download_manager = DownloadManager(jobs, self.resources_dir, verify=True, keep_file=keep_file, retries=retries)
    dowload_manager.download(download_test_suites)

TestSuite.download uses the manager as well (I think we can even remove this method from TestSuite, it shouldn't be needed)

    def download(
        self,
        jobs: int,
        out_dir: str,
        verify: bool,
        extract_all: bool = False,
        keep_file: bool = False,
        retries: int = 2,
    ) -> None:
        """
             Download the test suite resources.
        """
        manager = DownloadManager(out_dir, verify, extract_all, keep_file, retries)
        manager.download([self], jobs)

This approach simplifies things since all the download logic is kept in a single place, the DonwloadManager, and than Fluster uses that to download test suites.

[ylatuya]

@dabrain34, this is an example of how I would like the implementation to look like ylatuya@4bfa256. (There seems to be a bug in Github that does not include the changes in utils.py in the diff)

That way, we keep all the download logic in a single place, the DownloadManager, so that new features like #362 can be implemented in a single place.

[dabrain34]

I'm about to push something where utils.py has been replaced by download_manager.py

[dabrain34]

I think I addressed your request by moving all the downloadmager code to a specific file and move the test suite download to this module as well. Please give it a try and let me know.

[dabrain34]

so you merged #362 first which leads this PR to conflicts (again).

I think it would have been better to review and merge this first as it was modifying the download mechanism for real performance improvement in comparison to mirror.

The AV1 test suite download repeats the 6.5GB DL on every test vector, it has to be improved for CI purpose, see Mesa CI.

[dabrain34]

Please let me know if I should invest more time on rebasing and resolving conflicts as it might enter your roadmap.