Add download mirror functionality#362
Conversation
|
Unable to launch GitHub Actions jobs. ping: @ylatuya @rsanchez87 |
|
Thank you @mdimopoulos . Let me know how to proceed further with this PR. |
|
I was wondering if there is no license restriction to host such test suites in other places than the origin. Any though? |
I can only refer you to the terms and conditions of ISO that apply for most of the test suites of fluster. https://www.iso.org/terms-conditions-licence-agreement.html#Customer-Licence |
In that regard, 9d22655 which is part of #324 now ask users to accept the license before downloading resources. @mdimopoulos is there anything else blocking this PR? |
|
We have to take care this will be integrated cleanly with the logic of: #338 |
Add -m/--mirror option to the download subcommand that rewrites source URLs to point to a local HTTP mirror, with automatic fallback to the original internet source on failure. The mirror URL is constructed by stripping the scheme from the original URL and prepending the mirror base URL, e.g.: - https://storage.googleapis.com/aom-test-data/file.ivf - http://mirror.local:8080/storage.googleapis.com/aom-test-data/file.ivf Changes: - Add rewrite_url() and MIRROR_NETWORK_ERRORS to utils.py - Add mirror parameter with fallback logic to utils.download() - Add -m/--mirror CLI argument to the download subcommand - Propagate mirror through fluster.py and test_suite.py call chain - Add unit and integration tests for mirror URL rewriting and download Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Add scripts/mirror_sync.py which scans test suite JSON files for source URLs and downloads them into a directory tree suitable for serving as a local mirror (e.g. via nginx or python -m http.server). Supports parallel downloads and configurable retries. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Document the --mirror download option and mirror_sync.py script, including how the mirror URL rewriting works, usage examples, and instructions for setting up a mirror server. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
mdimopoulos
left a comment
mdimopoulos
left a comment
There was a problem hiding this comment.
Just an optional but advisable action point. PR can merged without though.
@ahresse feel free to decide on the action point
Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
3 participants
This pull request adds support for downloading test suite resources from a local HTTP mirror, improving speed and efficiency in CI or multi-machine environments. It introduces a new
--mirroroption to the download command, updates the documentation, and adds a script to help set up and sync a local mirror directory. The most important changes are:Local Mirror Support:
Added a
--mirroroption to thefluster.py downloadcommand, allowing users to specify a base URL for a local mirror. Fluster will try downloading from the mirror first and fall back to the original source if the mirror fails. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]Implemented mirror URL rewriting and robust error handling in the download utility, ensuring proper fallback to the original source if the mirror is unavailable or returns an error. [1] [2]
Documentation Updates:
README.mdto document the new--mirroroption, provide usage examples, and explain how to set up and use a local mirror for test suite resources. [1] [2] [3] [4]Mirror Sync Script:
scripts/mirror_sync.py, which scans all test suite JSON files and downloads every referenced resource into a local directory tree, making it easy to set up a mirror server. The script supports parallel downloads and skipping already-downloaded files.These changes make it much easier and faster to run fluster in environments where repeated downloads from the internet are undesirable or slow.