Add retry for downloading install_script_agent

[sarahchen6]

Motivation

dd-trace-java has had multiple flaky failures recently due to the install_script_agent failing to download, likely due to flaky infra issues. e.g.

• https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1857656432
• https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1851603017
• https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1858417690

Changes

Add a retry mechanism when pulling the install script. Fail if the script is not pulled - there's no point in continuing with the test if there's no agent installed.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

utils/build/ssi/base/download_with_retry.sh                             @DataDog/system-tests-core
utils/build/ssi/base/base_ssi.Dockerfile                                @DataDog/system-tests-core
utils/build/ssi/base/base_ssi_installer.Dockerfile                      @DataDog/system-tests-core
utils/build/ssi/base/install_script_ssi.sh                              @DataDog/system-tests-core
utils/build/ssi/base/install_script_ssi_installer.sh                    @DataDog/system-tests-core

[bric3]

Pre-approving, but I agree having dedicated functions could be useful. The backoff with jitter might be useful as well.

[mhdatie]

The premise of the PR makes sense to me - retrying for the install step to make this step less error prone.

The motivation however will not always be the case going forward:

... - there's no point in continuing with the test if there's no agent installed.

I believe the FFE team are adding a mechanism to run system tests without an agent installed. I just want to call this out in case it interferes with the new system tests setup being worked on this week #7298

[datadog-datadog-prod-us1]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ff8bb35 | Docs | Datadog PR Page | Give us feedback!}

[mhdatie]

LGTM, thanks for addressing all the comments!