fix(prow): add retry wrapper around cache step to survive interrupted cache save

[dillon-zheng]

Problem

Multiple TiFlash CI jobs (pull_unit_next_gen, pull_integration_next_gen, pull_integration_test, pull_unit_test) intermittently fail with the same error during the Checkout stage:

java.lang.InterruptedException
    at hudson.remoting.Request.call(Request.java:179)
    at hudson.remoting.Channel.call(Channel.java:1107)
    at hudson.FilePath.act(FilePath.java:1217)
    at CacheStep$CacheStepExecution$1.onSuccess(CacheStep.java:124)

The checkout itself completes successfully, but when the cache{} block exits and the CacheStep plugin tries to save cache state back to the agent (via remoting), the call is interrupted — likely due to spot node preemption or network instability.

Affected builds (last checked):

• pull_unit_next_gen refine makefile and tikv panic rules #68, Add CI tools #69, Automate tidb-test on DBaaS #72
• pull_integration_next_gen make arm image online #85
• pull_integration_test Compile config file v4.0 vs v5.0 #77
• pull_unit_test Unify binary/docker image/branch/repo's rule #67

Root Cause

In checkoutRefsWithCache and checkoutPrivateRefsWithCache, the retry(2) only covers the checkout inside the cache{} block. If the cache save fails (in onSuccess callback), there is no retry protection — the build fails immediately.

Fix

Add retry(3) around the cache{} block so that cache save failures (InterruptedException) trigger a retry. The inner retry(2) is removed since the outer retry already covers both cache and checkout failures.

On retry, the cache restore and checkout are nearly instant (data is already on disk), so the overhead is minimal.

// Before
cache(path: "./", includes: '**/*', key: cacheKey, restoreKeys: restoreKeys) {
    retry(2) {
        checkoutRefs(refs, ...)
    }
}

// After
retry(3) {
    cache(path: "./", includes: '**/*', key: cacheKey, restoreKeys: restoreKeys) {
        checkoutRefs(refs, ...)
    }
}

Validation

• git diff --check passed
• Jenkins pipeline syntax is mechanically valid (single-step replacement, no logic change outside the retry wrapper)

🤖 Generated with Claude Code

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wuhuizuo for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• libraries/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary
This PR addresses intermittent build failures caused by an interrupted cache save step during checkout in several TiFlash CI jobs. The fix introduces a retry wrapper around the entire cache{} block (instead of retrying only the checkout inside the cache block), ensuring that cache save failures trigger a retry. This approach is sound and aligns well with the described root cause. The change is minimal and focused, and the rationale is clearly documented. The code quality is good, but there are a few minor improvements to consider.

---

Critical Issues

• None found. The retry wrapper effectively addresses the interruption issue without introducing obvious regressions.

---

Code Improvements

• Remove redundant retry(2) blocks completely
  The PR removes the inner retry(2) by wrapping cache{} with retry(3). This is good, but ensure no other retry blocks remain nested elsewhere that could cause redundant retries or confusion.

• Consider parameterizing retry counts or adding comments about retry rationale
  The retry count of 3 is chosen to balance overhead and resilience. It may help future maintainers to explain why 3 retries and not more or less, or to extract this to a constant for easier tuning.

• Add handling or logging of retry failures
  Currently, if all retries fail, the failure propagates. It might be useful to add logging inside the retry block to capture failures or add a catch block to provide more context on failure after retries exhausted.

  Example:

  retry(3) {
      try {
          cache(path: "./", includes: '**/*', key: cacheKey, restoreKeys: restoreKeys) {
              checkoutRefs(...)
          }
      } catch (Exception e) {
          echo "Cache save or checkout failed on attempt ${currentAttempt}: ${e.getMessage()}"
          throw e
      }
  }

---

Best Practices

• Add comments inside checkoutRefsWithCache and checkoutPrivateRefsWithCache about retry scope
  Adding a concise comment explaining why the retry wraps the entire cache block (to cover cache save failures) will improve maintainability.

  Example comment:

  // Retry entire cache block to handle intermittent cache save interruptions (e.g., InterruptedException)
  retry(3) {
      cache(...) {
          ...
      }
  }

• Testing coverage
  The PR description does not mention any automated tests or pipeline runs that simulate or verify retry behavior explicitly. Consider adding unit or integration tests if feasible, or at least document manual validation steps or metrics showing reduced failures.

• Naming consistency
  The retry count changed from 2 to 3; confirm that this is consistent with retry counts elsewhere in the pipeline scripts or libraries if applicable.

---

File & lines reviewed:

• libraries/tipipeline/vars/prow.groovy lines 15-29 (both functions updated)

---

Summary of suggestions:

- Add inline comments explaining the retry scope around the cache block.
- Consider adding logging inside the retry block to capture retry attempts and failures.
- Ensure no other nested retries cause redundant retrying.
- If possible, add or document tests validating retry behavior.
- Optionally, parameterize retry count or extract to a constant for configurability.

This PR is a well-targeted fix for a known flaky failure mode and should improve CI stability as intended.