Skip to content

[WIP] ci: refactor 2.6 cache strategy#1718

Open
jamesgao-jpg wants to merge 2 commits into
zilliztech:2.6from
jamesgao-jpg:codex/knowhere-26-cache-refactor
Open

[WIP] ci: refactor 2.6 cache strategy#1718
jamesgao-jpg wants to merge 2 commits into
zilliztech:2.6from
jamesgao-jpg:codex/knowhere-26-cache-refactor

Conversation

@jamesgao-jpg

@jamesgao-jpg jamesgao-jpg commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Question

How can Knowhere 2.6 warm reusable release-branch caches while matching the cache behavior and code style introduced for main?

The 2.6 Unit Test and Analyzer workflows currently warm on pushes to main, so direct pushes to 2.6 produce neither branch runs nor refs/heads/2.6 caches. Existing cache keys are also split by job and assume a fixed CCache directory.

Solution

  • Warm Unit Test and Analyzer caches on direct pushes to 2.6.
  • Preserve all existing GitHub-hosted runner assignments.
  • Detect CCache's configured directory, keep compiler caches job-specific, and cap uploaded CCache at 1 GB.
  • Scope CCache and Conan 1 keys by stable runner class and target branch.
  • Share Conan across jobs and remove job fragmentation from new keys.
  • Start the new 2.6 cache namespace cold instead of retaining a migration-only OS/job fallback; successful 2.6 runs seed caches reusable by later PRs.
  • Update every cache caller, including UT, wheel, Analyzer, and Release.
  • Install CCache before Analyzer resolves its cache directory.
  • Document the rationale beside branch scoping, cold-start behavior, dynamic path detection, and size limiting.

Knowhere PR #1717 introduces the equivalent policy on main and is the direct behavior/style baseline for this PR. Cardinal PR https://github.com/zilliztech/cardinal/pull/889 applies the independently implemented and verifier-compared 2.6 counterpart; Cardinal PR https://github.com/zilliztech/cardinal/pull/887 provides the related Cardinal master change.

Known tradeoff

The shared Conan cache is immutable after the first producer saves it. On a cold branch cache, a non-UT job may save before UT/Analyzer and omit Catch2. Builds remain correct and common dependencies remain reusable, but Catch2 may be downloaded again until the Conan key changes. This matches the accepted policy in PR #1717; selecting one dependency-superset Conan producer is a possible follow-up.

Verification

  • Independent worker implementation followed by a separate cross-repository verifier agent
  • Verifier confirmed semantic and style parity with Cardinal 2.6 and Knowhere main
  • Restore/save key symmetry checked programmatically
  • All composite callers provide required inputs
  • No legacy OS/job restore references remain
  • All modified YAML parses successfully
  • Scoped Actionlint reports no cache-refactor errors
  • git diff --check
  • DCO sign-off verified exactly

Warm branch-scoped caches on 2.6 and align hosted CCache and Conan key behavior with the main-branch cache policy.

Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
@sre-ci-robot
sre-ci-robot requested a review from liliu-z July 11, 2026 14:21
@sre-ci-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign alexanderguzhva after the PR has been reviewed.
You can assign the PR to them by writing /assign @alexanderguzhva in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify

mergify Bot commented Jul 11, 2026

Copy link
Copy Markdown

@jamesgao-jpg 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

  1. If you're fixing a bug, label it as kind/bug.
  2. For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
  3. Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
  4. Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

@jamesgao-jpg jamesgao-jpg changed the title ci: refactor 2.6 cache strategy [WIP] ci: refactor 2.6 cache strategy Jul 11, 2026
@mergify mergify Bot added the ci-passed label Jul 11, 2026
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
@foxspy

foxspy commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

The CCache directory lookup can fail without failing the composite action in both .github/actions/cache-restore/action.yaml:31 and .github/actions/cache-save/action.yaml:33:

echo "dir=$(ccache --get-config cache_dir)" >> "${GITHUB_OUTPUT}"

Because the simple command is echo, its successful exit status masks a non-zero status from ccache, even with set -euo pipefail. This can silently publish an empty path and defer the failure to the cache step.

Please split the lookup from the output write so the query failure propagates:

cache_dir="$(ccache --get-config cache_dir)"
echo "dir=${cache_dir}" >> "${GITHUB_OUTPUT}"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants