[WIP] ci: refactor main branch cache strategy#1717
Conversation
Keep the persistent self-hosted compiler cache local, bound hosted-runner CCache uploads, and make Conan caches reusable by runner class and target branch. Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: jamesgao-jpg The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@jamesgao-jpg 🔍 Important: PR Classification Needed! For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:
For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”. Thanks for your efforts and contribution to the community!. |
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Question
How can Knowhere keep fast compilation without uploading persistent self-hosted runner caches, while still giving ephemeral GitHub-hosted jobs useful and reusable caches?
The current workflow hard-codes
~/.ccache. OnGithub-Action-Runner-Knowhere-2, the Actions service runs asubuntuand CCache reports/home/ubuntu/.cache/ccache; its CCache and Conan homes are both persistent and multi-GB. Transferring those local caches through GitHub adds latency and consumes repository cache quota without providing persistence.Conan cache keys also include the job and CPU feature bucket. That duplicates large dependency archives across hosted jobs and prevents maintained branches from having explicit cache identities.
Solution
mainruns seed caches reusable by later PRs.Cardinal PR https://github.com/zilliztech/cardinal/pull/887 introduces the related runner-class and target-branch cache strategy for Cardinal. Cardinal PR https://github.com/zilliztech/cardinal/pull/889 applies the same persistent-local-cache policy to its self-hosted
2.6compatibility job.Verification
git diff --checkrunner-classandcache-branchubuntuwith persistent 6.4 GB Conan and 5.2 GB CCache directories