From 6a6e247a0039802abe31590dd3c8ac2afd5df945 Mon Sep 17 00:00:00 2001 From: Matthias Bertschy Date: Mon, 7 Sep 2026 19:19:38 +0200 Subject: [PATCH] experiment(ci): validate storage's ConsolidateTimeSeries-delete shard-routing fix Throwaway validation PR -- do not merge. Follow-up to the residual component-tests write-timeout flakiness investigation (15-17/31 jobs failing even after kubescape/storage#397's sharded single-writer fix). Traced the root cause to deleteProcessedTimeSeries's DeleteContainerProfile call taking a raw pool connection outside storage's write-shard system entirely -- unlike SaveContainerProfile and Create/GuaranteedUpdate, which are all shard-routed. A raw connection has no way to yield to, or be yielded by, a live shard commit, so a genuine collision for SQLite's single writer lock blocks the loser for up to the full busy-timeout (60s in production). Confirmed locally against kubescape/storage (containerprofile_load_test.go, LOAD_CONSOLIDATORS=1): a pure 20-way concurrent write burst alone was flawless (9543/9543 ops), but adding one concurrent ConsolidateTimeSeries pass collapsed throughput by >250x. Routing that one delete call through a new singleWriter.runOnShard primitive restored throughput to ~5000+ ops/8s with near-zero failures in the same repro. This PR points the test storage deployment at that fix, manually built and pushed to quay.io/matthiasb_1/storage:test-shard-delete-fix-965bd1c1 (commit 965bd1c1 on kubescape/storage's fix/route-consolidation-delete- through-shard branch, not yet merged upstream), to empirically confirm it resolves this repo's residual component-tests failures before that fix merges. Expected result: further reduction in failure count from the 17-18/31 baseline (PRs #951/#952), ideally toward the historical 0-1/30 pre-regression baseline. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd --- .github/workflows/component-tests.yaml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/.github/workflows/component-tests.yaml b/.github/workflows/component-tests.yaml index b46c571537..cd9fc5118c 100644 --- a/.github/workflows/component-tests.yaml +++ b/.github/workflows/component-tests.yaml @@ -153,9 +153,14 @@ jobs: kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus -n monitoring --timeout=300s - name: Install Node Agent Chart run: | - STORAGE_TAG=$(./tests/scripts/storage-tag.sh) - echo "Storage tag that will be used: ${STORAGE_TAG}" - helm upgrade --install kubescape ./tests/chart --set clusterName=`kubectl config current-context` --set nodeAgent.image.tag=${{ needs.build-and-push-image.outputs.image_tag }} --set nodeAgent.image.repository=${{ needs.build-and-push-image.outputs.image_repo }} --set storage.image.tag=${STORAGE_TAG} -n kubescape --create-namespace --wait --timeout 10m --debug + # EXPERIMENT: validating a fix for the residual component-tests + # write-timeout flakiness (ConsolidateTimeSeries's Delete bypassing + # the write-shard system) before it merges upstream in + # kubescape/storage. Pins storage to a manually-built test image + # instead of the usual dynamic tag. Do not merge this override. + STORAGE_TAG=test-shard-delete-fix-965bd1c1 + echo "Storage tag that will be used: ${STORAGE_TAG} (EXPERIMENT override, repo quay.io/matthiasb_1/storage)" + helm upgrade --install kubescape ./tests/chart --set clusterName=`kubectl config current-context` --set nodeAgent.image.tag=${{ needs.build-and-push-image.outputs.image_tag }} --set nodeAgent.image.repository=${{ needs.build-and-push-image.outputs.image_repo }} --set storage.image.tag=${STORAGE_TAG} --set storage.image.repository=quay.io/matthiasb_1/storage -n kubescape --create-namespace --wait --timeout 10m --debug # Check that the node-agent pod is running kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=node-agent -n kubescape --timeout=600s sleep 5