From bb00afc48e681442aa77ca0df00e3a9d29b08571 Mon Sep 17 00:00:00 2001 From: Matthias Bertschy Date: Mon, 7 Sep 2026 20:18:12 +0200 Subject: [PATCH] experiment(ci): validate storage's combined shard-writer fixes (delete + WriteTimeSeriesEntry) Throwaway validation PR -- do not merge. Follow-up to #954 (closed): that PR validated routing ConsolidateTimeSeries's Delete through storage's write-shard system, which reduced failures from 17-18/31 to 13/31 and dropped "database is locked" occurrences to near-zero. Re-tracing the remaining 13 failures' logs found the SAME raw-connection bypass in AfterCreate's WriteTimeSeriesEntry call (fired on essentially every TS ContainerProfile create), surfacing as "sqlite: step: interrupted" -- initially misread as unrelated node-agent-side flakiness since the failing tests' own assertions don't mention storage. This PR points the test storage deployment at kubescape/storage's fix/route-consolidation-delete-through-shard branch (commit f3bf94e5, not yet merged upstream) with BOTH fixes applied, manually built and pushed to quay.io/matthiasb_1/storage:test-shard-writer-fixes-f3bf94e5, to empirically confirm the combined fix further reduces this repo's residual failures. Local repro (containerprofile_load_test.go, LOAD_CONSOLIDATORS=1) with both fixes: p50=55us, p99~4ms, only 1/8s of ops over 5s -- improved on the first fix's already-good numbers. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd --- .github/workflows/component-tests.yaml | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/.github/workflows/component-tests.yaml b/.github/workflows/component-tests.yaml index b46c571537..aa7ff5b754 100644 --- a/.github/workflows/component-tests.yaml +++ b/.github/workflows/component-tests.yaml @@ -153,9 +153,16 @@ jobs: kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus -n monitoring --timeout=300s - name: Install Node Agent Chart run: | - STORAGE_TAG=$(./tests/scripts/storage-tag.sh) - echo "Storage tag that will be used: ${STORAGE_TAG}" - helm upgrade --install kubescape ./tests/chart --set clusterName=`kubectl config current-context` --set nodeAgent.image.tag=${{ needs.build-and-push-image.outputs.image_tag }} --set nodeAgent.image.repository=${{ needs.build-and-push-image.outputs.image_repo }} --set storage.image.tag=${STORAGE_TAG} -n kubescape --create-namespace --wait --timeout 10m --debug + # EXPERIMENT: validating a second, combined fix for the residual + # component-tests write-timeout flakiness (routing both + # ConsolidateTimeSeries's Delete AND AfterCreate's + # WriteTimeSeriesEntry through the write-shard system) before it + # merges upstream in kubescape/storage. Pins storage to a + # manually-built test image instead of the usual dynamic tag. + # Do not merge this override. + STORAGE_TAG=test-shard-writer-fixes-f3bf94e5 + echo "Storage tag that will be used: ${STORAGE_TAG} (EXPERIMENT override, repo quay.io/matthiasb_1/storage)" + helm upgrade --install kubescape ./tests/chart --set clusterName=`kubectl config current-context` --set nodeAgent.image.tag=${{ needs.build-and-push-image.outputs.image_tag }} --set nodeAgent.image.repository=${{ needs.build-and-push-image.outputs.image_repo }} --set storage.image.tag=${STORAGE_TAG} --set storage.image.repository=quay.io/matthiasb_1/storage -n kubescape --create-namespace --wait --timeout 10m --debug # Check that the node-agent pod is running kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=node-agent -n kubescape --timeout=600s sleep 5