[EXPERIMENT] Validate kubescape/storage's ConsolidateTimeSeries-delete shard-routing fix

[matthyx]

Purpose

Throwaway validation PR — do not merge.

Follow-up to the residual component-tests write-timeout flakiness investigation (15-17/31 jobs failing even after kubescape/storage#397's sharded single-writer fix — see PRs #952/#953 for the CPU-starvation and pod-restart hypotheses, both ruled out).

Traced the actual root cause into kubescape/storage: ConsolidateTimeSeries's deleteProcessedTimeSeries step deletes each processed time-series profile via DeleteContainerProfile, which takes a raw pool connection outside storage's write-shard system entirely — unlike SaveContainerProfile and Create/GuaranteedUpdate's own commits, which are all routed through the 8-shard system (#397). A raw connection has no way to yield to, or be yielded by, a live shard commit, so a genuine collision for SQLite's single writer lock blocks the loser for up to the full busy-timeout (60s in production) instead of the microsecond in-process channel wait every shard-routed write enjoys.

Confirmed locally against kubescape/storage (containerprofile_load_test.go, LOAD_CONSOLIDATORS=1): a pure 20-way concurrent write burst alone was flawless (9543/9543 ops, p99=115ms), but adding one concurrent ConsolidateTimeSeries pass collapsed throughput by >250x (36 ops/8s, 14 failed, multi-second database is locked stalls) — the same fast, heterogeneous failure shape seen in this repo's CI, not the original catastrophic hang.

The fix: a new singleWriter.runOnShard(ctx, key, priority, fn) primitive routes that one delete call through the same shard a live commit for that key would use. (An earlier attempt at wrapping all of ConsolidateTimeSeries's per-key work this way self-deadlocked — it calls SaveContainerProfile for the same key/shard partway through — caught by kubescape/storage's own test suite before this PR, fixed by only routing the leaf delete operation.) Result in the same local repro: ~5000+ ops/8s, near-zero failures, p50 in microseconds and p99 ~5ms (down from 9.9s).

This PR points the test storage deployment at that fix, manually built and pushed to quay.io/matthiasb_1/storage:test-shard-delete-fix-965bd1c1 (commit 965bd1c1 on kubescape/storage's fix/route-consolidation-delete-through-shard branch, not yet merged upstream), to empirically confirm it resolves this repo's residual failures before that fix merges — the same methodology used throughout this investigation.

Test plan

• component-tests run on this PR — compare failure count against the 17/31 ([EXPERIMENT] Validate real kubescape/storage v0.0.333 release #951) and 18/31 ([EXPERIMENT] Test storage pod CPU starvation hypothesis for residual component-tests flakiness #952) baselines, ideally toward the historical 0-1/30 pre-regression baseline

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

AI-skills: none

Summary by CodeRabbit

• Chores
  • Updated component-test infrastructure to use a fixed storage image for more consistent validation.
  • Added additional installation diagnostics to help investigate intermittent test failures.
  • No user-facing product behavior changes are included in this update.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: af9796c9-21b5-41a0-830d-fe4a6ae0844b

📥 Commits

Reviewing files that changed from the base of the PR and between 1c450e5 and 6a6e247.

📒 Files selected for processing (1)

• .github/workflows/component-tests.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The component-tests workflow now installs Node Agent with a manually built storage image. The Helm command sets the image repository explicitly and enables debug output for experimental validation.

Changes

Component Test Storage Override

Layer / File(s)	Summary
Configure experimental storage image .github/workflows/component-tests.yaml	The workflow uses the fixed test-shard-delete-fix-965bd1c1 storage tag, sets quay.io/matthiasb_1/storage as the repository, and adds the Helm --debug flag.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: 🟡 Moderate · up to 6a6e2

Component tests now run an experimental storage image rather than the version-compatible image normally selected by the workflow. Compatibility should be established before merging so test results remain reliable and the workflow does not fail at runtime.

Suggested reviewers: entlein

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the experimental validation of the ConsolidateTimeSeries delete shard-routing fix, which matches the primary purpose of the changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch experiment/validate-consolidation-delete-shard-fix

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[matthyx]

Result: 13/31 failed — a real improvement over the 17/31 (#951) and 18/31 (#952) baselines, but not the near-total elimination the local repro suggested.

The targeted fix is confirmed working in real CI, not just locally. Checked "database is locked" occurrence counts across the failing jobs: 0, 2, 3, 4 — down from being the dominant, systemic failure mode. The specific mechanism (ConsolidateTimeSeries's Delete bypassing storage's write-shard system) is real and this fix genuinely reduces it in production-like conditions, not just the synthetic local load test.

However, the remaining 13 failures are overwhelmingly a different, unrelated flakiness source, now more visible with the storage contention mostly out of the way:

• Test_02: Expected alert 'Syscalls Anomalies in container' was not signaled — rule-engine/alert-detection timing, nothing to do with storage
• Test_08: Condition never satisfied — JSON-patch of the ContainerProfile must be accepted by storage — timing, not a lock error
• Test_11: Expected endpoint ... not found in the container profile — network/endpoint-detection timing
• Test_14: plain Not equal assertion mismatch

None of these show the write-timeout/lock-contention signature. This looks like a genuinely separate flakiness population in node-agent's own detection/reporting pipeline (eBPF event timing, rule evaluation latency), independent of storage's write path entirely — worth its own fresh investigation, but out of scope for this storage-locking thread.

Conclusion: the storage fix (kubescape/storage's fix/route-consolidation-delete-through-shard, commit 965bd1c1) is validated against real CI and worth landing as a real PR upstream. It measurably helps but does not fully resolve node-agent's residual component-tests flakiness — that requires a separate investigation into the detection-timing failures surfaced above.

Closing this throwaway validation PR. Not deleting the storage fix's branch (kept for opening the real kubescape/storage PR).

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com