[EXPERIMENT] Validate combined storage shard-writer fixes (delete + WriteTimeSeriesEntry)

[matthyx]

Purpose

Throwaway validation PR — do not merge.

Follow-up to #954 (closed): that PR validated routing ConsolidateTimeSeries's Delete through storage's write-shard system, which reduced failures from 17-18/31 to 13/31 and dropped database is locked occurrences to near-zero per job.

Re-tracing the remaining 13 failures' logs (Test_02, Test_08, Test_11 — initially assessed as an unrelated node-agent-side flakiness source, since their own assertion messages don't mention storage) found the same raw-connection bypass in AfterCreate's WriteTimeSeriesEntry call — fired on essentially every TS ContainerProfile create, i.e. most of node-agent's write traffic — surfacing as sqlite: step: interrupted instead of database is locked.

This PR points the test storage deployment at kubescape/storage#399 with a second commit applied (routing WriteTimeSeriesEntry through the same runOnShard mechanism), manually built and pushed to quay.io/matthiasb_1/storage:test-shard-writer-fixes-f3bf94e5, to empirically confirm the combined fix further reduces this repo's residual failures.

Local repro (containerprofile_load_test.go, LOAD_CONSOLIDATORS=1) with both fixes: p50=55µs, p99~4ms, only 1/8s of ops over 5s — improved on the first fix's already-good numbers.

Test plan

• component-tests run on this PR — compare failure count against 17/31 ([EXPERIMENT] Validate real kubescape/storage v0.0.333 release #951), 18/31 ([EXPERIMENT] Test storage pod CPU starvation hypothesis for residual component-tests flakiness #952), and 13/31 ([EXPERIMENT] Validate kubescape/storage's ConsolidateTimeSeries-delete shard-routing fix #954)

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

AI-skills: none

[coderabbitai]

Warning

Review limit reached

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: f9d9294f-8885-42d7-89a1-99091457e160

📥 Commits

Reviewing files that changed from the base of the PR and between 1c450e5 and bb00afc.

📒 Files selected for processing (1)

• .github/workflows/component-tests.yaml

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[matthyx]

Result: 17/31 failed — worse than #954's 13/31, roughly back to the original 17/31 baseline (#951). Mixed signal, not a clean win this time.

The targeted write-path errors stayed low, consistent with the fix still working: checked 3 failing jobs — database is locked (0-2 occurrences), sqlite: step: interrupted (0-6 occurrences) — nowhere near the systemic levels from before either fix landed.

But apiserver Handler timeout noise jumped substantially (108-300 occurrences per job, vs ~23 in #954's Test_08). Traced these: they're GET-resource-handler timeouts, mostly for unrelated cluster-infra pods (same background-polling noise identified earlier in this investigation), not specific to the WriteTimeSeriesEntry path this PR changed. Test_08's own failure this run was a PATCH request timing out over its full 30s retry window — the same symptom class as before, but the underlying storage error signature for it stayed low, so the actual bottleneck this run isn't obviously the write-lock contention either fix targeted.

Honest read: given this whole investigation has already seen 13-20/31 swings across nominally-similar runs (CI noise: runner variance, GC pauses, GH Actions infra), this could be normal variance rather than a regression introduced by the WriteTimeSeriesEntry fix — the specific error signature it targets did NOT come back. But I can't rule out a subtler interaction (e.g. the extra s.locks acquisition per create adding marginal read contention) without another run or more data, and I'm not confident enough either way to call this a clean confirmation.

Not chasing this further in isolation — a deeper, more careful pass (comprehensive audit of every remaining bypass, including the harder atomicity problem in processTimeSeriesInTransaction) is already underway via a planning workflow. Closing this throwaway PR; keeping the storage fix branch (both commits) as-is — they're validated locally and the write-path-specific evidence still supports them.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com