[EXPERIMENT] Test storage pod CPU starvation hypothesis for residual component-tests flakiness

[matthyx]

Purpose

Diagnostic experiment, not a proposed fix — do not merge.

Follow-up to the storage single-writer investigation (kubescape/storage#397, released v0.0.333). That fix eliminated the sustained 20-minute hangs, but component-tests still shows an elevated failure rate: 15-17/30 jobs failing (confirmed fresh today on #951: 17/31), vs a historical pre-regression baseline of 0-1/30.

#950 tested whether the residual failures were a shard/pool headroom problem (singleWriterShards 8→16, sqlitePoolSize 10→24) — result was 21/32 failed, worse, not better. That rules out shard/pool count as the lever and points at the storage pod's CPU quota instead: tests/chart/values.yaml's storage.resources has stayed at requests.cpu: 100m / limits.cpu: 500m across every experiment so far. Under a starved CPU quota, more concurrent write-path goroutines (the 8 fixed shards) add scheduling overhead without proportional throughput gain, which would explain why more shards didn't help.

This PR bumps storage.resources.requests.cpu 100m→500m and limits.cpu 500m→2000m, with no shard-count or other change, to isolate this variable. Looking for:

• Overall failure count vs the 17/31 baseline ([EXPERIMENT] Validate real kubescape/storage v0.0.333 release #951, same day)
• Whether storage-side context deadline exceeded / sqlite: step: interrupted errors specifically disappear from logs (vs the heterogeneous non-storage assertion failures, which this change should NOT affect)

Test plan

• component-tests run on this PR — compare failure count and error signature against [EXPERIMENT] Validate real kubescape/storage v0.0.333 release #951's 17/31 baseline

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd

AI-skills: none

Summary by CodeRabbit

• Chores
  • Increased the CPU capacity allocated to the storage component.
  • Memory allocation remains unchanged.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 4593f9cb-0ac5-4353-ba80-b5d974b60f50

📥 Commits

Reviewing files that changed from the base of the PR and between 1c450e5 and 8fa08e1.

📒 Files selected for processing (1)

• tests/chart/values.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The test chart increases the storage component’s CPU request from 100m to 500m and its CPU limit from 500m to 2000m. Memory settings remain unchanged.

Changes

Storage CPU Resources

Layer / File(s)	Summary
Update storage CPU resources tests/chart/values.yaml	The storage component now requests 500m CPU and permits up to 2000m CPU. Memory settings are unchanged.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 8fa08

This updates the storage workload used by component tests to request and allow more CPU for the starvation experiment. No concrete merge-blocking risk remains.

Suggested reviewers: alonliwsky

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the diagnostic experiment and its focus on storage pod CPU starvation as a possible cause of component-test flakiness. This matches the CPU resource changes in the pull req…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch experiment/storage-cpu-headroom

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Performance Benchmark Results

Node-Agent Resource Usage

Metric	BEFORE	AFTER	Delta
Avg CPU (cores)	0.175	0.176	+0.7%
Peak CPU (cores)	0.187	0.186	-0.2%
Peak CPU p95 (cores)	0.183	0.186	+1.7%
Avg Memory (MiB)	400.798	306.362	-23.6%
Peak Memory (MiB)	405.730	310.586	-23.5%

Dedup Effectiveness

No data available.

[matthyx]

Result: 18/31 failed — essentially identical to #951's same-day 17/31 baseline (if anything one worse), despite the storage pod's CPU limit going 500m→2000m (4x) and request 100m→500m (5x), confirmed applied in the actual helm install log for this run.

Hypothesis 1 (storage pod CPU starvation) is refuted. More CPU headroom on the storage pod's own cgroup made no measurable difference to either the failure count or the error signature.

Traced one failing job (Test_01_BasicAlertTest) end-to-end: the container profile write path itself is timing out — "failed to create container profile, requeuing", error: "Timeout: request did not complete within requested timeout - context deadline exceeded" — with the internal write queue visibly backing up (queueSize: 1 → queueSize: 5), causing the profile to never reach status: completed inside the test's window. Storage's own pod logs show no explicit errors (no sqlite: step: interrupted this time) — the timeouts are happening client-side against a server that isn't logging why it's slow.

New leading hypothesis: the bottleneck is node-level CPU oversubscription on the shared GitHub-hosted runner, not the storage pod's own cgroup limit. Each CI job's whole stack (Kind control-plane, storage, node-agent, kube-prometheus-stack's ~5 pods, kubelet/containerd, the go test process) shares one runner's fixed core count. Raising storage's limit raises its ceiling but can't manufacture free CPU cycles on an already-saturated node — which would also explain why #950's 8→16 shard bump made things no better (more goroutines contending for the same starved node, not the pod's own quota).

Not pursuing further per-pod resource tuning here. Closing this experiment; next step should instrument kubectl top nodes/kubectl describe node (or reduce the monitoring stack) during a live run to confirm node-level saturation before proposing a fix.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com