Conversation
Follow-up to the sharded single-writer validation (kubescape/storage#397): default 8 shards / 10 pool connections reduced failures substantially (15/30 vs 18-19/30 baseline, and no more sustained 20min hangs) but not to near-zero. Residual failures show occasional "sqlite: step: interrupted" errors and context-deadline timeouts under concurrent write bursts, suggesting 8 shards / pool size 10 may not give enough headroom for CI's actual burst size. Testing 16 shards / 24 pool connections via config.json overrides (no new storage image needed -- both are config-driven) to see if more headroom resolves the residual flakiness or if it points to a separate contributing cause. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LLN3CgGftcnAikV13qD6yJ
matthyx
added
the
ai-assisted
|
Warning Review limit reachedNext included review available in 17 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Result: 21 failed / 32 total — worse than #949's 8-shard/pool-10 result (15/30), not better. More shards and more pool connections did NOT reduce the residual flakiness; if anything it got slightly worse. Conclusion: the residual failures are not a shard/pool-headroom problem. My leading hypothesis now: the storage pod's CPU limit in tests/chart/values.yaml (500m, unchanged in this experiment) is the actual constraint — under a starved CPU quota, more concurrent goroutines add scheduling overhead without proportional throughput gain, which would explain why 16 shards performed no better (or slightly worse) than 8. The pod-churn/restart pattern observed in #949 is also still a candidate, independent of storage entirely. Not pursuing further shard/pool tuning — closing this experiment. |
Performance Benchmark ResultsNode-Agent Resource Usage
Dedup EffectivenessNo data available. |
1 participant
Purpose
This is a diagnostic experiment, not a proposed fix — do not merge.
Follow-up to #949 (default 8-shard validation of kubescape/storage#397): that run showed a big real improvement (15/30 failures vs 18-19/30 baseline, no more sustained 20-minute hangs, individual write latency dropped from minutes to seconds) but not the hoped-for near-zero. Traced the residual failures to occasional
context deadline exceededretries and one new error variant,write metadata: insert metadata: sqlite: step: interrupted, still occurring under concurrent write bursts even with 8-way sharding.This PR tests whether that residual contention is simply a matter of headroom: bumps
singleWriterShards8→16 andsqlitePoolSize10→24 viaconfig.jsonoverrides (both are config-driven in kubescape/storage, so no new image build needed — reuses the same validatedquay.io/kubescape/storage:test-sharded-writer-397bimage from #949).If this resolves the residual flakiness, it points to needing a higher default shard/pool count for write-heavy environments. If it doesn't help, the residual failures likely have a separate cause (e.g. CI pod churn/restarts observed in #949's investigation) worth investigating independently of the storage fix.
Test plan
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LLN3CgGftcnAikV13qD6yJ
AI-skills: none | cmds: /oh-my-claudecode:autopilot