[EXPERIMENT] Validate kubescape/storage#397 (sharded single-writer) fix, take 2

[matthyx]

Purpose

This is a validation experiment, not a proposed fix — do not merge.

Retry of #948, which was a false negative: that attempt's test image landed in quay.io/kubescape/storge (typo'd repo name from a pre-existing bug in kubescape/storage's manual build-image workflow), which turned out to be a brand-new, private-by-default repo. The Kind cluster couldn't pull it at all — the storage deployment never became ready, so all 31 test jobs failed on infrastructure grounds, not on anything related to the write-throughput regression.

This retry uses a corrected build (quay.io/kubescape/storage:test-sharded-writer-397b — confirmed public via curl https://quay.io/api/v1/repository/kubescape/storage/tag/ with no auth), built from kubescape/storage#397's fix/single-writer-sharded-commits via a throwaway build-workflow branch that fixes the repo-name typo and drops the :latest tag push so the real published tag is never touched.

Expected result: near-zero failures, matching the pre-regression baseline, since the fix restores per-key write parallelism (default 8 shards) while keeping every existing single-writer correctness guarantee.

Test plan

• component-tests run on this PR — compare failure count against the current baseline (18-19/30 failing)

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LLN3CgGftcnAikV13qD6yJ

AI-skills: none | cmds: /oh-my-claudecode:autopilot

Summary by CodeRabbit

• Tests
  • Component tests now use a fixed experimental storage image version.
  • Updated test output to reflect the selected experimental version.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 6b0f5fca-5118-442d-99cd-082457bd1193

📥 Commits

Reviewing files that changed from the base of the PR and between 1c450e5 and 2d774ce.

📒 Files selected for processing (1)

• .github/workflows/component-tests.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The component-test workflow now uses the experimental storage image tag test-sharded-writer-397b instead of resolving the tag dynamically.

Changes

Component test storage override

Layer / File(s)	Summary
Pin experimental storage image tag .github/workflows/component-tests.yaml	The workflow uses the fixed test-sharded-writer-397b storage image tag and labels the output as an experiment override.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 2d774

Component tests now use the specified experimental storage image for isolated validation. No concrete merge-blocking risk remains in the current change.

Suggested reviewers: entlein

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the validation experiment for the sharded single-writer fix and indicates that it is a second attempt. This matches the workflow change and PR objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch experiment/sharded-storage-writer-validation-2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[matthyx]

Result: 15/30 failed (build-and-push-image succeeded, so 15/30 test jobs, 19/32 total including infra jobs). Real, substantial improvement over the 18-19/30 baseline — no more sustained 20-min hangs, storage deployment comes up cleanly, individual container-profile writes went from minutes-long timeouts to 5-9 seconds for a 16-item burst. But not the hoped-for near-zero.

Traced the residual failures: still occasional context deadline exceeded retries plus a new error variant (sqlite: step: interrupted) under concurrent write bursts, self-healing within ~1-2 minutes rather than hanging for the full test timeout. Also observed the test pod's containers getting recreated multiple times within about a minute, adding extra write churn independent of storage.

Follow-up experiment testing higher shard/pool headroom (16 shards / 24 pool connections) in #950 before concluding whether this needs a higher default or has a separate contributing cause.