[EXPERIMENT] Try CUSTOM_REST_ENABLED to see if it avoids the single-writer CI regression

[matthyx]

Purpose

This is a diagnostic experiment, not a proposed fix — do not merge.

component-tests has been failing widely (18-19 of ~30 matrix jobs) since kubescape/helm-charts bumped storage.image.tag to v0.0.331, which flips kubescape/storage's single-writer write path on by default (storage#393). That path serializes every Create/GuaranteedUpdate for every container profile through one dedicated writer goroutine, collapsing write concurrency from up to 10-way (SqlitePoolSize) to strictly 1-way — a plausible cause of the sustained context deadline exceeded / failed to create container profile, requeuing errors seen throughout these runs (300+ occurrences, spanning the full ~19-minute test).

This PR flips CUSTOM_REST_ENABLED (storage#387) on in the test storage deployment to empirically check whether the new genericrest.Store-based REST path sidesteps the bottleneck.

Expected result: it won't help. Reading pkg/apiserver/apiserver.go in kubescape/storage@v0.0.331 shows the default REST (containerprofile.NewREST) and custom REST (containerprofile.NewCustomREST) are wired to the same containerProfileStorageImpl instance — both funnel through the identical single-writer path in pkg/registry/file/singlewriter.go. CUSTOM_REST_ENABLED only swaps the API-server registry layer, not the underlying storage write path.

Opening this PR to let CI confirm or refute that reading before pursuing a fix that targets the single-writer bottleneck directly.

Test plan

• component-tests run on this PR — compare failure count against the current baseline (18-19/30 failing)

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01LLN3CgGftcnAikV13qD6yJ

AI-skills: none | cmds: /oh-my-claudecode:autopilot

Summary by CodeRabbit

• New Features
  • Enabled custom REST functionality for the storage service.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: cf8bc922-8219-4f54-8459-692fb001a186

📥 Commits

Reviewing files that changed from the base of the PR and between 1c450e5 and 0e17c48.

📒 Files selected for processing (1)

• tests/chart/templates/storage/deployment.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The storage Deployment now sets CUSTOM_REST_ENABLED to "true" in the apiserver container environment.

Changes

Storage deployment configuration

Layer / File(s)	Summary
Apiserver environment configuration tests/chart/templates/storage/deployment.yaml	The apiserver container now defines CUSTOM_REST_ENABLED with the value "true".

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 0e17c

This updates the test storage deployment to enable the custom REST path. No concrete merge-blocking impact is identified in the supplied change context.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the change as an experiment involving CUSTOM_REST_ENABLED and states its purpose of investigating the single-writer CI regression.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch experiment/custom-rest-enabled-component-tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[matthyx]

Result: 18 failed / 32 total — statistically identical to the unmodified baseline (18-19/30). Confirms the code-level prediction: CUSTOM_REST_ENABLED only swaps the API-server REST registry layer; both the default and custom REST implementations share the same underlying StorageImpl/single-writer write path, so it doesn't touch the actual bottleneck.

Root cause and real fix are tracked in kubescape/storage#397 (shard the single-writer commit path by key hash instead of funneling every write through one global goroutine).

[github-actions]

Performance Benchmark Results

Node-Agent Resource Usage

Metric	BEFORE	AFTER	Delta
Avg CPU (cores)	0.195	0.212	+8.5%
Peak CPU (cores)	0.201	0.217	+8.3%
Peak CPU p95 (cores)	0.200	0.216	+8.3%
Avg Memory (MiB)	402.457	305.753	-24.0%
Peak Memory (MiB)	405.184	308.340	-23.9%

Dedup Effectiveness

No data available.