From 3905d1ce297e0aecbb62a79da614a4c6267f27da Mon Sep 17 00:00:00 2001
From: Matthias Bertschy <matthias.bertschy@gmail.com>
Date: Mon, 7 Sep 2026 15:56:40 +0200
Subject: [PATCH] experiment(ci): instrument component-tests to capture pod
 restarts and node pressure

Follow-up to the storage-CPU-headroom experiment (#952, refuted: bumping
the storage pod's own CPU limit 4x made no difference to the 17-18/31
residual failure rate). Traced one failing job's logs end-to-end: the
container-profile write path times out repeatedly ("failed to create
container profile, requeuing" / context deadline exceeded) with the
internal write queue backing up, independent of the storage pod's own
cgroup quota.

Two remaining candidates from the original investigation, neither yet
confirmed with hard data:
  1. Node-level CPU oversubscription on the shared GitHub-hosted runner
     (the storage pod's own limit doesn't matter if the whole node has
     no free cycles to give it).
  2. CI pod churn/restarts (liveness-probe failures under CI resource
     pressure) generating extra container-profile write traffic
     independent of storage, per #949's original observation of multiple
     container-instance IDs for the same logical container within ~1
     minute.

This is a diagnostic-only change: adds a step (always-run, after the
existing log dump) that captures `kubectl describe nodes` (allocatable
vs requested resources), `kubectl get pods -A -o wide` (restart counts),
a `kubectl describe pod` for any pod with a non-zero restart count, and
cluster events sorted by time -- across the full matrix, so failing vs
passing jobs can be correlated against actual restart/pressure evidence
instead of a single manually-traced job.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LCMGT6Po2tSr1VEDVrbbYd
---
 .github/workflows/component-tests.yaml | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/.github/workflows/component-tests.yaml b/.github/workflows/component-tests.yaml
index b46c571537..7fc16bd802 100644
--- a/.github/workflows/component-tests.yaml
+++ b/.github/workflows/component-tests.yaml
@@ -182,3 +182,22 @@ jobs:
           echo "-----------------------------------------"
           echo "Storage logs"
           kubectl logs $(kubectl get pods -n kubescape -o name | grep storage) -n kubescape
+      - name: Diagnose pod restarts and node resource pressure
+        if: always()
+        run: |
+          echo "Node describe (allocatable/capacity vs requests)"
+          kubectl describe nodes
+          echo "-----------------------------------------"
+          echo "Pods across all namespaces, with restart counts"
+          kubectl get pods -A -o wide
+          echo "-----------------------------------------"
+          echo "Pods with non-zero restarts (detail)"
+          for p in $(kubectl get pods -A --no-headers -o custom-columns=":metadata.namespace,:metadata.name,:status.containerStatuses[0].restartCount" | awk '$3!="" && $3!="<none>" && $3+0>0 {print $1"/"$2}'); do
+            ns=$(echo "$p" | cut -d/ -f1)
+            name=$(echo "$p" | cut -d/ -f2)
+            echo "=== $ns/$name ==="
+            kubectl describe pod "$name" -n "$ns"
+          done
+          echo "-----------------------------------------"
+          echo "Cluster events, sorted by time (liveness/readiness probe failures, OOMKilled, etc.)"
+          kubectl get events -A --sort-by=.lastTimestamp