DEVOP-579: add NetworkPolicy egress rollout plan (doc only)

[srt0422]

Summary

Adds tickets/devop-579-network-policy-rollout.md — a staged plan for rolling default-deny-egress NetworkPolicies across our 13 clusters.

Why doc-first

NetworkPolicy egress hardening is a 3-engineer-week project where the bulk of effort is discovery, not deployment. default-deny-egress silently breaks every workload that has an un-enumerated outbound dependency, so rushing it is production-impacting. Capturing the plan now (Phases 0-4, rollback procedure, dependencies on DEVOP-588/589) means subsequent loop runs or human owners can pick up execution without redoing the planning.

This PR adds only the plan document. No NetworkPolicy is deployed.

Test plan

• Document compiles, internal links resolve.
• Reviewer: confirm CNI assumption (Calico/Cilium across all 13 clusters?) and adjust Phase 0 if wrong.
• Reviewer: confirm priority ordering of namespaces in Phase 1 matches actual blast-radius.

Related

• Linear: https://linear.app/alloralabs/issue/DEVOP-579
• Hard dep: DEVOP-589 (Harbor proxy-cache) must land before Phase 2.
• Soft dep: DEVOP-588 (Kyverno on all clusters) for Phase 4.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

---

Summary by cubic

Adds a phased doc plan to roll out default-deny-egress with a parallel default-deny-ingress across all 13 clusters for DEVOP-579; no policies are deployed. Pins exact policy names for Kyverno/rollback, notes NetworkPolicies are stateful (conntrack), fixes Phase 0 with per‑CNI flow logging plus CoreDNS dnstap/Cilium DNS capture, enforces 48‑hour one‑namespace soaks with a clean gate, adds a suspect‑egress checklist, updates SECURITY-RUNBOOK.md with both rollback commands, and removes a phantom ingress baseline‑allow from the naming table.

• Dependencies
  • Hard: DEVOP-589 (Harbor proxy-cache) must land before Phase 2.
  • Soft: DEVOP-588 (Kyverno on all clusters) for Phase 4 enforcement.

^{Written for commit 058a8ea. Summary will update on new commits.}

[cubic-dev-ai]

4 issues found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tickets/devop-579-network-policy-rollout.md">

<violation number="1" location="tickets/devop-579-network-policy-rollout.md:33">
P2: The discovery checklist omits the DEVOP-579 requirement to explicitly flag suspect egress destinations (webhook/pastebin/ngrok/169.254.169.254).</violation>

<violation number="2" location="tickets/devop-579-network-policy-rollout.md:52">
P1: Phase 3 uses 24-hour soak windows, but linked Linear issue DEVOP-579 specifies 48-hour soaks for staged rollout.</violation>

<violation number="3" location="tickets/devop-579-network-policy-rollout.md:64">
P2: Phase 4 is missing the DEVOP-579 requirement to document the rollout/policies in SECURITY-RUNBOOK.md.</violation>

<violation number="4" location="tickets/devop-579-network-policy-rollout.md:74">
P2: This plan marks ingress NetworkPolicies as out of scope, but linked Linear issue DEVOP-579 requires default-deny for both egress and ingress.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant K8s as Kubernetes Clusters (13)
    participant CNI as CNI Plugin (Calico/Cilium)
    participant Hubble as Hubble/Flow Logs
    participant Harbor as Harbor Registry
    participant Kyverno as Kyverno Policy Engine
    participant Runbook as Rollback Runbook

    Note over K8s,Runbook: Phase 0 — Pre-flight Assessment
    K8s->>CNI: Confirm NetworkPolicy support
    alt CNI supports NetworkPolicy
        CNI-->>K8s: Calico, Cilium, or Antrea confirmed
    else Flannel without --network-policy
        CNI-->>K8s: Need to migrate CNI first
    end
    K8s->>Hubble: Enable flow logs on staging cluster
    Note over Hubble: Capture 7 days baseline traffic

    Note over K8s,Rollback: Phase 1 — Discovery (per namespace)
    loop For each namespace in priority order
        K8s->>Hubble: Query egress flow logs (7 days)
        Hubble-->>K8s: Destination CIDRs, DNS, ports
        K8s->>K8s: Categorize traffic (internal/infra/vendor/registries/customer)
        K8s->>K8s: Document in network-policies/discovery/<namespace>.md
    end

    Note over K8s,Rollback: Phase 2 — Allowlist Authoring
    K8s->>K8s: Create default-deny.yaml (deny all egress except DNS)
    K8s->>K8s: Create allowlist.yaml (derived from Phase 1)
    Note over K8s: DNS to kube-dns/coredns (53/udp, 53/tcp)
    Note over K8s: NTP always allowed (123/udp)
    Note over K8s: Cluster-internal pod-to-pod allowed by default
    K8s->>Harbor: Dependency on DEVOP-589 (Harbor proxy-cache)
    alt DEVOP-589 landed
        Note over K8s: Allowlists reference Harbor proxy instead of direct registries
    else Not yet landed
        Note over K8s: Allowlists must allow direct ghcr.io, docker.io, etc.
    end

    Note over K8s,Rollback: Phase 3 — Staged Rollout
    K8s->>K8s: Day 1: Apply to 1 staging namespace, observe 24h
    K8s->>K8s: Day 2: Apply to all staging namespaces, observe 24h
    K8s->>K8s: Day 3: Apply to 1 production namespace (lowest risk), observe 24h
    K8s->>K8s: Days 4-5: Roll forward remaining namespaces (lowest blast-radius first)
    alt Egress broken for workload
        K8s->>Runbook: kubectl delete networkpolicy default-deny -n <ns>
        Runbook-->>K8s: Egress restored immediately
    end

    Note over K8s,Rollback: Phase 4 — Steady State
    alt DEVOP-588 landed (Kyverno on all clusters)
        Kyverno->>K8s: Auto-flag new namespaces without default-deny
        K8s->>K8s: Monthly review of discovery documents
    else Not yet landed
        Note over K8s: Manual enforcement only
    end

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tickets/devop-579-network-policy-rollout.md">

<violation number="1" location="tickets/devop-579-network-policy-rollout.md:86">
P2: The new cohort definition serializes rollout to one namespace every two days, which cannot fit the stated Phase 3/week-3 and three-engineer-week schedule across 13 clusters. Define a batched cohort or revise the timeline/estimate so this plan is operationally schedulable.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The new cohort definition serializes rollout to one namespace every two days, which cannot fit the stated Phase 3/week-3 and three-engineer-week schedule across 13 clusters. Define a batched cohort or revise the timeline/estimate so this plan is operationally schedulable.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tickets/devop-579-network-policy-rollout.md, line 86:

<comment>The new cohort definition serializes rollout to one namespace every two days, which cannot fit the stated Phase 3/week-3 and three-engineer-week schedule across 13 clusters. Define a batched cohort or revise the timeline/estimate so this plan is operationally schedulable.</comment>

<file context>
@@ -83,7 +83,7 @@ whose egress only fires on cron/batch schedules.
 - [ ] Days 3–4: apply to all staging namespaces in 1 cluster. Soak 48h.
 - [ ] Days 5–6: apply to 1 production namespace (lowest-risk: docs site). Soak 48h.
-- [ ] Days 7+: roll forward through remaining production namespaces in priority-inverse order (lowest-blast-radius first), keeping a 48h soak between each cluster cohort.
+- [ ] Days 7+: roll forward through remaining production namespaces in priority-inverse order (lowest-blast-radius first), keeping a 48h soak between each cohort — one cohort is one namespace.
 
 A stage may only advance if the prior soak completed with zero
</file context>

[srt0422]

Final cross-provider review (agent-code-review, 3-pass)
verdict: Not ready
counts: P0=1 P1=2 P2=1 P3=1
coverage: Pass 1: kimi-k3 (orchestrator — substituted for Claude Opus 4.7 correctness+security and Claude Sonnet 5 style per user mandate; claude-code CLI not invoked). Pass 2: google/gemini-2.5-pro (correctness), deepseek/deepseek-v4-pro (security; 1 retry after initial timeout), meta-llama/llama-4-maverick (style). Pass 3: meta-llama/llama-4-maverick (correctness), z-ai/glm-5.2 (security — substituted for Claude Sonnet 5 per user mandate), meta-llama/llama-4-scout (style). No reductions. Every Pass 2/3 finding source-verified by the orchestrator before acceptance.

severity	file:line	category	reviewer	summary	suggested_fix
P0	tickets/devop-579-network-policy-rollout.md:94-95	correctness	both	The documented rollback does not restore traffic. After kubectl delete networkpolicy default-deny-egress -n <ns>, egress-baseline-allow (podSelector {} per the Phase 2 table) still selects every pod, so all pods remain egress-isolated with only DNS+NTP allowed; egress-allowlist likewise keeps its pods isolated. "Restores all egress instantly" is false, and the ingress command on line 95 has the same defect via ingress-allowlist. On-call executing this runbook during an incident leaves the namespace broken and escalates with a failed rollback. Confirmed by 4 independent passes (Gemini, DeepSeek, GLM, orchestrator).	Document deleting ALL egress policies: kubectl delete networkpolicy default-deny-egress egress-baseline-allow egress-allowlist -n <ns>; and for ingress: kubectl delete networkpolicy default-deny-ingress ingress-allowlist -n <ns>. Add a re-apply procedure (with SLA) so the namespace does not stay open indefinitely.
P1	tickets/devop-579-network-policy-rollout.md:61	security	both	The egress-baseline-allow DNS permit (53/udp+tcp to kube-dns/CoreDNS) leaves DNS tunneling / subdomain-encoded exfiltration open through the legitimate in-cluster resolver, defeating the stated C2-blocking goal (line 7); the suspect-destination review cannot see it because traffic terminates at the cluster resolver. Unacknowledged as residual risk. Confirmed by DeepSeek + GLM.	Acknowledge DNS tunneling as a residual risk in the plan and either (a) add a compensating control (Cilium DNS-proxy inspection of query patterns, CoreDNS query-rate/response-size anomaly detection), or (b) explicitly defer detection to DEVOP-570 (Falco) with the gap stated in the goal section.
P1	tickets/devop-579-network-policy-rollout.md:5	security	both	hostNetwork: true pods bypass NetworkPolicy enforcement entirely (Calico, Cilium, Antrea). No phase inventories or mitigates hostNetwork workloads, so the plan's goal (closing pod-to-C2 egress) is not achieved for that class; a compromised hostNetwork pod retains unrestricted egress and metadata access. Confirmed by DeepSeek + GLM.	Add a Phase 0/1 task to inventory all hostNetwork: true pods across the 13 clusters; then migrate them off hostNetwork where possible, apply host-level egress controls (iptables/eBPF) where not, or document them as accepted residual risk with compensating controls.
P2	tickets/devop-579-network-policy-rollout.md:101	security	persona	The Phase 4 Kyverno asserter only fails NEW namespaces missing a default-deny. After an emergency rollback deletes policies from an EXISTING namespace, no automated control detects or alerts on the resulting open-egress state; the namespace stays less-secure until manual re-apply.	Add a Kyverno audit-mode policy that periodically asserts ALL existing non-system namespaces contain default-deny-egress/default-deny-ingress by pinned metadata.name, alert on absence, and document a post-rollback re-apply SLA in SECURITY-RUNBOOK.md.
P3	tickets/devop-579-network-policy-rollout.md:62	doc	persona	egress-allowlist scoping is implied ("workload-specific") but not pinned; an author could write a single policy with namespace-wide podSelector: {} carrying broad allows, silently violating least privilege.	Add one line to the naming convention: egress-allowlist.yaml contains one NetworkPolicy object per workload, each with a specific spec.podSelector scoping rules to that workload.

Refuted after source-verification (14, recorded per the Pass 2/3 verification gate):

• "Cilium L7 DNS visibility works regardless of DNS proxy in-path" (pass3-correctness) — false; Hubble DNS events require the DNS proxy in-path, and the doc's statement is correct.
• "The rollback command is correct" (pass3-correctness) — refuted by the P0 finding above; its post-rollback policy-review concern is subsumed by the P2 finding.
• Line 60 "deny except baseline allows" called imprecise (pass2-correctness) — the Purpose column describes net effect under additive semantics, consistent with the other rows; no operator harm.
• "Phase 1 categories non-exhaustive" (pass3-correctness) — the list already signals openness via "etc." and the per-namespace bucket.
• Pass 2/3 style items (10): sink options already listed at line 18; DEVOP-571 already linked in the Links section; join predicate explained in prose at line 22; SECURITY-RUNBOOK.md sits at this repo's root so the bare name is unambiguous; "Direction/Purpose" headers self-explanatory; "emergency" used consistently in both rollback bullets; "do not roll suspect destinations by default" is disambiguated by the incident-review sentence in the same bullet; the ingress "mirror with substitutions" structure is a deliberate DRY choice (duplicating the full egress checklist was previously considered and rejected); "on-call escalation path" is a specification of what to document, not an unresolved reference.

Loop context: review-fix-loop applied 4 doc fixes across 3 iterations before this pass (conntrack/stateful wording, cohort=one-namespace definition, related-ticket links, phantom ingress baseline-allow reference). The blocking findings above were surfaced only by the cross-provider matrix; per the bounded-loop design they are recorded here as residuals for the next loop run or human decision — they were not auto-fixed in this pass.