Skip to content

fix(networking): Windows container egress is unenforceable on WinNAT+Hyper-V — evidence + VLAN recommendation - #142

Open
luthermonson wants to merge 1 commit into
mainfrom
fix/windows-egress-enforce
Open

fix(networking): Windows container egress is unenforceable on WinNAT+Hyper-V — evidence + VLAN recommendation#142
luthermonson wants to merge 1 commit into
mainfrom
fix/windows-egress-enforce

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Summary — honest negative result

I did NOT achieve a green Windows containment run. After genuine on-metal testing of every software mechanism available on the node, the evidence shows there is no per-container software egress filter that enforces on this host class (Windows Server 2025 build 26100, HNS NAT network, Hyper-V-isolated job containers). Real containment must come from outside the box — an upstream/switch VLAN whose router denies RFC1918.

This PR does not claim to fix egress. It replaces the code's false assurance ("refusing to start unfirewalled") with an honest, evidenced WARN + a windowsEgressNotEnforced pointer, so the daemon never implies a containment it does not provide, and so the next attempt does not repeat #1#4.

Phase 1 diagnosis — the ACLs are applied but NOT enforced

Caught a live job-container endpoint during a held containment run. The RFC1918 block ACLs buildEgressBlockPolicies produces are present on the live endpoint (10.88.82.163):

[{"Action":"Block","Direction":"Out","Priority":100,"Protocols":"6,17","RemoteAddresses":"10.0.0.0/8","RuleType":"Switch","Type":"ACL"},
 {"...":"172.16.0.0/12"},{"...":"192.168.0.0/16"},{"...":"169.254.0.0/16"}]

…yet the same run still reaches every plane (baseline, deployed v0.1.7):

FAIL: reached Proxmox coyotes at 192.168.5.1:8006
FAIL: reached Proxmox kings at 192.168.5.2:8006
FAIL: reached Incus daemon at 192.168.12.113:8443
FAIL: reached Grafana at 192.168.10.45:3000

So the endpoint RuleType:"Switch" (VFP) ACLs are stored but inert.

Root cause — VFP is not enforcing on the WinNAT switch

vfpctrl.exe /list-vmswitch-port enumerates the switch ports (host vNIC AEAC667C PortId 1; container synthetic ports), but every port- or NAT-scoped VFP operation fails:

vfpctrl /port <host-or-container> /get-port-state    -> ERROR (2) The system cannot find the file specified.
vfpctrl /port <...> /get-port-flow-settings          -> ERROR (2) ...
vfpctrl /port <...> /list-layer                      -> exit 1
vfpctrl /list-nat-range                              -> ERROR (87) The parameter is incorrect.

Only passive enumeration works. On a WinNAT nat network the VFP switch extension is not enforcing on the ports, so RuleType:"Switch" ACLs (and any vfpctrl-added rule, and the native vfpctrl "1:Block RFC1918" NAT flag) have no datapath to bite. Container→LAN egress is instead forwarded + SNATed by the host, a path the host Windows Firewall does not filter (it governs host-terminated traffic, not routing/forwarding).

Why every mechanism fails here (the enforcement matrix)

Mechanism Result on metal
HNS endpoint ACL, RuleType=Switch (current code) Present on live endpoint, not enforced — VFP not enforcing on NAT switch.
Direct VFP (vfpctrl) All port/NAT ops fail ("cannot find the file specified"); VFP not managing these ports.
Hyper-V firewall (New-NetFirewallHyperVRule, #140) Get-NetFirewallHyperVVMCreator and Get-NetFirewallHyperVPort are empty even with a live container → rules never attach.
Extended port ACL (Add-VMNetworkAdapterExtendedAcl) Cmdlet absent — Hyper-V PowerShell module not installed (Get-VMSwitch/Get-VMNetworkAdapter also absent).
Host WFP / netsh (#136) Evaluates NATed egress post-NAT (host source); Windows Firewall does not filter forwarded traffic; also disqualified (would block the host's own RFC1918).
In-container route blackhole (dead on-link next hop) Routes install, but planes still reached — defeated by the NAT gateway's proxy-ARP.
In-container route blackhole (route add … 127.0.0.1) The route addition failed: The parameter is incorrect.
In-container Windows Firewall (WFP dest-IP block) Get-NetFirewallProfileWindows System Error 1753 (BFE not running); Start-Service BFE/mpssvc"Cannot start service". Base Filtering Engine absent inside the container.

Every one of these was run on the live node and observed via the containment suite; the four planes stayed reachable in each case.

The intended allow-list (mirrors firewall_linux.go) was respected in every test: block only 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16 minus the container's own 10.88.0.0/16 (so gateway 10.88.0.1 / DNS / default route / container-to-container survive), leaving the public internet open. The gap is enforcement, not scope.

Recommendation

  1. Contain Windows-runner egress at an upstream VLAN (router/switch ACL that denies RFC1918 for the container-egress path). Job-proof, host-safe, and independent of the broken VFP/WFP/Hyper-V-firewall layers. This is what the negative result points to.
  2. The only remaining software avenue is a host-side WFP filter at FWPM_LAYER_IPFORWARD_V4 (the Windows analogue of the Linux iptables FORWARD chain), added via the WFP API. It is unverified (no CLI to test it; WinNAT's WFP-callout ordering is undocumented) and risky on the production node — a mis-scoped forward filter can sever the host's own management LAN, recoverable only through the fragile guest-agent. It should be prototyped in a lab, not shipped blind, rather than add a 4th untested attempt.

What this PR changes

  • network_windows.go: keep the fail-closed ACL apply, drop the false "refusing to start unfirewalled" claim, emit a per-container WARN with the metal-verified analysis.
  • firewall_windows.go: startup WARN that Windows egress filtering is best-effort and NOT enforced on this host class.
  • windowsEgressNotEnforced: single-line log pointer to the analysis.
  • (test/containment-suite, separate branch) added a Windows "internet must still be reachable" probe so a future working fix is checked for over-blocking too.

The ACLs are left in place — harmless, fail-closed, and correct on any future host where VFP or the Hyper-V firewall does enforce.

…yper-V, point to VLAN

Metal verification on Windows Server 2025 (build 26100) proved the RFC1918
block ACLs are stored on the HNS endpoint but not enforced, and that no
per-container software egress filter is available on this host class. Replace
the false 'refusing to start unfirewalled' assurance with an honest, evidenced
WARN and a windowsEgressNotEnforced pointer, so the daemon never implies a
containment it does not provide. Real containment must be an upstream VLAN.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant