feat(networking): Windows L2Bridge egress with router-safe VFP ACLs (opt-in) - #144
Draft
luthermonson wants to merge 1 commit into
Draft
feat(networking): Windows L2Bridge egress with router-safe VFP ACLs (opt-in)#144luthermonson wants to merge 1 commit into
luthermonson wants to merge 1 commit into
Conversation
…opt-in)
NAT cannot software-filter Windows container egress — VFP does not engage on
an HNS NAT network, so per-endpoint Switch ACLs are inert there. This adds an
opt-in L2Bridge path that binds containers to a host NIC via a NetAdapterName
network policy, giving each container a VFP-managed vSwitch port where ACLs
actually enforce. Proven on metal (static IP); DHCP is the v1 IPAM choice.
The router-safe ACL ladder (buildL2BridgeEgressACLPolicies, a pure function)
matches the Linux end-state: block ALL of 10/8, 172.16/12, 192.168/16,
169.254/16 — whole supernets, no gateway or own-subnet carve-out — and permit
only the internet. VFP is default-DENY once any ACL is present, so the ladder
is, by precedence (lower number wins):
90 Allow DHCP (UDP 67/68, Out+In) — lease/renew survives the block
95 Allow extra-allowed CIDRs (future use; default none)
100 Block the RFC1918 + link-local supernets, Out
65500 Allow 0.0.0.0/0 Out AND In — both mandatory (the In allow keeps
TCP SYN-ACK return traffic alive; without the pair the port
default-denies everything, internet included)
DNS is set to public resolvers on the endpoint so the container never needs
the LAN router for name resolution. The rule set is static (independent of the
leased gateway/DNS) and applied at endpoint creation, before the container
starts — fail-closed, no post-lease window. Any ApplyPolicy error tears down
the endpoint and refuses the job.
NAT stays the default and is untouched: buildEgressBlockPolicies and the NAT
init/setup path are unchanged, and the new ladder is only reached when a pool
sets network.l2bridge_egress = true. The NAT block-only builder is deliberately
NOT repurposed — its whole-supernet block with no carve-out would blackhole the
NAT gateway (10.88.0.1, inside 10/8) if VFP ever engaged.
Config: network.l2bridge_egress (bool), network.host_nic (required when on),
network.public_dns (default 1.1.1.1/8.8.8.8), network.extra_allowed_destinations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Windows container egress cannot be software-filtered on the current HNS NAT
network. VFP (the Virtual Filter Platform) does not engage on a NAT network, so
the per-endpoint Switch ACLs ephemerd already applies are inert — enforcement
there rests entirely on the Hyper-V firewall. We need an enforcement point where
software ACLs actually bite.
Fix (opt-in, NAT stays default)
Add an L2Bridge networking path, gated behind a per-pool flag
(
network.l2bridge_egress). On L2Bridge the container gets a VFP-managedvSwitch port, so per-endpoint ACLs enforce. This was proven on metal
(node
mfl-win-amd64-1, Server 2025): with the correct ACL model, aHyper-V-isolated container reached
1.1.1.1/ public HTTPS whilecontainer→Grafana/Incus/Proxmox (RFC1918) was blocked.
Merging this flips nothing — NAT remains the default and the NAT init/setup/
ACL path is untouched. A pool only reaches L2Bridge by setting
l2bridge_egress = true.The router-safe ACL model (the load-bearing change)
VFP is default-DENY the moment any ACL is present — a pure-Block set (today's
buildEgressBlockPolicies) would black-hole the internet too.buildL2BridgeEgressACLPolicies(a pure, unit-tested function) emits this precedence ladder as
RuleType=SwitchVFP ACLs (lower Priority = higher precedence, first match wins):
This matches the Linux end-state (
firewall_linux.go): block all of RFC1918including the LAN router, permit only the internet.
On L2Bridge the container is a LAN peer, so carving out the subnet would expose
the same-subnet management plane and the router. This is the fix for the
same-subnet management-plane bug.
default-denies everything; the inbound allow is the non-obvious one — drop it
and every outbound TCP connect fails when the SYN-ACK returns.
Dns.ServerListis set to aconfigurable public resolver (default
1.1.1.1,8.8.8.8), so DNS rides the0.0.0.0/0allow and the container never needs the LAN router. No allow tothe gateway/router.
access, which we block.
Fail-closed, no post-lease window
The ladder is static — it does not depend on the leased gateway or DNS (DNS is
public, the router gets no allow) — so the ACLs are applied at endpoint creation,
before the container starts. Any
ApplyPolicyerror tears down the endpoint andrefuses the job.
Network shape
L2Bridge network is
hcn.L2Bridgebound to the host NIC via ahcn.NetAdapterNamenetwork policy (recipe §1). IPAM = DHCP: the network declares no static subnet/
routes and the endpoint carries no
IpConfigurations, so the container leases itsIP, gateway, and default route from the LAN DHCP server. Network name is
ephemerd-l2bridge(distinct from the NATephemerd).Config surface (what mayfly must render)
New
[network]keys (per-pool → rendered into the per-VM ephemerdconfig.toml,same flow as the existing
[vm.linux]toggle andwindows_runnersizing):l2bridge_egress(bool, defaultfalse) — the opt-in.host_nic(string) — required whenl2bridge_egress = true; the host adaptername to bridge onto. No default (host-specific; do not hardcode "Ethernet 2",
the spike's hot-added test NIC). Missing → init fails closed.
public_dns([]string, default["1.1.1.1","8.8.8.8"]).extra_allowed_destinations([]string, default[]) — future use.Wired through
networking.Config(L2BridgeEgress,HostNIC,PublicDNS,ExtraAllowedCIDRs) at bothnetworking.Newcall sites incmd/ephemerd/main.go.Documented in
config.example.toml. mayfly rendering is a follow-up (not inthis PR): add the four keys to the per-pool Windows config schema and render them
into
[network];host_nicshould be a required field when the pool opts in.Honest caveats
enforcement with a static IP. DHCP is the chosen v1 IPAM and will be validated
at the gated cutover. If DHCP-on-L2Bridge misbehaves there, static-pool IPAM is
the proven fallback (recipe §6 recommends static-first).
the 2-tier (high-prio blocks over low-prio allow-any) was. The default posture here
has no carve-outs (empty
extra_allowed_destinations), so the default set is theproven shape plus the DHCP allow.
v0.14.0-rc.1'shcndoes not expose aready source-IP/MAC anti-spoof endpoint/port policy in a form I could apply without
guessing the schema, so it is not implemented here rather than faked. On L2Bridge
the container is a routable LAN peer that could source-spoof within the L2 domain;
egress enforcement rests on the VFP ACLs. Follow-up: evaluate an HNS
L2BridgeMAC/IP anti-spoof port policy.
Deviations from the recipe
buildEgressBlockPoliciesfor NAT. Rewriting it into theladder (or applying the ladder on NAT) would blackhole the NAT gateway
10.88.0.1(inside the whole-
10/8block, no carve-out) if VFP ever engaged. Kept the NATblock-only builder and path exactly as-is; the ladder is a new function on
the L2Bridge path only. This better satisfies "NAT stays intact".
IpConfigurations— per the user's DHCP v1 choice; see caveat above.Tests
pkg/networking/network_windows_test.go(pure, no HCN calls):TestL2BridgeEgressACLPolicies_LadderShape— both allow-any (Out+In), both DHCPallows, all four supernets blocked; exact protocols/priorities/directions.
TestL2BridgeEgressACLPolicies_NoGatewayOrSubnetCarveOut— no allow to any RFC1918address, blocks are plain CIDRs (no exclusion ranges leaked in).
TestL2BridgeEgressACLPolicies_Precedence— DHCP/extra < block < allow-any.TestL2BridgeEgressACLPolicies_ExtraAllowed— carve-outs emitted Out, above block.TestBuildEgressBlockPolicies(NAT) unchanged and passing.go build ./pkg/networking/... ./pkg/config/...,go vet ./pkg/networking/..., andgo test ./pkg/networking/...all clean (nativeGOOS=windows). (go build ./...hits a pre-existing missing-embed-artifact error in
pkg/vm, unrelated to this change.)Not in this PR
No live node touched. The live cutover is a separate, gated step.