Skip to content

WIP: Migrate iptables to nftables in networking test suite #3935

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
weliang1 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

WIP: Migrate iptables to nftables in networking test suite #3935

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions features/networking/egress-ip.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ Feature: Egress IP related features
@s390x @ppc64le @heterogeneous @arm64 @amd64
@hypershift-hosted
@inactive
Scenario: OCP-15473:SDN The related iptables/openflow rules will be removed once the egressIP gets removed from netnamespace
Scenario: OCP-15473:SDN The related nftables/openflow rules will be removed once the egressIP gets removed from netnamespace
Given the env is using "OpenShiftSDN" networkType
Given the valid egress IP is added to the node
And I have a project
Expand All @@ -286,9 +286,9 @@ Feature: Egress IP related features
Given as admin I successfully merge patch resource "netnamespace/<%= project.name %>" with:
| {"egressIPs": ["<%= cb.valid_ip %>"]} |

#Check related iptables added
#Check related nftables rules added
When I run command on the "<%= node.name%>" node's sdn pod:
| bash | -c | iptables-save \| grep "<%= cb.valid_ip %>" |
| bash | -c | nft list ruleset \| grep "<%= cb.valid_ip %>" |
Then the step should succeed
And the output should contain:
| OPENSHIFT-MASQUERADE |
Expand All @@ -305,11 +305,11 @@ Feature: Egress IP related features
Given as admin I successfully merge patch resource "netnamespace/<%= project.name %>" with:
| {"egressIPs": null} |

# Check related iptables and openlows removed
# Check related nftables rules and openflows removed
And I wait up to 30 seconds for the steps to pass:
"""
When I run command on the "<%= node.name%>" node's sdn pod:
| bash | -c | iptables-save \| egrep "OPENSHIFT-FIREWALL-ALLOW\|OPENSHIFT-MASQUERADE" |
| bash | -c | nft list ruleset \| egrep "OPENSHIFT-FIREWALL-ALLOW\|OPENSHIFT-MASQUERADE" |
Then the step should succeed
And the output should not contain:
| <%= cb.valid_ip %> |
Expand Down
11 changes: 6 additions & 5 deletions features/networking/ovn.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,20 +455,21 @@ Feature: OVN related networking scenarios
When I store the ovnkube-master "south" leader pod in the clipboard
Then the step should succeed

# ipv6 uses ip6tables binary
Given evaluation of `(cb.south_leader.ip.include? ":") ? "ip6tables" : "iptables"` is stored in the :iptables_command clipboard
# Determine IP family for nftables (ip or ip6)
Given evaluation of `(cb.south_leader.ip.include? ":") ? "ip6" : "ip"` is stored in the :nft_family clipboard
Given evaluation of `(cb.south_leader.ip.include? ":") ? "ip6 saddr" : "ip saddr"` is stored in the :nft_saddr clipboard

Given I store the masters in the clipboard excluding "<%= cb.south_leader.node_name %>"
And I use the "<%= cb.nodes[0].name %>" node
# make sure to unblock after the test
And I register clean-up steps:
"""
When I run commands on the host:
| <%= cb.iptables_command %> -t filter -D INPUT -s <%= cb.south_leader.ip %> -p tcp --dport 9643:9644 -j DROP |
| bash | -c | nft delete rule <%= cb.nft_family %> filter INPUT handle $(nft -a list chain <%= cb.nft_family %> filter INPUT \| grep '<%= cb.south_leader.ip %>' \| grep 'tcp dport 9643-9644 drop' \| awk '{print $NF}') |
"""
# don't block all traffic that breaks etcd, just block the OVN ssl ports
When I run commands on the host:
| <%= cb.iptables_command %> -t filter -A INPUT -s <%= cb.south_leader.ip %> -p tcp --dport 9643:9644 -j DROP |
| bash | -c | nft add rule <%= cb.nft_family %> filter INPUT <%= cb.nft_saddr %> <%= cb.south_leader.ip %> tcp dport 9643-9644 drop |
Then the step should succeed

# election timer is 1 second by default but the RAFT JSON-RPC probe might take 5 seconds to notice
Expand All @@ -485,7 +486,7 @@ Feature: OVN related networking scenarios
"""
# try to get the isolated leader for debug, it might not work
When I run commands on the host:
| <%= cb.iptables_command %> -t filter -D INPUT -s <%= cb.south_leader.ip %> -p tcp --dport 9643:9644 -j DROP |
| bash | -c | nft delete rule <%= cb.nft_family %> filter INPUT handle $(nft -a list chain <%= cb.nft_family %> filter INPUT \| grep '<%= cb.south_leader.ip %>' \| grep 'tcp dport 9643-9644 drop' \| awk '{print $NF}') |
# wait for OVN to reconverge
# wait 120 seconds for convergence due to election timer as described above.
And I wait up to 120 seconds for the steps to pass:
Expand Down
22 changes: 12 additions & 10 deletions features/networking/pod.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -570,11 +570,10 @@ Feature: Pod related networking scenarios
Then evaluation of `pod.name` is stored in the :client_pod clipboard
Given I store the masters in the :masters clipboard
And I use the "<%= cb.masters[0].name %>" node
#Making sure the iptables on latest centos are actually nf_tables as the xt_u32 is only supported with that
#Making sure nftables is available for u32 matching functionality
When I run commands on the host:
| iptables --version |
| nft --version |
Then the step should succeed
Then the output should contain "nf_tables"

Given I obtain test data file "networking/centos_latest_admin.yaml"
When I run oc create as admin over "centos_latest_admin.yaml" replacing paths:
Expand All @@ -590,23 +589,26 @@ Feature: Pod related networking scenarios
| modprobe xt_u32 |
Then the step should succeed

#Checking whether server pod also has nf_tables version installed
#Checking whether server pod has nftables installed
When admin executes on the "<%= cb.server_pod.name %>" pod:
| bash | -c | iptables --version |
| bash | -c | nft --version |
Then the step should succeed
And the output should contain "nf_tables"

#Creating iptable rule on server pod with module u32 and Successful pings will represent a successful xt_u32 match
#Creating nftables rule on server pod with u32 matching - allow ICMP echo requests (type 8)
Given admin executes on the "<%= cb.server_pod.name %>" pod:
| bash | -c | iptables -t filter -A INPUT -i eth0 -m u32 --u32 '6 & 0xFF = 1 && 4 & 0x3FFF = 0 && 0 >> 22 & 0x3C @ 0 >> 24 = 8' |
| bash | -c | nft add table ip filter |
And admin executes on the "<%= cb.server_pod.name %>" pod:
| bash | -c | nft add chain ip filter INPUT { type filter hook input priority 0 \; } |
And admin executes on the "<%= cb.server_pod.name %>" pod:
| bash | -c | nft add rule ip filter INPUT iifname eth0 ip protocol 1 @nh,32,32 0x08000000 accept |
Then the step should succeed
When I execute on the "<%= cb.client_pod %>" pod:
| ping |-s 2000 | -c1 | <%= cb.server_pod.ip %> |
Then the step should succeed

#Successful pings will represent a successful xt_u32 match. Here we allow ping for less than 256 bytes packet else Reject
#Add rule to reject packets with total length >= 256 bytes (0x100). Here we allow ping for less than 256 bytes packet else Reject
Given admin executes on the "<%= cb.server_pod.name %>" pod:
| bash | -c | iptables -t filter -A INPUT -i eth0 -m u32 --u32 '0 & 0xFFFF = 0x100:0xFFFF' -j REJECT |
| bash | -c | nft add rule ip filter INPUT iifname eth0 @nh,32,16 >= 0x100 reject |
Then the step should succeed
When I execute on the "<%= cb.client_pod %>" pod:
| ping | -s 64 |-c1 | <%= cb.server_pod.ip %> |
Expand Down
45 changes: 22 additions & 23 deletions features/networking/sdn.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Feature: SDN related networking scenarios
"""
Given 15 seconds have passed
When I get the networking components logs of the node since "90s" ago
And the output should contain "Using iptables Proxier"
And the output should contain "Using nftables Proxier"
"""

Given I restart the network components on the node after scenario
Expand Down Expand Up @@ -51,34 +51,34 @@ Feature: SDN related networking scenarios
@s390x @ppc64le @heterogeneous @arm64 @amd64
@hypershift-hosted
@critical
Scenario: OCP-11286:SDN iptables rules will be repaired automatically once it gets destroyed
Scenario: OCP-11286:SDN nftables rules will be repaired automatically once it gets destroyed
# we do not detect incomplete rule removal since ~4.3, BZ-1810316
# so only test on >= 4.3
Given the master version >= "4.3"
Given the env is using "OpenShiftSDN" networkType
Given I select a random node's host
And the node iptables config is checked
And the node nftables config is checked
And the step succeeded
And I restart the network components on the node after scenario
And I register clean-up steps:
"""
When the node iptables config is checked
When the node nftables config is checked
Then the step succeeded
"""

When the node standard iptables rules are removed
# wait full iptablesSyncPeriod, which is 30 seconds by default
When the node standard nftables rules are removed
# wait full nftablesSyncPeriod, which is 30 seconds by default
# This step is a negative check, so we have to wait the full period to make sure the rules were not restored.
And 35 seconds have passed
When the node iptables config is checked
When the node nftables config is checked
# Removing individual rules will not trigger automatic repair on < 4.3
# the check should fail
Then the step failed
# >= 4.3 we have to flush all the rules and tables to trigger a repair
When the node standard iptables rules are completely flushed
When the node standard nftables rules are completely flushed
And I wait up to 60 seconds for the steps to pass:
"""
Given the node iptables config is checked
Given the node nftables config is checked
Then the step succeeded
"""

Expand All @@ -103,11 +103,11 @@ Feature: SDN related networking scenarios
Then evaluation of `pod.node_name` is stored in the :node_name clipboard

When I run command on the "<%= cb.node_name %>" node's sdn pod:
| iptables | -S | -t | filter |
| bash | -c | nft list table ip filter |
Then the step should succeed
And the output should contain:
| -N OPENSHIFT-ADMIN-OUTPUT-RULES |
| -A FORWARD -i tun0 ! -o tun0 -m comment --comment "administrator overrides" -j OPENSHIFT-ADMIN-OUTPUT-RULES |
| chain OPENSHIFT-ADMIN-OUTPUT-RULES |
| iifname "tun0" oifname != "tun0" comment "administrator overrides" jump OPENSHIFT-ADMIN-OUTPUT-RULES |

# @author yadu@redhat.com
# @case_id OCP-15251
Expand Down Expand Up @@ -231,18 +231,18 @@ Feature: SDN related networking scenarios
@proxy @noproxy
@s390x @ppc64le @heterogeneous @arm64 @amd64
@hypershift-hosted
Scenario: OCP-23543:SDN The iptables binary on sdn containers should be the same as host
Scenario: OCP-23543:SDN The nftables binary on sdn containers should be the same as host
Given I select a random node's host
When I run commands on the host:
| iptables-save --version |
| nft --version |
Then the step should succeed
And evaluation of `@result[:stdout].scan(/\d\.\d.\d/)[0]` is stored in the :iptables_version_host clipboard
#Comparing host and sdn container version for iptables binary
And evaluation of `@result[:stdout].scan(/v\d+\.\d+\.\d+/)[0]` is stored in the :nft_version_host clipboard
#Comparing host and sdn container version for nftables binary
When I run command on the node's sdn pod:
| iptables-save | --version |
| nft | --version |
Then the step should succeed
And the output should contain:
| <%= cb.iptables_version_host %> |
| <%= cb.nft_version_host %> |

# @author huirwang@redhat.com
# @case_id OCP-25707
Expand Down Expand Up @@ -549,10 +549,9 @@ Feature: SDN related networking scenarios
Given the env is using "OpenShiftSDN" networkType
Given I select a random node's host
And I run commands on the host:
| iptables -t raw -S |
| bash | -c | nft list table ip raw |
Then the step should succeed
And the output should contain:
| -N OPENSHIFT-NOTRACK |
| -A PREROUTING -m comment --comment "disable conntrack for vxlan" -j OPENSHIFT-NOTRACK |
| -A OUTPUT -m comment --comment "disable conntrack for vxlan" -j OPENSHIFT-NOTRACK |
| -A OPENSHIFT-NOTRACK -p udp -m udp --dport 4789 -j NOTRACK |
| chain OPENSHIFT-NOTRACK |
| comment "disable conntrack for vxlan" jump OPENSHIFT-NOTRACK |
| udp dport 4789 notrack |
20 changes: 10 additions & 10 deletions features/networking/service.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -662,7 +662,7 @@ Feature: Service related networking scenarios
@proxy @noproxy
@s390x @ppc64le @heterogeneous @arm64 @amd64
@hypershift-hosted
Scenario: OCP-10216:SDN The iptables rules for the service should be DNAT or REDIRECT to node after being idled
Scenario: OCP-10216:SDN The nftables rules for the service should be DNAT or REDIRECT to node after being idled
Given I have a project
And evaluation of `project.name` is stored in the :proj_name clipboard
Given I obtain test data file "networking/list_for_pods.json"
Expand All @@ -686,11 +686,11 @@ Feature: Service related networking scenarios
And the output should match:
| test-service.*none |
When I run commands on the host:
| iptables -S -t nat \| grep <%= cb.proj_name %>/test-service |
| bash | -c | nft list ruleset \| grep '<%= cb.proj_name %>/test-service' |
Then the step should succeed
And the output should match:
| KUBE-PORTALS-CONTAINER -d <%= cb.service_ip %>/32 -p tcp .* -m tcp --dport 27017 -j (DNAT --to-destination <%= cb.hostip %>:\d+\|REDIRECT --to-ports \d+) |
| KUBE-PORTALS-HOST -d <%= cb.service_ip %>/32 -p tcp .* -m tcp --dport 27017 -j DNAT --to-destination <%= cb.hostip %>:\d+ |
| KUBE-PORTALS-CONTAINER.*<%= cb.service_ip %>/32.*tcp.*dport 27017.*(dnat to <%= cb.hostip %>:\d+\|redirect to :\d+) |
| KUBE-PORTALS-HOST.*<%= cb.service_ip %>/32.*tcp.*dport 27017.*dnat to <%= cb.hostip %>:\d+ |

Then I wait up to 60 seconds for the steps to pass:
"""
Expand All @@ -707,14 +707,14 @@ Feature: Service related networking scenarios
And the output should match:
| test-service\s+<%= cb.pod_ip %>:8080 |
When I run commands on the host:
| iptables -S -t nat \| grep <%= cb.proj_name %>/test-service |
| bash | -c | nft list ruleset \| grep '<%= cb.proj_name %>/test-service' |
Then the step should succeed
And the output should not contain "REDIRECT"
And the output should not contain "redirect"
And the output should match:
| KUBE-SEP-.+ -s <%= cb.pod_ip %>/32 .* -j KUBE-MARK-MASQ |
| KUBE-SEP-.+ -p tcp .* -m tcp -j DNAT --to-destination <%= cb.pod_ip %>:8080 |
| KUBE-SERVICES -d <%= cb.service_ip %>/32 -p tcp .* -m tcp --dport 27017 -j KUBE-SVC-.+ |
| KUBE-SVC-.+ .* -j KUBE-SEP-.+ |
| KUBE-SEP.*<%= cb.pod_ip %>/32.*KUBE-MARK-MASQ |
| KUBE-SEP.*tcp.*dnat to <%= cb.pod_ip %>:8080 |
| KUBE-SERVICES.*<%= cb.service_ip %>/32.*tcp.*dport 27017.*KUBE-SVC |
| KUBE-SVC.*KUBE-SEP |


# @author jechen@redhat.com
Expand Down