Bypass Tests
The following scenario shows different configuration alternatives to improve the OSDx firewall performance.
Test Local Bypass
Description
Builds a scenario with three DUTs in which a performance test is carried out between DUT1 and DUT2, and DUT0 is the router running the firewall. “Local bypass” is set to allow the firewall to internally skips packets belonging to a flow that must be bypassed. The performance test may produce better results than the general tests.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 17308 0 --:--:-- --:--:-- --:--:-- 18428
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0 :
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic queue FW_Q elements 1
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.584 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.584/0.584/0.584/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.493 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.493/0.493/0.493/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 53500 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 60.9 MBytes 511 Mbits/sec 35 1.58 MBytes [ 5] 1.00-2.00 sec 45.0 MBytes 377 Mbits/sec 0 1.75 MBytes [ 5] 2.00-3.00 sec 66.2 MBytes 556 Mbits/sec 0 1.88 MBytes [ 5] 3.00-4.00 sec 47.5 MBytes 398 Mbits/sec 0 1.98 MBytes [ 5] 4.00-5.00 sec 51.2 MBytes 430 Mbits/sec 0 2.06 MBytes [ 5] 5.00-6.00 sec 48.8 MBytes 409 Mbits/sec 0 2.12 MBytes [ 5] 6.00-7.00 sec 47.5 MBytes 398 Mbits/sec 0 2.17 MBytes [ 5] 7.00-8.00 sec 50.0 MBytes 419 Mbits/sec 0 2.19 MBytes [ 5] 8.00-9.00 sec 51.2 MBytes 430 Mbits/sec 0 2.21 MBytes [ 5] 9.00-10.00 sec 50.0 MBytes 419 Mbits/sec 0 2.21 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 518 MBytes 435 Mbits/sec 35 sender [ 5] 0.00-10.03 sec 517 MBytes 433 Mbits/sec receiver iperf Done.
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
06/25/2025-10:09:22.857945 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:53484 -> 40.0.0.2:5001 06/25/2025-10:09:22.858943 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:53500 -> 40.0.0.2:5001
Test Capture Bypass Using Packet Mark
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. “Capture bypass” is set to allow the firewall to mark packets. An external tool can then decide what to do with the flow when the mark is seen. For this example, when packet marks are detected, the traffic is assigned a label, thereby allowing the possibility of classifying traffic. In particular, labeling avoids traffic from entering the firewall.
Performance must improve considerably compared to the Local Bypass test.
The test is extended by using other packet marks that we have customized for the firewall.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 780 0 --:--:-- --:--:-- --:--:-- 777
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0 :
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass mark 129834765 set service firewall FW stream bypass mask 129834765 set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic label BYPASS set traffic policy FW-SKIP rule 1 log prefix SKIP set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS set traffic policy FW-SKIP rule 1 set label BYPASS set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE set traffic queue FW_Q elements 1 set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS set traffic selector MARKED-PACKETS rule 1 mark 129834765
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.538 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.538/0.538/0.538/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.495 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.495/0.495/0.495/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 57986 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 76.1 MBytes 639 Mbits/sec 97 1.32 MBytes [ 5] 1.00-2.00 sec 115 MBytes 965 Mbits/sec 0 1.46 MBytes [ 5] 2.00-3.00 sec 251 MBytes 2.11 Gbits/sec 0 1.59 MBytes [ 5] 3.00-4.00 sec 282 MBytes 2.37 Gbits/sec 0 1.68 MBytes [ 5] 4.00-5.00 sec 90.0 MBytes 755 Mbits/sec 1 2.42 MBytes [ 5] 5.00-6.00 sec 285 MBytes 2.39 Gbits/sec 58 1.96 MBytes [ 5] 6.00-7.00 sec 211 MBytes 1.77 Gbits/sec 1 1.41 KBytes [ 5] 7.00-8.00 sec 336 MBytes 2.82 Gbits/sec 0 2.55 MBytes [ 5] 8.00-9.00 sec 321 MBytes 2.69 Gbits/sec 25 1.95 MBytes [ 5] 9.00-10.00 sec 302 MBytes 2.54 Gbits/sec 0 2.11 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.22 GBytes 1.91 Gbits/sec 182 sender [ 5] 0.00-10.00 sec 2.22 GBytes 1.90 Gbits/sec receiver iperf Done.
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
06/25/2025-10:09:56.536109 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:57974 -> 40.0.0.2:5001 06/25/2025-10:09:56.537337 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:57986 -> 40.0.0.2:5001
Step 8: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:
(?m)^.*\[SKIP\-1\].*$Show output
Jun 25 10:09:48.000333 osdx systemd-timedated[360309]: Changed local time to Wed 2025-06-25 10:09:48 UTC Jun 25 10:09:48.001908 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'set date 2025-06-25 10:09:48'. Jun 25 10:09:48.003890 osdx systemd-journald[1763]: Time jumped backwards, rotating. Jun 25 10:09:48.327960 osdx systemd-journald[1763]: Runtime Journal (/run/log/journal/68a40460747548beabf18e9bf53e3a28) is 2.1M, max 15.3M, 13.1M free. Jun 25 10:09:48.331895 osdx systemd-journald[1763]: Received client request to rotate journal, rotating. Jun 25 10:09:48.331959 osdx systemd-journald[1763]: Vacuuming done, freed 0B of archived journals from /run/log/journal/68a40460747548beabf18e9bf53e3a28. Jun 25 10:09:48.339011 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'system journal clear'. Jun 25 10:09:48.586202 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'system coredump delete all'. Jun 25 10:09:48.818447 osdx OSDxCLI[359893]: User 'admin' entered the configuration menu. Jun 25 10:09:49.040685 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'. Jun 25 10:09:49.099696 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'. Jun 25 10:09:49.196482 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'. Jun 25 10:09:49.252549 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'. Jun 25 10:09:49.356799 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'. Jun 25 10:09:49.415259 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'. Jun 25 10:09:49.537491 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'show working'. Jun 25 10:09:49.606746 osdx ubnt-cfgd[360341]: inactive Jun 25 10:09:49.641581 osdx INFO[360351]: FRR daemons did not change Jun 25 10:09:49.663893 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0 Jun 25 10:09:49.723887 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1 Jun 25 10:09:49.750789 osdx (udev-worker)[360471]: Network interface NamePolicy= disabled on kernel command line. Jun 25 10:09:49.925495 osdx cfgd[1463]: [359893]Completed change to active configuration Jun 25 10:09:49.940216 osdx OSDxCLI[359893]: User 'admin' committed the configuration. Jun 25 10:09:49.958599 osdx OSDxCLI[359893]: User 'admin' left the configuration menu. Jun 25 10:09:52.067151 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'. Jun 25 10:09:52.183456 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'. Jun 25 10:09:52.263219 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'. Jun 25 10:09:52.810781 osdx file_operation[360630]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running:// Jun 25 10:09:53.000532 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'. Jun 25 10:09:53.140165 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'file show running://test-performance.rules'. Jun 25 10:09:53.289927 osdx OSDxCLI[359893]: User 'admin' entered the configuration menu. Jun 25 10:09:53.354850 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'. Jun 25 10:09:53.495054 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'. Jun 25 10:09:53.587556 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'. Jun 25 10:09:53.646458 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'. Jun 25 10:09:53.746902 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'. Jun 25 10:09:53.806739 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'. Jun 25 10:09:53.927736 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'. Jun 25 10:09:53.984462 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'. Jun 25 10:09:54.078628 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'. Jun 25 10:09:54.168796 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'. Jun 25 10:09:54.225644 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic label BYPASS'. Jun 25 10:09:54.316597 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'. Jun 25 10:09:54.374727 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'. Jun 25 10:09:54.524610 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'. Jun 25 10:09:54.583612 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'. Jun 25 10:09:54.686190 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'. Jun 25 10:09:54.744749 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'. Jun 25 10:09:54.844821 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'. Jun 25 10:09:54.915548 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'. Jun 25 10:09:55.024013 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'. Jun 25 10:09:55.116340 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'show working'. Jun 25 10:09:55.213700 osdx ubnt-cfgd[360677]: inactive Jun 25 10:09:55.289712 osdx INFO[360714]: FRR daemons did not change Jun 25 10:09:55.587491 osdx systemd[1]: Reloading. Jun 25 10:09:55.635891 osdx systemd-sysv-generator[360767]: stat() failed on /etc/init.d/README, ignoring: No such file or directory Jun 25 10:09:55.780167 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service... Jun 25 10:09:55.805346 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service. Jun 25 10:09:56.034737 osdx INFO[360749]: Rules successfully loaded Jun 25 10:09:56.035395 osdx cfgd[1463]: [359893]Completed change to active configuration Jun 25 10:09:56.037555 osdx OSDxCLI[359893]: User 'admin' committed the configuration. Jun 25 10:09:56.165397 osdx OSDxCLI[359893]: User 'admin' left the configuration menu. Jun 25 10:09:56.539901 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=35126 DF PROTO=TCP SPT=57974 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d Jun 25 10:09:56.539980 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=32194 DF PROTO=TCP SPT=57986 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d Jun 25 10:10:01.455903 osdx kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 1009205779 wd_nsec: 1009205801 Jun 25 10:10:06.732083 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.
Note
The following steps are just a reiteration of the previous test, but with the difference that the packet mark is an extra mark.
Step 9: Modify the following configuration lines in DUT0 :
set service firewall FW stream bypass extra-mark 1 mask 3294967295 set service firewall FW stream bypass extra-mark 1 value 3294967295 set traffic policy FW-SKIP rule 1 selector FW_SEL_EXTRA_MARK set traffic selector FW_SEL_EXTRA_MARK rule 1 extra-mark 1 value 3294967295
Step 10: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 35896 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 244 MBytes 2.05 Gbits/sec 311 1.35 MBytes [ 5] 1.00-2.00 sec 274 MBytes 2.30 Gbits/sec 0 1.49 MBytes [ 5] 2.00-3.00 sec 291 MBytes 2.44 Gbits/sec 0 1.63 MBytes [ 5] 3.00-4.00 sec 266 MBytes 2.23 Gbits/sec 0 1.74 MBytes [ 5] 4.00-5.00 sec 304 MBytes 2.55 Gbits/sec 31 1.35 MBytes [ 5] 5.00-6.00 sec 318 MBytes 2.66 Gbits/sec 0 1.49 MBytes [ 5] 6.00-7.00 sec 268 MBytes 2.24 Gbits/sec 113 1.12 MBytes [ 5] 7.00-8.00 sec 244 MBytes 2.04 Gbits/sec 0 1.27 MBytes [ 5] 8.00-9.00 sec 251 MBytes 2.11 Gbits/sec 0 1.40 MBytes [ 5] 9.00-10.00 sec 312 MBytes 2.62 Gbits/sec 0 1.55 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.71 GBytes 2.32 Gbits/sec 455 sender [ 5] 0.00-10.00 sec 2.71 GBytes 2.32 Gbits/sec receiver iperf Done.
Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
06/25/2025-10:09:56.536109 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:57974 -> 40.0.0.2:5001 06/25/2025-10:09:56.537337 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:57986 -> 40.0.0.2:5001 06/25/2025-10:10:11.383563 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35886 -> 40.0.0.2:5001 06/25/2025-10:10:11.384580 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35896 -> 40.0.0.2:5001
Step 12: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:
(?m)^.*\[SKIP\-1\].*$Show output
Jun 25 10:09:48.000333 osdx systemd-timedated[360309]: Changed local time to Wed 2025-06-25 10:09:48 UTC Jun 25 10:09:48.001908 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'set date 2025-06-25 10:09:48'. Jun 25 10:09:48.003890 osdx systemd-journald[1763]: Time jumped backwards, rotating. Jun 25 10:09:48.327960 osdx systemd-journald[1763]: Runtime Journal (/run/log/journal/68a40460747548beabf18e9bf53e3a28) is 2.1M, max 15.3M, 13.1M free. Jun 25 10:09:48.331895 osdx systemd-journald[1763]: Received client request to rotate journal, rotating. Jun 25 10:09:48.331959 osdx systemd-journald[1763]: Vacuuming done, freed 0B of archived journals from /run/log/journal/68a40460747548beabf18e9bf53e3a28. Jun 25 10:09:48.339011 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'system journal clear'. Jun 25 10:09:48.586202 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'system coredump delete all'. Jun 25 10:09:48.818447 osdx OSDxCLI[359893]: User 'admin' entered the configuration menu. Jun 25 10:09:49.040685 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'. Jun 25 10:09:49.099696 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'. Jun 25 10:09:49.196482 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'. Jun 25 10:09:49.252549 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'. Jun 25 10:09:49.356799 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'. Jun 25 10:09:49.415259 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'. Jun 25 10:09:49.537491 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'show working'. Jun 25 10:09:49.606746 osdx ubnt-cfgd[360341]: inactive Jun 25 10:09:49.641581 osdx INFO[360351]: FRR daemons did not change Jun 25 10:09:49.663893 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0 Jun 25 10:09:49.723887 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1 Jun 25 10:09:49.750789 osdx (udev-worker)[360471]: Network interface NamePolicy= disabled on kernel command line. Jun 25 10:09:49.925495 osdx cfgd[1463]: [359893]Completed change to active configuration Jun 25 10:09:49.940216 osdx OSDxCLI[359893]: User 'admin' committed the configuration. Jun 25 10:09:49.958599 osdx OSDxCLI[359893]: User 'admin' left the configuration menu. Jun 25 10:09:52.067151 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'. Jun 25 10:09:52.183456 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'. Jun 25 10:09:52.263219 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'. Jun 25 10:09:52.810781 osdx file_operation[360630]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running:// Jun 25 10:09:53.000532 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'. Jun 25 10:09:53.140165 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'file show running://test-performance.rules'. Jun 25 10:09:53.289927 osdx OSDxCLI[359893]: User 'admin' entered the configuration menu. Jun 25 10:09:53.354850 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'. Jun 25 10:09:53.495054 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'. Jun 25 10:09:53.587556 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'. Jun 25 10:09:53.646458 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'. Jun 25 10:09:53.746902 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'. Jun 25 10:09:53.806739 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'. Jun 25 10:09:53.927736 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'. Jun 25 10:09:53.984462 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'. Jun 25 10:09:54.078628 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'. Jun 25 10:09:54.168796 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'. Jun 25 10:09:54.225644 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic label BYPASS'. Jun 25 10:09:54.316597 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'. Jun 25 10:09:54.374727 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'. Jun 25 10:09:54.524610 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'. Jun 25 10:09:54.583612 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'. Jun 25 10:09:54.686190 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'. Jun 25 10:09:54.744749 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'. Jun 25 10:09:54.844821 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'. Jun 25 10:09:54.915548 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'. Jun 25 10:09:55.024013 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'. Jun 25 10:09:55.116340 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'show working'. Jun 25 10:09:55.213700 osdx ubnt-cfgd[360677]: inactive Jun 25 10:09:55.289712 osdx INFO[360714]: FRR daemons did not change Jun 25 10:09:55.587491 osdx systemd[1]: Reloading. Jun 25 10:09:55.635891 osdx systemd-sysv-generator[360767]: stat() failed on /etc/init.d/README, ignoring: No such file or directory Jun 25 10:09:55.780167 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service... Jun 25 10:09:55.805346 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service. Jun 25 10:09:56.034737 osdx INFO[360749]: Rules successfully loaded Jun 25 10:09:56.035395 osdx cfgd[1463]: [359893]Completed change to active configuration Jun 25 10:09:56.037555 osdx OSDxCLI[359893]: User 'admin' committed the configuration. Jun 25 10:09:56.165397 osdx OSDxCLI[359893]: User 'admin' left the configuration menu. Jun 25 10:09:56.539901 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=35126 DF PROTO=TCP SPT=57974 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d Jun 25 10:09:56.539980 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=32194 DF PROTO=TCP SPT=57986 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d Jun 25 10:10:01.455903 osdx kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 1009205779 wd_nsec: 1009205801 Jun 25 10:10:06.732083 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'. Jun 25 10:10:06.828248 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'system journal show | cat'. Jun 25 10:10:06.974846 osdx OSDxCLI[359893]: User 'admin' entered the configuration menu. Jun 25 10:10:07.035453 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'. Jun 25 10:10:07.048189 osdx systemd[1]: Starting logrotate.service - Rotate log files... Jun 25 10:10:07.073207 osdx systemd[1]: logrotate.service: Deactivated successfully. Jun 25 10:10:07.073325 osdx systemd[1]: Finished logrotate.service - Rotate log files. Jun 25 10:10:07.135189 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'. Jun 25 10:10:07.191049 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'. Jun 25 10:10:07.290251 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'. Jun 25 10:10:07.348616 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'. Jun 25 10:10:07.437244 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'. Jun 25 10:10:07.529915 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'. Jun 25 10:10:07.607373 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'. Jun 25 10:10:07.705593 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'. Jun 25 10:10:07.769046 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'. Jun 25 10:10:07.868939 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic label BYPASS'. Jun 25 10:10:07.932182 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'. Jun 25 10:10:08.033618 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'. Jun 25 10:10:08.093387 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'. Jun 25 10:10:08.186338 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'. Jun 25 10:10:08.244952 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'. Jun 25 10:10:08.343915 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'. Jun 25 10:10:08.402242 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'. Jun 25 10:10:08.502611 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'. Jun 25 10:10:08.558510 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'. Jun 25 10:10:08.657773 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass extra-mark 1 value 3294967295'. Jun 25 10:10:08.715223 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass extra-mark 1 mask 3294967295'. Jun 25 10:10:08.817832 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector FW_SEL_EXTRA_MARK'. Jun 25 10:10:08.877593 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_EXTRA_MARK rule 1 extra-mark 1 value 3294967295'. Jun 25 10:10:09.007783 osdx OSDxCLI[359893]: User 'admin' added a new cfg line: 'show changes'. Jun 25 10:10:09.096500 osdx ubnt-cfgd[360895]: inactive Jun 25 10:10:09.136980 osdx INFO[360911]: FRR daemons did not change Jun 25 10:10:09.402651 osdx systemd[1]: Stopping suricata@FW.service - Suricata client "FW" service... Jun 25 10:10:10.868500 osdx systemd[1]: suricata@FW.service: Deactivated successfully. Jun 25 10:10:10.868637 osdx systemd[1]: Stopped suricata@FW.service - Suricata client "FW" service. Jun 25 10:10:10.868676 osdx systemd[1]: suricata@FW.service: Consumed 1.535s CPU time. Jun 25 10:10:10.896308 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service... Jun 25 10:10:10.919653 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service. Jun 25 10:10:11.136297 osdx INFO[360936]: Rules successfully loaded Jun 25 10:10:11.136825 osdx cfgd[1463]: [359893]Completed change to active configuration Jun 25 10:10:11.139003 osdx OSDxCLI[359893]: User 'admin' committed the configuration. Jun 25 10:10:11.157922 osdx OSDxCLI[359893]: User 'admin' left the configuration menu. Jun 25 10:10:11.383905 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=55928 DF PROTO=TCP SPT=35886 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d EMARK1=0xc46535ff Jun 25 10:10:11.388190 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=28252 DF PROTO=TCP SPT=35896 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d EMARK1=0xc46535ff Jun 25 10:10:18.032033 osdx systemd[1]: systemd-timedated.service: Deactivated successfully. Jun 25 10:10:21.560564 osdx OSDxCLI[359893]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.
Test Capture Bypass Using Conntrack Mark
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later.
Performance must improve considerably compared to the Local Bypass test.
Then this test is broadened by using other conntrack marks that we have customized for the firewall.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 22595 0 --:--:-- --:--:-- --:--:-- 25800
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0 :
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass mark 129834765 set service firewall FW stream bypass mask 129834765 set service firewall FW stream bypass set-connmark set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 2 action enqueue FW_Q set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE set traffic queue FW_Q elements 1 set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.543 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.543/0.543/0.543/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.460 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.460/0.460/0.460/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 58670 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 48.9 MBytes 410 Mbits/sec 1 1.65 MBytes [ 5] 1.00-2.00 sec 181 MBytes 1.52 Gbits/sec 50 1.74 MBytes [ 5] 2.00-3.00 sec 271 MBytes 2.28 Gbits/sec 0 1.89 MBytes [ 5] 3.00-4.00 sec 255 MBytes 2.14 Gbits/sec 0 2.01 MBytes [ 5] 4.00-5.00 sec 265 MBytes 2.22 Gbits/sec 0 2.11 MBytes [ 5] 5.00-6.00 sec 274 MBytes 2.30 Gbits/sec 0 2.18 MBytes [ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec 426 1.72 MBytes [ 5] 7.00-8.00 sec 295 MBytes 2.47 Gbits/sec 0 1.87 MBytes [ 5] 8.00-9.00 sec 338 MBytes 2.83 Gbits/sec 0 1.98 MBytes [ 5] 9.00-10.00 sec 311 MBytes 2.61 Gbits/sec 0 2.07 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.46 GBytes 2.11 Gbits/sec 477 sender [ 5] 0.00-10.00 sec 2.46 GBytes 2.11 Gbits/sec receiver iperf Done.
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
06/25/2025-10:10:43.822797 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:58654 -> 40.0.0.2:5001 06/25/2025-10:10:43.823672 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:58670 -> 40.0.0.2:5001
Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*mark=129834765.*$Show output
icmp 1 19 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=39 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=39 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 tcp 6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=58654 dport=5001 packets=16 bytes=1303 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=58654 packets=13 bytes=1016 [ASSURED] (Sc: not-bypass) mark=129834765 use=1 icmp 1 19 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=261 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=261 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 tcp 6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=58670 dport=5001 packets=1825391 bytes=2738081265 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=58670 packets=203358 bytes=10569848 [ASSURED] (Sc: not-bypass) mark=129834765 use=1 conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.
Note
The following steps are just a reiteration of the previous test, but with the difference that the conntrack mark used is an extra connmark.
Step 9: Modify the following configuration lines in DUT0 :
set service firewall FW stream bypass extra-mark 2 mask 3294967295 set service firewall FW stream bypass extra-mark 2 set-extra-connmark set service firewall FW stream bypass extra-mark 2 value 3294967295 set traffic policy FW_PLAN rule 2 selector FW_SEL_EXTRA_MARK set traffic selector FW_SEL_EXTRA_MARK rule 1 not extra-connmark 2 value 3294967295
Step 10: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 39942 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 315 MBytes 2.64 Gbits/sec 18 1.95 MBytes [ 5] 1.00-2.00 sec 284 MBytes 2.38 Gbits/sec 50 1.48 MBytes [ 5] 2.00-3.00 sec 299 MBytes 2.51 Gbits/sec 0 1.62 MBytes [ 5] 3.00-4.00 sec 311 MBytes 2.61 Gbits/sec 0 1.75 MBytes [ 5] 4.00-5.00 sec 229 MBytes 1.92 Gbits/sec 1 1.41 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 1.87 MBytes [ 5] 6.00-7.00 sec 265 MBytes 2.22 Gbits/sec 248 969 KBytes [ 5] 7.00-8.00 sec 262 MBytes 2.20 Gbits/sec 0 1.16 MBytes [ 5] 8.00-9.00 sec 269 MBytes 2.25 Gbits/sec 0 1.36 MBytes [ 5] 9.00-10.00 sec 269 MBytes 2.25 Gbits/sec 0 1.51 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.44 GBytes 2.10 Gbits/sec 317 sender [ 5] 0.00-10.00 sec 2.44 GBytes 2.10 Gbits/sec receiver iperf Done.
Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
06/25/2025-10:10:43.822797 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:58654 -> 40.0.0.2:5001 06/25/2025-10:10:43.823672 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:58670 -> 40.0.0.2:5001 06/25/2025-10:10:58.864277 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:39936 -> 40.0.0.2:5001 06/25/2025-10:10:58.865110 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:39942 -> 40.0.0.2:5001
Step 12: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*emark2=3294967295.*$Show output
tcp 6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=39936 dport=5001 packets=16 bytes=1299 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=39936 packets=13 bytes=1016 [ASSURED] (Sc: not-bypass) mark=129834765 emark2=3294967295 use=1 tcp 6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=39942 dport=5001 packets=1812635 bytes=2718941201 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=39942 packets=320789 bytes=16674100 [ASSURED] (Sc: not-bypass) mark=129834765 emark2=3294967295 use=1 conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.
Test Bypass-Drop Using Conntrack Marks
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test is aimed at configuring “Capture bypass drop” to avoid dropped packets from entering the firewall.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 200 100 200 0 0 1916 0 --:--:-- --:--:-- --:--:-- 1923
Step 2: Run command file show running://drop-performance.rules at DUT0 and expect this output:
Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;) drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)
Step 3: Set the following configuration in DUT0 :
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW bypass action drop set connmark mark 147652983 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://drop-performance.rules' set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 1 action drop set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP set traffic policy FW_PLAN rule 2 action enqueue FW_Q set traffic queue FW_Q elements 1 set traffic selector FW_SEL_DROP rule 1 connmark 147652983
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.528 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.528/0.528/0.528/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.440 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.440/0.440/0.440/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5000 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1Expect this output in
DUT2:^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated admin@osdx$
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Dropping TCP performance test traffic).+$Show output
06/25/2025-10:11:30.507530 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:46328 -> 40.0.0.2:5000
Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*mark=147652983.*$Show output
tcp 6 29 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=46328 dport=5000 packets=8 bytes=610 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=46328 packets=5 bytes=270 [ASSURED] (Sc: not-bypass) mark=147652983 use=1 icmp 1 26 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=42 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=42 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 icmp 1 26 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=264 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=264 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 conntrack v1.4.7 (conntrack-tools): 3 flow entries have been shown.
Step 9: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:
(?m)^1\s+FW_SEL_DROP\s+[1-9].*$Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high ------------------------------------------------------------------ rule selector pkts match pkts eval bytes match bytes eval ------------------------------------------------------------------ 1 FW_SEL_DROP 4 7 210 438 2 - 3 3 228 228 ------------------------------------------------------------------ Total 7 7 438 438 Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high ------------------------------------------------------------------ rule selector pkts match pkts eval bytes match bytes eval ------------------------------------------------------------------ 1 FW_SEL_DROP 6 11 499 868 2 - 5 5 369 369 ------------------------------------------------------------------ Total 11 11 868 868
Note
Testing with another conntrack mark.
Step 10: Modify the following configuration lines in DUT0 :
delete service firewall FW bypass action drop set connmark mark set service firewall FW bypass action drop set connmark extra-mark 2 value 3967295294 set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP_EM set traffic selector FW_SEL_DROP_EM rule 1 extra-connmark 2 value 3967295294
Step 11: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5000 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1Expect this output in
DUT2:^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated admin@osdx$
Step 12: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Dropping TCP performance test traffic).+$Show output
06/25/2025-10:11:30.507530 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:46328 -> 40.0.0.2:5000 06/25/2025-10:11:36.870062 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:46328 -> 40.0.0.2:5000 06/25/2025-10:11:38.397276 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:57804 -> 40.0.0.2:5000
Step 13: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*emark2=3967295294.*$Show output
tcp 6 30 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=57804 dport=5000 packets=8 bytes=610 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=57804 packets=5 bytes=270 [ASSURED] (Sc: not-bypass) mark=0 emark2=3967295294 use=1 conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.
Step 14: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:
(?m)^1\s+FW_SEL_DROP_EM\s+[1-9].*$Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high --------------------------------------------------------------------- rule selector pkts match pkts eval bytes match bytes eval --------------------------------------------------------------------- 1 FW_SEL_DROP_EM 4 7 210 376 2 - 3 3 166 166 --------------------------------------------------------------------- Total 7 7 376 376 Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high --------------------------------------------------------------------- rule selector pkts match pkts eval bytes match bytes eval --------------------------------------------------------------------- 1 FW_SEL_DROP_EM 6 11 499 830 2 - 5 5 331 331 --------------------------------------------------------------------- Total 11 11 830 830
Test Capture And Offload
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later. In addition, OSDx is instructed to accelerate the flow using internal accelerators.
Performance must improve considerably compared to the previous test, to reach its top value.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 18804 0 --:--:-- --:--:-- --:--:-- 21500
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0 :
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass action accept set conntrack offload-flag set service firewall FW stream bypass mark 129834765 set service firewall FW stream bypass mask 129834765 set service firewall FW stream bypass set-connmark set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic policy FW_PLAN rule 2 action enqueue FW_Q set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE set traffic queue FW_Q elements 1 set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.656 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.656/0.656/0.656/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.450 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.450/0.450/0.450/0.000 ms
Step 6: Initiate a background bandwidth test from DUT2 to DUT1. The control is returned back allowing to perform another tasks while test is running
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
06/25/2025-10:12:01.883697 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54568 -> 40.0.0.2:5001 06/25/2025-10:12:01.884812 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54576 -> 40.0.0.2:5001
Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$Show output
icmp 1 29 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=267 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=267 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 tcp 6 src=20.0.0.2 dst=40.0.0.2 sport=54568 dport=5001 packets=8 bytes=589 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=54568 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=2 bytes=104 packets=4 bytes=211] mark=129834765 use=3 icmp 1 29 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=45 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=45 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 tcp 6 src=20.0.0.2 dst=40.0.0.2 sport=54576 dport=5001 packets=18609 bytes=27907753 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=54576 packets=2613 bytes=135884 [ASSURED] [OFFLOAD, packets=18596 bytes=27892552 packets=2611 bytes=135772] mark=129834765 use=2 conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.
Step 9: Stop the current bandwidth test between DUT2 and DUT1
Step 10: Run command file copy http://10.215.168.1/~robot/test-performance-udp.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 133 100 133 0 0 3955 0 --:--:-- --:--:-- --:--:-- 4030
Step 11: Run command file show running://test-performance-udp.rules at DUT0 and expect this output:
Show output
alert udp any any -> any 5001 (msg: "Skipping test network performance UDP traffic"; bypass; flow: established, to_server; sid: 41;)
Step 12: Modify the following configuration lines in DUT0 :
set service firewall FW ruleset file 'running://test-performance-udp.rules'
Step 13: Initiate a background bandwidth test from DUT2 to DUT1. The control is returned back allowing to perform another tasks while test is running
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 udp port 5001 parallel 1
Step 14: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance UDP traffic).+$Show output
06/25/2025-10:12:01.883697 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54568 -> 40.0.0.2:5001 06/25/2025-10:12:01.884812 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54576 -> 40.0.0.2:5001 06/25/2025-10:12:04.904000 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54578 -> 40.0.0.2:5001 06/25/2025-10:12:04.906437 [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:56764 -> 40.0.0.2:5001 06/25/2025-10:12:04.916548 [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:56764 -> 40.0.0.2:5001 06/25/2025-10:12:04.928580 [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:56764 -> 40.0.0.2:5001
Step 15: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^udp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$Show output
tcp 6 src=20.0.0.2 dst=40.0.0.2 sport=54578 dport=5001 packets=8 bytes=607 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=54578 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=2 bytes=104 packets=4 bytes=211] mark=129834765 use=3 udp 17 src=20.0.0.2 dst=40.0.0.2 sport=56764 dport=5001 packets=13 bytes=17744 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=56764 packets=1 bytes=32 [OFFLOAD, packets=9 bytes=13284 packets=0 bytes=0] mark=129834765 use=2 conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.
Step 16: Stop the current bandwidth test between DUT2 and DUT1