Bypass Tests

The following scenario shows different configuration alternatives to improve the OSDx firewall performance.

../../../_images/topology32.svg

Test Local Bypass

Description

Builds a scenario with three DUTs in which a performance test is carried out between DUT1 and DUT2, and DUT0 is the router running the firewall. “Local bypass” is set to allow the firewall to internally skips packets belonging to a flow that must be bypassed. The performance test may produce better results than the general tests.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  11598      0 --:--:-- --:--:-- --:--:-- 11727

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic queue FW_Q elements 1

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.586 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.586/0.586/0.586/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.430 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.430/0.430/0.430/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 35838 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  71.1 MBytes   597 Mbits/sec    0   2.54 MBytes
[  5]   1.00-2.00   sec  38.8 MBytes   325 Mbits/sec  132   2.14 MBytes
[  5]   2.00-3.00   sec  12.5 MBytes   105 Mbits/sec   40   1.58 MBytes
[  5]   3.00-4.00   sec  28.8 MBytes   241 Mbits/sec  136   1.41 KBytes
[  5]   4.00-5.00   sec  7.50 MBytes  62.9 Mbits/sec    0   1.98 MBytes
[  5]   5.00-6.00   sec  72.5 MBytes   608 Mbits/sec   21   1.85 MBytes
[  5]   6.00-7.00   sec  50.0 MBytes   419 Mbits/sec    0   2.01 MBytes
[  5]   7.00-8.00   sec  48.8 MBytes   409 Mbits/sec    0   2.15 MBytes
[  5]   8.00-9.00   sec  50.0 MBytes   419 Mbits/sec    0   2.25 MBytes
[  5]   9.00-10.00  sec  65.0 MBytes   545 Mbits/sec    6   1.64 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   445 MBytes   373 Mbits/sec  335             sender
[  5]   0.00-10.01  sec   443 MBytes   371 Mbits/sec                  receiver

iperf Done.

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/23/2025-08:59:39.176722  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35830 -> 40.0.0.2:5001
06/23/2025-08:59:39.177741  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35838 -> 40.0.0.2:5001

Test Capture Bypass Using Packet Mark

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. “Capture bypass” is set to allow the firewall to mark packets. An external tool can then decide what to do with the flow when the mark is seen. For this example, when packet marks are detected, the traffic is assigned a label, thereby allowing the possibility of classifying traffic. In particular, labeling avoids traffic from entering the firewall.

Performance must improve considerably compared to the Local Bypass test.

The test is extended by using other packet marks that we have customized for the firewall.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0   3610      0 --:--:-- --:--:-- --:--:--  3685

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass mark 129834765
set service firewall FW stream bypass mask 129834765
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic label BYPASS
set traffic policy FW-SKIP rule 1 log prefix SKIP
set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS
set traffic policy FW-SKIP rule 1 set label BYPASS
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS
set traffic selector MARKED-PACKETS rule 1 mark 129834765

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.499 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.499/0.499/0.499/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.454 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.454/0.454/0.454/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 54434 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  44.4 MBytes   372 Mbits/sec    1   1.63 MBytes
[  5]   1.00-2.00   sec   184 MBytes  1.54 Gbits/sec   45   1.87 MBytes
[  5]   2.00-3.00   sec   320 MBytes  2.68 Gbits/sec    0   2.05 MBytes
[  5]   3.00-4.00   sec   305 MBytes  2.56 Gbits/sec    0   2.18 MBytes
[  5]   4.00-5.00   sec   308 MBytes  2.58 Gbits/sec   14   1.64 MBytes
[  5]   5.00-6.00   sec   315 MBytes  2.64 Gbits/sec    0   1.77 MBytes
[  5]   6.00-7.00   sec   306 MBytes  2.57 Gbits/sec    0   1.89 MBytes
[  5]   7.00-8.00   sec  27.5 MBytes   231 Mbits/sec    1   2.05 MBytes
[  5]   8.00-9.00   sec   278 MBytes  2.33 Gbits/sec   10   1.94 MBytes
[  5]   9.00-10.00  sec   264 MBytes  2.21 Gbits/sec    0   2.10 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.30 GBytes  1.97 Gbits/sec   71             sender
[  5]   0.00-10.01  sec  2.29 GBytes  1.97 Gbits/sec                  receiver

iperf Done.

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/23/2025-09:00:09.697332  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54426 -> 40.0.0.2:5001
06/23/2025-09:00:09.698305  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54434 -> 40.0.0.2:5001

Step 8: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:

(?m)^.*\[SKIP\-1\].*$
Show output
Jun 23 09:00:01.357624 osdx systemd-journald[1747]: Runtime Journal (/run/log/journal/76ff399027db4a3db160ed8651bac52f) is 2.2M, max 15.3M, 13.1M free.
Jun 23 09:00:01.360213 osdx systemd-journald[1747]: Received client request to rotate journal, rotating.
Jun 23 09:00:01.360293 osdx systemd-journald[1747]: Vacuuming done, freed 0B of archived journals from /run/log/journal/76ff399027db4a3db160ed8651bac52f.
Jun 23 09:00:01.369619 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'system journal clear'.
Jun 23 09:00:01.618821 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'system coredump delete all'.
Jun 23 09:00:01.866155 osdx OSDxCLI[269307]: User 'admin' entered the configuration menu.
Jun 23 09:00:01.979034 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'.
Jun 23 09:00:02.056935 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'.
Jun 23 09:00:02.161100 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'.
Jun 23 09:00:02.230061 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'.
Jun 23 09:00:02.321419 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'.
Jun 23 09:00:02.389436 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'.
Jun 23 09:00:02.533144 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'show working'.
Jun 23 09:00:02.642753 osdx INFO[269664]: FRR daemons did not change
Jun 23 09:00:02.664217 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0
Jun 23 09:00:02.756223 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1
Jun 23 09:00:02.786515 osdx (udev-worker)[269767]: Network interface NamePolicy= disabled on kernel command line.
Jun 23 09:00:03.002515 osdx cfgd[1453]: [269307]Completed change to active configuration
Jun 23 09:00:03.028361 osdx OSDxCLI[269307]: User 'admin' committed the configuration.
Jun 23 09:00:03.044807 osdx OSDxCLI[269307]: User 'admin' left the configuration menu.
Jun 23 09:00:05.424354 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'.
Jun 23 09:00:05.505269 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'.
Jun 23 09:00:05.587430 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'.
Jun 23 09:00:06.132580 osdx file_operation[269855]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running://
Jun 23 09:00:06.193373 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'.
Jun 23 09:00:06.373489 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'file show running://test-performance.rules'.
Jun 23 09:00:06.525209 osdx OSDxCLI[269307]: User 'admin' entered the configuration menu.
Jun 23 09:00:06.598027 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'.
Jun 23 09:00:06.701188 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'.
Jun 23 09:00:06.756407 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'.
Jun 23 09:00:06.857221 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'.
Jun 23 09:00:06.936910 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'.
Jun 23 09:00:07.037782 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'.
Jun 23 09:00:07.119722 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'.
Jun 23 09:00:07.217335 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'.
Jun 23 09:00:07.292322 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'.
Jun 23 09:00:07.381098 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'.
Jun 23 09:00:07.466403 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic label BYPASS'.
Jun 23 09:00:07.568877 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'.
Jun 23 09:00:07.628477 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'.
Jun 23 09:00:07.736330 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'.
Jun 23 09:00:07.795904 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'.
Jun 23 09:00:07.902143 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'.
Jun 23 09:00:07.960331 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'.
Jun 23 09:00:08.061581 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'.
Jun 23 09:00:08.142119 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'.
Jun 23 09:00:08.234373 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'.
Jun 23 09:00:08.342027 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'show working'.
Jun 23 09:00:08.495130 osdx INFO[269937]: FRR daemons did not change
Jun 23 09:00:08.774549 osdx systemd[1]: Reloading.
Jun 23 09:00:08.892213 osdx systemd-sysv-generator[269990]: stat() failed on /etc/init.d/README, ignoring: No such file or directory
Jun 23 09:00:09.040647 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service...
Jun 23 09:00:09.065420 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service.
Jun 23 09:00:09.268030 osdx INFO[269972]: Rules successfully loaded
Jun 23 09:00:09.268815 osdx cfgd[1453]: [269307]Completed change to active configuration
Jun 23 09:00:09.270980 osdx OSDxCLI[269307]: User 'admin' committed the configuration.
Jun 23 09:00:09.295496 osdx OSDxCLI[269307]: User 'admin' left the configuration menu.
Jun 23 09:00:09.700241 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=24133 DF PROTO=TCP SPT=54426 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 23 09:00:09.700330 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=37539 DF PROTO=TCP SPT=54434 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 23 09:00:19.908477 osdx systemd[1]: Starting logrotate.service - Rotate log files...
Jun 23 09:00:19.941042 osdx systemd[1]: logrotate.service: Deactivated successfully.
Jun 23 09:00:19.941176 osdx systemd[1]: Finished logrotate.service - Rotate log files.
Jun 23 09:00:19.945208 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.

Note

The following steps are just a reiteration of the previous test, but with the difference that the packet mark is an extra mark.

Step 9: Modify the following configuration lines in DUT0 :

set service firewall FW stream bypass extra-mark 1 mask 3294967295
set service firewall FW stream bypass extra-mark 1 value 3294967295
set traffic policy FW-SKIP rule 1 selector FW_SEL_EXTRA_MARK
set traffic selector FW_SEL_EXTRA_MARK rule 1 extra-mark 1 value 3294967295

Step 10: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 51978 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   230 MBytes  1.93 Gbits/sec  480   1.62 MBytes
[  5]   1.00-2.00   sec   286 MBytes  2.40 Gbits/sec    0   1.76 MBytes
[  5]   2.00-3.00   sec   289 MBytes  2.42 Gbits/sec    0   1.88 MBytes
[  5]   3.00-4.00   sec   194 MBytes  1.62 Gbits/sec    1   1.41 KBytes
[  5]   4.00-5.00   sec   271 MBytes  2.28 Gbits/sec    5   1.93 MBytes
[  5]   5.00-6.00   sec   188 MBytes  1.57 Gbits/sec  492    365 KBytes
[  5]   6.00-7.00   sec   276 MBytes  2.32 Gbits/sec    0    744 KBytes
[  5]   7.00-8.00   sec   249 MBytes  2.09 Gbits/sec    0   1.03 MBytes
[  5]   8.00-9.00   sec   231 MBytes  1.94 Gbits/sec    0   1.32 MBytes
[  5]   9.00-10.01  sec   159 MBytes  1.32 Gbits/sec   39   1.75 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  2.32 GBytes  1.99 Gbits/sec  1017             sender
[  5]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec                  receiver

iperf Done.

Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/23/2025-09:00:09.697332  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54426 -> 40.0.0.2:5001
06/23/2025-09:00:09.698305  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:54434 -> 40.0.0.2:5001
06/23/2025-09:00:24.732780  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:51964 -> 40.0.0.2:5001
06/23/2025-09:00:24.733704  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:51978 -> 40.0.0.2:5001

Step 12: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:

(?m)^.*\[SKIP\-1\].*$
Show output
Jun 23 09:00:01.357624 osdx systemd-journald[1747]: Runtime Journal (/run/log/journal/76ff399027db4a3db160ed8651bac52f) is 2.2M, max 15.3M, 13.1M free.
Jun 23 09:00:01.360213 osdx systemd-journald[1747]: Received client request to rotate journal, rotating.
Jun 23 09:00:01.360293 osdx systemd-journald[1747]: Vacuuming done, freed 0B of archived journals from /run/log/journal/76ff399027db4a3db160ed8651bac52f.
Jun 23 09:00:01.369619 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'system journal clear'.
Jun 23 09:00:01.618821 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'system coredump delete all'.
Jun 23 09:00:01.866155 osdx OSDxCLI[269307]: User 'admin' entered the configuration menu.
Jun 23 09:00:01.979034 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'.
Jun 23 09:00:02.056935 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'.
Jun 23 09:00:02.161100 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'.
Jun 23 09:00:02.230061 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'.
Jun 23 09:00:02.321419 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'.
Jun 23 09:00:02.389436 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'.
Jun 23 09:00:02.533144 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'show working'.
Jun 23 09:00:02.642753 osdx INFO[269664]: FRR daemons did not change
Jun 23 09:00:02.664217 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0
Jun 23 09:00:02.756223 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1
Jun 23 09:00:02.786515 osdx (udev-worker)[269767]: Network interface NamePolicy= disabled on kernel command line.
Jun 23 09:00:03.002515 osdx cfgd[1453]: [269307]Completed change to active configuration
Jun 23 09:00:03.028361 osdx OSDxCLI[269307]: User 'admin' committed the configuration.
Jun 23 09:00:03.044807 osdx OSDxCLI[269307]: User 'admin' left the configuration menu.
Jun 23 09:00:05.424354 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'.
Jun 23 09:00:05.505269 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'.
Jun 23 09:00:05.587430 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'.
Jun 23 09:00:06.132580 osdx file_operation[269855]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running://
Jun 23 09:00:06.193373 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'.
Jun 23 09:00:06.373489 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'file show running://test-performance.rules'.
Jun 23 09:00:06.525209 osdx OSDxCLI[269307]: User 'admin' entered the configuration menu.
Jun 23 09:00:06.598027 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'.
Jun 23 09:00:06.701188 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'.
Jun 23 09:00:06.756407 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'.
Jun 23 09:00:06.857221 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'.
Jun 23 09:00:06.936910 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'.
Jun 23 09:00:07.037782 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'.
Jun 23 09:00:07.119722 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'.
Jun 23 09:00:07.217335 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'.
Jun 23 09:00:07.292322 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'.
Jun 23 09:00:07.381098 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'.
Jun 23 09:00:07.466403 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic label BYPASS'.
Jun 23 09:00:07.568877 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'.
Jun 23 09:00:07.628477 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'.
Jun 23 09:00:07.736330 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'.
Jun 23 09:00:07.795904 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'.
Jun 23 09:00:07.902143 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'.
Jun 23 09:00:07.960331 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'.
Jun 23 09:00:08.061581 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'.
Jun 23 09:00:08.142119 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'.
Jun 23 09:00:08.234373 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'.
Jun 23 09:00:08.342027 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'show working'.
Jun 23 09:00:08.495130 osdx INFO[269937]: FRR daemons did not change
Jun 23 09:00:08.774549 osdx systemd[1]: Reloading.
Jun 23 09:00:08.892213 osdx systemd-sysv-generator[269990]: stat() failed on /etc/init.d/README, ignoring: No such file or directory
Jun 23 09:00:09.040647 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service...
Jun 23 09:00:09.065420 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service.
Jun 23 09:00:09.268030 osdx INFO[269972]: Rules successfully loaded
Jun 23 09:00:09.268815 osdx cfgd[1453]: [269307]Completed change to active configuration
Jun 23 09:00:09.270980 osdx OSDxCLI[269307]: User 'admin' committed the configuration.
Jun 23 09:00:09.295496 osdx OSDxCLI[269307]: User 'admin' left the configuration menu.
Jun 23 09:00:09.700241 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=24133 DF PROTO=TCP SPT=54426 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 23 09:00:09.700330 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=37539 DF PROTO=TCP SPT=54434 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 23 09:00:19.908477 osdx systemd[1]: Starting logrotate.service - Rotate log files...
Jun 23 09:00:19.941042 osdx systemd[1]: logrotate.service: Deactivated successfully.
Jun 23 09:00:19.941176 osdx systemd[1]: Finished logrotate.service - Rotate log files.
Jun 23 09:00:19.945208 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.
Jun 23 09:00:20.050268 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'system journal show | cat'.
Jun 23 09:00:20.204638 osdx OSDxCLI[269307]: User 'admin' entered the configuration menu.
Jun 23 09:00:20.297607 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'.
Jun 23 09:00:20.355403 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'.
Jun 23 09:00:20.453988 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'.
Jun 23 09:00:20.555269 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'.
Jun 23 09:00:20.614970 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'.
Jun 23 09:00:20.731625 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'.
Jun 23 09:00:20.818248 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'.
Jun 23 09:00:20.914042 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'.
Jun 23 09:00:20.971710 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'.
Jun 23 09:00:21.080447 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'.
Jun 23 09:00:21.137462 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic label BYPASS'.
Jun 23 09:00:21.281365 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'.
Jun 23 09:00:21.341308 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'.
Jun 23 09:00:21.448755 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'.
Jun 23 09:00:21.503914 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'.
Jun 23 09:00:21.606714 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'.
Jun 23 09:00:21.667077 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'.
Jun 23 09:00:21.948882 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'.
Jun 23 09:00:22.005977 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'.
Jun 23 09:00:22.106086 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'.
Jun 23 09:00:22.166293 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass extra-mark 1 value 3294967295'.
Jun 23 09:00:22.266228 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass extra-mark 1 mask 3294967295'.
Jun 23 09:00:22.325464 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector FW_SEL_EXTRA_MARK'.
Jun 23 09:00:22.428794 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_EXTRA_MARK rule 1 extra-mark 1 value 3294967295'.
Jun 23 09:00:22.551016 osdx OSDxCLI[269307]: User 'admin' added a new cfg line: 'show changes'.
Jun 23 09:00:22.678847 osdx INFO[270128]: FRR daemons did not change
Jun 23 09:00:22.956956 osdx systemd[1]: Stopping suricata@FW.service - Suricata client "FW" service...
Jun 23 09:00:24.255054 osdx systemd[1]: suricata@FW.service: Deactivated successfully.
Jun 23 09:00:24.255167 osdx systemd[1]: Stopped suricata@FW.service - Suricata client "FW" service.
Jun 23 09:00:24.255196 osdx systemd[1]: suricata@FW.service: Consumed 1.379s CPU time.
Jun 23 09:00:24.272474 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service...
Jun 23 09:00:24.289126 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service.
Jun 23 09:00:24.465793 osdx INFO[270153]: Rules successfully loaded
Jun 23 09:00:24.466446 osdx cfgd[1453]: [269307]Completed change to active configuration
Jun 23 09:00:24.468318 osdx OSDxCLI[269307]: User 'admin' committed the configuration.
Jun 23 09:00:24.498507 osdx OSDxCLI[269307]: User 'admin' left the configuration menu.
Jun 23 09:00:24.737065 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=47806 DF PROTO=TCP SPT=51964 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d EMARK1=0xc46535ff
Jun 23 09:00:24.737152 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=54292 DF PROTO=TCP SPT=51978 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d EMARK1=0xc46535ff
Jun 23 09:00:31.031442 osdx systemd[1]: systemd-timedated.service: Deactivated successfully.
Jun 23 09:00:35.055110 osdx OSDxCLI[269307]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.

Test Capture Bypass Using Conntrack Mark

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later.

Performance must improve considerably compared to the Local Bypass test.

Then this test is broadened by using other conntrack marks that we have customized for the firewall.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  34902      0 --:--:-- --:--:-- --:--:-- 43000

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass mark 129834765
set service firewall FW stream bypass mask 129834765
set service firewall FW stream bypass set-connmark
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 2 action enqueue FW_Q
set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.509 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.509/0.509/0.509/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.524 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.524/0.524/0.524/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 37726 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  56.5 MBytes   474 Mbits/sec    0   1.92 MBytes
[  5]   1.00-2.00   sec  35.0 MBytes   294 Mbits/sec  226   1.41 KBytes
[  5]   2.00-3.00   sec   132 MBytes  1.11 Gbits/sec   87   1.68 MBytes
[  5]   3.00-4.00   sec   262 MBytes  2.20 Gbits/sec   87   1.30 MBytes
[  5]   4.00-5.00   sec   240 MBytes  2.01 Gbits/sec    0   1.43 MBytes
[  5]   5.00-6.00   sec   244 MBytes  2.04 Gbits/sec    0   1.54 MBytes
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0   1.67 MBytes
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec    0   1.78 MBytes
[  5]   8.00-9.00   sec   294 MBytes  2.46 Gbits/sec    0   1.90 MBytes
[  5]   9.00-10.00  sec   284 MBytes  2.38 Gbits/sec   75   1.47 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.05 GBytes  1.76 Gbits/sec  475             sender
[  5]   0.00-10.00  sec  2.05 GBytes  1.76 Gbits/sec                  receiver

iperf Done.

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/23/2025-09:00:56.998669  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37714 -> 40.0.0.2:5001
06/23/2025-09:00:56.999650  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37726 -> 40.0.0.2:5001

Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*mark=129834765.*$
Show output
icmp     1 19 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=150 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=150 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
tcp      6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=37714 dport=5001 packets=16 bytes=1300 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=37714 packets=13 bytes=1020 [ASSURED] (Sc: not-bypass) mark=129834765 use=1
tcp      6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=37726 dport=5001 packets=1523382 bytes=2285063605 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=37726 packets=238872 bytes=12414812 [ASSURED] (Sc: not-bypass) mark=129834765 use=1
icmp     1 19 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=56 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=56 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.

Note

The following steps are just a reiteration of the previous test, but with the difference that the conntrack mark used is an extra connmark.

Step 9: Modify the following configuration lines in DUT0 :

set service firewall FW stream bypass extra-mark 2 mask 3294967295
set service firewall FW stream bypass extra-mark 2 set-extra-connmark
set service firewall FW stream bypass extra-mark 2 value 3294967295
set traffic policy FW_PLAN rule 2 selector FW_SEL_EXTRA_MARK
set traffic selector FW_SEL_EXTRA_MARK rule 1 not extra-connmark 2 value 3294967295

Step 10: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 38746 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   166 MBytes  1.40 Gbits/sec   16   1.79 MBytes
[  5]   1.00-2.00   sec   294 MBytes  2.46 Gbits/sec    0   1.96 MBytes
[  5]   2.00-3.00   sec   300 MBytes  2.52 Gbits/sec    0   2.10 MBytes
[  5]   3.00-4.00   sec   299 MBytes  2.51 Gbits/sec  326    559 KBytes
[  5]   4.00-5.00   sec   269 MBytes  2.25 Gbits/sec    0    851 KBytes
[  5]   5.00-6.00   sec   274 MBytes  2.30 Gbits/sec    0   1.04 MBytes
[  5]   6.00-7.00   sec   296 MBytes  2.49 Gbits/sec    0   1.23 MBytes
[  5]   7.00-8.00   sec   300 MBytes  2.52 Gbits/sec    0   1.40 MBytes
[  5]   8.00-9.00   sec   270 MBytes  2.26 Gbits/sec    0   1.53 MBytes
[  5]   9.00-10.00  sec   285 MBytes  2.39 Gbits/sec    0   1.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.69 GBytes  2.31 Gbits/sec  342             sender
[  5]   0.00-10.00  sec  2.69 GBytes  2.31 Gbits/sec                  receiver

iperf Done.

Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/23/2025-09:00:56.998669  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37714 -> 40.0.0.2:5001
06/23/2025-09:00:56.999650  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37726 -> 40.0.0.2:5001
06/23/2025-09:01:11.913316  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:38744 -> 40.0.0.2:5001
06/23/2025-09:01:11.914144  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:38746 -> 40.0.0.2:5001

Step 12: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*emark2=3294967295.*$
Show output
tcp      6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=38746 dport=5001 packets=1993149 bytes=2989714065 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=38746 packets=308769 bytes=16049308 [ASSURED] (Sc: not-bypass) mark=129834765 emark2=3294967295 use=1
tcp      6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=38744 dport=5001 packets=16 bytes=1302 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=38744 packets=13 bytes=1019 [ASSURED] (Sc: not-bypass) mark=129834765 emark2=3294967295 use=1
conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.

Test Bypass-Drop Using Conntrack Marks

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test is aimed at configuring “Capture bypass drop” to avoid dropped packets from entering the firewall.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   200  100   200    0     0   1022      0 --:--:-- --:--:-- --:--:--  1025

Step 2: Run command file show running://drop-performance.rules at DUT0 and expect this output:

Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)
drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW bypass action drop set connmark mark 147652983
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://drop-performance.rules'
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action drop
set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP
set traffic policy FW_PLAN rule 2 action enqueue FW_Q
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_DROP rule 1 connmark 147652983

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.557 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.557/0.557/0.557/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.449 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.449/0.449/0.449/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5000
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
admin@osdx$

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping TCP performance test traffic).+$
Show output
06/23/2025-09:01:44.743398  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35162 -> 40.0.0.2:5000

Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*mark=147652983.*$
Show output
icmp     1 26 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=153 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=153 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
icmp     1 26 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=59 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=59 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
tcp      6 29 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=35162 dport=5000 packets=8 bytes=610 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=35162 packets=5 bytes=270 [ASSURED] (Sc: not-bypass) mark=147652983 use=1
conntrack v1.4.7 (conntrack-tools): 3 flow entries have been shown.

Step 9: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:

(?m)^1\s+FW_SEL_DROP\s+[1-9].*$
Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high

------------------------------------------------------------------
rule    selector    pkts match  pkts eval  bytes match  bytes eval
------------------------------------------------------------------
1      FW_SEL_DROP           4          7          210         438
2      -                     3          3          228         228
------------------------------------------------------------------
Total                        7          7          438         438

Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high

------------------------------------------------------------------
rule    selector    pkts match  pkts eval  bytes match  bytes eval
------------------------------------------------------------------
1      FW_SEL_DROP           6         11          499         868
2      -                     5          5          369         369
------------------------------------------------------------------
Total                       11         11          868         868

Note

Testing with another conntrack mark.

Step 10: Modify the following configuration lines in DUT0 :

delete service firewall FW bypass action drop set connmark mark
set service firewall FW bypass action drop set connmark extra-mark 2 value 3967295294
set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP_EM
set traffic selector FW_SEL_DROP_EM rule 1 extra-connmark 2 value 3967295294

Step 11: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5000
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
admin@osdx$

Step 12: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping TCP performance test traffic).+$
Show output
06/23/2025-09:01:44.743398  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35162 -> 40.0.0.2:5000
06/23/2025-09:01:51.125015  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:35162 -> 40.0.0.2:5000
06/23/2025-09:01:52.701085  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:41730 -> 40.0.0.2:5000

Step 13: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*emark2=3967295294.*$
Show output
tcp      6 29 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=41730 dport=5000 packets=8 bytes=610 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=41730 packets=5 bytes=270 [ASSURED] (Sc: not-bypass) mark=0 emark2=3967295294 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

Step 14: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:

(?m)^1\s+FW_SEL_DROP_EM\s+[1-9].*$
Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high

---------------------------------------------------------------------
rule      selector     pkts match  pkts eval  bytes match  bytes eval
---------------------------------------------------------------------
1      FW_SEL_DROP_EM           4          7          210         376
2      -                        3          3          166         166
---------------------------------------------------------------------
Total                           7          7          376         376

Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high

---------------------------------------------------------------------
rule      selector     pkts match  pkts eval  bytes match  bytes eval
---------------------------------------------------------------------
1      FW_SEL_DROP_EM           6         11          499         830
2      -                        5          5          331         331
---------------------------------------------------------------------
Total                          11         11          830         830

Test Capture And Offload

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later. In addition, OSDx is instructed to accelerate the flow using internal accelerators.

Performance must improve considerably compared to the previous test, to reach its top value.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  33602      0 --:--:-- --:--:-- --:--:-- 43000

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass action accept set conntrack offload-flag
set service firewall FW stream bypass mark 129834765
set service firewall FW stream bypass mask 129834765
set service firewall FW stream bypass set-connmark
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic policy FW_PLAN rule 2 action enqueue FW_Q
set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.517 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.517/0.517/0.517/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.482 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.482/0.482/0.482/0.000 ms

Step 6: Initiate a background bandwidth test from DUT2 to DUT1. The control is returned back allowing to perform another tasks while test is running

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/23/2025-09:02:15.909899  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45374 -> 40.0.0.2:5001
06/23/2025-09:02:15.910936  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45378 -> 40.0.0.2:5001

Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$
Show output
tcp      6 src=20.0.0.2 dst=40.0.0.2 sport=45374 dport=5001 packets=8 bytes=589 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=45374 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=2 bytes=104 packets=4 bytes=211] mark=129834765 use=3
icmp     1 29 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=62 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=62 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
icmp     1 29 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=156 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=156 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
tcp      6 src=20.0.0.2 dst=40.0.0.2 sport=45378 dport=5001 packets=23799 bytes=35694201 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=45378 packets=2818 bytes=146556 [ASSURED] [OFFLOAD, packets=23787 bytes=35680500 packets=2816 bytes=146444] mark=129834765 use=2
conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.

Step 9: Stop the current bandwidth test between DUT2 and DUT1

Step 10: Run command file copy http://10.215.168.1/~robot/test-performance-udp.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   133  100   133    0     0  11594      0 --:--:-- --:--:-- --:--:-- 12090

Step 11: Run command file show running://test-performance-udp.rules at DUT0 and expect this output:

Show output
alert udp any any -> any 5001 (msg: "Skipping test network performance UDP traffic"; bypass; flow: established, to_server; sid: 41;)

Step 12: Modify the following configuration lines in DUT0 :

set service firewall FW ruleset file 'running://test-performance-udp.rules'

Step 13: Initiate a background bandwidth test from DUT2 to DUT1. The control is returned back allowing to perform another tasks while test is running

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 udp port 5001 parallel 1

Step 14: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance UDP traffic).+$
Show output
06/23/2025-09:02:15.909899  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45374 -> 40.0.0.2:5001
06/23/2025-09:02:15.910936  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45378 -> 40.0.0.2:5001
06/23/2025-09:02:18.210133  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45384 -> 40.0.0.2:5001
06/23/2025-09:02:18.211606  [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:39660 -> 40.0.0.2:5001
06/23/2025-09:02:18.222340  [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:39660 -> 40.0.0.2:5001
06/23/2025-09:02:18.234401  [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:39660 -> 40.0.0.2:5001

Step 15: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^udp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$
Show output
tcp      6 src=20.0.0.2 dst=40.0.0.2 sport=45384 dport=5001 packets=8 bytes=607 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=45384 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=2 bytes=104 packets=4 bytes=211] mark=129834765 use=3
udp      17 src=20.0.0.2 dst=40.0.0.2 sport=39660 dport=5001 packets=16 bytes=22172 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=39660 packets=1 bytes=32 [OFFLOAD, packets=12 bytes=17712 packets=0 bytes=0] mark=129834765 use=2
conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.

Step 16: Stop the current bandwidth test between DUT2 and DUT1


Test Traffic Early Dropping

Description

Builds a scenario with three DUTs and a simple ruleset to drop TCP traffic between DUT1 and DUT2. Such traffic must pass through port 5000 for the rule to match. Later, XDP is queried to check if packets are being dropped at the specified interface.

The contents of the rule file are:

drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)

This rule allows the connection to be established and traffic to be dropped later.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running://drop-performance.rules force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   200  100   200    0     0  38963      0 --:--:-- --:--:-- --:--:-- 40000

Step 2: Run command file show running://drop-performance.rules at DUT0 and expect this output:

Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)
drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://drop-performance.rules'
set service firewall FW stream bypass action drop set xdp-early-drop eth1
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic queue FW_Q elements 1

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.577 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.577/0.577/0.577/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.512 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.512/0.512/0.512/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5000
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
admin@osdx$

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping TCP performance test traffic).+$
Show output
06/23/2025-09:02:38.064543  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:34720 -> 40.0.0.2:5000

Step 8: Run command service firewall FW show early-drop-stats eth1 at DUT0 and check if output matches the following regular expressions:

yes\s+201\s+\d+\s+[1-9]\d*\s+[1-9]\d*
Show output
------------------------------------------------------------------------
  src       dst     src port  dst port  tcp  vlan_0  vlan_1  pkts  bytes
------------------------------------------------------------------------
20.0.0.2  40.0.0.2     34720      5000  yes     201       0     9    731
40.0.0.2  20.0.0.2      5000     34720  yes     201       0     0      0

Step 9: Run command interfaces ethernet eth1 monitor xdp-stats times 1 at DUT0 and expect this output:

Show output
Pin path: /sys/fs/bpf/eth1
Period of 0.250133s ending at 1750669361.838414
XDP_DROP              10 pkts (         4 pps)           0 KiB (     0 Mbits/s)
XDP_PASS        17558495 pkts (        12 pps)    22651771 KiB (     0 Mbits/s)
XDP_TX                 0 pkts (         0 pps)           0 KiB (     0 Mbits/s)
XDP_REDIRECT           0 pkts (         0 pps)           0 KiB (     0 Mbits/s)

Step 10: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 30 udp port 5001 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
iperf3: interrupt - the client has terminated
admin@osdx$

Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping UDP performance test traffic).+$
Show output
06/23/2025-09:02:38.064543  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:34720 -> 40.0.0.2:5000
06/23/2025-09:02:41.982447  [Drop] [**] [1:2:0] Dropping UDP performance test traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:36850 -> 40.0.0.2:5001

Step 12: Run command service firewall FW show early-drop-stats eth1 at DUT0 and check if output matches the following regular expressions:

yes\s+201\s+\d+\s+[1-9]\d*\s+[1-9]\d*
Show output
------------------------------------------------------------------------
  src       dst     src port  dst port  tcp  vlan_0  vlan_1  pkts  bytes
------------------------------------------------------------------------
40.0.0.2  20.0.0.2      5001     36850  no      201       0     0      0
20.0.0.2  40.0.0.2     34720      5000  yes     201       0    12    905
20.0.0.2  40.0.0.2     36850      5001  no      201       0     0      0
40.0.0.2  20.0.0.2      5000     34720  yes     201       0     0      0

Step 13: Run command interfaces ethernet eth1 monitor xdp-stats times 1 at DUT0 and expect this output:

Show output
Pin path: /sys/fs/bpf/eth1
Period of 0.250133s ending at 1750669365.728246
XDP_DROP              12 pkts (         0 pps)           0 KiB (     0 Mbits/s)
XDP_PASS        17558552 pkts (         0 pps)    22651776 KiB (     0 Mbits/s)
XDP_TX                 0 pkts (         0 pps)           0 KiB (     0 Mbits/s)
XDP_REDIRECT           0 pkts (         0 pps)           0 KiB (     0 Mbits/s)