Bypass Tests

The following scenario shows different configuration alternatives to improve the OSDx firewall performance.

../../../_images/topology26.svg

Test Local Bypass

Description

Builds a scenario with three DUTs in which a performance test is carried out between DUT1 and DUT2, and DUT0 is the router running the firewall. “Local bypass” is set to allow the firewall to internally skips packets belonging to a flow that must be bypassed. The performance test may produce better results than the general tests.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  18156      0 --:--:-- --:--:-- --:--:-- 18428

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic queue FW_Q elements 1

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.709 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.709/0.709/0.709/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.587 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.587/0.587/0.587/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 41404 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   154 MBytes  1.29 Gbits/sec    7   1.84 MBytes
[  5]   1.00-2.00   sec   148 MBytes  1.24 Gbits/sec    0   2.01 MBytes
[  5]   2.00-3.00   sec   145 MBytes  1.22 Gbits/sec   12   1.51 MBytes
[  5]   3.00-4.00   sec   140 MBytes  1.17 Gbits/sec    0   1.60 MBytes
[  5]   4.00-5.00   sec   150 MBytes  1.26 Gbits/sec    0   1.67 MBytes
[  5]   5.00-6.00   sec   149 MBytes  1.25 Gbits/sec    0   1.72 MBytes
[  5]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec  121   1.28 MBytes
[  5]   7.00-8.00   sec  81.2 MBytes   682 Mbits/sec    0   1.35 MBytes
[  5]   8.00-9.00   sec   132 MBytes  1.11 Gbits/sec    0   1.40 MBytes
[  5]   9.00-10.00  sec   150 MBytes  1.26 Gbits/sec   56   1.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.37 GBytes  1.17 Gbits/sec  196             sender
[  5]   0.00-10.01  sec  1.36 GBytes  1.17 Gbits/sec                  receiver

iperf Done.

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/04/2026-19:59:29.287640  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:41398 -> 40.0.0.2:5001
06/04/2026-19:59:29.288610  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:41404 -> 40.0.0.2:5001

Test Capture Bypass Using Packet Mark

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. “Capture bypass” is set to allow the firewall to mark packets. An external tool can then decide what to do with the flow when the mark is seen. For this example, when packet marks are detected, the traffic is assigned a label, thereby allowing the possibility of classifying traffic. In particular, labeling avoids traffic from entering the firewall.

Performance must improve considerably compared to the Local Bypass test.

The test is extended by using other packet marks that we have customized for the firewall.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  20286      0 --:--:-- --:--:-- --:--:-- 21500

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass mark 129834765
set service firewall FW stream bypass mask 129834765
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic label BYPASS
set traffic policy FW-SKIP rule 1 log prefix SKIP
set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS
set traffic policy FW-SKIP rule 1 set label BYPASS
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS
set traffic selector MARKED-PACKETS rule 1 mark 129834765

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.942 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.942/0.942/0.942/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.549 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.549/0.549/0.549/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 38404 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   320 MBytes  2.69 Gbits/sec  143   1.33 MBytes
[  5]   1.00-2.00   sec   336 MBytes  2.82 Gbits/sec    0   1.50 MBytes
[  5]   2.00-3.00   sec   311 MBytes  2.61 Gbits/sec    0   1.65 MBytes
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   1.77 MBytes
[  5]   4.00-5.00   sec   270 MBytes  2.26 Gbits/sec  276   1.35 MBytes
[  5]   5.00-6.00   sec   308 MBytes  2.58 Gbits/sec    0   1.50 MBytes
[  5]   6.00-7.00   sec   299 MBytes  2.50 Gbits/sec   85   1.14 MBytes
[  5]   7.00-8.00   sec   270 MBytes  2.27 Gbits/sec    0   1.30 MBytes
[  5]   8.00-9.00   sec   309 MBytes  2.59 Gbits/sec    0   1.47 MBytes
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0   1.60 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.91 GBytes  2.50 Gbits/sec  504             sender
[  5]   0.00-10.00  sec  2.91 GBytes  2.50 Gbits/sec                  receiver

iperf Done.

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/04/2026-20:00:04.050084  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:38398 -> 40.0.0.2:5001
06/04/2026-20:00:04.051261  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:38404 -> 40.0.0.2:5001

Step 8: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:

(?m)^.*\[SKIP\-1\].*$
Show output
Jun 04 19:59:56.397239 osdx systemd-journald[172397]: Runtime Journal (/run/log/journal/d7b8b58d13984d0bb9acdb87ea6c32f4) is 1.7M, max 13.8M, 12.0M free.
Jun 04 19:59:56.397707 osdx systemd-journald[172397]: Received client request to rotate journal, rotating.
Jun 04 19:59:56.397752 osdx systemd-journald[172397]: Vacuuming done, freed 0B of archived journals from /run/log/journal/d7b8b58d13984d0bb9acdb87ea6c32f4.
Jun 04 19:59:56.406610 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'system journal clear'.
Jun 04 19:59:56.631387 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'system coredump delete all'.
Jun 04 19:59:56.837146 osdx OSDxCLI[1151222]: User 'admin' entered the configuration menu.
Jun 04 19:59:56.925661 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'.
Jun 04 19:59:57.018782 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'.
Jun 04 19:59:57.080164 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'.
Jun 04 19:59:57.163531 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'.
Jun 04 19:59:57.219500 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'.
Jun 04 19:59:57.310580 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'.
Jun 04 19:59:57.376141 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'show working'.
Jun 04 19:59:57.476892 osdx ubnt-cfgd[1151733]: inactive
Jun 04 19:59:57.503146 osdx INFO[1151739]: FRR daemons did not change
Jun 04 19:59:57.537715 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1
Jun 04 19:59:57.557384 osdx WARNING[1151779]: No supported link modes on interface eth1
Jun 04 19:59:57.559039 osdx modulelauncher[1151779]: osdx.utils.xos cmd error: /sbin/ethtool -A eth1 autoneg on
Jun 04 19:59:57.559051 osdx modulelauncher[1151779]: Command '/sbin/ethtool -A eth1 autoneg on' returned non-zero exit status 76.
Jun 04 19:59:57.560278 osdx modulelauncher[1151779]: osdx.utils.xos cmd error: /sbin/ethtool -s eth1 autoneg on advertise Asym_Pause off Pause off --
Jun 04 19:59:57.560288 osdx modulelauncher[1151779]: Command '/sbin/ethtool -s eth1 autoneg on advertise Asym_Pause off Pause off --' returned non-zero exit status 75.
Jun 04 19:59:57.597720 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0
Jun 04 19:59:57.644017 osdx WARNING[1151859]: No supported link modes on interface eth0
Jun 04 19:59:57.645359 osdx modulelauncher[1151859]: osdx.utils.xos cmd error: /sbin/ethtool -A eth0 autoneg on
Jun 04 19:59:57.645374 osdx modulelauncher[1151859]: Command '/sbin/ethtool -A eth0 autoneg on' returned non-zero exit status 76.
Jun 04 19:59:57.646514 osdx modulelauncher[1151859]: osdx.utils.xos cmd error: /sbin/ethtool -s eth0 autoneg on advertise Asym_Pause off Pause off --
Jun 04 19:59:57.646523 osdx modulelauncher[1151859]: Command '/sbin/ethtool -s eth0 autoneg on advertise Asym_Pause off Pause off --' returned non-zero exit status 75.
Jun 04 19:59:57.655837 osdx (udev-worker)[1151875]: Network interface NamePolicy= disabled on kernel command line.
Jun 04 19:59:57.693070 osdx (udev-worker)[1151887]: Network interface NamePolicy= disabled on kernel command line.
Jun 04 19:59:57.822075 osdx cfgd[1850]: [1151222]Completed change to active configuration
Jun 04 19:59:57.836362 osdx OSDxCLI[1151222]: User 'admin' committed the configuration.
Jun 04 19:59:57.862340 osdx OSDxCLI[1151222]: User 'admin' left the configuration menu.
Jun 04 19:59:59.942518 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'.
Jun 04 20:00:00.029189 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'.
Jun 04 20:00:00.120991 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'.
Jun 04 20:00:00.631926 osdx file_operation[1152055]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running://
Jun 04 20:00:00.661652 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'.
Jun 04 20:00:00.806516 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'file show running://test-performance.rules'.
Jun 04 20:00:00.959785 osdx OSDxCLI[1151222]: User 'admin' entered the configuration menu.
Jun 04 20:00:01.030847 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'.
Jun 04 20:00:01.134963 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'.
Jun 04 20:00:01.193932 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'.
Jun 04 20:00:01.289946 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'.
Jun 04 20:00:01.348753 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'.
Jun 04 20:00:01.444379 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'.
Jun 04 20:00:01.540526 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'.
Jun 04 20:00:01.611824 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'.
Jun 04 20:00:01.705298 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'.
Jun 04 20:00:01.761220 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'.
Jun 04 20:00:01.870008 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic label BYPASS'.
Jun 04 20:00:01.942989 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'.
Jun 04 20:00:02.047555 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'.
Jun 04 20:00:02.123513 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'.
Jun 04 20:00:02.224754 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'.
Jun 04 20:00:02.302086 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'.
Jun 04 20:00:02.405761 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'.
Jun 04 20:00:02.472701 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'.
Jun 04 20:00:02.609684 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'.
Jun 04 20:00:02.662942 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'.
Jun 04 20:00:02.801809 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'show working'.
Jun 04 20:00:02.895449 osdx ubnt-cfgd[1152110]: inactive
Jun 04 20:00:02.974911 osdx INFO[1152155]: FRR daemons did not change
Jun 04 20:00:03.221985 osdx systemd[1]: Reloading.
Jun 04 20:00:03.297721 osdx systemd-sysv-generator[1152213]: stat() failed on /etc/init.d/README, ignoring: No such file or directory
Jun 04 20:00:03.422147 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service...
Jun 04 20:00:03.442617 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service.
Jun 04 20:00:03.633775 osdx INFO[1152192]: Rules successfully loaded
Jun 04 20:00:03.634357 osdx cfgd[1850]: [1151222]Completed change to active configuration
Jun 04 20:00:03.636568 osdx OSDxCLI[1151222]: User 'admin' committed the configuration.
Jun 04 20:00:03.654585 osdx OSDxCLI[1151222]: User 'admin' left the configuration menu.
Jun 04 20:00:04.053729 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=23924 DF PROTO=TCP SPT=38398 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 04 20:00:04.053814 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=15882 DF PROTO=TCP SPT=38404 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 04 20:00:14.214737 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.

Note

The following steps are just a reiteration of the previous test, but with the difference that the packet mark is an extra mark.

Step 9: Modify the following configuration lines in DUT0 :

set service firewall FW stream bypass extra-mark 1 mask 3294967295
set service firewall FW stream bypass extra-mark 1 value 3294967295
set traffic policy FW-SKIP rule 1 selector FW_SEL_EXTRA_MARK
set traffic selector FW_SEL_EXTRA_MARK rule 1 extra-mark 1 value 3294967295

Step 10: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 59024 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   280 MBytes  2.35 Gbits/sec    8   1.32 MBytes
[  5]   1.00-2.00   sec   302 MBytes  2.54 Gbits/sec    0   1.47 MBytes
[  5]   2.00-3.00   sec   332 MBytes  2.79 Gbits/sec    0   1.63 MBytes
[  5]   3.00-4.00   sec   339 MBytes  2.84 Gbits/sec    0   1.78 MBytes
[  5]   4.00-5.00   sec   318 MBytes  2.66 Gbits/sec    0   1.90 MBytes
[  5]   5.00-6.00   sec   334 MBytes  2.80 Gbits/sec    0   2.02 MBytes
[  5]   6.00-7.00   sec   308 MBytes  2.58 Gbits/sec  278   1.50 MBytes
[  5]   7.00-8.00   sec   281 MBytes  2.36 Gbits/sec    0   1.66 MBytes
[  5]   8.00-9.00   sec   308 MBytes  2.58 Gbits/sec   26   1.26 MBytes
[  5]   9.00-10.00  sec   334 MBytes  2.80 Gbits/sec    0   1.43 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.06 GBytes  2.63 Gbits/sec  312             sender
[  5]   0.00-10.00  sec  3.06 GBytes  2.63 Gbits/sec                  receiver

iperf Done.

Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/04/2026-20:00:04.050084  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:38398 -> 40.0.0.2:5001
06/04/2026-20:00:04.051261  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:38404 -> 40.0.0.2:5001
06/04/2026-20:00:18.932716  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:59020 -> 40.0.0.2:5001
06/04/2026-20:00:18.933817  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:59024 -> 40.0.0.2:5001

Step 12: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:

(?m)^.*\[SKIP\-1\].*$
Show output
Jun 04 19:59:56.397239 osdx systemd-journald[172397]: Runtime Journal (/run/log/journal/d7b8b58d13984d0bb9acdb87ea6c32f4) is 1.7M, max 13.8M, 12.0M free.
Jun 04 19:59:56.397707 osdx systemd-journald[172397]: Received client request to rotate journal, rotating.
Jun 04 19:59:56.397752 osdx systemd-journald[172397]: Vacuuming done, freed 0B of archived journals from /run/log/journal/d7b8b58d13984d0bb9acdb87ea6c32f4.
Jun 04 19:59:56.406610 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'system journal clear'.
Jun 04 19:59:56.631387 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'system coredump delete all'.
Jun 04 19:59:56.837146 osdx OSDxCLI[1151222]: User 'admin' entered the configuration menu.
Jun 04 19:59:56.925661 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'.
Jun 04 19:59:57.018782 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'.
Jun 04 19:59:57.080164 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'.
Jun 04 19:59:57.163531 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'.
Jun 04 19:59:57.219500 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'.
Jun 04 19:59:57.310580 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'.
Jun 04 19:59:57.376141 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'show working'.
Jun 04 19:59:57.476892 osdx ubnt-cfgd[1151733]: inactive
Jun 04 19:59:57.503146 osdx INFO[1151739]: FRR daemons did not change
Jun 04 19:59:57.537715 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1
Jun 04 19:59:57.557384 osdx WARNING[1151779]: No supported link modes on interface eth1
Jun 04 19:59:57.559039 osdx modulelauncher[1151779]: osdx.utils.xos cmd error: /sbin/ethtool -A eth1 autoneg on
Jun 04 19:59:57.559051 osdx modulelauncher[1151779]: Command '/sbin/ethtool -A eth1 autoneg on' returned non-zero exit status 76.
Jun 04 19:59:57.560278 osdx modulelauncher[1151779]: osdx.utils.xos cmd error: /sbin/ethtool -s eth1 autoneg on advertise Asym_Pause off Pause off --
Jun 04 19:59:57.560288 osdx modulelauncher[1151779]: Command '/sbin/ethtool -s eth1 autoneg on advertise Asym_Pause off Pause off --' returned non-zero exit status 75.
Jun 04 19:59:57.597720 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0
Jun 04 19:59:57.644017 osdx WARNING[1151859]: No supported link modes on interface eth0
Jun 04 19:59:57.645359 osdx modulelauncher[1151859]: osdx.utils.xos cmd error: /sbin/ethtool -A eth0 autoneg on
Jun 04 19:59:57.645374 osdx modulelauncher[1151859]: Command '/sbin/ethtool -A eth0 autoneg on' returned non-zero exit status 76.
Jun 04 19:59:57.646514 osdx modulelauncher[1151859]: osdx.utils.xos cmd error: /sbin/ethtool -s eth0 autoneg on advertise Asym_Pause off Pause off --
Jun 04 19:59:57.646523 osdx modulelauncher[1151859]: Command '/sbin/ethtool -s eth0 autoneg on advertise Asym_Pause off Pause off --' returned non-zero exit status 75.
Jun 04 19:59:57.655837 osdx (udev-worker)[1151875]: Network interface NamePolicy= disabled on kernel command line.
Jun 04 19:59:57.693070 osdx (udev-worker)[1151887]: Network interface NamePolicy= disabled on kernel command line.
Jun 04 19:59:57.822075 osdx cfgd[1850]: [1151222]Completed change to active configuration
Jun 04 19:59:57.836362 osdx OSDxCLI[1151222]: User 'admin' committed the configuration.
Jun 04 19:59:57.862340 osdx OSDxCLI[1151222]: User 'admin' left the configuration menu.
Jun 04 19:59:59.942518 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'.
Jun 04 20:00:00.029189 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'.
Jun 04 20:00:00.120991 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'.
Jun 04 20:00:00.631926 osdx file_operation[1152055]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running://
Jun 04 20:00:00.661652 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'.
Jun 04 20:00:00.806516 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'file show running://test-performance.rules'.
Jun 04 20:00:00.959785 osdx OSDxCLI[1151222]: User 'admin' entered the configuration menu.
Jun 04 20:00:01.030847 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'.
Jun 04 20:00:01.134963 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'.
Jun 04 20:00:01.193932 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'.
Jun 04 20:00:01.289946 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'.
Jun 04 20:00:01.348753 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'.
Jun 04 20:00:01.444379 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'.
Jun 04 20:00:01.540526 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'.
Jun 04 20:00:01.611824 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'.
Jun 04 20:00:01.705298 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'.
Jun 04 20:00:01.761220 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'.
Jun 04 20:00:01.870008 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic label BYPASS'.
Jun 04 20:00:01.942989 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'.
Jun 04 20:00:02.047555 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'.
Jun 04 20:00:02.123513 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'.
Jun 04 20:00:02.224754 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'.
Jun 04 20:00:02.302086 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'.
Jun 04 20:00:02.405761 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'.
Jun 04 20:00:02.472701 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'.
Jun 04 20:00:02.609684 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'.
Jun 04 20:00:02.662942 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'.
Jun 04 20:00:02.801809 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'show working'.
Jun 04 20:00:02.895449 osdx ubnt-cfgd[1152110]: inactive
Jun 04 20:00:02.974911 osdx INFO[1152155]: FRR daemons did not change
Jun 04 20:00:03.221985 osdx systemd[1]: Reloading.
Jun 04 20:00:03.297721 osdx systemd-sysv-generator[1152213]: stat() failed on /etc/init.d/README, ignoring: No such file or directory
Jun 04 20:00:03.422147 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service...
Jun 04 20:00:03.442617 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service.
Jun 04 20:00:03.633775 osdx INFO[1152192]: Rules successfully loaded
Jun 04 20:00:03.634357 osdx cfgd[1850]: [1151222]Completed change to active configuration
Jun 04 20:00:03.636568 osdx OSDxCLI[1151222]: User 'admin' committed the configuration.
Jun 04 20:00:03.654585 osdx OSDxCLI[1151222]: User 'admin' left the configuration menu.
Jun 04 20:00:04.053729 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=23924 DF PROTO=TCP SPT=38398 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 04 20:00:04.053814 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=15882 DF PROTO=TCP SPT=38404 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d
Jun 04 20:00:14.214737 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.
Jun 04 20:00:14.320437 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'system journal show | cat'.
Jun 04 20:00:14.555853 osdx OSDxCLI[1151222]: User 'admin' entered the configuration menu.
Jun 04 20:00:14.620971 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'.
Jun 04 20:00:14.716112 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'.
Jun 04 20:00:14.770129 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'.
Jun 04 20:00:14.868062 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'.
Jun 04 20:00:14.936612 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'.
Jun 04 20:00:15.059052 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'.
Jun 04 20:00:15.160263 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'.
Jun 04 20:00:15.223327 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'.
Jun 04 20:00:15.314720 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'.
Jun 04 20:00:15.368841 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'.
Jun 04 20:00:15.465369 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic label BYPASS'.
Jun 04 20:00:15.548087 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'.
Jun 04 20:00:15.640229 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'.
Jun 04 20:00:15.698295 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'.
Jun 04 20:00:15.799658 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'.
Jun 04 20:00:15.866004 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'.
Jun 04 20:00:15.961142 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'.
Jun 04 20:00:16.037837 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'.
Jun 04 20:00:16.127042 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'.
Jun 04 20:00:16.180584 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'.
Jun 04 20:00:16.274465 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass extra-mark 1 value 3294967295'.
Jun 04 20:00:16.331183 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass extra-mark 1 mask 3294967295'.
Jun 04 20:00:16.428167 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector FW_SEL_EXTRA_MARK'.
Jun 04 20:00:16.508144 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_EXTRA_MARK rule 1 extra-mark 1 value 3294967295'.
Jun 04 20:00:16.620490 osdx OSDxCLI[1151222]: User 'admin' added a new cfg line: 'show changes'.
Jun 04 20:00:16.738099 osdx systemd[1]: Starting logrotate.service - Rotate log files...
Jun 04 20:00:16.738529 osdx ubnt-cfgd[1152333]: inactive
Jun 04 20:00:16.763628 osdx systemd[1]: logrotate.service: Deactivated successfully.
Jun 04 20:00:16.763879 osdx systemd[1]: Finished logrotate.service - Rotate log files.
Jun 04 20:00:16.797181 osdx INFO[1152356]: FRR daemons did not change
Jun 04 20:00:17.062027 osdx systemd[1]: Stopping suricata@FW.service - Suricata client "FW" service...
Jun 04 20:00:18.433407 osdx systemd[1]: suricata@FW.service: Deactivated successfully.
Jun 04 20:00:18.433529 osdx systemd[1]: Stopped suricata@FW.service - Suricata client "FW" service.
Jun 04 20:00:18.433565 osdx systemd[1]: suricata@FW.service: Consumed 1.525s CPU time.
Jun 04 20:00:18.466090 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service...
Jun 04 20:00:18.483215 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service.
Jun 04 20:00:18.699663 osdx INFO[1152383]: Rules successfully loaded
Jun 04 20:00:18.700291 osdx cfgd[1850]: [1151222]Completed change to active configuration
Jun 04 20:00:18.702764 osdx OSDxCLI[1151222]: User 'admin' committed the configuration.
Jun 04 20:00:18.718434 osdx OSDxCLI[1151222]: User 'admin' left the configuration menu.
Jun 04 20:00:18.933725 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=31972 DF PROTO=TCP SPT=59020 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d EMARK1=0xc46535ff
Jun 04 20:00:18.938133 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=47635 DF PROTO=TCP SPT=59024 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d EMARK1=0xc46535ff
Jun 04 20:00:26.038416 osdx systemd[1]: systemd-timedated.service: Deactivated successfully.
Jun 04 20:00:29.082826 osdx OSDxCLI[1151222]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.

Test Capture Bypass Using Conntrack Mark

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later.

Performance must improve considerably compared to the Local Bypass test.

Then this test is broadened by using other conntrack marks that we have customized for the firewall.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  22395      0 --:--:-- --:--:-- --:--:-- 25800

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass mark 129834765
set service firewall FW stream bypass mask 129834765
set service firewall FW stream bypass set-connmark
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 2 action enqueue FW_Q
set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.640 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.640/0.640/0.640/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.556 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.556/0.556/0.556/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 44882 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   320 MBytes  2.69 Gbits/sec   89   1.85 MBytes
[  5]   1.00-2.00   sec   336 MBytes  2.82 Gbits/sec    0   2.01 MBytes
[  5]   2.00-3.00   sec   350 MBytes  2.94 Gbits/sec    0   2.14 MBytes
[  5]   3.00-4.00   sec   334 MBytes  2.80 Gbits/sec   13   1.59 MBytes
[  5]   4.00-5.00   sec   365 MBytes  3.06 Gbits/sec    0   1.75 MBytes
[  5]   5.00-6.00   sec   342 MBytes  2.87 Gbits/sec    0   1.89 MBytes
[  5]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec    0   1.99 MBytes
[  5]   7.00-8.00   sec   332 MBytes  2.79 Gbits/sec   48   1.52 MBytes
[  5]   8.00-9.00   sec   329 MBytes  2.76 Gbits/sec    0   1.67 MBytes
[  5]   9.00-10.00  sec   314 MBytes  2.63 Gbits/sec   56   1.27 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.23 GBytes  2.77 Gbits/sec  206             sender
[  5]   0.00-10.00  sec  3.22 GBytes  2.77 Gbits/sec                  receiver

iperf Done.

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/04/2026-20:00:56.115862  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:44880 -> 40.0.0.2:5001
06/04/2026-20:00:56.116964  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:44882 -> 40.0.0.2:5001

Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*mark=129834765.*$
Show output
icmp     1 19 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=122 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=122 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
tcp      6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=44882 dport=5001 packets=2392359 bytes=3588518281 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=44882 packets=321548 bytes=16717044 [ASSURED] (Sc: not-bypass) mark=129834765 use=1
icmp     1 19 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=927 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=927 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
tcp      6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=44880 dport=5001 packets=16 bytes=1300 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=44880 packets=13 bytes=1020 [ASSURED] (Sc: not-bypass) mark=129834765 use=1
conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.

Note

The following steps are just a reiteration of the previous test, but with the difference that the conntrack mark used is an extra connmark.

Step 9: Modify the following configuration lines in DUT0 :

set service firewall FW stream bypass extra-mark 2 mask 3294967295
set service firewall FW stream bypass extra-mark 2 set-extra-connmark
set service firewall FW stream bypass extra-mark 2 value 3294967295
set traffic policy FW_PLAN rule 2 selector FW_SEL_EXTRA_MARK
set traffic selector FW_SEL_EXTRA_MARK rule 1 not extra-connmark 2 value 3294967295

Step 10: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1
Expect this output in DUT2:
Connecting to host 40.0.0.2, port 5001
[  5] local 20.0.0.2 port 39102 connected to 40.0.0.2 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   274 MBytes  2.30 Gbits/sec   40   1.31 MBytes
[  5]   1.00-2.00   sec   314 MBytes  2.63 Gbits/sec    0   1.47 MBytes
[  5]   2.00-3.00   sec   325 MBytes  2.73 Gbits/sec    0   1.62 MBytes
[  5]   3.00-4.00   sec   321 MBytes  2.69 Gbits/sec    0   1.76 MBytes
[  5]   4.00-5.00   sec   339 MBytes  2.84 Gbits/sec   31   1.36 MBytes
[  5]   5.00-6.00   sec   331 MBytes  2.78 Gbits/sec    0   1.52 MBytes
[  5]   6.00-7.00   sec   339 MBytes  2.84 Gbits/sec    0   1.68 MBytes
[  5]   7.00-8.00   sec   330 MBytes  2.77 Gbits/sec    0   1.82 MBytes
[  5]   8.00-9.00   sec   341 MBytes  2.86 Gbits/sec   80   1.36 MBytes
[  5]   9.00-10.00  sec   331 MBytes  2.78 Gbits/sec    0   1.53 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.17 GBytes  2.72 Gbits/sec  151             sender
[  5]   0.00-10.00  sec  3.17 GBytes  2.72 Gbits/sec                  receiver

iperf Done.

Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/04/2026-20:00:56.115862  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:44880 -> 40.0.0.2:5001
06/04/2026-20:00:56.116964  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:44882 -> 40.0.0.2:5001
06/04/2026-20:01:11.192640  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:39092 -> 40.0.0.2:5001
06/04/2026-20:01:11.193757  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:39102 -> 40.0.0.2:5001

Step 12: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*emark2=3294967295.*$
Show output
tcp      6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=39092 dport=5001 packets=16 bytes=1301 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=39092 packets=13 bytes=1019 [ASSURED] (Sc: not-bypass) mark=129834765 emark2=3294967295 use=1
tcp      6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=39102 dport=5001 packets=2350119 bytes=3525158913 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=39102 packets=300304 bytes=15606408 [ASSURED] (Sc: not-bypass) mark=129834765 emark2=3294967295 use=1
conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.

Test Bypass-Drop Using Conntrack Marks

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test is aimed at configuring “Capture bypass drop” to avoid dropped packets from entering the firewall.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   200  100   200    0     0   4499      0 --:--:-- --:--:-- --:--:--  4545

Step 2: Run command file show running://drop-performance.rules at DUT0 and expect this output:

Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)
drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW bypass action drop set connmark mark 147652983
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://drop-performance.rules'
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action drop
set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP
set traffic policy FW_PLAN rule 2 action enqueue FW_Q
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_DROP rule 1 connmark 147652983

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.586 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.586/0.586/0.586/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.748 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.748/0.748/0.748/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5000
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
admin@osdx$

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping TCP performance test traffic).+$
Show output
06/04/2026-20:01:42.816635  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37086 -> 40.0.0.2:5000

Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*mark=147652983.*$
Show output
tcp      6 29 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=37086 dport=5000 packets=7 bytes=557 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=37086 packets=4 bytes=217 [ASSURED] (Sc: not-bypass) mark=147652983 use=1
icmp     1 26 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=125 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=125 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
icmp     1 26 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=930 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=930 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
conntrack v1.4.7 (conntrack-tools): 3 flow entries have been shown.

Step 9: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:

(?m)^1\s+FW_SEL_DROP\s+[1-9].*$
Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high

------------------------------------------------------------------
rule    selector    pkts match  pkts eval  bytes match  bytes eval
------------------------------------------------------------------
1      FW_SEL_DROP           4          7          210         438
2      -                     3          3          228         228
------------------------------------------------------------------
Total                        7          7          438         438

Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high

------------------------------------------------------------------
rule    selector    pkts match  pkts eval  bytes match  bytes eval
------------------------------------------------------------------
1      FW_SEL_DROP           5         10          445         814
2      -                     5          5          369         369
------------------------------------------------------------------
Total                       10         10          814         814

Note

Testing with another conntrack mark.

Step 10: Modify the following configuration lines in DUT0 :

delete service firewall FW bypass action drop set connmark mark
set service firewall FW bypass action drop set connmark extra-mark 2 value 3967295294
set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP_EM
set traffic selector FW_SEL_DROP_EM rule 1 extra-connmark 2 value 3967295294

Step 11: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5000
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
admin@osdx$

Step 12: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping TCP performance test traffic).+$
Show output
06/04/2026-20:01:42.816635  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37086 -> 40.0.0.2:5000
06/04/2026-20:01:49.113264  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:37086 -> 40.0.0.2:5000
06/04/2026-20:01:50.626299  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:57746 -> 40.0.0.2:5000

Step 13: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*emark2=3967295294.*$
Show output
tcp      6 29 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=57746 dport=5000 packets=7 bytes=557 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=57746 packets=4 bytes=217 [ASSURED] (Sc: not-bypass) mark=0 emark2=3967295294 use=1
conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.

Step 14: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:

(?m)^1\s+FW_SEL_DROP_EM\s+[1-9].*$
Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high

---------------------------------------------------------------------
rule      selector     pkts match  pkts eval  bytes match  bytes eval
---------------------------------------------------------------------
1      FW_SEL_DROP_EM           4          7          210         376
2      -                        3          3          166         166
---------------------------------------------------------------------
Total                           7          7          376         376

Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high

---------------------------------------------------------------------
rule      selector     pkts match  pkts eval  bytes match  bytes eval
---------------------------------------------------------------------
1      FW_SEL_DROP_EM           4          9          356         686
2      -                        5          5          330         330
---------------------------------------------------------------------
Total                           9          9          686         686

Test Capture And Offload

Description

Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later. In addition, OSDx is instructed to accelerate the flow using internal accelerators.

Performance must improve considerably compared to the previous test, to reach its top value.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  41266      0 --:--:-- --:--:-- --:--:-- 43000

Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:

Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://test-performance.rules'
set service firewall FW stream bypass action accept set conntrack offload-flag
set service firewall FW stream bypass mark 129834765
set service firewall FW stream bypass mask 129834765
set service firewall FW stream bypass set-connmark
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic policy FW_PLAN rule 2 action enqueue FW_Q
set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE
set traffic queue FW_Q elements 1
set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.603 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.603/0.603/0.603/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.484 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.484/0.484/0.484/0.000 ms

Step 6: Initiate a background bandwidth test from DUT2 to DUT1. The control is returned back allowing to perform another tasks while test is running

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance traffic).+$
Show output
06/04/2026-20:02:15.168932  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:49004 -> 40.0.0.2:5001
06/04/2026-20:02:15.169962  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:49020 -> 40.0.0.2:5001

Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$
Show output
icmp     1 29 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=933 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=933 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
icmp     1 29 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=128 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=128 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1
tcp      6 src=20.0.0.2 dst=40.0.0.2 sport=49004 dport=5001 packets=7 bytes=537 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=49004 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=1 bytes=52 packets=4 bytes=211] mark=129834765 use=3
tcp      6 src=20.0.0.2 dst=40.0.0.2 sport=49020 dport=5001 packets=57624 bytes=86427237 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=49020 packets=7889 bytes=411468 [ASSURED] [OFFLOAD, packets=57611 bytes=86412036 packets=7887 bytes=411356] mark=129834765 use=2
conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.

Step 9: Stop the current bandwidth test between DUT2 and DUT1

Step 10: Run command file copy http://10.215.168.1/~robot/test-performance-udp.rules running:// force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   133  100   133    0     0   4274      0 --:--:-- --:--:-- --:--:--  4290

Step 11: Run command file show running://test-performance-udp.rules at DUT0 and expect this output:

Show output
alert udp any any -> any 5001 (msg: "Skipping test network performance UDP traffic"; bypass; flow: established, to_server; sid: 41;)

Step 12: Modify the following configuration lines in DUT0 :

set service firewall FW ruleset file 'running://test-performance-udp.rules'

Step 13: Initiate a background bandwidth test from DUT2 to DUT1. The control is returned back allowing to perform another tasks while test is running

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 udp port 5001 parallel 1

Step 14: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Skipping test network performance UDP traffic).+$
Show output
06/04/2026-20:02:15.168932  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:49004 -> 40.0.0.2:5001
06/04/2026-20:02:15.169962  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:49020 -> 40.0.0.2:5001
06/04/2026-20:02:16.965095  [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:49022 -> 40.0.0.2:5001
06/04/2026-20:02:16.967052  [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:44724 -> 40.0.0.2:5001
06/04/2026-20:02:16.977565  [**] [1:41:0] Skipping test network performance UDP traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:44724 -> 40.0.0.2:5001

Step 15: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:

(?m)^udp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$
Show output
tcp      6 src=20.0.0.2 dst=40.0.0.2 sport=49022 dport=5001 packets=8 bytes=607 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=49022 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=2 bytes=104 packets=4 bytes=211] mark=129834765 use=3
udp      17 src=20.0.0.2 dst=40.0.0.2 sport=44724 dport=5001 packets=11 bytes=14792 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=44724 packets=1 bytes=32 [OFFLOAD, packets=7 bytes=10332 packets=0 bytes=0] mark=129834765 use=2
conntrack v1.4.7 (conntrack-tools): 2 flow entries have been shown.

Step 16: Stop the current bandwidth test between DUT2 and DUT1

Test Traffic Early Dropping

Description

Builds a scenario with three DUTs and a simple ruleset to drop TCP traffic between DUT1 and DUT2. Such traffic must pass through port 5000 for the rule to match. Later, XDP is queried to check if packets are being dropped at the specified interface.

The contents of the rule file are:

drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)

This rule allows the connection to be established and traffic to be dropped later.

Scenario

Step 1: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running://drop-performance.rules force at DUT0 and expect this output:

Show output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   200  100   200    0     0  35435      0 --:--:-- --:--:-- --:--:-- 40000

Step 2: Run command file show running://drop-performance.rules at DUT0 and expect this output:

Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)
drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)

Step 3: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.215.168.64/24
set interfaces ethernet eth0 traffic nat source rule 1 address masquerade
set interfaces ethernet eth1 vif 101 address 40.0.0.1/8
set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN
set interfaces ethernet eth1 vif 201 address 20.0.0.1/8
set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN
set protocols static route 0.0.0.0/0 next-hop 10.215.168.1
set service dns static host-name WAN inet 10.215.168.1
set service firewall FW logging level config
set service firewall FW logging outputs fast
set service firewall FW mode inline queue FW_Q
set service firewall FW ruleset file 'running://drop-performance.rules'
set service firewall FW stream bypass action drop set xdp-early-drop eth1
set service firewall FW validator-timeout 20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'
set traffic policy FW_PLAN rule 1 action enqueue FW_Q
set traffic queue FW_Q elements 1

Step 4: Ping IP address 20.0.0.2 from DUT1:

admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1
Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data.
64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=1.03 ms

--- 20.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.030/1.030/1.030/0.000 ms

Step 5: Ping IP address 40.0.0.2 from DUT2:

admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1
Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data.
64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.695 ms

--- 40.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.695/0.695/0.695/0.000 ms

Step 6: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5000
admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
iperf3: interrupt - the client has terminated
admin@osdx$

Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping TCP performance test traffic).+$
Show output
06/04/2026-20:02:37.244398  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:33502 -> 40.0.0.2:5000

Step 8: Run command service firewall FW show early-drop-stats eth1 at DUT0 and check if output matches the following regular expressions:

yes\s+201\s+\d+\s+[1-9]\d*\s+[1-9]\d*
Show output
------------------------------------------------------------------------
  src       dst     src port  dst port  tcp  vlan_0  vlan_1  pkts  bytes
------------------------------------------------------------------------
40.0.0.2  20.0.0.2      5000     33502  yes     201       0     0      0
20.0.0.2  40.0.0.2     33502      5000  yes     201       0     7    602

Step 9: Run command interfaces ethernet eth1 monitor xdp-stats times 1 at DUT0 and expect this output:

Show output
Pin path: /sys/fs/bpf/eth1
Period of 0.250140s ending at 1780603361.046928
XDP_DROP               8 pkts (         4 pps)           0 KiB (     0 Mbits/s)
XDP_PASS        23176770 pkts (        12 pps)    30600512 KiB (     0 Mbits/s)
XDP_TX                 0 pkts (         0 pps)           0 KiB (     0 Mbits/s)
XDP_REDIRECT           0 pkts (         0 pps)           0 KiB (     0 Mbits/s)

Step 10: Initiate a bandwidth test from DUT2 to DUT1

admin@DUT1$ monitor test performance server port 5001
admin@DUT2$ monitor test performance client 40.0.0.2 duration 30 udp port 5001 parallel 1
Expect this output in DUT2:
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
iperf3: interrupt - the client has terminated
admin@osdx$

Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:

(?m)^.+(Dropping UDP performance test traffic).+$
Show output
06/04/2026-20:02:37.244398  [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:33502 -> 40.0.0.2:5000
06/04/2026-20:02:41.212033  [Drop] [**] [1:2:0] Dropping UDP performance test traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:43704 -> 40.0.0.2:5001

Step 12: Run command service firewall FW show early-drop-stats eth1 at DUT0 and check if output matches the following regular expressions:

yes\s+201\s+\d+\s+[1-9]\d*\s+[1-9]\d*
Show output
------------------------------------------------------------------------
  src       dst     src port  dst port  tcp  vlan_0  vlan_1  pkts  bytes
------------------------------------------------------------------------
40.0.0.2  20.0.0.2      5000     33502  yes     201       0     0      0
20.0.0.2  40.0.0.2     43704      5001  no      201       0     0      0
40.0.0.2  20.0.0.2      5001     43704  no      201       0     0      0
20.0.0.2  40.0.0.2     33502      5000  yes     201       0    10    776

Step 13: Run command interfaces ethernet eth1 monitor xdp-stats times 1 at DUT0 and expect this output:

Show output
Pin path: /sys/fs/bpf/eth1
Period of 0.250140s ending at 1780603364.822131
XDP_DROP              10 pkts (         0 pps)           0 KiB (     0 Mbits/s)
XDP_PASS        23176827 pkts (         0 pps)    30600516 KiB (     0 Mbits/s)
XDP_TX                 0 pkts (         0 pps)           0 KiB (     0 Mbits/s)
XDP_REDIRECT           0 pkts (         0 pps)           0 KiB (     0 Mbits/s)