Bypass Tests
The following scenario shows different configuration alternatives to improve the OSDx firewall performance.
Test Local Bypass
Description
Builds a scenario with three DUTs in which a performance test is carried out between DUT1 and DUT2, and DUT0 is the router running the firewall. “Local bypass” is set to allow the firewall to internally skips packets belonging to a flow that must be bypassed. The performance test may produce better results than the general tests.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 15096 0 --:--:-- --:--:-- --:--:-- 16125
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0:
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic queue FW_Q elements 1
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.775 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.775/0.775/0.775/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.767 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.767/0.767/0.767/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 40580 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.02 sec 68.1 MBytes 562 Mbits/sec 79 1.44 MBytes [ 5] 1.02-2.00 sec 70.0 MBytes 597 Mbits/sec 137 1.10 MBytes [ 5] 2.00-3.00 sec 73.8 MBytes 618 Mbits/sec 0 1.18 MBytes [ 5] 3.00-4.01 sec 65.0 MBytes 539 Mbits/sec 0 1.24 MBytes [ 5] 4.01-5.00 sec 71.2 MBytes 606 Mbits/sec 0 1.28 MBytes [ 5] 5.00-6.00 sec 63.8 MBytes 535 Mbits/sec 0 1.31 MBytes [ 5] 6.00-7.00 sec 65.0 MBytes 545 Mbits/sec 0 1.32 MBytes [ 5] 7.00-8.00 sec 48.8 MBytes 409 Mbits/sec 0 1.33 MBytes [ 5] 8.00-9.00 sec 41.2 MBytes 346 Mbits/sec 0 1.35 MBytes [ 5] 9.00-10.00 sec 46.2 MBytes 388 Mbits/sec 0 1.38 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 613 MBytes 514 Mbits/sec 216 sender [ 5] 0.00-10.01 sec 611 MBytes 512 Mbits/sec receiver iperf Done.
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
05/11/2025-22:54:51.012286 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:40566 -> 40.0.0.2:5001 05/11/2025-22:54:51.013862 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:40580 -> 40.0.0.2:5001
Test Capture Bypass Using Packet Mark
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. “Capture bypass” is set to allow the firewall to mark packets. An external tool can then decide what to do with the flow when the mark is seen. For this example, when packet marks are detected, the traffic is assigned a label, thereby allowing the possibility of classifying traffic. In particular, labeling avoids traffic from entering the firewall.
Performance must improve considerably compared to the Local Bypass test.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 290 0 --:--:-- --:--:-- --:--:-- 290
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0:
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass mark 129834765 set service firewall FW stream bypass mask 129834765 set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic label BYPASS set traffic policy FW-SKIP rule 1 log prefix SKIP set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS set traffic policy FW-SKIP rule 1 set label BYPASS set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE set traffic queue FW_Q elements 1 set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS set traffic selector MARKED-PACKETS rule 1 mark 129834765
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.784 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.784/0.784/0.784/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.772 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.772/0.772/0.772/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 42836 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec 76 1.64 MBytes [ 5] 1.00-2.00 sec 82.5 MBytes 693 Mbits/sec 53 1.24 MBytes [ 5] 2.00-3.00 sec 86.2 MBytes 724 Mbits/sec 0 1.31 MBytes [ 5] 3.00-4.00 sec 93.8 MBytes 786 Mbits/sec 0 1.36 MBytes [ 5] 4.00-5.00 sec 73.8 MBytes 619 Mbits/sec 0 1.39 MBytes [ 5] 5.00-6.00 sec 78.8 MBytes 661 Mbits/sec 0 1.41 MBytes [ 5] 6.00-7.00 sec 83.8 MBytes 703 Mbits/sec 42 1.01 MBytes [ 5] 7.00-8.00 sec 71.2 MBytes 598 Mbits/sec 0 1.08 MBytes [ 5] 8.00-9.00 sec 73.8 MBytes 619 Mbits/sec 0 1.14 MBytes [ 5] 9.00-10.01 sec 86.2 MBytes 720 Mbits/sec 0 1.17 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 841 MBytes 705 Mbits/sec 171 sender [ 5] 0.00-10.01 sec 839 MBytes 704 Mbits/sec receiver iperf Done.
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
05/11/2025-22:55:32.232302 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:42830 -> 40.0.0.2:5001 05/11/2025-22:55:32.234573 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:42836 -> 40.0.0.2:5001
Step 8: Run command system journal show | cat at DUT0 and check if output matches the following regular expressions:
(?m)^.*\[SKIP\-1\].*$Show output
May 11 22:55:18.001241 osdx systemd-timedated[199658]: Changed local time to Sun 2025-05-11 22:55:18 UTC May 11 22:55:18.003030 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'set date 2025-05-11 22:55:18'. May 11 22:55:18.004961 osdx systemd-journald[118835]: Time jumped backwards, rotating. May 11 22:55:18.443989 osdx systemd-journald[118835]: Runtime Journal (/run/log/journal/f55f446d40464b198e70fbabb9c21674) is 2.1M, max 15.3M, 13.1M free. May 11 22:55:18.444967 osdx systemd-journald[118835]: Received client request to rotate journal, rotating. May 11 22:55:18.445027 osdx systemd-journald[118835]: Vacuuming done, freed 0B of archived journals from /run/log/journal/f55f446d40464b198e70fbabb9c21674. May 11 22:55:18.463210 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'system journal clear'. May 11 22:55:19.012305 osdx osdx-coredump[199675]: Deleting all coredumps in /opt/vyatta/etc/config/coredump... May 11 22:55:19.024335 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'system coredump delete all'. May 11 22:55:19.807485 osdx OSDxCLI[199323]: User 'admin' entered the configuration menu. May 11 22:55:19.945232 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 address 10.215.168.64/24'. May 11 22:55:20.027988 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth0 traffic nat source rule 1 address masquerade'. May 11 22:55:20.147645 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set protocols static route 0.0.0.0/0 next-hop 10.215.168.1'. May 11 22:55:20.268790 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service dns static host-name WAN inet 10.215.168.1'. May 11 22:55:20.406040 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 address 40.0.0.1/8'. May 11 22:55:20.523250 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 address 20.0.0.1/8'. May 11 22:55:20.671933 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'show working'. May 11 22:55:20.844974 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth0 May 11 22:55:20.952981 osdx kernel: 8021q: adding VLAN 0 to HW filter on device eth1 May 11 22:55:21.003592 osdx (udev-worker)[199797]: Network interface NamePolicy= disabled on kernel command line. May 11 22:55:21.345786 osdx cfgd[1244]: [199323]Completed change to active configuration May 11 22:55:21.385107 osdx OSDxCLI[199323]: User 'admin' committed the configuration. May 11 22:55:21.442003 osdx OSDxCLI[199323]: User 'admin' left the configuration menu. May 11 22:55:24.787815 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'ping 10.215.168.1 count 1 size 56 timeout 1'. May 11 22:55:24.927052 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'ping 40.0.0.2 count 1 size 56 timeout 1'. May 11 22:55:25.041537 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'ping 20.0.0.2 count 1 size 56 timeout 1'. May 11 22:55:25.826504 osdx file_operation[199881]: using src url: http://10.215.168.1/~robot/test-performance.rules dst url: running:// May 11 22:55:26.305607 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'file copy http://10.215.168.1/~robot/test-performance.rules running:// force'. May 11 22:55:26.555840 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'file show running://test-performance.rules'. May 11 22:55:26.819398 osdx OSDxCLI[199323]: User 'admin' entered the configuration menu. May 11 22:55:26.946275 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW ruleset file running://test-performance.rules'. May 11 22:55:27.106724 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW mode inline queue FW_Q'. May 11 22:55:27.230320 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW logging outputs fast'. May 11 22:55:27.348191 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW logging level config'. May 11 22:55:27.512913 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW validator-timeout 20'. May 11 22:55:27.670649 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic queue FW_Q elements 1'. May 11 22:55:27.846333 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN'. May 11 22:55:27.995337 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN'. May 11 22:55:28.128331 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mark 129834765'. May 11 22:55:28.254960 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set service firewall FW stream bypass mask 129834765'. May 11 22:55:28.431557 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic label BYPASS'. May 11 22:55:28.593364 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 log prefix SKIP'. May 11 22:55:28.695447 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 selector MARKED-PACKETS'. May 11 22:55:28.815908 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic policy FW-SKIP rule 1 set label BYPASS'. May 11 22:55:28.943981 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic selector MARKED-PACKETS rule 1 mark 129834765'. May 11 22:55:29.068892 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic selector FW_SEL_ENQUEUE rule 1 not label BYPASS'. May 11 22:55:29.227890 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 selector FW_SEL_ENQUEUE'. May 11 22:55:29.380408 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 101 traffic policy out FW-SKIP'. May 11 22:55:29.489397 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set interfaces ethernet eth1 vif 201 traffic policy out FW-SKIP'. May 11 22:55:29.671121 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'set traffic policy FW_PLAN rule 1 action enqueue FW_Q'. May 11 22:55:29.932531 osdx OSDxCLI[199323]: User 'admin' added a new cfg line: 'show working'. May 11 22:55:30.665648 osdx systemd[1]: Reloading. May 11 22:55:30.832969 osdx systemd-sysv-generator[200009]: stat() failed on /etc/init.d/README, ignoring: No such file or directory May 11 22:55:31.013482 osdx systemd[1]: Starting suricata@FW.service - Suricata client "FW" service... May 11 22:55:31.047413 osdx systemd[1]: Started suricata@FW.service - Suricata client "FW" service. May 11 22:55:31.412902 osdx INFO[199991]: Rules successfully loaded May 11 22:55:31.414617 osdx cfgd[1244]: [199323]Completed change to active configuration May 11 22:55:31.418187 osdx OSDxCLI[199323]: User 'admin' committed the configuration. May 11 22:55:31.464905 osdx OSDxCLI[199323]: User 'admin' left the configuration menu. May 11 22:55:32.232971 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=13550 DF PROTO=TCP SPT=42830 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d May 11 22:55:32.236970 osdx kernel: [SKIP-1] ACCEPT IN=eth1.201 OUT=eth1.101 MAC=de:ad:be:ef:6c:01:de:ad:be:ef:6c:20:08:00:45:00:00:59 SRC=20.0.0.2 DST=40.0.0.2 LEN=89 TOS=0x00 PREC=0x00 TTL=63 ID=12690 DF PROTO=TCP SPT=42836 DPT=5001 WINDOW=502 RES=0x00 ACK PSH URGP=0 MARK=0x7bd1f0d May 11 22:55:42.538013 osdx OSDxCLI[199323]: User 'admin' executed a new command: 'service firewall FW show logging fast | tail'.
Test Capture Bypass Using Conntrack Mark
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later.
Performance must improve considerably compared to the Local Bypass test.
Then, the test is broaden by configuring “Capture bypass drop” to also avoid dropped packets from entering the firewall.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 17352 0 --:--:-- --:--:-- --:--:-- 18428
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0:
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass mark 129834765 set service firewall FW stream bypass mask 129834765 set service firewall FW stream bypass set-connmark set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 2 action enqueue FW_Q set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE set traffic queue FW_Q elements 1 set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.740 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.740/0.740/0.740/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.853 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.853/0.853/0.853/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5001 parallel 1Expect this output in
DUT2:Connecting to host 40.0.0.2, port 5001 [ 5] local 20.0.0.2 port 55444 connected to 40.0.0.2 port 5001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 125 MBytes 1.05 Gbits/sec 154 1.08 MBytes [ 5] 1.00-2.00 sec 136 MBytes 1.14 Gbits/sec 0 1.16 MBytes [ 5] 2.00-3.00 sec 132 MBytes 1.11 Gbits/sec 0 1.24 MBytes [ 5] 3.00-4.00 sec 152 MBytes 1.28 Gbits/sec 0 1.32 MBytes [ 5] 4.00-5.00 sec 134 MBytes 1.12 Gbits/sec 0 1.39 MBytes [ 5] 5.00-6.00 sec 122 MBytes 1.03 Gbits/sec 0 1.45 MBytes [ 5] 6.00-7.00 sec 132 MBytes 1.11 Gbits/sec 33 1.11 MBytes [ 5] 7.00-8.00 sec 118 MBytes 986 Mbits/sec 0 1.22 MBytes [ 5] 8.00-9.00 sec 136 MBytes 1.14 Gbits/sec 0 1.30 MBytes [ 5] 9.00-10.00 sec 156 MBytes 1.31 Gbits/sec 0 1.36 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.31 GBytes 1.13 Gbits/sec 187 sender [ 5] 0.00-10.02 sec 1.31 GBytes 1.12 Gbits/sec receiver iperf Done.
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
05/11/2025-22:56:11.876848 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:55440 -> 40.0.0.2:5001 05/11/2025-22:56:11.883951 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:55444 -> 40.0.0.2:5001
Step 8: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.*mark=129834765.*$Show output
icmp 1 19 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=301 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=301 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 icmp 1 19 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=43 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=43 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 tcp 6 19 TIME_WAIT src=20.0.0.2 dst=40.0.0.2 sport=55440 dport=5001 packets=15 bytes=1537 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=55440 packets=13 bytes=1030 [ASSURED] (Sc: not-bypass) mark=129834765 use=1 tcp 6 9 CLOSE src=20.0.0.2 dst=40.0.0.2 sport=55444 dport=5001 packets=973789 bytes=1460667113 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=55444 packets=65104 bytes=3377184 [ASSURED] (Sc: not-bypass) mark=129834765 use=1 conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.
Step 9: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 200 100 200 0 0 23755 0 --:--:-- --:--:-- --:--:-- 25000
Step 10: Run command file show running://drop-performance.rules at DUT0 and expect this output:
Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;) drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)
Step 11: Modify the following configuration lines in DUT0:
set service firewall FW bypass action drop set connmark mark 147652983 set service firewall FW ruleset file 'running://drop-performance.rules' set traffic policy FW_PLAN rule 1 action drop set traffic policy FW_PLAN rule 1 selector FW_SEL_DROP set traffic selector FW_SEL_DROP rule 1 connmark 147652983
Step 12: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5000 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1Expect this output in
DUT2:^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated admin@osdx$
Step 13: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Dropping TCP performance test traffic).+$Show output
05/11/2025-22:56:11.876848 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:55440 -> 40.0.0.2:5001 05/11/2025-22:56:11.883951 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:55444 -> 40.0.0.2:5001 05/11/2025-22:56:26.645135 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:56466 -> 40.0.0.2:5000
Step 14: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5000.*mark=147652983.*$Show output
tcp 6 29 LAST_ACK src=20.0.0.2 dst=40.0.0.2 sport=56466 dport=5000 packets=9 bytes=700 src=40.0.0.2 dst=20.0.0.2 sport=5000 dport=56466 packets=5 bytes=270 [ASSURED] (Sc: not-bypass) mark=147652983 use=1 conntrack v1.4.7 (conntrack-tools): 1 flow entries have been shown.
Step 15: Run command traffic policy FW_PLAN show at DUT0 and check if output matches the following regular expressions:
(?m)^1\s+FW_SEL_DROP\s+[1-9].*$Show output
Policy FW_PLAN -- ifc eth1.101 -- hook in prio very-high --------------------------------------------------------------------- rule selector pkts match pkts eval bytes match bytes eval --------------------------------------------------------------------- 1 FW_SEL_DROP 4 5 210 270 2 FW_SEL_ENQUEUE 1 1 60 60 --------------------------------------------------------------------- Total 5 5 270 270 Policy FW_PLAN -- ifc eth1.201 -- hook in prio very-high --------------------------------------------------------------------- rule selector pkts match pkts eval bytes match bytes eval --------------------------------------------------------------------- 1 FW_SEL_DROP 6 9 499 700 2 FW_SEL_ENQUEUE 3 3 201 201 --------------------------------------------------------------------- Total 9 9 700 700
Test Capture And Offload
Description
Builds a scenario with three DUTs in which a performance test is conducted between DUT1 and DUT2, and DUT0 is the router running the firewall. This test sets the conntrack mark directly, thus skipping all the steps required to set it later. In addition, OSDx is instructed to accelerate the flow using internal accelerators.
Performance must improve considerably compared to the previous test, to reach its top value.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/test-performance.rules running:// force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 16054 0 --:--:-- --:--:-- --:--:-- 16125
Step 2: Run command file show running://test-performance.rules at DUT0 and expect this output:
Show output
alert tcp any any -> any 5001 (msg: "Skipping test network performance traffic"; bypass; flow: established, to_server; sid: 40;)
Step 3: Set the following configuration in DUT0:
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://test-performance.rules' set service firewall FW stream bypass action accept set conntrack offload-flag set service firewall FW stream bypass mark 129834765 set service firewall FW stream bypass mask 129834765 set service firewall FW stream bypass set-connmark set service firewall FW validator-timeout 20 set system conntrack timeout tcp established 60 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set system offload timeout 60 set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic policy FW_PLAN rule 2 action enqueue FW_Q set traffic policy FW_PLAN rule 2 selector FW_SEL_ENQUEUE set traffic queue FW_Q elements 1 set traffic selector FW_SEL_ENQUEUE rule 1 not connmark 129834765
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.738 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.738/0.738/0.738/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.702 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.702/0.702/0.702/0.000 ms
Step 6: Run command monitor test performance server port 5001 at DUT1.
Step 7: Run command monitor test performance client 40.0.0.2 duration 10 port 5001 at DUT2.
Step 8: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Skipping test network performance traffic).+$Show output
05/11/2025-22:57:01.386932 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:44344 -> 40.0.0.2:5001 05/11/2025-22:57:01.392257 [**] [1:40:0] Skipping test network performance traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:44360 -> 40.0.0.2:5001
Step 9: Run command system conntrack show at DUT0 and check if output matches the following regular expressions:
(?m)^tcp\s+.*src=20.0.0.2 dst=40.0.0.2.+dport=5001.+OFFLOAD.+mark=129834765.*$Show output
tcp 6 src=20.0.0.2 dst=40.0.0.2 sport=44344 dport=5001 packets=8 bytes=589 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=44344 packets=7 bytes=376 [ASSURED] [OFFLOAD, packets=5 bytes=388 packets=4 bytes=211] mark=129834765 use=2 icmp 1 29 src=40.0.0.2 dst=20.0.0.2 type=8 code=0 id=304 packets=1 bytes=84 src=20.0.0.2 dst=40.0.0.2 type=0 code=0 id=304 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 tcp 6 src=20.0.0.2 dst=40.0.0.2 sport=44360 dport=5001 packets=18909 bytes=28359201 src=40.0.0.2 dst=20.0.0.2 sport=5001 dport=44360 packets=2225 bytes=115708 [ASSURED] [OFFLOAD, packets=18896 bytes=28344000 packets=2223 bytes=115596] mark=129834765 use=2 icmp 1 29 src=20.0.0.2 dst=40.0.0.2 type=8 code=0 id=46 packets=1 bytes=84 src=40.0.0.2 dst=20.0.0.2 type=0 code=0 id=46 packets=1 bytes=84 (Sc: not-bypass) mark=0 use=1 conntrack v1.4.7 (conntrack-tools): 4 flow entries have been shown.
Test Traffic Early Dropping
Description
Builds a scenario with three DUTs and a simple ruleset to drop TCP traffic between DUT1 and DUT2. Such traffic must pass through port 5000 for the rule to match. Later, XDP is queried to check if packets are being dropped at the specified interface.
The contents of the rule file are:
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;)
This rule allows the connection to be established and traffic to be dropped later.
Scenario
Step 1: Run command file copy http://10.215.168.1/~robot/drop-performance.rules running://drop-performance.rules force at DUT0 and expect this output:
Show output
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 200 100 200 0 0 25913 0 --:--:-- --:--:-- --:--:-- 28571
Step 2: Run command file show running://drop-performance.rules at DUT0 and expect this output:
Show output
drop tcp any any -> any 5000 (msg: "Dropping TCP performance test traffic"; sid: 1; flow: established, to_server;) drop udp any any -> any 5001 (msg: "Dropping UDP performance test traffic"; sid: 2;)
Step 3: Set the following configuration in DUT0:
set interfaces ethernet eth0 address 10.215.168.64/24 set interfaces ethernet eth0 traffic nat source rule 1 address masquerade set interfaces ethernet eth1 vif 101 address 40.0.0.1/8 set interfaces ethernet eth1 vif 101 traffic policy in FW_PLAN set interfaces ethernet eth1 vif 201 address 20.0.0.1/8 set interfaces ethernet eth1 vif 201 traffic policy in FW_PLAN set protocols static route 0.0.0.0/0 next-hop 10.215.168.1 set service dns static host-name WAN inet 10.215.168.1 set service firewall FW logging level config set service firewall FW logging outputs fast set service firewall FW mode inline queue FW_Q set service firewall FW ruleset file 'running://drop-performance.rules' set service firewall FW stream bypass action drop set xdp-early-drop eth1 set service firewall FW validator-timeout 20 set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0' set traffic policy FW_PLAN rule 1 action enqueue FW_Q set traffic queue FW_Q elements 1
Step 4: Ping IP address 20.0.0.2 from DUT1:
admin@DUT1$ ping 20.0.0.2 count 1 size 56 timeout 1Show output
PING 20.0.0.2 (20.0.0.2) 56(84) bytes of data. 64 bytes from 20.0.0.2: icmp_seq=1 ttl=63 time=0.803 ms --- 20.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.803/0.803/0.803/0.000 ms
Step 5: Ping IP address 40.0.0.2 from DUT2:
admin@DUT2$ ping 40.0.0.2 count 1 size 56 timeout 1Show output
PING 40.0.0.2 (40.0.0.2) 56(84) bytes of data. 64 bytes from 40.0.0.2: icmp_seq=1 ttl=63 time=0.642 ms --- 40.0.0.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.642/0.642/0.642/0.000 ms
Step 6: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5000 admin@DUT2$ monitor test performance client 40.0.0.2 duration 10 port 5000 parallel 1Expect this output in
DUT2:^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr iperf3: interrupt - the client has terminated admin@osdx$
Step 7: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Dropping TCP performance test traffic).+$Show output
05/11/2025-22:57:33.450838 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45020 -> 40.0.0.2:5000
Step 8: Run command service firewall FW show early-drop-stats eth1 at DUT0 and check if output matches the following regular expressions:
yes\s+201\s+\d+\s+[1-9]\d*\s+[1-9]\d*Show output
------------------------------------------------------------------------ src dst src port dst port tcp vlan_0 vlan_1 pkts bytes ------------------------------------------------------------------------ 20.0.0.2 40.0.0.2 45020 5000 yes 201 0 8 673 40.0.0.2 20.0.0.2 5000 45020 yes 201 0 0 0
Step 9: Run command interfaces ethernet eth1 monitor xdp-stats times 1 at DUT0 and expect this output:
Show output
Pin path: /sys/fs/bpf/eth1 Period of 0.250482s ending at 1747004257.272229 XDP_DROP 9 pkts ( 4 pps) 0 KiB ( 0 Mbits/s) XDP_PASS 14 pkts ( 4 pps) 1 KiB ( 0 Mbits/s) XDP_TX 0 pkts ( 0 pps) 0 KiB ( 0 Mbits/s) XDP_REDIRECT 0 pkts ( 0 pps) 0 KiB ( 0 Mbits/s)
Step 10: Initiate a bandwidth test from DUT2 to DUT1
admin@DUT1$ monitor test performance server port 5001 admin@DUT2$ monitor test performance client 40.0.0.2 duration 30 udp port 5001 parallel 1Expect this output in
DUT2:^C- - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams iperf3: interrupt - the client has terminated admin@osdx$
Step 11: Run command service firewall FW show logging fast | tail at DUT0 and check if output matches the following regular expressions:
(?m)^.+(Dropping UDP performance test traffic).+$Show output
05/11/2025-22:57:33.450838 [Drop] [**] [1:1:0] Dropping TCP performance test traffic [**] [Classification: (null)] [Priority: 3] {TCP} 20.0.0.2:45020 -> 40.0.0.2:5000 05/11/2025-22:57:37.575524 [Drop] [**] [1:2:0] Dropping UDP performance test traffic [**] [Classification: (null)] [Priority: 3] {UDP} 20.0.0.2:59540 -> 40.0.0.2:5001
Step 12: Run command service firewall FW show early-drop-stats eth1 at DUT0 and check if output matches the following regular expressions:
yes\s+201\s+\d+\s+[1-9]\d*\s+[1-9]\d*Show output
------------------------------------------------------------------------ src dst src port dst port tcp vlan_0 vlan_1 pkts bytes ------------------------------------------------------------------------ 40.0.0.2 20.0.0.2 5001 59540 no 201 0 0 0 20.0.0.2 40.0.0.2 45020 5000 yes 201 0 11 847 20.0.0.2 40.0.0.2 59540 5001 no 201 0 0 0 40.0.0.2 20.0.0.2 5000 45020 yes 201 0 0 0
Step 13: Run command interfaces ethernet eth1 monitor xdp-stats times 1 at DUT0 and expect this output:
Show output
Pin path: /sys/fs/bpf/eth1 Period of 0.250190s ending at 1747004261.286362 XDP_DROP 11 pkts ( 0 pps) 0 KiB ( 0 Mbits/s) XDP_PASS 33 pkts ( 0 pps) 2 KiB ( 0 Mbits/s) XDP_TX 0 pkts ( 0 pps) 0 KiB ( 0 Mbits/s) XDP_REDIRECT 0 pkts ( 0 pps) 0 KiB ( 0 Mbits/s)