Failover-Split-Brain

Test suite for DHCP Failover split-brain scenarios.

Split-brain occurs when both DHCP servers lose communication with each other but both remain operational. Each server thinks the other is down and may enter “communications-interrupted” state.

This test suite verifies: - Both servers detect communications-interrupted state - Clients can still obtain IPs during split-brain - Servers recover to normal state when communication is restored

Test Split-Brain Detection

Description

This scenario verifies that when communication between primary and secondary DHCP servers is blocked, both servers detect the issue and enter “communications-interrupted” state.

The failover protocol uses port 647 for peer communication. By blocking this port, we simulate a network partition.

Expected behavior: - Initially both servers are in “normal” state - After blocking port 647, both detect “communications-interrupted” - Servers may also show “partner down” after timeout

Scenario

Step 1: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.0.0.1/24
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover local-address 10.0.0.1
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover name failover-split
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover peer-address 10.0.0.2
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover status primary
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 start 10.0.0.10 stop 10.0.0.20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 2: Set the following configuration in DUT1 :

set interfaces ethernet eth0 address 10.0.0.2/24
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover local-address 10.0.0.2
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover name failover-split
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover peer-address 10.0.0.1
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover status secondary
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 start 10.0.0.10 stop 10.0.0.20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 3: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Local state\s+normal

Step 4: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Partner state\s+normal

Step 5: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Local state\s+normal

Step 6: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Partner state\s+normal

Step 7: Modify the following configuration lines in DUT0 :

set interfaces ethernet eth0 traffic policy link-in BLOCK_FAILOVER
set traffic policy BLOCK_FAILOVER rule 1 action drop
set traffic policy BLOCK_FAILOVER rule 1 selector BLOCK_PEER
set traffic policy BLOCK_FAILOVER rule 2 action accept
set traffic selector BLOCK_PEER rule 1 source address 10.0.0.2/32

Step 8: Modify the following configuration lines in DUT1 :

set interfaces ethernet eth0 traffic policy link-in BLOCK_FAILOVER
set traffic policy BLOCK_FAILOVER rule 1 action drop
set traffic policy BLOCK_FAILOVER rule 1 selector BLOCK_PEER
set traffic policy BLOCK_FAILOVER rule 2 action accept
set traffic selector BLOCK_PEER rule 1 source address 10.0.0.1/32

Step 9: Run command service dhcp-server show failover at DUT0 and expect this output:

Step 10: Run command service dhcp-server show failover at DUT1 and expect this output:

Test Split-Brain Client Still Gets IP

Description

This scenario verifies that during a split-brain condition, DHCP clients can still obtain IP addresses from one of the servers. The failover protocol is designed to allow both servers to serve clients during communication interruption.

Expected behavior: - Split-brain is active (servers can’t communicate) - Client requests DHCP lease - Client successfully obtains IP from available server

Scenario

Step 1: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.0.0.1/24
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover local-address 10.0.0.1
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover name failover-split
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover peer-address 10.0.0.2
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover status primary
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 start 10.0.0.10 stop 10.0.0.20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 2: Set the following configuration in DUT1 :

set interfaces ethernet eth0 address 10.0.0.2/24
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover local-address 10.0.0.2
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover name failover-split
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover peer-address 10.0.0.1
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover status secondary
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 start 10.0.0.10 stop 10.0.0.20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 3: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Local state\s+normal

Step 4: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Partner state\s+normal

Step 5: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Local state\s+normal

Step 6: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Partner state\s+normal

Step 7: Modify the following configuration lines in DUT0 :

set interfaces ethernet eth0 traffic policy link-in BLOCK_FAILOVER
set traffic policy BLOCK_FAILOVER rule 1 action drop
set traffic policy BLOCK_FAILOVER rule 1 selector BLOCK_PEER
set traffic policy BLOCK_FAILOVER rule 2 action accept
set traffic selector BLOCK_PEER rule 1 source address 10.0.0.2/32

Step 8: Modify the following configuration lines in DUT1 :

set interfaces ethernet eth0 traffic policy link-in BLOCK_FAILOVER
set traffic policy BLOCK_FAILOVER rule 1 action drop
set traffic policy BLOCK_FAILOVER rule 1 selector BLOCK_PEER
set traffic policy BLOCK_FAILOVER rule 2 action accept
set traffic selector BLOCK_PEER rule 1 source address 10.0.0.1/32

Step 9: Run command service dhcp-server show failover at DUT0 and expect this output:

Step 10: Set the following configuration in DUT2 :

set interfaces ethernet eth0 address dhcp
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 11: Run command interfaces ethernet eth0 show at DUT2 and check if output contains the following tokens:

10.0.0.

Step 12: Run command service dhcp-server show leases at DUT0 and expect this output: Step 13: Run command service dhcp-server show leases at DUT1 and expect this output:

Test Split-Brain Recovery

Description

This scenario verifies that after a split-brain condition is resolved (communication restored), both servers recover to normal state and synchronize their lease databases.

Expected behavior: - Start in split-brain state - Remove traffic blocking rules - Both servers recover to “normal” state - Lease databases are synchronized

Scenario

Step 1: Set the following configuration in DUT0 :

set interfaces ethernet eth0 address 10.0.0.1/24
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover local-address 10.0.0.1
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover name failover-split
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover peer-address 10.0.0.2
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover status primary
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 start 10.0.0.10 stop 10.0.0.20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 2: Set the following configuration in DUT1 :

set interfaces ethernet eth0 address 10.0.0.2/24
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover local-address 10.0.0.2
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover name failover-split
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover peer-address 10.0.0.1
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 failover status secondary
set service dhcp-server shared-network dhcp subnet 10.0.0.0/24 start 10.0.0.10 stop 10.0.0.20
set system login user admin authentication encrypted-password '$6$GSjsCj8gHLv$/VcqU6FLi6CT2Oxn0MJQ2C2tqnRDrYKNF8HIYWJp68nvXvPdFccDsT04.WtigUONbKYrgKg8d6rEs8PjljMkH0'

Step 3: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Local state\s+normal

Step 4: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Partner state\s+normal

Step 5: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Local state\s+normal

Step 6: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Partner state\s+normal

Step 7: Modify the following configuration lines in DUT0 :

set interfaces ethernet eth0 traffic policy link-in BLOCK_FAILOVER
set traffic policy BLOCK_FAILOVER rule 1 action drop
set traffic policy BLOCK_FAILOVER rule 1 selector BLOCK_PEER
set traffic policy BLOCK_FAILOVER rule 2 action accept
set traffic selector BLOCK_PEER rule 1 source address 10.0.0.2/32

Step 8: Modify the following configuration lines in DUT1 :

set interfaces ethernet eth0 traffic policy link-in BLOCK_FAILOVER
set traffic policy BLOCK_FAILOVER rule 1 action drop
set traffic policy BLOCK_FAILOVER rule 1 selector BLOCK_PEER
set traffic policy BLOCK_FAILOVER rule 2 action accept
set traffic selector BLOCK_PEER rule 1 source address 10.0.0.1/32

Step 9: Run command service dhcp-server show failover at DUT0 and expect this output:

Step 10: Modify the following configuration lines in DUT0 :

delete interfaces ethernet eth0 traffic

Step 11: Modify the following configuration lines in DUT0 :

delete traffic policy

Step 12: Modify the following configuration lines in DUT0 :

delete traffic

Step 13: Modify the following configuration lines in DUT1 :

delete interfaces ethernet eth0 traffic

Step 14: Modify the following configuration lines in DUT1 :

delete traffic policy

Step 15: Modify the following configuration lines in DUT1 :

delete traffic

Step 16: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Local state\s+normal

Step 17: Run command service dhcp-server show failover at DUT0 and check if output matches the following regular expressions:

Partner state\s+normal

Step 18: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Local state\s+normal

Step 19: Run command service dhcp-server show failover at DUT1 and check if output matches the following regular expressions:

Partner state\s+normal