Inspecting Network Layers
In this chapter, we will detail some useful commands to rule out problems per network layer (from the most physical to the most abstract).
Step 1: Physical Level
First of all, we must check all defined interfaces are working at the physical level. To do this, there are several useful commands:
interfaces show: checks global information.
Example:
admin@osdx$ interfaces show
-----------------------------------------------------------------
Name IP Address Admin Oper Vrf Description
-----------------------------------------------------------------
br0 192.168.100.10/24 up up
fe80::9007:dbff:fe85:fa8/64
eth0 fe80::dcad:beff:feef:6c10/64 up up
eth1 down down
interfaces show detailed: checks global information in greater detail.
Example:
admin@osdx$ interfaces show detailed
----------------------------------------------------------------------------------------------------------------
Name Idx IP Address Admin Oper Link MTU Vrf Upper Lower Type Phys addr
----------------------------------------------------------------------------------------------------------------
br0 4 192.168.100.10/24 up up up 1500 bridge de:ad:be:ef:6c:10
fe80::9007:dbff:fe85:fa8/64
eth0 2 fe80::dcad:beff:feef:6c10/64 up up up 1500 br0 ethernet de:ad:be:ef:6c:10
eth1 3 down down down 1500 ethernet de:ad:be:ef:6c:11
interfaces show counters: checks all interface counters.
Example:
admin@osdx$ interfaces show counters
----------------------------------------------------------------------------
Name Oper Rx Packets Rx Bytes Rx Errors Tx Packets Tx Bytes Tx Errors
----------------------------------------------------------------------------
br0 up 3 140 0 16 1460 0
eth0 up 13 854 0 20 1820 0
eth1 down 0 0 0 0 0 0
interfaces <interface_type> show: checks the global information pertaining to a given interface type.
Example:
admin@osdx$ interfaces ethernet show
-----------------------------------------------------------------
Name IP Address Admin Oper Vrf Description
-----------------------------------------------------------------
eth0 fe80::dcad:beff:feef:6c10/64 up up
eth1 down down
Step 2: Link Level
Next, we will check the information at the link level. Different commands can be used for this task:
system ip neighbors show: checks information about neighbors.
Example:
admin@osdx$ system ip neighbors show
192.168.100.20 dev br0 lladdr de:ad:be:ef:6c:20 REACHABLE
system ip neighbors show interface <ifc>: checks information about neighbors per interface.
Example:
admin@osdx$ system ip neighbors show interface br0
192.168.100.20 lladdr de:ad:be:ef:6c:20 REACHABLE
Step 3: Network Level
Now we are going to check if the routing information is OK. The following commands are useful:
protocols ip show route: checks the main VRF routing table.
Example:
admin@osdx$ protocols ip show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:11
C>* 192.168.100.0/24 is directly connected, br0, 00:01:11
protocols ip show route <ipv4|ipv4net>: checks routing table entries per type.
Example:
admin@osdx$ protocols ip show route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:57
protocols ip show route summary: checks the summary of routing table entries.
Example:
admin@osdx$ protocols ip show route summary
Route Source Routes FIB (vrf default)
connected 1 1
static 1 1
------
Totals 2 2
protocols vrf <value> ip show route: checks a given VRF routing table
Example:
admin@osdx$ protocols vrf BLUE ip show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:06:31
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:06:31
protocols vrf <value> ip show route <ipv4|ipv4net>: checks selected VRF routing table entries by type.
Example:
admin@osdx$ protocols vrf BLUE ip show route connected
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37
protocols vrf <value> ip show route summary: checks the counter of selected VRF routing table entries.
Example:
admin@osdx$ protocols vrf BLUE ip show route connected
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37
admin@osdx$ protocols vrf BLUE ip show route summary
Route Source Routes FIB (vrf BLUE)
kernel 1 1
connected 1 1
------
Totals 2 2
For each protocol, the commands to check the connection and routing status are shown below in the corresponding section.
Step 4: Transport Level
This section shows the commands used to obtain information regarding the transport layer.
CONNTRACK info
system conntrack show: checks the conntrack table.
Example:
admin@osdx$ system conntrack show
udp 17 22 src=10.0.0.2 dst=10.0.0.1 sport=40128 dport=2055 packets=1 bytes=146 [UNREPLIED] src=10.0.0.1 dst=10.0.0.2 sport=2055 dport=40128 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055]
icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=13 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=13 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1]
tcp 6 15 TIME_WAIT src=10.0.0.2 dst=10.0.0.1 sport=43850 dport=8080 packets=6 bytes=338 src=10.0.0.1 dst=10.0.0.2 sport=8080 dport=43850 packets=5 bytes=286 [ASSURED] mark=0 use=3 appdetect[L4:8080]
udp 17 22 src=127.0.0.1 dst=127.0.0.1 sport=48253 dport=2055 packets=1 bytes=146 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=2055 dport=48253 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055]
icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=12 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=12 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1]
conntrack v1.4.5 (conntrack-tools): 5 flow entries have been shown.
Conntrack information can be filtered by protocol, source, destination, IP family, and NAT.
system conntrack show protocol <value>: only shows entries with a specific protocol
Example:
system conntrack show protocol udp
udp 17 src=11.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=11.0.0.2 sport=1234 dport=2345 vrf=wan3 packets=5 bytes=240 [OFFLOAD, packets=3 bytes=144 packets=4 bytes=192] mark=0 use=2 appdetect[L4:1234]
udp 17 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 vrf=tenant2 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 vrf=wan2 packets=5 bytes=240 [OFFLOAD, packets=3 bytes=144 packets=4 bytes=192] mark=0 use=2 appdetect[L4:1234]
udp 17 28 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 vrf=tenant1 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 vrf=wan1 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 26 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 10 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 vrf=wan2 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 vrf=wan2 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 8 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 vrf=wan1 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 vrf=wan1 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 5 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 packets=5 bytes=240 mark=0 use=2 appdetect[L4:1234]
conntrack v1.4.5 (conntrack-tools): 7 flow entries have been shown.
system conntrack show family <value>: only shows IPv4/IPv6 entries.system conntrack show source <ipv4|ipv6|ipv4net|ipv6net>: only shows entries with source IPsystem conntrack show destination <ipv4|ipv6|ipv4net|ipv6net>: only shows entries with destination IPsystem conntrack show nat: only shows NAT entriessystem conntrack show source-nat: only shows source NAT entriessystem conntrack show destination-nat: only shows destination NAT entries