.. _inspecting_network_layers: ========================= Inspecting Network Layers ========================= .. sidebar:: Contents .. contents:: :depth: 3 :local: In this chapter, we will detail some useful commands to rule out problems per network layer (from the most physical to the most abstract). Step 1: Physical Level ====================== First of all, we must check all defined interfaces are working at the physical level. To do this, there are several useful commands: * :osdx:op:`interfaces show`: checks global information. *Example:* .. code-block:: none admin@osdx$ interfaces show ----------------------------------------------------------------- Name IP Address Admin Oper Vrf Description ----------------------------------------------------------------- br0 192.168.100.10/24 up up fe80::9007:dbff:fe85:fa8/64 eth0 fe80::dcad:beff:feef:6c10/64 up up eth1 down down * :osdx:op:`interfaces show detailed`: checks global information in greater detail. *Example:* .. code-block:: none admin@osdx$ interfaces show detailed ---------------------------------------------------------------------------------------------------------------- Name Idx IP Address Admin Oper Link MTU Vrf Upper Lower Type Phys addr ---------------------------------------------------------------------------------------------------------------- br0 4 192.168.100.10/24 up up up 1500 bridge de:ad:be:ef:6c:10 fe80::9007:dbff:fe85:fa8/64 eth0 2 fe80::dcad:beff:feef:6c10/64 up up up 1500 br0 ethernet de:ad:be:ef:6c:10 eth1 3 down down down 1500 ethernet de:ad:be:ef:6c:11 * :osdx:op:`interfaces show counters`: checks all interface counters. *Example:* .. code-block:: none admin@osdx$ interfaces show counters ---------------------------------------------------------------------------- Name Oper Rx Packets Rx Bytes Rx Errors Tx Packets Tx Bytes Tx Errors ---------------------------------------------------------------------------- br0 up 3 140 0 16 1460 0 eth0 up 13 854 0 20 1820 0 eth1 down 0 0 0 0 0 0 * ``interfaces show``: checks the global information pertaining to a given interface type. *Example:* .. code-block:: none admin@osdx$ interfaces ethernet show ----------------------------------------------------------------- Name IP Address Admin Oper Vrf Description ----------------------------------------------------------------- eth0 fe80::dcad:beff:feef:6c10/64 up up eth1 down down Step 2: Link Level ================== Next, we will check the information at the link level. Different commands can be used for this task: * :osdx:op:`system ip neighbors show`: checks information about neighbors. *Example:* .. code-block:: none admin@osdx$ system ip neighbors show 192.168.100.20 dev br0 lladdr de:ad:be:ef:6c:20 REACHABLE * :osdx:op:`system ip neighbors show interface *`: checks information about neighbors per interface. *Example:* .. code-block:: none admin@osdx$ system ip neighbors show interface br0 192.168.100.20 lladdr de:ad:be:ef:6c:20 REACHABLE Step 3: Network Level ===================== Now we are going to check if the routing information is OK. The following commands are useful: * :osdx:op:`protocols ip show route`: checks the main VRF routing table. *Example:* .. code-block:: none admin@osdx$ protocols ip show route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:11 C>* 192.168.100.0/24 is directly connected, br0, 00:01:11 * :osdx:op:`protocols ip show route *`: checks routing table entries per type. *Example:* .. code-block:: none admin@osdx$ protocols ip show route static Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:57 * :osdx:op:`protocols ip show route summary`: checks the summary of routing table entries. *Example:* .. code-block:: none admin@osdx$ protocols ip show route summary Route Source Routes FIB (vrf default) connected 1 1 static 1 1 ------ Totals 2 2 * :osdx:op:`protocols vrf * ip show route`: checks a given VRF routing table *Example:* .. code-block:: none admin@osdx$ protocols vrf BLUE ip show route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF BLUE: K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:06:31 C>* 192.168.200.0/24 is directly connected, eth1.102, 00:06:31 * :osdx:op:`protocols vrf * ip show route *`: checks selected VRF routing table entries by type. *Example:* .. code-block:: none admin@osdx$ protocols vrf BLUE ip show route connected Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF BLUE: C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37 * :osdx:op:`protocols vrf * ip show route summary`: checks the counter of selected VRF routing table entries. *Example:* .. code-block:: none admin@osdx$ protocols vrf BLUE ip show route connected Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF BLUE: C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37 admin@osdx$ protocols vrf BLUE ip show route summary Route Source Routes FIB (vrf BLUE) kernel 1 1 connected 1 1 ------ Totals 2 2 For each protocol, the commands to check the connection and routing status are shown below in the corresponding section. .. toctree:: :titlesonly: :glob: protocols/*/index Step 4: Transport Level ======================= This section shows the commands used to obtain information regarding the transport layer. CONNTRACK info -------------- * :osdx:op:`system conntrack show`: checks the conntrack table. *Example:* .. code-block:: none admin@osdx$ system conntrack show udp 17 22 src=10.0.0.2 dst=10.0.0.1 sport=40128 dport=2055 packets=1 bytes=146 [UNREPLIED] src=10.0.0.1 dst=10.0.0.2 sport=2055 dport=40128 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055] icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=13 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=13 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1] tcp 6 15 TIME_WAIT src=10.0.0.2 dst=10.0.0.1 sport=43850 dport=8080 packets=6 bytes=338 src=10.0.0.1 dst=10.0.0.2 sport=8080 dport=43850 packets=5 bytes=286 [ASSURED] mark=0 use=3 appdetect[L4:8080] udp 17 22 src=127.0.0.1 dst=127.0.0.1 sport=48253 dport=2055 packets=1 bytes=146 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=2055 dport=48253 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055] icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=12 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=12 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1] conntrack v1.4.5 (conntrack-tools): 5 flow entries have been shown. Conntrack information can be filtered by protocol, source, destination, IP family, and NAT. * :osdx:op:`system conntrack show protocol *`: only shows entries with a specific protocol *Example:* .. code-block:: none system conntrack show protocol udp udp 17 src=11.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=11.0.0.2 sport=1234 dport=2345 vrf=wan3 packets=5 bytes=240 [OFFLOAD, packets=3 bytes=144 packets=4 bytes=192] mark=0 use=2 appdetect[L4:1234] udp 17 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 vrf=tenant2 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 vrf=wan2 packets=5 bytes=240 [OFFLOAD, packets=3 bytes=144 packets=4 bytes=192] mark=0 use=2 appdetect[L4:1234] udp 17 28 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 vrf=tenant1 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 vrf=wan1 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234] udp 17 26 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234] udp 17 10 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 vrf=wan2 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 vrf=wan2 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234] udp 17 8 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 vrf=wan1 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 vrf=wan1 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234] udp 17 5 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 packets=5 bytes=240 mark=0 use=2 appdetect[L4:1234] conntrack v1.4.5 (conntrack-tools): 7 flow entries have been shown. * :osdx:op:`system conntrack show family *`: only shows IPv4/IPv6 entries. * :osdx:op:`system conntrack show source *`: only shows entries with source IP * :osdx:op:`system conntrack show destination *`: only shows entries with destination IP * :osdx:op:`system conntrack show nat`: only shows NAT entries * :osdx:op:`system conntrack show source-nat`: only shows source NAT entries * :osdx:op:`system conntrack show destination-nat`: only shows destination NAT entries