.. _intro-troubleshooting: =============== Troubleshooting =============== In this chapter, we will show some useful commands to help the admin deal with problems. Physical Level ============== First of all, we must check all defined interfaces are working at the physical level. To do this, there are several useful commands: * :osdx:op:`interfaces show`: checks global information. *Example:* .. code-block:: none admin@osdx$ interfaces show ----------------------------------------------------------------- Name IP Address Admin Oper Vrf Description ----------------------------------------------------------------- br0 192.168.100.10/24 up up fe80::9007:dbff:fe85:fa8/64 eth0 fe80::dcad:beff:feef:6c10/64 up up eth1 down down * :osdx:op:`interfaces show detailed`: checks global information in greater detail. *Example:* .. code-block:: none admin@osdx$ interfaces show detailed ---------------------------------------------------------------------------------------------------------------- Name Idx IP Address Admin Oper Link MTU Vrf Upper Lower Type Phys addr ---------------------------------------------------------------------------------------------------------------- br0 4 192.168.100.10/24 up up up 1500 bridge de:ad:be:ef:6c:10 fe80::9007:dbff:fe85:fa8/64 eth0 2 fe80::dcad:beff:feef:6c10/64 up up up 1500 br0 ethernet de:ad:be:ef:6c:10 eth1 3 down down down 1500 ethernet de:ad:be:ef:6c:11 * :osdx:op:`interfaces show counters`: checks all interface counters. *Example:* .. code-block:: none admin@osdx$ interfaces show counters ---------------------------------------------------------------------------- Name Oper Rx Packets Rx Bytes Rx Errors Tx Packets Tx Bytes Tx Errors ---------------------------------------------------------------------------- br0 up 3 140 0 16 1460 0 eth0 up 13 854 0 20 1820 0 eth1 down 0 0 0 0 0 0 * ``interfaces show``: checks the global information pertaining to a given interface type. *Example:* .. code-block:: none admin@osdx$ interfaces ethernet show ----------------------------------------------------------------- Name IP Address Admin Oper Vrf Description ----------------------------------------------------------------- eth0 fe80::dcad:beff:feef:6c10/64 up up eth1 down down Link Level ========== Next, we will check the information at the link level. Different commands can be used for this task: * :osdx:op:`system ip neighbors show`: checks information about neighbors. *Example:* .. code-block:: none admin@osdx$ system ip neighbors show 192.168.100.20 dev br0 lladdr de:ad:be:ef:6c:20 REACHABLE * :osdx:op:`system ip neighbors show interface *`: checks information about neighbors per interface. *Example:* .. code-block:: none admin@osdx$ system ip neighbors show interface br0 192.168.100.20 lladdr de:ad:be:ef:6c:20 REACHABLE Network Level ============= Now we are going to check if the routing information is OK. The following commands are useful: * :osdx:op:`protocols ip show route`: checks the main VRF routing table. *Example:* .. code-block:: none admin@osdx$ protocols ip show route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:11 C>* 192.168.100.0/24 is directly connected, br0, 00:01:11 * :osdx:op:`protocols ip show route *`: checks routing table entries per type. *Example:* .. code-block:: none admin@osdx$ protocols ip show route static Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:57 * :osdx:op:`protocols ip show route summary`: checks the summary of routing table entries. *Example:* .. code-block:: none admin@osdx$ protocols ip show route summary Route Source Routes FIB (vrf default) connected 1 1 static 1 1 ------ Totals 2 2 * :osdx:op:`protocols vrf * ip show route`: checks a given VRF routing table *Example:* .. code-block:: none admin@osdx$ protocols vrf BLUE ip show route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF BLUE: K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:06:31 C>* 192.168.200.0/24 is directly connected, eth1.102, 00:06:31 * :osdx:op:`protocols vrf * ip show route *`: checks selected VRF routing table entries by type. *Example:* .. code-block:: none admin@osdx$ protocols vrf BLUE ip show route connected Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF BLUE: C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37 * :osdx:op:`protocols vrf * ip show route summary`: checks the counter of selected VRF routing table entries. *Example:* .. code-block:: none admin@osdx$ protocols vrf BLUE ip show route connected Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF BLUE: C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37 admin@osdx$ protocols vrf BLUE ip show route summary Route Source Routes FIB (vrf BLUE) kernel 1 1 connected 1 1 ------ Totals 2 2 IPsec protocol -------------- Checks whether the IPsec protocol information is correct. * :osdx:op:`vpn ipsec show policy`: checks the information available on kernel crypto policies. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show policy src 10.0.0.1/32 dst 10.0.0.2/32 dir out priority 367231 tmpl src 10.0.0.1 dst 10.0.0.2 proto esp spi 0xcde9784b reqid 1 mode tunnel src 10.0.0.2/32 dst 10.0.0.1/32 dir fwd priority 367231 tmpl src 10.0.0.2 dst 10.0.0.1 proto esp reqid 1 mode tunnel src 10.0.0.2/32 dst 10.0.0.1/32 dir in priority 367231 tmpl src 10.0.0.2 dst 10.0.0.1 proto esp reqid 1 mode tunnel src 0.0.0.0/0 dst 0.0.0.0/0 socket in priority 0 src 0.0.0.0/0 dst 0.0.0.0/0 socket out priority 0 src 0.0.0.0/0 dst 0.0.0.0/0 socket in priority 0 src 0.0.0.0/0 dst 0.0.0.0/0 socket out priority 0 src ::/0 dst ::/0 socket in priority 0 src ::/0 dst ::/0 socket out priority 0 src ::/0 dst ::/0 socket in priority 0 src ::/0 dst ::/0 socket out priority 0 * :osdx:op:`vpn ipsec show sa`: checks information related to IPsec SA. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show sa vpn-peer-SITE1: #1, ESTABLISHED, IKEv2, 0fd20672a782d852_i* 0aab0776adbd3fc1_r local '10.0.0.1' @ 10.0.0.1[500] remote '10.0.0.2' @ 10.0.0.2[500] NULL/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_3072 established 1479s ago, rekeying in 25550s peer-SITE1-tunnel-1: #1, reqid 1, INSTALLED, TUNNEL, ESP:NULL/HMAC_SHA1_96 installed 1479s ago, rekeying in 1942s, expires in 2481s in c7130959, 168 bytes, 2 packets, 1479s ago out cde9784b, 168 bytes, 2 packets, 1479s ago local 10.0.0.1/32 remote 10.0.0.2/32 * :osdx:op:`vpn ipsec show sa local *`: checks information related to IPsec SA in a selected local peer. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show sa local 10.0.0.1 vpn-peer-SITE1: #1, ESTABLISHED, IKEv2, 0fd20672a782d852_i* 0aab0776adbd3fc1_r local '10.0.0.1' @ 10.0.0.1[500] remote '10.0.0.2' @ 10.0.0.2[500] NULL/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_3072 established 1544s ago, rekeying in 25485s peer-SITE1-tunnel-1: #1, reqid 1, INSTALLED, TUNNEL, ESP:NULL/HMAC_SHA1_96 installed 1544s ago, rekeying in 1877s, expires in 2416s in c7130959, 168 bytes, 2 packets, 1544s ago out cde9784b, 168 bytes, 2 packets, 1544s ago local 10.0.0.1/32 remote 10.0.0.2/32 * :osdx:op:`vpn ipsec show sa remote *`: checks information related to IPsec SA in a selected peer. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show sa remote 10.0.0.2 vpn-peer-SITE1: #1, ESTABLISHED, IKEv2, 0fd20672a782d852_i* 0aab0776adbd3fc1_r local '10.0.0.1' @ 10.0.0.1[500] remote '10.0.0.2' @ 10.0.0.2[500] NULL/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_3072 established 1581s ago, rekeying in 25448s peer-SITE1-tunnel-1: #1, reqid 1, INSTALLED, TUNNEL, ESP:NULL/HMAC_SHA1_96 installed 1581s ago, rekeying in 1840s, expires in 2379s in c7130959, 168 bytes, 2 packets, 1581s ago out cde9784b, 168 bytes, 2 packets, 1581s ago local 10.0.0.1/32 remote 10.0.0.2/32 * :osdx:op:`vpn ipsec show state`: checks the kernel cryptostate. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show state src 10.0.0.1 dst 10.0.0.2 proto esp spi 0xcde9784b reqid 1 mode tunnel replay-window 0 flag af-unspec auth-trunc hmac(sha1) 0x6e924c645c189d0176cb1dba5a445d5078749249 96 enc ecb(cipher_null) anti-replay context: seq 0x0, oseq 0x2, bitmap 0x00000000 src 10.0.0.2 dst 10.0.0.1 proto esp spi 0xc7130959 reqid 1 mode tunnel replay-window 32 flag af-unspec auth-trunc hmac(sha1) 0x4721395ffe9e83a8f77de8eed16bdea194b4b8a0 96 enc ecb(cipher_null) anti-replay context: seq 0x2, oseq 0x0, bitmap 0x00000003 * :osdx:op:`vpn ipsec show ike status`: checks the IKE process status. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show ike status IKE Process Running PID: 4140 NHRP protocol ------------- Checks whether the NHRP protocol information is correct. * :osdx:op:`protocols ip show nhrp`: checks the status of tunnel interfaces. *Example:* .. code-block:: none admin@osdx$ protocols ip show nhrp Iface Type Protocol NBMA Claimed NBMA Expires(s) Flags Identity tun1 local 10.1.0.1 172.1.0.1 172.1.0.1 - - tun1 dynamic 10.1.0.3 172.1.0.3 172.1.0.3 6784 UT 172.1.0.3 DMVPN protocol -------------- Checks whether the DMVPN protocol information is correct. * :osdx:op:`vpn ipsec show dmvpn`: checks the information on dmvpn connections. *Example:* .. code-block:: none admin@osdx$ vpn ipsec show dmvpn Src Dst Flags SAs Identity 172.1.0.1 172.1.0.3 n 1 172.1.0.3 BGP protocol ------------ Checks whether the BGP protocol information is correct. * :osdx:op:`protocols bgp show ip`: checks information on BGP-learned routes. *Example:* .. code-block:: none admin@osdx$ protocols bgp show ip BGP table version is 3, local router ID is 10.10.0.100, vrf id 0 Default local pref 100, local AS 10 local address - Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 1.1.1.0/24 10.10.0.200 0 0 20 ? Displayed 1 routes and 1 total paths OSPF protocol ------------- Checks whether the OSPF protocol information is correct. * :osdx:op:`protocols ospf show`: checks the general information on OSPF. *Example:* .. code-block:: none admin@osdx$ protocols ospf show OSPF Routing Process, Router ID: 10.215.200.50 Supports only single TOS (TOS0) routes This implementation conforms to RFC2328 RFC1583Compatibility flag is disabled OpaqueCapability flag is disabled Initial SPF scheduling delay 0 millisec(s) Minimum hold time between consecutive SPFs 50 millisec(s) Maximum hold time between consecutive SPFs 5000 millisec(s) Hold time multiplier is currently 1 SPF algorithm last executed 2m28s ago Last SPF duration 22 usecs SPF timer is inactive LSA minimum interval 5000 msecs LSA minimum arrival 1000 msecs Write Multiplier set to 20 Refresh timer 10 secs Maximum multiple paths(ECMP) supported 32 Administrative distance 110 Number of external LSA 1. Checksum Sum 0x00009616 Number of opaque AS LSA 0. Checksum Sum 0x00000000 Number of areas attached to this router: 1 Area ID: 0.0.0.1 Shortcutting mode: Default, S-bit consensus: no Number of interfaces in this area: Total: 1, Active: 1 Number of fully adjacent neighbors in this area: 1 Area has no authentication Number of full virtual adjacencies going through this area: 0 SPF algorithm executed 3 times Number of LSA 3 Number of router LSA 2. Checksum Sum 0x00013586 Number of network LSA 1. Checksum Sum 0x0000a9a6 Number of summary LSA 0. Checksum Sum 0x00000000 Number of ASBR summary LSA 0. Checksum Sum 0x00000000 Number of NSSA LSA 0. Checksum Sum 0x00000000 Number of opaque link LSA 0. Checksum Sum 0x00000000 Number of opaque area LSA 0. Checksum Sum 0x00000000 * :osdx:op:`protocols ospf show border-routers`: checks the information on OSPF border routers. *Example:* .. code-block:: none admin@osdx$ protocols ospf show border-routers ============ OSPF router routing table ============= R 10.215.200.100 [1] area: 0.0.0.1, ASBR via 10.215.200.100, eth0 * :osdx:op:`protocols ospf show database`: checks OSPF routing database information. *Example:* .. code-block:: none admin@osdx$ protocols ospf show database OSPF Router with ID (10.215.200.50) Router Link States (Area 0.0.0.1) Link ID ADV Router Age Seq# CkSum Link count 10.215.200.50 10.215.200.50 1056 0x80000004 0x4a60 1 10.215.200.100 10.215.200.100 1056 0x80000004 0xeb26 1 Net Link States (Area 0.0.0.1) Link ID ADV Router Age Seq# CkSum 10.215.200.100 10.215.200.100 1057 0x80000001 0xa9a6 AS External Link States Link ID ADV Router Age Seq# CkSum Route 1.1.1.0 10.215.200.100 1097 0x80000001 0x9616 E2 1.1.1.0/24 [0x0] * :osdx:op:`protocols ospf show interface *`: checks information on OSPF interfaces. *Example:* .. code-block:: none admin@osdx$ protocols ospf show interface eth0 eth0 is up ifindex 2, MTU 1500 bytes, BW 4294967295 Mbit Internet Address 10.215.200.50/24, Broadcast 10.215.200.255, Area 0.0.0.1 MTU mismatch detection: enabled Router ID 10.215.200.50, Network Type BROADCAST, Cost: 1 Transmit Delay is 1 sec, State Backup, Priority 1 Designated Router (ID) 10.215.200.100 Interface Address 10.215.200.100/24 Backup Designated Router (ID) 10.215.200.50, Interface Address 10.215.200.50 Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5 Hello due in 2.274s Neighbor Count is 1, Adjacent neighbor count is 1 * :osdx:op:`protocols ospf show neighbor *`: checks information on OSPF neighbors. *Example:* .. code-block:: none admin@osdx$ protocols ospf show neighbor eth0 Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL 10.215.200.100 1 Full/DR 19m14s 35.623s 10.215.200.100 eth0:10.215.200.50 0 0 0 * :osdx:op:`protocols ospf show route`: checks OSPF routing information. *Example:* .. code-block:: none admin@osdx$ protocols ospf show route ============ OSPF network routing table ============ N 10.215.200.0/24 [1] area: 0.0.0.1 directly attached to eth0 ============ OSPF router routing table ============= R 10.215.200.100 [1] area: 0.0.0.1, ASBR via 10.215.200.100, eth0 ============ OSPF external routing table =========== N E2 1.1.1.0/24 [1/20] tag: 0 via 10.215.200.100, eth0 Most of the :osdx:op:`protocols ospf show` commands can be executed with VRF support. The following are just a sample: * :osdx:op:`protocols vrf * ospf show`: checks the general information on OSPF through a specific VRF. * :osdx:op:`protocols vrf * ospf show border-routers`: checks the information on OSPF border routers through a specific VRF. * :osdx:op:`protocols vrf * ospf show database`: checks OSPF routing database information through a specific VRF. * :osdx:op:`protocols vrf * ospf show interface *`: checks information on OSPF interfaces through a specific VRF. * :osdx:op:`protocols vrf * ospf show neighbor *`: checks information on OSPF neighbors through a specific VRF. * :osdx:op:`protocols vrf * ospf show route`: checks OSPF routing information through a specific VRF. Transport Level =============== This section shows the commands used to obtain information regarding the transport layer. CONNTRACK info -------------- * :osdx:op:`system conntrack show`: checks the conntrack table. *Example:* .. code-block:: none admin@osdx$ system conntrack show udp 17 22 src=10.0.0.2 dst=10.0.0.1 sport=40128 dport=2055 packets=1 bytes=146 [UNREPLIED] src=10.0.0.1 dst=10.0.0.2 sport=2055 dport=40128 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055] icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=13 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=13 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1] tcp 6 15 TIME_WAIT src=10.0.0.2 dst=10.0.0.1 sport=43850 dport=8080 packets=6 bytes=338 src=10.0.0.1 dst=10.0.0.2 sport=8080 dport=43850 packets=5 bytes=286 [ASSURED] mark=0 use=3 appdetect[L4:8080] udp 17 22 src=127.0.0.1 dst=127.0.0.1 sport=48253 dport=2055 packets=1 bytes=146 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=2055 dport=48253 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055] icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=12 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=12 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1] conntrack v1.4.5 (conntrack-tools): 5 flow entries have been shown.