Troubleshooting
In this chapter, we will show some useful commands to help the admin deal with problems.
Physical Level
First of all, we must check all defined interfaces are working at the physical level. To do this, there are several useful commands:
interfaces show: checks global information.
Example:
admin@osdx$ interfaces show
-----------------------------------------------------------------
Name IP Address Admin Oper Vrf Description
-----------------------------------------------------------------
br0 192.168.100.10/24 up up
fe80::9007:dbff:fe85:fa8/64
eth0 fe80::dcad:beff:feef:6c10/64 up up
eth1 down down
interfaces show detailed: checks global information in greater detail.
Example:
admin@osdx$ interfaces show detailed
----------------------------------------------------------------------------------------------------------------
Name Idx IP Address Admin Oper Link MTU Vrf Upper Lower Type Phys addr
----------------------------------------------------------------------------------------------------------------
br0 4 192.168.100.10/24 up up up 1500 bridge de:ad:be:ef:6c:10
fe80::9007:dbff:fe85:fa8/64
eth0 2 fe80::dcad:beff:feef:6c10/64 up up up 1500 br0 ethernet de:ad:be:ef:6c:10
eth1 3 down down down 1500 ethernet de:ad:be:ef:6c:11
interfaces show counters: checks all interface counters.
Example:
admin@osdx$ interfaces show counters
----------------------------------------------------------------------------
Name Oper Rx Packets Rx Bytes Rx Errors Tx Packets Tx Bytes Tx Errors
----------------------------------------------------------------------------
br0 up 3 140 0 16 1460 0
eth0 up 13 854 0 20 1820 0
eth1 down 0 0 0 0 0 0
interfaces <interface_type> show: checks the global information pertaining to a given interface type.
Example:
admin@osdx$ interfaces ethernet show
-----------------------------------------------------------------
Name IP Address Admin Oper Vrf Description
-----------------------------------------------------------------
eth0 fe80::dcad:beff:feef:6c10/64 up up
eth1 down down
Link Level
Next, we will check the information at the link level. Different commands can be used for this task:
system ip neighbors show: checks information about neighbors.
Example:
admin@osdx$ system ip neighbors show
192.168.100.20 dev br0 lladdr de:ad:be:ef:6c:20 REACHABLE
system ip neighbors show interface <ifc>: checks information about neighbors per interface.
Example:
admin@osdx$ system ip neighbors show interface br0
192.168.100.20 lladdr de:ad:be:ef:6c:20 REACHABLE
Network Level
Now we are going to check if the routing information is OK. The following commands are useful:
protocols ip show route: checks the main VRF routing table.
Example:
admin@osdx$ protocols ip show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:11
C>* 192.168.100.0/24 is directly connected, br0, 00:01:11
protocols ip show route <ipv4|ipv4net>: checks routing table entries per type.
Example:
admin@osdx$ protocols ip show route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:57
protocols ip show route summary: checks the summary of routing table entries.
Example:
admin@osdx$ protocols ip show route summary
Route Source Routes FIB (vrf default)
connected 1 1
static 1 1
------
Totals 2 2
protocols vrf <value> ip show route: checks a given VRF routing table
Example:
admin@osdx$ protocols vrf BLUE ip show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:06:31
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:06:31
protocols vrf <value> ip show route <ipv4|ipv4net>: checks selected VRF routing table entries by type.
Example:
admin@osdx$ protocols vrf BLUE ip show route connected
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37
protocols vrf <value> ip show route summary: checks the counter of selected VRF routing table entries.
Example:
admin@osdx$ protocols vrf BLUE ip show route connected
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37
admin@osdx$ protocols vrf BLUE ip show route summary
Route Source Routes FIB (vrf BLUE)
kernel 1 1
connected 1 1
------
Totals 2 2
IPsec protocol
Checks whether the IPsec protocol information is correct.
vpn ipsec show policy: checks the information available on kernel crypto policies.
Example:
admin@osdx$ vpn ipsec show policy
src 10.0.0.1/32 dst 10.0.0.2/32
dir out priority 367231
tmpl src 10.0.0.1 dst 10.0.0.2
proto esp spi 0xcde9784b reqid 1 mode tunnel
src 10.0.0.2/32 dst 10.0.0.1/32
dir fwd priority 367231
tmpl src 10.0.0.2 dst 10.0.0.1
proto esp reqid 1 mode tunnel
src 10.0.0.2/32 dst 10.0.0.1/32
dir in priority 367231
tmpl src 10.0.0.2 dst 10.0.0.1
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
vpn ipsec show sa: checks information related to IPsec SA.
Example:
admin@osdx$ vpn ipsec show sa
vpn-peer-SITE1: #1, ESTABLISHED, IKEv2, 0fd20672a782d852_i* 0aab0776adbd3fc1_r
local '10.0.0.1' @ 10.0.0.1[500]
remote '10.0.0.2' @ 10.0.0.2[500]
NULL/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_3072
established 1479s ago, rekeying in 25550s
peer-SITE1-tunnel-1: #1, reqid 1, INSTALLED, TUNNEL, ESP:NULL/HMAC_SHA1_96
installed 1479s ago, rekeying in 1942s, expires in 2481s
in c7130959, 168 bytes, 2 packets, 1479s ago
out cde9784b, 168 bytes, 2 packets, 1479s ago
local 10.0.0.1/32
remote 10.0.0.2/32
vpn ipsec show sa local <value>: checks information related to IPsec SA in a selected local peer.
Example:
admin@osdx$ vpn ipsec show sa local 10.0.0.1
vpn-peer-SITE1: #1, ESTABLISHED, IKEv2, 0fd20672a782d852_i* 0aab0776adbd3fc1_r
local '10.0.0.1' @ 10.0.0.1[500]
remote '10.0.0.2' @ 10.0.0.2[500]
NULL/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_3072
established 1544s ago, rekeying in 25485s
peer-SITE1-tunnel-1: #1, reqid 1, INSTALLED, TUNNEL, ESP:NULL/HMAC_SHA1_96
installed 1544s ago, rekeying in 1877s, expires in 2416s
in c7130959, 168 bytes, 2 packets, 1544s ago
out cde9784b, 168 bytes, 2 packets, 1544s ago
local 10.0.0.1/32
remote 10.0.0.2/32
vpn ipsec show sa remote <value>: checks information related to IPsec SA in a selected peer.
Example:
admin@osdx$ vpn ipsec show sa remote 10.0.0.2
vpn-peer-SITE1: #1, ESTABLISHED, IKEv2, 0fd20672a782d852_i* 0aab0776adbd3fc1_r
local '10.0.0.1' @ 10.0.0.1[500]
remote '10.0.0.2' @ 10.0.0.2[500]
NULL/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_3072
established 1581s ago, rekeying in 25448s
peer-SITE1-tunnel-1: #1, reqid 1, INSTALLED, TUNNEL, ESP:NULL/HMAC_SHA1_96
installed 1581s ago, rekeying in 1840s, expires in 2379s
in c7130959, 168 bytes, 2 packets, 1581s ago
out cde9784b, 168 bytes, 2 packets, 1581s ago
local 10.0.0.1/32
remote 10.0.0.2/32
vpn ipsec show state: checks the kernel cryptostate.
Example:
admin@osdx$ vpn ipsec show state
src 10.0.0.1 dst 10.0.0.2
proto esp spi 0xcde9784b reqid 1 mode tunnel
replay-window 0 flag af-unspec
auth-trunc hmac(sha1) 0x6e924c645c189d0176cb1dba5a445d5078749249 96
enc ecb(cipher_null)
anti-replay context: seq 0x0, oseq 0x2, bitmap 0x00000000
src 10.0.0.2 dst 10.0.0.1
proto esp spi 0xc7130959 reqid 1 mode tunnel
replay-window 32 flag af-unspec
auth-trunc hmac(sha1) 0x4721395ffe9e83a8f77de8eed16bdea194b4b8a0 96
enc ecb(cipher_null)
anti-replay context: seq 0x2, oseq 0x0, bitmap 0x00000003
vpn ipsec show ike status: checks the IKE process status.
Example:
admin@osdx$ vpn ipsec show ike status
IKE Process Running
PID: 4140
NHRP protocol
Checks whether the NHRP protocol information is correct.
protocols ip show nhrp: checks the status of tunnel interfaces.
Example:
admin@osdx$ protocols ip show nhrp
Iface Type Protocol NBMA Claimed NBMA Expires(s) Flags Identity
tun1 local 10.1.0.1 172.1.0.1 172.1.0.1 - -
tun1 dynamic 10.1.0.3 172.1.0.3 172.1.0.3 6784 UT 172.1.0.3
DMVPN protocol
Checks whether the DMVPN protocol information is correct.
vpn ipsec show dmvpn: checks the information on dmvpn connections.
Example:
admin@osdx$ vpn ipsec show dmvpn
Src Dst Flags SAs Identity
172.1.0.1 172.1.0.3 n 1 172.1.0.3
BGP protocol
Checks whether the BGP protocol information is correct.
protocols bgp show ip: checks information on BGP-learned routes.
Example:
admin@osdx$ protocols bgp show ip
BGP table version is 3, local router ID is 10.10.0.100, vrf id 0
Default local pref 100, local AS 10
local address -
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 1.1.1.0/24 10.10.0.200 0 0 20 ?
Displayed 1 routes and 1 total paths
OSPF protocol
Checks whether the OSPF protocol information is correct.
protocols ospf show: checks the general information on OSPF.
Example:
admin@osdx$ protocols ospf show
OSPF Routing Process, Router ID: 10.215.200.50
Supports only single TOS (TOS0) routes
This implementation conforms to RFC2328
RFC1583Compatibility flag is disabled
OpaqueCapability flag is disabled
Initial SPF scheduling delay 0 millisec(s)
Minimum hold time between consecutive SPFs 50 millisec(s)
Maximum hold time between consecutive SPFs 5000 millisec(s)
Hold time multiplier is currently 1
SPF algorithm last executed 2m28s ago
Last SPF duration 22 usecs
SPF timer is inactive
LSA minimum interval 5000 msecs
LSA minimum arrival 1000 msecs
Write Multiplier set to 20
Refresh timer 10 secs
Maximum multiple paths(ECMP) supported 32
Administrative distance 110
Number of external LSA 1. Checksum Sum 0x00009616
Number of opaque AS LSA 0. Checksum Sum 0x00000000
Number of areas attached to this router: 1
Area ID: 0.0.0.1
Shortcutting mode: Default, S-bit consensus: no
Number of interfaces in this area: Total: 1, Active: 1
Number of fully adjacent neighbors in this area: 1
Area has no authentication
Number of full virtual adjacencies going through this area: 0
SPF algorithm executed 3 times
Number of LSA 3
Number of router LSA 2. Checksum Sum 0x00013586
Number of network LSA 1. Checksum Sum 0x0000a9a6
Number of summary LSA 0. Checksum Sum 0x00000000
Number of ASBR summary LSA 0. Checksum Sum 0x00000000
Number of NSSA LSA 0. Checksum Sum 0x00000000
Number of opaque link LSA 0. Checksum Sum 0x00000000
Number of opaque area LSA 0. Checksum Sum 0x00000000
protocols ospf show border-routers: checks the information on OSPF border routers.
Example:
admin@osdx$ protocols ospf show border-routers
============ OSPF router routing table =============
R 10.215.200.100 [1] area: 0.0.0.1, ASBR
via 10.215.200.100, eth0
protocols ospf show database: checks OSPF routing database information.
Example:
admin@osdx$ protocols ospf show database
OSPF Router with ID (10.215.200.50)
Router Link States (Area 0.0.0.1)
Link ID ADV Router Age Seq# CkSum Link count
10.215.200.50 10.215.200.50 1056 0x80000004 0x4a60 1
10.215.200.100 10.215.200.100 1056 0x80000004 0xeb26 1
Net Link States (Area 0.0.0.1)
Link ID ADV Router Age Seq# CkSum
10.215.200.100 10.215.200.100 1057 0x80000001 0xa9a6
AS External Link States
Link ID ADV Router Age Seq# CkSum Route
1.1.1.0 10.215.200.100 1097 0x80000001 0x9616 E2 1.1.1.0/24 [0x0]
protocols ospf show interface <ifc>: checks information on OSPF interfaces.
Example:
admin@osdx$ protocols ospf show interface eth0
eth0 is up
ifindex 2, MTU 1500 bytes, BW 4294967295 Mbit <UP,BROADCAST,RUNNING,MULTICAST>
Internet Address 10.215.200.50/24, Broadcast 10.215.200.255, Area 0.0.0.1
MTU mismatch detection: enabled
Router ID 10.215.200.50, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State Backup, Priority 1
Designated Router (ID) 10.215.200.100 Interface Address 10.215.200.100/24
Backup Designated Router (ID) 10.215.200.50, Interface Address 10.215.200.50
Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters
Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5
Hello due in 2.274s
Neighbor Count is 1, Adjacent neighbor count is 1
protocols ospf show neighbor <ifc|ipv4>: checks information on OSPF neighbors.
Example:
admin@osdx$ protocols ospf show neighbor eth0
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
10.215.200.100 1 Full/DR 19m14s 35.623s 10.215.200.100 eth0:10.215.200.50 0 0 0
protocols ospf show route: checks OSPF routing information.
Example:
admin@osdx$ protocols ospf show route
============ OSPF network routing table ============
N 10.215.200.0/24 [1] area: 0.0.0.1
directly attached to eth0
============ OSPF router routing table =============
R 10.215.200.100 [1] area: 0.0.0.1, ASBR
via 10.215.200.100, eth0
============ OSPF external routing table ===========
N E2 1.1.1.0/24 [1/20] tag: 0
via 10.215.200.100, eth0
Most of the protocols ospf show commands can be executed with VRF support. The following are just a sample:
protocols vrf <value> ospf show: checks the general information on OSPF through a specific VRF.protocols vrf <value> ospf show border-routers: checks the information on OSPF border routers through a specific VRF.protocols vrf <value> ospf show database: checks OSPF routing database information through a specific VRF.protocols vrf <value> ospf show interface <ifc>: checks information on OSPF interfaces through a specific VRF.protocols vrf <value> ospf show neighbor <ifc|ipv4>: checks information on OSPF neighbors through a specific VRF.protocols vrf <value> ospf show route: checks OSPF routing information through a specific VRF.
Transport Level
This section shows the commands used to obtain information regarding the transport layer.
CONNTRACK info
system conntrack show: checks the conntrack table.
Example:
admin@osdx$ system conntrack show
udp 17 22 src=10.0.0.2 dst=10.0.0.1 sport=40128 dport=2055 packets=1 bytes=146 [UNREPLIED] src=10.0.0.1 dst=10.0.0.2 sport=2055 dport=40128 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055]
icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=13 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=13 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1]
tcp 6 15 TIME_WAIT src=10.0.0.2 dst=10.0.0.1 sport=43850 dport=8080 packets=6 bytes=338 src=10.0.0.1 dst=10.0.0.2 sport=8080 dport=43850 packets=5 bytes=286 [ASSURED] mark=0 use=3 appdetect[L4:8080]
udp 17 22 src=127.0.0.1 dst=127.0.0.1 sport=48253 dport=2055 packets=1 bytes=146 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=2055 dport=48253 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055]
icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=12 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=12 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1]
conntrack v1.4.5 (conntrack-tools): 5 flow entries have been shown.