Inspecting Network Layers
In this chapter, we will detail some useful commands to rule out problems per network layer (from the most physical to the most abstract).
Step 1: Physical Level
First of all, we must check all defined interfaces are working at the physical level. All interfaces share a group of commands to gain information about these interfaces:
interfaces show: checks global information.interfaces show detailed: checks global information in greater detail.interfaces show counters: checks all interface counters.interfaces <interface_type> show: checks the global information pertaining to a given interface type.
Examples:
admin@osdx$ interfaces show
-----------------------------------------------------------------
Name IP Address Admin Oper Vrf Description
-----------------------------------------------------------------
br0 192.168.100.10/24 up up
fe80::9007:dbff:fe85:fa8/64
eth0 fe80::dcad:beff:feef:6c10/64 up up
eth1 down down
admin@osdx$ interfaces show detailed
----------------------------------------------------------------------------------------------------------------
Name Idx IP Address Admin Oper Link MTU Vrf Upper Lower Type Phys addr
----------------------------------------------------------------------------------------------------------------
br0 4 192.168.100.10/24 up up up 1500 bridge de:ad:be:ef:6c:10
fe80::9007:dbff:fe85:fa8/64
eth0 2 fe80::dcad:beff:feef:6c10/64 up up up 1500 br0 ethernet de:ad:be:ef:6c:10
eth1 3 down down down 1500 ethernet de:ad:be:ef:6c:11
admin@osdx$ interfaces show counters
----------------------------------------------------------------------------
Name Oper Rx Packets Rx Bytes Rx Errors Tx Packets Tx Bytes Tx Errors
----------------------------------------------------------------------------
br0 up 3 140 0 16 1460 0
eth0 up 13 854 0 20 1820 0
eth1 down 0 0 0 0 0 0
admin@osdx$ interfaces ethernet show
-----------------------------------------------------------------
Name IP Address Admin Oper Vrf Description
-----------------------------------------------------------------
eth0 fe80::dcad:beff:feef:6c10/64 up up
eth1 down down
These tables show the name of the interface, the IPv4 and IPv6 address associated to that interface (if any), if the interface is administratively up or down (Admin), if the interface is physically up or down (Link), if the interface is functionally up and can be used (Oper), the maximum transfer unit (MTU) of the interface, if it has any vrf associated to that physical interface, if the interface has a master (Upper), the link type (lower), the MAC address of the interface (Phys addr) and the number of Errors and Packets transmitted and received.
Apart from these commands, others are associated to just an interface type.
Bridge
For bridges interfaces, there are four relevant commands:
interfaces bridge show ports: Displays information about the port devices of the bridge.interfaces bridge show vlan: Illustrates known VLAN IDs for a link.interfaces bridge show fdb: Shows known Ethernet addresses on a link.interfaces bridge show stp: Displays Spanning Tree Protocol information
Examples:
admin@osdx$ interfaces bridge show ports
----------------------
bridge port state
----------------------
br0 eth2 disabled
admin@osdx$ interfaces bridge show vlan
----------------------------------------
bridge port vid flags
----------------------------------------
all br0 1 PVID, Egress Untagged
br0 eth2 102 PVID, Egress Untagged
admin@osdx$ interfaces bridge show fdb
-------------------------------------------------
bridge port mac state flags
-------------------------------------------------
br0 eth2 00:a0:26:0e:00:62 permanent
br0 eth2 00:a0:26:0e:00:62 permanent
br0 br0 33:33:00:00:00:01 permanent self
br0 br0 33:33:00:00:00:02 permanent self
br0 br0 01:00:5e:00:00:6a permanent self
br0 br0 33:33:00:00:00:6a permanent self
br0 br0 01:00:5e:00:00:01 permanent self
br0 br0 33:33:ff:5f:81:09 permanent self
br0 br0 33:33:ff:00:00:00 permanent self
br0 br0 00:a0:26:0e:00:62 permanent
admin@osdx$ interfaces bridge show stp
-----------------------------------------------------------------------------------------------
bridge id version ageing hello topology changes last topology change
-----------------------------------------------------------------------------------------------
br0 8.000.00:A0:26:0E:3A:A4 stp 300s 2s 0 842s ago
Cellular
There are several commands to gain information and check the status of cellular interfaces:
interfaces cellular show capabilities: Show capabilities of cellular interfaces
Example:
admin@osdx$ interfaces cellular show capabilities
cell0
Module Manufacturer: Quectel
Module Model: EC25E
Module Firmware: EC25EFAR06A01M4G_TEL
Hardware Revision: 10000
IMEI: 866308064953342
Radio Interfaces: gsm, umts, lte
Data Service Capability: non-simultaneous-cs-ps
Maximum TX/RX rate supported: 50000/100000 Kbps
Bands: gsm-dcs-1800, gsm-900-extended, gsm-900-primary, wcdma-2100, wcdma-850-us, wcdma-900
LTE bands: 1, 3, 5, 7, 8, 20, 38, 40, 41
interfaces cellular <value> show device-info: Displays information about a specific cellular interface
Example:
admin@osdx$ interfaces cellular cell0 show device-info
Module Manufacturer = Quectel
Module Model = EC25E
Module Firmware = EC25EFAR06A01M4G_TEL
Hardware Revision = 10000
IMEI = 866308064953342
Radio Interfaces = gsm, umts, lte
Data Service Capability = non-simultaneous-cs-ps
Maximum TX/RX rate supported = 50000/100000 Kbps
Temperature = 44
interfaces cellular <value> show supported-bands: Displays the supported bands
Example:
admin@osdx$ interfaces cellular cell0 show supported-bands
Bands = gsm-dcs-1800, gsm-900-extended, gsm-900-primary, wcdma-2100, wcdma-850-us, wcdma-900
LTE bands = 1, 3, 5, 7, 8, 20, 38, 40, 41
interfaces cellular <value> show preferred-bands: Displays the preferred bands
Example:
admin@osdx$ interfaces cellular cell0 show preferred-bands
Bands preference = gsm-dcs-1800, gsm-900-extended, gsm-900-primary, wcdma-2100, wcdma-850-us, wcdma-900
LTE bands preference = 1, 3, 5, 7, 8, 20, 38, 40, 41
interfaces cellular <value> show cell-info: Displays information about the serving cell and neighboring cells
Example:
admin@osdx$ interfaces cellular cell0 show cell-info
GSM/EDGE info:
Cell ID = 13087
PLMN ID coded = 21401
Location Area Code = 17166
Absolute RF channel number = 102
Base Station Identity Code (BSIC) = 6
Timing advance (bit periods) = 0
Serving cell Rx Level range = 23
GSM/EDGE Cell #0
Cell ID = unavailable
PLMN ID coded = unavailable
Location Area Code = unavailable
Absolute RF channel number = 117
Base Station Identity Code (BSIC) = 59
Serving cell Rx Level range = 5
interfaces cellular <value> show signal-quality: Displays information about the quality of the signal received by the module
Example:
admin@osdx$ interfaces cellular cell0 show signal-quality
GSM/GPRS:
RSSI (dBm) = -92
Rx Chain #0
Radio tuned = yes
Rx Level (dBm) = -87.500000
interfaces cellular <value> show network-status: Displays information about the state of the radio link
Example:
admin@osdx$ interfaces cellular cell0 show network-status
SIM status = LOCKED (SIM not inserted)
Registration state = not-registered-searching
Public Land Mobile Network code = HOME[21401] ACT[17354]
Public Land Mobile Network name = Vodafone
Network technology currently in use = gsm
Current Service Domain registered = none (capable cs-ps)
Current Roaming status = on
3GPP Location Area Code = 17166
3GPP Cell ID = 13087
Radio Band = gsm-900-extended
Channel = 102
GSM E-GPRS supported = yes
GSM Dual Transfer Mode supported = yes
RX level (dBm) = -92
Coverage level = 2 (** )
Ethernet
When it comes to ethernet interfaces, the interfaces ethernet <value> show physical command shows physical device information
and the interfaces ethernet <value> show statistics command its statistics for the specified ethernet interface. In addition,
the interfaces ethernet <value> show plugin-module command displays information from plugin modules such as SFP+ or QSFP.
Examples:
admin@osdx$ interfaces ethernet eth0p3 show physical
Settings for eth0p3:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: external
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Link detected: yes
driver: dsa
version: 6.1.55
firmware-version: N/A
expansion-rom-version:
bus-info: platform
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
admin@RS420$ interfaces ethernet eth0p3 show statistics
NIC statistics:
tx_packets: 251
tx_bytes: 38706
rx_packets: 1317
rx_bytes: 133319
in_good_octets: 138651
in_bad_octets: 0
in_unicast: 1070
in_broadcasts: 0
in_multicasts: 248
in_pause: 0
in_undersize: 0
in_fragments: 0
in_oversize: 0
in_jabber: 0
in_rx_error: 0
in_fcs_error: 0
out_octets: 41172
out_unicast: 245
out_broadcasts: 1
out_multicasts: 18
out_pause: 0
excessive: 0
collisions: 0
deferred: 0
single: 0
multiple: 0
out_fcs_error: 0
late: 0
hist_64bytes: 805
hist_65_127bytes: 451
hist_128_255bytes: 301
hist_256_511bytes: 10
hist_512_1023bytes: 5
hist_1024_max_bytes: 10
sw_in_discards: 0
sw_in_filtered: 0
sw_out_filtered: 1318
atu_member_violation: 0
atu_miss_violation: 0
atu_full_violation: 0
vtu_member_violation: 0
vtu_miss_violation: 0
admin@osdx$ interfaces ethernet eth2 show plugin-module
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Transceiver type : Ethernet: 1000BASE-SX
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x02 (8/4/2G Rx Rate_Select only)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 300m
Length (62.5um) : 300m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : EDGE
Vendor OUI : 00:1b:21
Vendor PN : 10G-SFP-300I-IN
Vendor rev : A
Option values : 0x00 0x3a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
Option : RATE_SELECT implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : E021903020001
Date code : 190302
Optical diagnostics support : Yes
Laser bias current : 6.734 mA
Laser output power : 0.5763 mW / -2.39 dBm
Receiver signal average optical power : 0.0785 mW / -11.05 dBm
Module temperature : 36.12 degrees C / 97.01 degrees F
Module voltage : 3.3253 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : Off
Laser rx power high warning : Off
Laser rx power low warning : Off
Laser bias current high alarm threshold : 100.000 mA
Laser bias current low alarm threshold : 0.000 mA
Laser bias current high warning threshold : 90.000 mA
Laser bias current low warning threshold : 0.100 mA
Laser output power high alarm threshold : 1.0000 mW / 0.00 dBm
Laser output power low alarm threshold : 0.2511 mW / -6.00 dBm
Laser output power high warning threshold : 0.7943 mW / -1.00 dBm
Laser output power low warning threshold : 0.3162 mW / -5.00 dBm
Module temperature high alarm threshold : 90.00 degrees C / 194.00 degrees F
Module temperature low alarm threshold : -45.00 degrees C / -49.00 degrees F
Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
Module temperature low warning threshold : -40.00 degrees C / -40.00 degrees F
Module voltage high alarm threshold : 3.8000 V
Module voltage low alarm threshold : 2.7000 V
Module voltage high warning threshold : 3.7000 V
Module voltage low warning threshold : 2.8000 V
Laser rx power high alarm threshold : 1.0000 mW / 0.00 dBm
Laser rx power low alarm threshold : 0.0500 mW / -13.01 dBm
Laser rx power high warning threshold : 0.7943 mW / -1.00 dBm
Laser rx power low warning threshold : 0.0630 mW / -12.01 dBm
Wlan
Finally, specific commands for wlan interfaces are as follows:
interfaces wlan show status: Shows the status of wlan interfaces
Example
admin@osdx$ interfaces wlan show status
Interface: wlan0
Mode: station
Phy: wifi0
Status: scanning
Address: 50:5a:65:42:e3:7c
BSSID:
SSID:
Interface: wlan1
Status: down
Interface: wlan2
Status: down
Interface: wlan3
Status: down
Interface: wlan4
Status: down
Interface: wlan5
Status: down
interfaces wlan <value> show stations <value>: Shows associated stations
Example
admin@osdx$ interfaces wlan wlan1 show stations
dev_name:wlan1
Number of STA = 0
interfaces wlan <value> show stats: Shows wlan stats
Example
admin@osdx$ interfaces wlan wlan0 show stats
driver_name = "wlan"
driver_version = PCIE9098--17.68.1.p136.38-MXM5X17366.p23-GPL-(FP68)
interface_name="wlan0"
firmware_major_version=17.68.1
bss_mode ="Managed"
media_state="Disconnected"
mac_address="50:5a:65:42:e3:7c"
multicast_count="3"
essid=""
bssid="00:00:00:00:00:00"
channel="0"
region_code = "00"
multicast_address[0]="33:33:00:00:00:01"
multicast_address[1]="33:33:00:00:00:02"
multicast_address[2]="01:00:5e:00:00:01"
num_tx_bytes = 0
num_rx_bytes = 0
num_tx_pkts = 0
num_rx_pkts = 0
num_tx_pkts_dropped = 0
num_rx_pkts_dropped = 0
num_tx_pkts_err = 0
num_rx_pkts_err = 0
carrier off
tx queue 0: stopped
tx queue 1: stopped
tx queue 2: stopped
tx queue 3: stopped
Step 2: Link Level
Next, we will check the information at the link level. Different commands can be used for this task:
system ip neighbors show: checks information about neighbors.
Example:
admin@osdx$ system ip neighbors show
192.168.100.20 dev br0 lladdr de:ad:be:ef:6c:20 REACHABLE
system ip neighbors show interface <ifc>: checks information about neighbors per interface.
Example:
admin@osdx$ system ip neighbors show interface br0
192.168.100.20 lladdr de:ad:be:ef:6c:20 REACHABLE
Step 3: Network Level
Now we are going to check if the routing information is OK. The following commands are useful:
protocols ip show route: checks the main VRF routing table.
Example:
admin@osdx$ protocols ip show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:11
C>* 192.168.100.0/24 is directly connected, br0, 00:01:11
protocols ip show route <ipv4|ipv4net>: checks routing table entries per type.
Example:
admin@osdx$ protocols ip show route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, br0, weight 1, 00:01:57
protocols ip show route summary: checks the summary of routing table entries.
Example:
admin@osdx$ protocols ip show route summary
Route Source Routes FIB (vrf default)
connected 1 1
static 1 1
------
Totals 2 2
protocols vrf <value> ip show route: checks a given VRF routing table
Example:
admin@osdx$ protocols vrf BLUE ip show route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 00:06:31
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:06:31
protocols vrf <value> ip show route <ipv4|ipv4net>: checks selected VRF routing table entries by type.
Example:
admin@osdx$ protocols vrf BLUE ip show route connected
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37
protocols vrf <value> ip show route summary: checks the counter of selected VRF routing table entries.
Example:
admin@osdx$ protocols vrf BLUE ip show route connected
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF BLUE:
C>* 192.168.200.0/24 is directly connected, eth1.102, 00:07:37
admin@osdx$ protocols vrf BLUE ip show route summary
Route Source Routes FIB (vrf BLUE)
kernel 1 1
connected 1 1
------
Totals 2 2
For each protocol, the commands to check the connection and routing status are shown below in the corresponding section.
Step 4: Transport Level
This section shows the commands used to obtain information regarding the transport layer using the conntrack tool. Conntrack utility provides a featured userspace interface to the Netfilter connection tracking system. It can be used to dump a list of all (or a filtered selection of) currently tracked connections.
CONNTRACK info
system conntrack show: checks the conntrack table.
Example:
admin@osdx$ system conntrack show
udp 17 22 src=10.0.0.2 dst=10.0.0.1 sport=40128 dport=2055 packets=1 bytes=146 [UNREPLIED] src=10.0.0.1 dst=10.0.0.2 sport=2055 dport=40128 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055]
icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=13 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=13 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1]
tcp 6 15 TIME_WAIT src=10.0.0.2 dst=10.0.0.1 sport=43850 dport=8080 packets=6 bytes=338 src=10.0.0.1 dst=10.0.0.2 sport=8080 dport=43850 packets=5 bytes=286 [ASSURED] mark=0 use=3 appdetect[L4:8080]
udp 17 22 src=127.0.0.1 dst=127.0.0.1 sport=48253 dport=2055 packets=1 bytes=146 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=2055 dport=48253 packets=0 bytes=0 mark=0 use=1 appdetect[L4:2055]
icmp 1 22 src=10.0.0.2 dst=10.0.0.1 type=8 code=0 id=12 packets=1 bytes=84 src=10.0.0.1 dst=10.0.0.2 type=0 code=0 id=12 packets=1 bytes=84 mark=0 use=1 appdetect[L3:1]
conntrack v1.4.5 (conntrack-tools): 5 flow entries have been shown.
Conntrack information can be filtered by protocol, source, destination, IP family, and NAT.
system conntrack show protocol <value>: only shows entries with a specific protocol
Example:
system conntrack show protocol udp
udp 17 src=11.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=11.0.0.2 sport=1234 dport=2345 vrf=wan3 packets=5 bytes=240 [OFFLOAD, packets=3 bytes=144 packets=4 bytes=192] mark=0 use=2 appdetect[L4:1234]
udp 17 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 vrf=tenant2 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 vrf=wan2 packets=5 bytes=240 [OFFLOAD, packets=3 bytes=144 packets=4 bytes=192] mark=0 use=2 appdetect[L4:1234]
udp 17 28 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 vrf=tenant1 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 vrf=wan1 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 26 src=10.0.0.2 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=10.0.0.2 sport=1234 dport=2345 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 10 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 vrf=wan2 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 vrf=wan2 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 8 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 vrf=wan1 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 vrf=wan1 packets=5 bytes=240 mark=0 use=1 appdetect[L4:1234]
udp 17 5 src=20.0.0.1 dst=20.0.0.2 sport=2345 dport=1234 packets=5 bytes=240 src=20.0.0.2 dst=20.0.0.1 sport=1234 dport=2345 packets=5 bytes=240 mark=0 use=2 appdetect[L4:1234]
conntrack v1.4.5 (conntrack-tools): 7 flow entries have been shown.
system conntrack show family <value>: only shows IPv4/IPv6 entries.system conntrack show source <ipv4|ipv6|ipv4net|ipv6net>: only shows entries with source IPsystem conntrack show destination <ipv4|ipv6|ipv4net|ipv6net>: only shows entries with destination IPsystem conntrack show nat: only shows NAT entriessystem conntrack show source-nat: only shows source NAT entriessystem conntrack show destination-nat: only shows destination NAT entries
More information about conntrack and its multiple options can be found here.