================== Network Monitoring ================== OSDx provides many tools to inspect network traffic and provide insightful statistics about how traffic is being processed in the system. .. note:: Although all of the examples included below use an IPV4 address, using an IPV6 address is also possible. If more information about IPV6 is needed, please refer to :doc:`chapter `. Ping ==== Ping is a basic Internet command that allows a user to test and verify whether a given destination IP address exists and can accept requests in computer network administration. Ping is also used to confirm that the computer the user tries to reach is operational. An example of a good request/response will be the following: .. code-block:: none PING 192.168.122.1 (192.168.122.1) 56(84) bytes of data. 64 bytes from 192.168.122.1: icmp_seq=1 ttl=64 time=0.581 ms 64 bytes from 192.168.122.1: icmp_seq=2 ttl=64 time=0.091 ms 64 bytes from 192.168.122.1: icmp_seq=3 ttl=64 time=0.102 ms 64 bytes from 192.168.122.1: icmp_seq=4 ttl=64 time=0.103 ms 64 bytes from 192.168.122.1: icmp_seq=5 ttl=64 time=0.127 ms 64 bytes from 192.168.122.1: icmp_seq=6 ttl=64 time=0.110 ms 64 bytes from 192.168.122.1: icmp_seq=7 ttl=64 time=0.129 ms --- 192.168.122.1 ping statistics --- 7 packets transmitted, 7 received, 0% packet loss, time 6127ms rtt min/avg/max/mdev = 0.091/0.177/0.581/0.165 ms where 192.168.122.1 is the destination IP, icmp_seq are the ICMP packets sequence send to destination, ttl is the time-to-live, and time is the duration it takes to reach 192.168.122.1. On the other hand, if the ping command is dropped, we will see something like: .. code-block:: none PING 192.168.122.4 (192.168.122.4) 56(84) bytes of data. From 192.168.122.3 icmp_seq=1 Destination Host Unreachable From 192.168.122.3 icmp_seq=2 Destination Host Unreachable From 192.168.122.3 icmp_seq=3 Destination Host Unreachable From 192.168.122.3 icmp_seq=4 Destination Host Unreachable From 192.168.122.3 icmp_seq=5 Destination Host Unreachable From 192.168.122.3 icmp_seq=6 Destination Host Unreachable --- 192.168.122.4 ping statistics --- 7 packets transmitted, 0 received, +6 errors, 100% packet loss, time 6130ms In this case, all packets are lost because the destination host is unreachable. If dns service is up and working, we can ping a FQDN instead of an IP address: .. code-block:: none PING www.teldat.com (20.107.224.3) 56(84) bytes of data. 64 bytes from 20.107.224.3 (20.107.224.3): icmp_seq=1 ttl=104 time=138 ms 64 bytes from 20.107.224.3 (20.107.224.3): icmp_seq=2 ttl=104 time=155 ms 64 bytes from 20.107.224.3 (20.107.224.3): icmp_seq=3 ttl=104 time=61.7 ms 64 bytes from 20.107.224.3 (20.107.224.3): icmp_seq=4 ttl=104 time=56.6 ms --- www.teldat.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms If our network topology looks like the following image (where source is **A**, destination is **B**, and the ping command is failing between **A** and **B**), instead of doing a ping to **B**, pinging an intermediate node (such as **C** or **D**) will be better to check if the ICMP packet is reaching these nodes (this may be useful too with the other tools described below). .. image:: ping_graph.png :width: 600 :alt: network topology Finnally, this ping feature can be combined with **VRFs** (virtual routing forwarding). In this case, the command will be ``ping vrf ``, where the ipv4/ipv6 destination address must be reachable from the VRF routing table. *Example*: .. code-block:: none admin@osdx$ show running | grep VRF0 set interfaces ethernet eth0p0 vrf VRF0 set service dhcp-server shared-network TEST local-vrf VRF0 set system vrf VRF0 .. code-block:: none admin@osdx$ protocols vrf VRF0 ip show route Codes: K - kernel route, C - connected, L - local, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, t - Table-Direct, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF VRF0: K>* 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), weight 1, 16:56:26 C>* 100.1.1.0/24 is directly connected, eth0p0, weight 1, 16:56:22 L>* 100.1.1.1/32 is directly connected, eth0p0, weight 1, 16:56:22 .. code-block:: none admin@osdx$ ping 100.1.1.10 vrf VRF0 ping: Warning: source address might be selected on device other than: VRF0 PING 100.1.1.10 (100.1.1.10) from 100.1.1.1 VRF0: 56(84) bytes of data. 64 bytes from 100.1.1.10: icmp_seq=1 ttl=64 time=0.517 ms 64 bytes from 100.1.1.10: icmp_seq=2 ttl=64 time=0.369 ms 64 bytes from 100.1.1.10: icmp_seq=3 ttl=64 time=0.380 ms 64 bytes from 100.1.1.10: icmp_seq=4 ttl=64 time=0.404 ms --- 100.1.1.10 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3043ms rtt min/avg/max/mdev = 0.369/0.417/0.517/0.058 ms .. note:: Despite this VRF example using the ping command, it can also work with the other tools defined in this section. ARPing ====== ARPing works as Ping does, but it only sends an ARP REQUEST to a neighbour host. An example of this connection is: .. code-block:: none ARPING 192.168.122.1 from 192.168.122.3 eth0 Unicast reply from 192.168.122.1 [55:51:3D:42:0E:FC] 0.664ms Unicast reply from 192.168.122.1 [55:51:3D:42:0E:FC] 0.632ms Unicast reply from 192.168.122.1 [55:51:3D:42:0E:FC] 0.634ms Unicast reply from 192.168.122.1 [55:51:3D:42:0E:FC] 0.624ms Unicast reply from 192.168.122.1 [55:51:3D:42:0E:FC] 0.622ms Sent 5 probes (1 broadcast(s)) Received 5 response(s) Traceroute ========== :osdx:op:`traceroute *` is a tool used to track the route packets taken from an IP network on their way to a given host. It uses the IP protocol's time to live (TTL) field and attempts to elicit an ICMP TIME_EXCEEDED response from each gateway to the host. This program attempts to trace the route an IP packet would follow to some internet host by launching probe packets with a small ttl (time to live) then listening for an ICMP "time exceeded" reply from a gateway. We start our probes with a ttl of one and increase by one until we get an ICMP "port unreachable" (or TCP reset), which means we reached the "host" or hit a max (30 hops by default). *Examples:* .. code-block:: none admin@osdx$ traceroute 12.0.0.1 traceroute to 12.0.0.1 (12.0.0.1), 30 hops max, 60 byte packets 1 12.0.0.1 1.084 ms 0.436 ms 0.260 ms .. code-block:: none admin@osdx$ traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 12.0.0.1 0.615 ms 0.477 ms 0.322 ms 2 192.168.214.50 0.491 ms 0.401 ms 0.523 ms 3 192.168.191.129 0.643 ms 0.573 ms 0.500 ms 4 192.168.191.1 0.802 ms 0.688 ms 0.730 ms 5 172.24.10.96 1.189 ms 1.113 ms 1.018 ms 6 195.53.109.241 1.186 ms 1.305 ms 1.146 ms 7 193.152.72.129 3.248 ms 3.582 ms 3.206 ms 8 81.46.14.161 7.641 ms 7.674 ms 7.605 ms 9 81.41.246.145 7.421 ms 6.405 ms 6.807 ms 10 81.46.0.17 5.863 ms 5.790 ms 5.717 ms 11 * * * 12 213.140.50.43 4.970 ms 72.14.219.20 5.540 ms 213.140.50.43 5.101 ms 13 192.178.110.85 6.676 ms 6.092 ms 108.170.249.11 13.534 ms 14 142.251.49.55 5.098 ms 142.251.49.53 12.670 ms 142.250.239.25 12.440 ms 15 8.8.8.8 11.998 ms 12.279 ms 12.183 ms Nslookup ======== :osdx:op:`nslookup *` is a feature to query Internet domain name servers. This is useful when an ERR_NAME_NOT_RESOLVED message is received, in order to verify if our dns server is correctly resolving the domain name or not. *Example:* .. code-block:: none admin@osdx$ nslookup www.teldat.com Server: 192.168.212.3 Address: 192.168.212.3#53 Name: www.teldat.com Address: 20.107.224.3 Traffic Dump Monitor ==================== This tool is an analyzer to monitor incoming or outgoing traffic based on different filters, such as an interface, vrf, packets or any `pcap-filter `_. This is a really powerful tool to determine what type of traffic is going through our device and test if our network is working correctly. By way of example, we are able to monitor all icmp or arp traffic coming from IP address 192.168.122.1 (as shown below): .. code-block:: none admin@OSDx$ traffic dump monitor interface eth0 filter "(src host 192.168.122.1) and (icmp or arp)" tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:56:38.942725 IP 192.168.122.1 > 192.168.122.3: ICMP echo request, id 33358, seq 1, length 64 12:56:39.958309 IP 192.168.122.1 > 192.168.122.3: ICMP echo request, id 33358, seq 2, length 64 12:56:40.982331 IP 192.168.122.1 > 192.168.122.3: ICMP echo request, id 33358, seq 3, length 64 12:56:42.006337 IP 192.168.122.1 > 192.168.122.3: ICMP echo request, id 33358, seq 4, length 64 12:57:56.117992 ARP, Request who-has 192.168.122.3 tell 192.168.122.1, length 28 In addition, we are able to save the captured traffic in a ``.pcap`` file using the ``save `` option as follows: .. code-block:: none admin@osdx$ traffic dump monitor filter icmp save capture tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes ^C9 packets captured 9 packets received by filter 0 packets dropped by kernel Output file was generated in '/support/capture.pcap' Traffic Bandwidth Monitor ========================= The :osdx:op:`traffic bandwidth monitor interface *` command shows a graph displaying the amount of traffic going through the selected interface. .. figure:: bandwidth.png :width: 400px *Bandwidth graph* The graph may be exited by pressing ``Ctrl+C``. Traffic Flow Monitor ==================== The :osdx:op:`traffic flow monitor interface *` command shows a graph displaying the currently active network flows. .. figure:: flow.png :width: 400px *Flow graph* The graph may be exited by pressing ``Ctrl+C``. .. warning:: This command requires high cpu usage. Use it with care in high traffic scenarios. Conntrack Monitor ================= Another useful command to check flows is :osdx:op:`system conntrack monitor`, which shows all state changes for every active network flow. *Example:* .. code-block:: none admin@osdx$ system conntrack monitor [NEW] icmp 1 30 src=10.215.168.1 dst=10.215.168.2 type=8 code=0 id=26362 [UNREPLIED] src=10.215.168.2 dst=10.215.168.1 type=0 code=0 id=26362 [UPDATE] icmp 1 30 src=10.215.168.1 dst=10.215.168.2 type=8 code=0 id=26362 src=10.215.168.2 dst=10.215.168.1 type=0 code=0 id=26362 [DESTROY] icmp 1 src=10.215.168.1 dst=10.215.168.2 type=8 code=0 id=9835 packets=59 bytes=4956 src=10.215.168.2 dst=10.215.168.1 type=0 code=0 id=9835 packets=59 bytes=4956 conntrack v1.4.5 (conntrack-tools): 3 flow events have been shown. The monitor session can be ended by pressing ``Ctrl+C``. Monitor Test Connection ======================= The ``monitor test connection`` command performs a connection test through the transport layer. This tool is useful to monitor the status of a connection for different protocols. These protocols are ``tcp``, ``udp`` and ``ssl``. We also need to specify if the device will work as ``server`` or ``client`` (for client mode, address server is mandatory) and the ``port`` the server will be listening to. For more information, please refer to `site `_. *Example* .. figure:: test-tcp-conn.png :width: 400px *Tcp Connection* The monitor session may be ended by pressing ``Ctrl+C``. Monitor Test Performance ======================== The ``monitor test performance`` command does a bandwidth test via tcp or udp, to check the throughput in a given interface. As with the ``monitor test connection`` command, it is necessary to specify which device will work as ``server``, which as ``client`` and the ``port`` to listen. A ``tcp`` connection will be used as default, but using ``udp`` is also possible (this option must be set at the client's side). Results are displayed on both the client and server. At least one line of output will be generated per measurement interval. Each line of output includes the time that has gone by since the test started, the amount of data transferred during the interval, and the average bitrate of said interval. Note that the values for each measurement interval are taken from the point of view of the endpoint process emitting that output (in other words, the output on the client shows the measurement interval data for the client). *Example* .. figure:: test-performance.gif :width: 1000px Test Performance The monitor session may be ended by pressing ``Ctrl+C``. We can combine this tool with the :osdx:op:`system processes monitor` command (explained in this :ref:`section `) in order to see how is the CPU consumption when an elevated traffic is incoming or outgoing. In the following example, we can check how the Idle CPU (amount of time the CPU is not busy) decreases a lot when the test start and increases again when the test stop. *Example* .. figure:: test-cpu-idle.gif :width: 900px Test Performance Click on `page `_ to learn more about this tool.