.. _logging: ======= Logging ======= Logging is the act of keeping an append-only record of events which can later be read for troubleshooting purposes or forensic analysis. OSDx provides two ways to view logs: :doc:`System Journal ` and :doc:`/articles/system/syslog/index`, but it is important to note that the **syslog tool will only work with information that also appears under the journal**. Therefore, if (for instance) we want to send logs to a remote syslog server or a local file, we must first include that information in the journal. Otherwise, no logs will appear. Events ====== Events are occurrences located at a single point in time and contain a plain-text message with information about its cause and context. We can split them in three categories: * Events associated with running services * Events displayed while interacting with the device via CLI * Events related to the OSDx engine and subsystems The first category relates to logs that show us information on what a **service** is doing and whether it encounters any problem. Examples of OSDx services could be ssh, snmp, dns, dhcp, firewall, conntrack, syslog and so on. The logs associated to **CLI** interactions focus on activities within the device. This includes everything: configuring the device, checking the status of an interface, pinging a device, login or logout... Finally, the **engine and subsystems** logs are those illustrating what an OSDx device needs to accomplish a task. A task could be to retrieve the information of a configuration node, obtain data from monitor-db, check and validate nodes, apply changes to active configurations and much more. Understanding the flow of these logs and what they mean can be the difference between knowing what is happening in our device or not (if the issue is system-related or service-related). Log levels ========== Events have a severity level (*log level*) that hints at how the message should be interpreted and helps react to it. Priority-wise, events fall in the following order: * **Emergency**: Panic messages. They indicate the system cannot be used and requires immediate attention. * **Alert**: Alert messages requiring immediate action to resolve a critical issue. * **Critical**: Critical error messages that demand intervention to prevent system failure. * **Error**: Error messages for non-critical conditions that may affect normal operations. * **Warning**: Warnings that may lead to errors or unexpected behavior if not addressed. * **Notice**: Information messages that may hint at an issue or abnormal state. * **Informational**: Normal information messages seen in a working system. * **Debug**: Descriptive messages aimed at debugging specific issues. Lower level (lower priority) configurations show more information than higher ones. If a low log level is configured, all the logs related to that level will be shown (together with all the logs related to upper levels). For example, if the info level is set, notice, warning and err messages will be shown. In OSDx, notice or info level is set by default to prevent too many messages from appearing at the journal. .. _Enable Logs: Enable Logs =========== To activate/deactivate logs, we must first run :osdx:op:`configure` at the configuration menu. Logs belonging to a specific service/module: -------------------------------------------- Most of the services have the same log activation structure: .. code-block:: none admin@osdx# set service/system log-level/logging * **Examples:** .. code-block:: none admin@osdx# set service ssh log-level debug admin@osdx# .. code-block:: none admin@osdx# set system syslog logging log-level info admin@osdx# .. code-block:: none admin@osdx# set vpn ipsec logging log-types any log-level 1 admin@osdx# There are several modules in which you can enable the log level. Here are some examples: * :osdx:cfg:`service ssh log-level *` * :osdx:cfg:`service snmp log-level *` * :osdx:cfg:`service ssm log-level *` * :osdx:cfg:`service cfm log-level *` * :osdx:cfg:`service cnm log-level *` * :osdx:cfg:`service dhcp-client log-level debug` * :osdx:cfg:`service dns proxy log level *` * :osdx:cfg:`service firewall * logging level *` * :osdx:cfg:`service nsm log-level *` * :osdx:cfg:`system syslog logging log-level *` * :osdx:cfg:`system conntrack logging log-level *` * :osdx:cfg:`vpn ipsec logging log-types any log-level *` For ``vpn ipsec loging``, instead of using ``any`` as a log type, we can be more specific. `Here `_ we can find the meaning of all of them. Tipically, the default log level for all services (such as syslog or cfm) is ``notice`` or ``info`` (for ssh or conntrack). In the ``protocols`` path, the way logs are enabled is a little bit different. We must specify which protocol we want to log, the different parameters for that protocol, and the log level associated. To start with, use ``set protocols logging parameter`` together with ``set protocols logging level `` (info level by default). For example, command :osdx:cfg:`protocols logging bgp keepalives neighbor *` allows us to debug the BGP keepalives for the specified neighbor. CLI-related logs: -------------------- .. code-block:: none admin@osdx# set system cli configuration logging cli debug Logs for engine and subsystems: ------------------------------- .. code-block:: none admin@osdx# set system cli configuration logging engine debug .. code-block:: none admin@osdx# set system cli configuration logging launcher debug Global logs: ------------ Runnning :osdx:cfg:`system cli configuration logging global *` will enable cli, engine and launcher logs, but will not apply to other services like ssh, cnm, etc. The predefined value is ``notice``. By default, the ``global`` option is the one configured at an OSDx device because it handles all these logs (cli, engine and launcher). However, once cli, engine or launcher are configured manually (using the commands mentioned above), the device will take these as reference. This means that, for instance, if we configured the ``info`` level for *cli* events (:osdx:cfg:`system cli configuration logging cli *`), then both *engine* and *launcher* events will keep ``notice`` as their level, but *cli* would be updated to ``info``. If we then delete this path (:osdx:cfg:`system cli configuration logging cli *`), ``notice`` will go back to being the log level for *cli* events. Journal ======= The system journal is a volatile in-memory log, meaning it gets reset every time the system is rebooted. It is a useful tool to quickly inspect the state of the system and investigate possible issues due to a faulty configuration. To view the system journal, run the :osdx:op:`system journal show` command. Navigating the journal ---------------------- The journal uses a pager to help the user navigate the potentially massive log. While it might be cumbersome to navigate at first, using some basic shortcuts it becomes an extremely powerful tool. The most important shortcut to remember is the command for quitting the pager. This is done by simply pressing the ``q`` key. Navigating the log can be done using the up and down arrow keys. If the log messages are cut off for being too long, the right and left arrow keys can be used for horizontal scrolling. The ``g`` and ``G`` keys can be respectively used to go to the beginning and end of the journal. Finally, logs can be searched through by pressing ``/``, followed by the text and the ``Enter`` key. Jumping to the next search result is done by pressing ``n``, whilst the previous result can be reached by pressing ``N``. To learn more, press ``h`` for a full description of every shortcut available. Journal without pager --------------------- Sometimes, it may be useful to show the entire journal without using a pager. This can be done using the ``system journal show | cat`` command. Monitoring the journal ---------------------- If the goal is to show the journaled events as they arrive, showing the system journal will not suffice (since it only shows a snapshot of the events when the command is run). Instead, use the :osdx:op:`system journal monitor` command to view the events as they are being logged. Syslog ====== OSDx supports the syslog message logging standard, which allows events to be sent to remote syslog servers and to continuously store logs in the system. By way of example, we are going to explain how to send **ssh** events to a syslog server. First, we need to enable ssh logs in order to capture them through syslog. Use the :osdx:cfg:`service ssh log-level *` command and set the ``info`` level. Then, we need to configure an appropriate syslog server and specify we only want to send ssh events. The configuration is as follows: .. code-block:: none set service ssh log-level info set system syslog host 12.0.0.1 filter SSH_FILTER app ssh set system syslog host 12.0.0.1 filter SSH_FILTER level info set system syslog host 12.0.0.1 port 514 set system syslog host 12.0.0.1 protocol udp In this case, ``12.0.0.1`` is the syslog server we connect to, ``SSH_FILTER`` is the filter applied to journal entries to let them know we only want the events for the ``app`` (process name) *ssh* with ``level`` *info* (or higher), ``514`` is the port the syslog server is listening to and ``udp`` is the protocol that must be used to send these logs. Finally, if we log into this OSDx device via ssh (to capture some ssh traffic) and we then check our server log file, we should see somethin along the lines of: .. code-block:: none 2025-02-12T15:32:40.192384+01:00 12.0.0.2 2025-02-12T11:40:56.181232+00:00 auth-info osdx sshd[2916063]: Accepted password for admin from 12.0.0.1 port 48800 ssh2 2025-02-12T15:32:40.205112+01:00 12.0.0.2 2025-02-12T11:40:56.193739+00:00 authpriv-info osdx sshd[2916063]: pam_unix(sshd:session): session opened for user admin(uid=1000) by (uid=0) For more information about syslog and filters, check out the :doc:`/articles/system/syslog/index` chapter. Understanding Logs ================== The goal of this section is to try and explain basic concepts pertaining to the different logs that appear and how to interpret them. In terms of device interaction, *cli* logs are really important because they show how the user is handling the product. Examples of these logs are: .. code-block:: none OSDxCLI[2208]: User 'admin' has logged out. OSDxCLI[2000]: User 'admin' has logged in. OSDxCLI[2000]: User 'admin' entered the configuration menu. OSDxCLI[2000]: User 'admin' added a new cfg line: 'set system alarm TEST'. OSDxCLI[2000]: User 'admin' committed the configuration. OSDxCLI[2000]: User 'admin' left the configuration menu. OSDxCLI[2000]: User 'admin' executed a new command: 'ping teldat.es'. OSDxCLI[2000]: User 'admin' entered an invalid command: 'invalid command'. They are really descriptive of what a user is doing (log in and log out the device, enter and leave the configuration menu, add a configuration line, commit the configuration, run commands and enter invalid ones). Other logs related to user interaction are those that signal an error has occurred (**CLI error**) because an entered value does not match the defined syntax (**value validation error**), a configuration path is not valid due to dependencies with other nodes, or because certain operations are required (**commit validation error**). Examples of these events are described below. If the command is ``set system login user :invalid_name``, then the error is: .. code-block:: none osdx cfgd[1235]: [2000]Command output: Username contains unusual characters: only letters, numbers, underscores and dashes allowed Value validation failed If the committed command is ``set interfaces ethernet eth0 vrf RED``, then the commit error is: .. code-block:: none osdx cfgd[1235]: [2000]Command output: [ interfaces ethernet eth0 vrf ] \"system vrf RED\" does not exist Commit validation failed The process name of these events is different from *cli* logs. That is because **OSDxCLI** refers to *cli* events while **cfgd** refers to *engine* events. However, we can see that **cfgd** logs display the process number (2000) of **OSDxCLI**, meaning both logs are generated by an action triggered at the **OSDxCLI** service. In fact, **the OSDx engine will add the process number of all the services that need to send information to cfgd to most of its logs at debug level**. This means that, if we want to follow the flow of a user operation at the journal, we can just track the process number of OSDxCLI at **cfgd** logs to see what operations the engine is doing and the results when returning to CLI. Traffic Logs ============ As we haven seen, there are a lot logs we can enable/disable to understand what is happening at the device. Apart from the logs we have discussed, there are others related to traffic. In particular, an OSDx device is able to **log any incoming or outgoing traffic flow in the device**. To do so, a special configuration related to the ``traffic policy`` and ``traffic selector`` commands is needed. To better explain this, an example will be given of how to log traffic to the journal filtered by the source IP address. First, the ``traffic selector`` command configures the source IP address we want to use to filter the incoming traffic: .. code-block:: none set traffic selector SEL rule 1 source address 12.0.0.1 We then apply this selector to the ``traffic policy`` path, together with the ``log`` option. For this last option, we can also include the ``log-level`` of those logs (warning, by default) and a ``prefix``. .. code-block:: none set traffic policy POL rule 1 log prefix MY_PREFIX set traffic policy POL rule 1 selector SEL We also need to attach this new policy to a specific interface. Since we want to log incoming packets, the command is: .. code-block:: none set interfaces ethernet eth2 traffic policy in POL Finally, if we ping our device from the source IP address and, at the same time, we monitor the journal, we will see something like: .. figure:: traffic-log.png :width: 400px Traffic log For more information about the ``traffic`` tool and how to use it, please check :doc:`article `. Click on :doc:`one ` if you just need information on ``traffic policy``.