Logging

Logging is the act of keeping an append-only record of events which can later be read for troubleshooting purposes or forensic analysis. OSDx provides two ways to view logs: System Journal and Syslog, but it is important to note that the syslog tool will only work with information that also appears under the journal. Therefore, if (for instance) we want to send logs to a remote syslog server or a local file, we must first include that information in the journal. Otherwise, no logs will appear.

Events

Events are occurrences located at a single point in time and contain a plain-text message with information about its cause and context. We can split them in three categories:

• Events associated with running services

• Events displayed while interacting with the device via CLI

• Events related to the OSDx engine and subsystems

The first category relates to logs that show us information on what a service is doing and whether it encounters any problem. Examples of OSDx services could be ssh, snmp, dns, dhcp, firewall, conntrack, syslog and so on.

The logs associated to CLI interactions focus on activities within the device. This includes everything: configuring the device, checking the status of an interface, pinging a device, login or logout…

Finally, the engine and subsystems logs are those illustrating what an OSDx device needs to accomplish a task. A task could be to retrieve the information of a configuration node, obtain data from monitor-db, check and validate nodes, apply changes to active configurations and much more. Understanding the flow of these logs and what they mean can be the difference between knowing what is happening in our device or not (if the issue is system-related or service-related).

Log levels

Events have a severity level (log level) that hints at how the message should be interpreted and helps react to it. Priority-wise, events fall in the following order:

• Emergency: Panic messages. They indicate the system cannot be used and requires immediate attention.

• Alert: Alert messages requiring immediate action to resolve a critical issue.

• Critical: Critical error messages that demand intervention to prevent system failure.

• Error: Error messages for non-critical conditions that may affect normal operations.

• Warning: Warnings that may lead to errors or unexpected behavior if not addressed.

• Notice: Information messages that may hint at an issue or abnormal state.

• Informational: Normal information messages seen in a working system.

• Debug: Descriptive messages aimed at debugging specific issues.

Lower level (lower priority) configurations show more information than higher ones. If a low log level is configured, all the logs related to that level will be shown (together with all the logs related to upper levels). For example, if the info level is set, notice, warning and err messages will be shown. In OSDx, notice or info level is set by default to prevent too many messages from appearing at the journal.

Enable Logs

To activate/deactivate logs, we must first run configure at the configuration menu.

Logs belonging to a specific service/module:

Most of the services have the same log activation structure:

admin@osdx# set service/system <service> log-level/logging *

Examples:

admin@osdx# set service ssh log-level debug
admin@osdx#

admin@osdx# set system syslog logging log-level info
admin@osdx#

admin@osdx# set vpn ipsec logging log-types any log-level 1
admin@osdx#

There are several modules in which you can enable the log level. Here are some examples:

• service ssh log-level <txt>

• service snmp log-level <txt>

• service ssm log-level <txt>

• service cfm log-level <txt>

• service cnm log-level <txt>

• service dhcp-client log-level debug

• service dns proxy log level <u32>

• service firewall <id> logging level <id>

• service nsm log-level <txt>

• system syslog logging log-level <txt>

• system conntrack logging log-level <txt>

• vpn ipsec logging log-types any log-level <txt>

For vpn ipsec loging, instead of using any as a log type, we can be more specific. Here we can find the meaning of all of them.

Tipically, the default log level for all services (such as syslog or cfm) is notice or info (for ssh or conntrack).

In the protocols path, the way logs are enabled is a little bit different. We must specify which protocol we want to log, the different parameters for that protocol, and the log level associated. To start with, use set protocols logging <prot> parameter together with set protocols logging level <level> (info level by default). For example, command protocols logging bgp keepalives neighbor <ipv4|ipv6|id> allows us to debug the BGP keepalives for the specified neighbor.

CLI-related logs:

admin@osdx# set system cli configuration logging cli debug

Logs for engine and subsystems:

admin@osdx# set system cli configuration logging engine debug

admin@osdx# set system cli configuration logging launcher debug

Global logs:

Runnning system cli configuration logging global <txt> will enable cli, engine and launcher logs, but will not apply to other services like ssh, cnm, etc. The predefined value is notice.

By default, the global option is the one configured at an OSDx device because it handles all these logs (cli, engine and launcher). However, once cli, engine or launcher are configured manually (using the commands mentioned above), the device will take these as reference. This means that, for instance, if we configured the info level for cli events (system cli configuration logging cli <txt>), then both engine and launcher events will keep notice as their level, but cli would be updated to info. If we then delete this path (system cli configuration logging cli <txt>), notice will go back to being the log level for cli events.

Journal

The system journal is a volatile in-memory log, meaning it gets reset every time the system is rebooted. It is a useful tool to quickly inspect the state of the system and investigate possible issues due to a faulty configuration.

To view the system journal, run the system journal show command.

Navigating the journal

The journal uses a pager to help the user navigate the potentially massive log. While it might be cumbersome to navigate at first, using some basic shortcuts it becomes an extremely powerful tool.

The most important shortcut to remember is the command for quitting the pager. This is done by simply pressing the q key.

Navigating the log can be done using the up and down arrow keys. If the log messages are cut off for being too long, the right and left arrow keys can be used for horizontal scrolling. The g and G keys can be respectively used to go to the beginning and end of the journal.

Finally, logs can be searched through by pressing /, followed by the text and the Enter key. Jumping to the next search result is done by pressing n, whilst the previous result can be reached by pressing N.

To learn more, press h for a full description of every shortcut available.

Journal without pager

Sometimes, it may be useful to show the entire journal without using a pager. This can be done using the system journal show | cat command.

Monitoring the journal

If the goal is to show the journaled events as they arrive, showing the system journal will not suffice (since it only shows a snapshot of the events when the command is run). Instead, use the system journal monitor command to view the events as they are being logged.

Syslog

OSDx supports the syslog message logging standard, which allows events to be sent to remote syslog servers and to continuously store logs in the system.

By way of example, we are going to explain how to send ssh events to a syslog server.

First, we need to enable ssh logs in order to capture them through syslog. Use the service ssh log-level <txt> command and set the info level.

Then, we need to configure an appropriate syslog server and specify we only want to send ssh events. The configuration is as follows:

set service ssh log-level info
set system syslog host 12.0.0.1 filter SSH_FILTER app ssh
set system syslog host 12.0.0.1 filter SSH_FILTER level info
set system syslog host 12.0.0.1 port 514
set system syslog host 12.0.0.1 protocol udp

In this case, 12.0.0.1 is the syslog server we connect to, SSH_FILTER is the filter applied to journal entries to let them know we only want the events for the app (process name) ssh with level info (or higher), 514 is the port the syslog server is listening to and udp is the protocol that must be used to send these logs.

Finally, if we log into this OSDx device via ssh (to capture some ssh traffic) and we then check our server log file, we should see somethin along the lines of:

2025-02-12T15:32:40.192384+01:00 12.0.0.2  2025-02-12T11:40:56.181232+00:00 auth-info osdx sshd[2916063]:  Accepted password for admin from 12.0.0.1 port 48800 ssh2
2025-02-12T15:32:40.205112+01:00 12.0.0.2  2025-02-12T11:40:56.193739+00:00 authpriv-info osdx sshd[2916063]:  pam_unix(sshd:session): session opened for user admin(uid=1000) by (uid=0)

For more information about syslog and filters, check out the Syslog chapter.

Understanding Logs

The goal of this section is to try and explain basic concepts pertaining to the different logs that appear and how to interpret them.

In terms of device interaction, cli logs are really important because they show how the user is handling the product. Examples of these logs are:

OSDxCLI[2208]: User 'admin' has logged out.
OSDxCLI[2000]: User 'admin' has logged in.
OSDxCLI[2000]: User 'admin' entered the configuration menu.
OSDxCLI[2000]: User 'admin' added a new cfg line: 'set system alarm TEST'.
OSDxCLI[2000]: User 'admin' committed the configuration.
OSDxCLI[2000]: User 'admin' left the configuration menu.
OSDxCLI[2000]: User 'admin' executed a new command: 'ping teldat.es'.
OSDxCLI[2000]: User 'admin' entered an invalid command: 'invalid command'.

They are really descriptive of what a user is doing (log in and log out the device, enter and leave the configuration menu, add a configuration line, commit the configuration, run commands and enter invalid ones).

Other logs related to user interaction are those that signal an error has occurred (CLI error) because an entered value does not match the defined syntax (value validation error), a configuration path is not valid due to dependencies with other nodes, or because certain operations are required (commit validation error). Examples of these events are described below.

If the command is set system login user :invalid_name, then the error is:

osdx cfgd[1235]: [2000]Command output:
                 Username contains unusual characters: only letters, numbers, underscores and dashes allowed
                 Value validation failed

If the committed command is set interfaces ethernet eth0 vrf RED, then the commit error is:

osdx cfgd[1235]: [2000]Command output:
                 [ interfaces ethernet eth0 vrf ]
                 \"system vrf RED\" does not exist
                 Commit validation failed

The process name of these events is different from cli logs. That is because OSDxCLI refers to cli events while cfgd refers to engine events. However, we can see that cfgd logs display the process number (2000) of OSDxCLI, meaning both logs are generated by an action triggered at the OSDxCLI service. In fact, the OSDx engine will add the process number of all the services that need to send information to cfgd to most of its logs at debug level. This means that, if we want to follow the flow of a user operation at the journal, we can just track the process number of OSDxCLI at cfgd logs to see what operations the engine is doing and the results when returning to CLI.

Traffic Logs

As we haven seen, there are a lot logs we can enable/disable to understand what is happening at the device. Apart from the logs we have discussed, there are others related to traffic. In particular, an OSDx device is able to log any incoming or outgoing traffic flow in the device. To do so, a special configuration related to the traffic policy and traffic selector commands is needed. To better explain this, an example will be given of how to log traffic to the journal filtered by the source IP address.

First, the traffic selector command configures the source IP address we want to use to filter the incoming traffic:

set traffic selector SEL rule 1 source address 12.0.0.1

We then apply this selector to the traffic policy path, together with the log option. For this last option, we can also include the log-level of those logs (warning, by default) and a prefix.

set traffic policy POL rule 1 log prefix MY_PREFIX
set traffic policy POL rule 1 selector SEL

We also need to attach this new policy to a specific interface. Since we want to log incoming packets, the command is:

set interfaces ethernet eth2 traffic policy in POL

Finally, if we ping our device from the source IP address and, at the same time, we monitor the journal, we will see something like:

Traffic log

For more information about the traffic tool and how to use it, please check article. Click on one if you just need information on traffic policy.