========= Conntrack ========= Overview ======== The :osdx:cfg:`system conntrack` feature allows network connections to be tracked through OSDx devices. The conntrack feature gives the Netfilter connection tracking system a userspace interface, thus enabling the monitoring and management of all active connections. This chapter covers the basics of connection tracking, application detection capabilities, and advanced traffic classification features for policy-based routing and traffic filtering. Basic Connection Tracking ========================== Connection Format ----------------- Connections displayed include the source and reply information, along with metadata. Here's an example: .. code-block:: bash icmp 1 25 src=10.0.0.1 dst=10.0.0.2 type=8 code=0 id=18723 vrf=RED packets=2 bytes=168 src=10.0.0.2 dst=10.0.0.1 type=0 code=0 id=18723 vrf=RED packets=2 bytes=168 mark=33 use=1 appdetect[L3:1] **Connection Fields:** * **Protocol**: ``icmp`` - Protocol identifier (icmp, tcp, udp, etc.) * **Protocol Number**: ``1`` - Numeric protocol identifier * **TTL**: ``25`` - Time remaining before connection expires * **Origin**: ``src=10.0.0.1 dst=10.0.0.2`` - Source and destination of original packet * **Reply**: ``src=10.0.0.2 dst=10.0.0.1`` - Source and destination of reply packet * **VRF**: ``vrf=RED`` - Virtual routing and forwarding instance * **Statistics**: ``packets=2 bytes=168`` - Packet and byte counters * **Metadata**: * ``mark=33`` - Packet mark for policy routing * ``use=1`` - Usage counter * ``appdetect[L3:1]`` - Application detection data * ``[OFFLOAD]`` or ``[HW_OFFLOAD]`` - Hardware offload status * ``Sc: not-bypass`` - Security bypass status For more information on traffic policies and marks, see :doc:`Traffic Routing ` and :doc:`Firewall Service `. Connection States ----------------- Running :osdx:op:`system conntrack monitor` displays connections with status information: * **NEW**: Newly established connections * **UPDATE**: Updated connection information * **DESTROY**: Terminated connections Logging ======= The :osdx:cfg:`system conntrack logging` feature sends connection tracking information to syslog, enabling centralized monitoring and auditing. Configuration ------------- **Event Types** Configure which connection events to log (default: **destroy**): * **NEW**: Logs new connections * **UPDATE**: Logs connection updates * **DESTROY**: Logs connection termination * **ALL**: Logs all events **Log Level** The **log-level** option controls the syslog severity level for logged messages (default: **info**): View logs using :osdx:op:`system journal monitor` or :osdx:op:`system journal show`: .. code-block:: bash osdx ulogd[9431]: [NEW] ORIG: SRC=10.0.0.1 DST=10.0.0.2 VRF=RED PROTO=TCP SPT=55406 DPT=80 PKTS=6 BYTES=577 , REPLY: SRC=10.0.0.2 DST=10.0.0.1 VRF=RED PROTO=TCP SPT=80 DPT=55406 PKTS=4 BYTES=653 MARK=33 LABELS=LOCAL APPDETECT[L4:80 http-host:10.0.0.1] Log messages include: * Origin and reply connection information * Traffic policy labels * Application detection data (app-id and detected protocols) **Custom Identity** Customize the log identifier using :osdx:cfg:`system conntrack logging identity *`: .. code-block:: bash # Changes "ulogd" to "OSDx_DUT0" set system conntrack logging identity OSDx_DUT0 Examples: :doc:`Conntrack Logging Examples ` Application Detection (App-Detect) =================================== Overview -------- :osdx:cfg:`system conntrack app-detect` is an implementation based on `RFC6759 `_ that allows applications to be identified in connection tracking flows using Deep Packet Inspection (DPI). Application detection provides traffic flows with: * **Application ID (App-ID)**: Identifies the application or protocol, the APP-ID consists of: * **Engine ID**: Network layer (L3, L4, U) * **Selector ID**: Protocol or service identifier (protocol number, port number, custom ID, provider ID) * **Detected Data**: Protocol-specific information (hostnames, URLs, etc.) The App-ID assigned can be used by traffic selectors for traffic policy matching (see :doc:`Traffic Policy `). Display application detection information using: * :osdx:op:`system conntrack show` * :osdx:op:`system conntrack monitor` * :osdx:cfg:`system conntrack logging` Output format: ``appdetect[engineID:selectorID "detected data"]`` Application ID Structure ------------------------- The Application ID is made up of two components: Engine ID ********* Indicates the network layer associated with the flow: * **L3**: Layer 3 Internet Protocol Number (`IANA-L3 `_) - E.g., ICMP, protocol code 1 per `RFC792 `_ * **L4**: Layer 4 Transport Protocol Port Number (`IANA-L4 `_) - E.g., HTTP on port 80 * **U**: Upper-layer application number (Layer 7 applications) - Custom or provider application identifiers (e.g. U6 for custom dictionary) Selector ID *********** Specific identifier within the engine type: * **L3**: Protocol number (e.g., 1 for ICMP) * **L4**: Service port number (e.g., 80 for HTTP, 443 for HTTPS) * Application detectors can override port-based assignment (e.g., L4:80 assigned to traffic on port 8080 if HTTP is detected) * **U**: Upper-layer application identifier number set via traffic policy or dictionary System Architecture ------------------- Session-Based Packet Processing ******************************** OSDx conntrack implements session-based packet processing to efficiently classify and handle IP packets. When enabled, the system automatically associates IP (IPv4/IPv6) packets with connection tracking sessions. **Session Creation:** When a packet is processed: 1. The system attempts to find the packet's corresponding session among existing connections 2. If no matching session exists, a new session is created and assigned to the packet 3. Subsequent packets belonging to the same flow are associated with this session **Session Definition:** Sessions are identified by: * Source IP address * Destination IP address * Protocol (e.g., TCP, UDP, ICMP) * Source and destination ports (for TCP/UDP protocols) This means a session can represent various types of traffic flows, such as: * A TCP connection between devices * ICMP echo requests and replies Application Identification Integration *************************************** To boost session identification, an Application ID (App-ID) is linked to each session through the application detection subsystem (app-detect). The App-ID assigned provides: * Session classification based on detected applications * Traffic policy matching via traffic selectors * Enhanced visibility in connection tracking logs and netflow exportation **How Detection Works:** When app-detect is configured via :osdx:cfg:`system conntrack app-detect`, OSDx automatically enables application detection for all network traffic passing through the device. The system processes both incoming traffic and locally-generated traffic. The detection process works as follows: 1. **Cache Lookup**: When a new connection starts, OSDx first checks if the destination IP address has a recently detected cached App-ID: * If found, the cached App-ID is assigned to the connection 2. **Deep Packet Inspection**: DPI runs on each packet until the application is identified or detection completes. Exceptionally, DNS inspection could continue detecting the host even after initial resolution. The number of packets required, if fragmentation isn't applied, varies by protocol: * **DNS (UDP)**: 2 packets (query and response) * **DNS (TCP)**: 5 packets (TCP handshake, query, and response) * **HTTP**: 4 packets (TCP handshake and HTTP request) * **SSL/TLS**: 4 packets (TCP handshake and Client Hello) During inspection, the system: * Examines protocol-specific patterns (HTTP requests, SSL/TLS handshakes, DNS messages) * Extracts relevant information (hostnames, URLs, query names) * Searches configured application dictionaries for matching patterns * Continues inspection until the application is identified or deemed undetectable 3. **App-ID Assignment**: Once the application is identified, the App-ID is: * Stored in the connection tracking entry * Cached for the destination IP address to speed up detection in future connections and perform First Packet Detection (FPD) * Immediately available for use in traffic policy rules 4. **Subsequent Packets**: All remaining packets in the connection inherit the assigned App-ID without further inspection. The assigned App-ID can be referenced in traffic policy rules to enable application-aware classification for policy-based routing, QoS, and security filtering. .. note:: When :osdx:cfg:`system offload` is configured, once an application is detected for a session, subsequent packets can be offloaded to the fast path. Application Detectors ---------------------- Application detectors use DPI to identify specific applications by analyzing packet content. The system examines protocol-specific patterns and structures to identify DNS, HTTP, and SSL/TLS traffic. DNS Detection ************* Detects DNS protocol traffic by analyzing the DNS message structure and validating message formatting. * **Protocol**: DNS (UDP/TCP) * **Detection Method**: Analyzes DNS header fields and message structure * **App-ID Assignment**: L4:53 (or L4:5353 for multicast DNS), regardless of actual server port * **Detection Mode**: * Continuously inspects DNS flows by default to detect each query in the session * When :osdx:cfg:`system conntrack app-detect dns-host disable-continuous-resolution` is enabled, only the first query is inspected. HTTP Detection ************** Detects HTTP protocol traffic (versions 1.0 and 1.1) by analyzing HTTP request methods and headers. * **Protocol**: HTTP on TCP connections * **Detection Method**: Validates HTTP request methods and parses request headers * **App-ID Assignment**: L4:80, regardless of actual server port * **Detection Scope**: Only takes into account the initial session request (persistent connections show first request only) SSL/TLS Detection ***************** Detects SSL/TLS encrypted traffic by analyzing the TLS handshake. * **Protocol**: SSL/TLS (all versions) * **Detection Method**: Parses the TLS Client Hello message to identify encrypted connections * **App-ID Assignment**: Port-based (e.g., L4:443 for HTTPS) * SSL/TLS is a transport layer protocol, not a final application identifier * The server port provides better application context * **Detection Scope**: Only shows the initial Client Hello message Data Extraction --------------- In addition to protocol detection, app-detect can extract protocol-specific metadata from detected traffic. The data extracted is displayed alongside the App-ID in connection tracking output and logs. DNS Data Extraction ******************* Extracts domain names from DNS queries and responses. * **Configuration**: :osdx:cfg:`system conntrack app-detect dns-host` * **Extracted Data**: * DNS query names (hostnames being resolved) HTTP Data Extraction ******************** Extracts various fields from HTTP request headers. * **Available Fields**: * :osdx:cfg:`system conntrack app-detect http-host` - Extracts Host header (domain name) * :osdx:cfg:`system conntrack app-detect http-url` - Extracts complete request URL * :osdx:cfg:`system conntrack app-detect http-referer` - Extracts Referer header * :osdx:cfg:`system conntrack app-detect http-user-agent` - Extracts User-Agent string SSL/TLS Data Extraction *********************** Extracts the server hostname from SSL/TLS handshakes. * **Configuration**: :osdx:cfg:`system conntrack app-detect ssl-host` * **Extracted Data**: Server hostname from Server Name Indication (SNI) extension Application ID Storage Modes ============================ OSDx supports two storage modes for Application IDs, configured via :osdx:cfg:`system conntrack app-detect app-id-storage`. Override Mode (Default) ----------------------- In override mode (:osdx:cfg:`system conntrack app-detect app-id-storage override`), only the highest layer Application ID is stored for each traffic session. When a higher-layer application is detected, the previous App-ID is replaced. **Example:** A connection initially classified as L4:443 (HTTPS port) may be reclassified as U128:42 when a dictionary match identifies the specific application. Chained Mode ------------ In chained mode (:osdx:cfg:`system conntrack app-detect app-id-storage chained`), all detected Application IDs are stored for each traffic session. This preserves the detection history in full, allowing traffic selectors to match any of the detected App-IDs in the chain. **Use Cases:** * Track App-ID transitions (e.g., HTTP detection followed by dictionary-based reclassification) * Match traffic based on detection identifier tuples (e.g., together L3 protocol, L4 port, custom dictionary and provider dictionary properties like category and reputation) * Policy enforcement requiring multiple classification criteria * Enhanced visibility for logging and Netflow exportation **Example:** A connection might have an App-ID chain of ``[L3:6;L4:443;U130:15;U131:85]`` where: * L3:6 identifies the TCP protocol * L4:443 identifies the HTTPS protocol * U130:15 is the category from a remote dictionary * U131:85 is the reputation score from a remote dictionary .. note:: **Dictionary Priority:** The dictionary number in :osdx:cfg:`system conntrack app-detect dictionary *` serves as the priority value, affecting the search order and App-ID resolution when multiple dictionaries are configured. Lower numbers indicate higher priority (dictionary 1 has the highest priority). In **override mode**, higher priority dictionaries are preferred. In **chained mode**, priority has no effect since all resolved App-IDs are stored. Remote Dictionary ================= Overview -------- Remote dictionaries enable cloud-based application categorization by connecting to external classification services. When OSDx detects traffic requiring classification (via DNS, HTTP or SSL detectors), it communicates with the remote dictionary server for remote App-ID resolution. Remote dictionaries support the retrieval of the following classification properties: * **Category**: Application category ID (Classification Engine ID U130) * **Reputation**: Reputation score (Classification Engine ID U131) Configuration ------------- Configure a remote dictionary using :osdx:cfg:`system conntrack app-detect dictionary * remote`: **Required Parameters:** * ``url`` / ``encrypted-url``: Remote dictionary server URL * ``key`` / ``encrypted-key``: API key for authentication * ``property category`` or ``property reputation``: Classification property to retrieve **Optional Parameters:** * ``ssl-allow-insecure``: Disable SSL certificate verification (not recommended for production) * ``alarm connection-error``: Alarm to trigger on connection failures **Binding Options:** Control the network interface or address used to connect to the remote server: * ``local-address``: Bind to a specific IPv4 address * ``local-interface``: Bind to a specific network interface * ``local-vrf``: Bind to a specific VRF **Traffic Marking:** Mark outgoing remote dictionary traffic for routing or QoS purposes: * ``mark``: Set a specific numeric mark * ``vrf-mark``: Set the mark based on a VRF Traffic Selector Integration ============================ The ``app-detect`` traffic selector provides comprehensive matching capabilities for application detection data, including App-ID values and detection states. App-ID Matching --------------- Match traffic based on Application ID values using :osdx:cfg:`traffic selector * rule * app-detect app-id`: .. code-block:: bash # Match by L3 protocol (e.g., ICMP = 1) set traffic selector ICMP rule 1 app-detect app-id l3 1 # Match by L4 port (e.g., HTTPS = 443) set traffic selector HTTPS rule 1 app-detect app-id l4 443 # Match by custom dictionary App-ID set traffic selector CUSTOM rule 1 app-detect app-id custom 42 # Match by specific classification engine and selector set traffic selector ENGINE rule 1 app-detect app-id engine 128 selector 30 # Match by remote dictionary category set traffic selector CATEGORY rule 1 app-detect app-id category 15 # Match by remote dictionary reputation set traffic selector REPUTATION rule 1 app-detect app-id reputation greater-than 70 Category and Reputation Classification -------------------------------------- Remote dictionaries assign Application IDs using Classification Engine IDs 130 (category) and 131 (reputation). **Category (Engine 130):** Categories are numeric identifiers representing application types (e.g., social media, streaming, business). The specific category values depend on the remote dictionary service provider. **Reputation (Engine 131):** Reputation scores indicate the trustworthiness or risk level of a destination. Higher values typically indicate lower risk. **Traffic Selector Matching:** Use traffic selectors to match category and reputation values: .. code-block:: bash # Match traffic with category 15 set traffic selector SOCIAL rule 1 app-detect app-id category 15 # Match traffic with reputation below 50 (higher risk) set traffic selector RISKY rule 1 app-detect app-id reputation less-than 50 # Match traffic with reputation above 80 (lower risk) set traffic selector TRUSTED rule 1 app-detect app-id reputation greater-than 80 Detection State Matching ------------------------ Match traffic based on the current detection state using :osdx:cfg:`traffic selector * rule * app-detect state`: * **detecting**: Traffic session currently being analyzed by the detection engine * **detected**: Traffic session where an application has been successfully identified * **host-detected**: Traffic session where DPI has detected a host (FQDN) **Example:** .. code-block:: bash # Match traffic still being analyzed set traffic selector ANALYZING rule 1 app-detect state detecting # Match traffic identified by any DNS, HTTP or SSL application detectors set traffic selector IDENTIFIED rule 1 app-detect state detected # Match traffic where a host has been detected set traffic selector HOST_FOUND rule 1 app-detect state host-detected Application Identification Methods =================================== Assign App-ID via Traffic Policy --------------------------------- Assign App-IDs through interface-applied traffic policies. **Example:** .. code-block:: none set traffic policy Test rule 1 set app-id custom 33 set interfaces ethernet eth4 traffic policy in Test Send traffic and view the result: .. code-block:: bash # system conntrack show icmp 1 28 src=10.215.168.1 dst=10.215.168.64 type=8 code=0 id=17504 packets=2 bytes=168 src=10.215.168.64 dst=10.215.168.1 type=0 code=0 id=17504 packets=2 bytes=168 mark=0 use=1 appdetect[U6:33] The custom App-ID (U6:33) is displayed. Dictionary-Based Classification -------------------------------- Application dictionaries provide structured sets of identifiable applications with names, App-IDs, and matching criteria (FQDN patterns or IP ranges). CLI Dictionary Configuration ***************************** Create custom dictionaries directly via CLI commands: .. code-block:: bash # Define custom app with FQDN pattern set system conntrack app-detect dictionary 1 custom app-id 42 fqdn enterprise.opentok.com # Add application name set system conntrack app-detect dictionary 1 custom app-id 42 name "Teldat Test" * **Classification Engine ID**: 6 (for CLI-defined dictionaries) XML Dictionary Files ******************** Define dictionaries in XML format (with optional GZIP compression): .. code-block:: xml 10.215.168.1 10.215.168.64 255.255.255.192 **Application Elements:** * ``engine_id``: Classification engine identifier (custom value, e.g., 128) * ``id``: Application selector ID * ``name``: Human-readable application name * ``version``: Application definition version * ``fqdn_list``: FQDN patterns for matching * ``address_list``: IP address ranges for matching **Load Dictionary:** .. code-block:: bash set system conntrack app-detect dictionary 1 filename running://path/to/dict.xml **Example Result:** After loading the dictionary and enabling private IP matching, establish an SSH connection to 10.215.168.64: .. code-block:: bash tcp 6 3597 ESTABLISHED src=10.215.168.1 dst=10.215.168.64 sport=52360 dport=22 packets=18 bytes=1016 src=10.215.168.64 dst=10.215.168.1 sport=22 dport=52360 packets=21 bytes=2452 [ASSURED] mark=0 use=1 appdetect[U128:31] The flow is classified with App-ID U128:31 based on the destination IP range match. .. note:: Dictionary-based classification can override default port-based assignment. In this example, SSH (normally L4:22) is reclassified as U128:31 per the dictionary definition. Configuration Commands ---------------------- .. osdx:cmdtree:: cfg :maxdepth: 2 system conntrack Operational Commands -------------------- .. osdx:cmdtree:: op :maxdepth: 2 system conntrack