Traffic Control

This chapter covers some aspects related to traffic control, which is the core element in configuring Quality of Service (QoS).

To process network traffic, three different objects can be combined: qdiscs, classes and filters:

  • A qdisc (traffic control on OSDx) is a traffic scheduler. E.g., FIFO (first-in first-out) is a common Qdisc example.

  • Some qdiscs can contain classes, which can be seen as containers for further Qdiscs. These are called classful qdiscs. These objects are very useful because network traffic can be prioritized by specifying the classes that should be dequeued first.

  • Filters (match on OSDx) can be used to determine in which class a packet will be queued. They are always attached to classes.

Configuration

The following types of traffic control are currently supported:

  • fifo: the simplest traffic control measure. Pure first-in, first-out behavior. It can be limited in both packets and bytes. This is a classless qdisc.

  • fq-codel: fair queuing controlled delay uses a stochastic model to classify incoming packets into different flows. This traffic control measure assigns a fair share of the bandwidth to every flow. This is a classless qdisc.

  • network-emulator: this traffic control can be used to add or modify some network characteristics, such as delay, packet loss, duplication, etc. This is a classless qdisc.

  • tbf: token bucket filter is useful when slowing traffic down to a precisely configured rate. A different traffic control can be specified as a TBF child. This is a classless qdisc.

  • htb: hierarchy token bucket implements a rich linksharing hierarchy of classes. HTB can prioritize classes and each one can contain traffic control as a child. This is a classful qdisc.

  • wfq: weighted fair queuing is a network-scheduling algorithm that allows users to specify what fraction of the total capacity is given to each class, depending on their weight value. This qdisc can be offloaded to hardware on some devices, provided that the Ethernet port used is not part of the router’s switch, if the router is equipped with one. If hardware offload is not enabled, htb is used internally. This is a classful qdisc.

  • lwdrr: lockless weighted deficit round robin is a classful qdisc that combines hierarchical DRR scheduling with optional token bucket rate limiting at every level. Classes are organized in a parent-child hierarchy (up to 8 levels deep) and scheduled using weighted deficit round robin with strict priority support (0–7). This is a classful qdisc.

  • mqprio: multiqueue priority qdisc is a hardware-offloaded queueing discipline designed for modern network interface cards supporting multiple transmit queues. MQPRIO provides traffic class-based scheduling with per-class bandwidth rate limiting. Unlike other classful qdiscs, mqprio does not support traditional match filters. Traffic classification must be performed externally using traffic policies with the set class action. This is a classful qdisc.

The syntax to create a traffic control is as follows:

set traffic control <qdisc_name> type <qdisc_type> [ ... ]

A traffic control discipline can be assigned to an interface. There are two modes: ingress and egress. The former is used to control inbound traffic (in) and the latter to control outbound traffic (out).

In order to assign a traffic control to an interface, you have to use the following command:

set interfaces <if_type> <if_name> traffic control <in / out> <qdisc_name>

Tip

When testing bandwidth, the monitor test performance operational command is very useful.

HTB (Hierarchy Token Bucket)

HTB is the most versatile classful qdisc available. It organizes classes in a parent-child hierarchy where each class has a guaranteed bandwidth and an optional upper limit.

Class hierarchy

An HTB qdisc requires defining the total available bandwidth for the link and, at least, one default-class where unmatched traffic is sent. Classes can be organized in a tree by assigning a parent to each class. Classes without an explicit parent are direct children of the root qdisc.

root (bandwidth 100 Mbps)
├── class 1                     bandwidth 50%, ceiling 100%
│   ├── class 2 (TCP 8080)      parent 1, bandwidth 30 Mbps
│   └── class 3 (TCP other)     parent 1, bandwidth 20 Mbps
└── class 4 (default)           bandwidth 10%

set traffic control QDISC type htb bandwidth 100
set traffic control QDISC type htb default-class 4
set traffic control QDISC type htb class 1 bandwidth percentage 50
set traffic control QDISC type htb class 1 ceiling percentage 100
set traffic control QDISC type htb class 2 bandwidth rate 30
set traffic control QDISC type htb class 2 parent 1
set traffic control QDISC type htb class 3 bandwidth rate 20
set traffic control QDISC type htb class 3 parent 1
set traffic control QDISC type htb class 4 bandwidth percentage 10

The hierarchy can be nested to any depth.

The system validates the hierarchy at commit time: self-references, non-existent parents, and cycles are rejected. A class with children cannot be deleted until all its children are removed first.

Bandwidth, ceiling and borrowing

Each class has two bandwidth parameters:

  • bandwidth: the guaranteed minimum the class will receive, even under congestion.

  • ceiling: the maximum the class is allowed to use. When sibling classes are idle, a class may borrow their unused bandwidth up to its ceiling value.

Both parameters can be specified in two ways:

  • bandwidth rate / ceiling rate: absolute value in Mbps. For example, bandwidth rate 30 guarantees 30 Mbps.

  • bandwidth percentage / ceiling percentage: percentage of the overall qdisc bandwidth (not the parent class bandwidth). For example, on a qdisc with bandwidth 100, bandwidth percentage 50 is 50 Mbps (regardless of what the parent class bandwidth is).

If no ceiling is set, it defaults to the class guaranteed bandwidth. This means the class will not borrow any excess bandwidth from idle siblings. To allow borrowing, set the ceiling higher than the guaranteed bandwidth.

Warning

Setting the ceiling below the guaranteed bandwidth will limit the class to the ceiling value, effectively reducing its throughput below the guaranteed rate. For example, a class with bandwidth rate 30 and ceiling percentage 20 on a 100 Mbps link will be capped at ~20 Mbps instead of the guaranteed 30 Mbps. Setting ceiling percentage 0 disables borrowing entirely, limiting the class to exactly its guaranteed bandwidth.

Priority

The priority parameter (0–7) controls which class gets to use excess bandwidth first. A lower value means higher priority. When multiple classes compete for surplus bandwidth, the one with the lowest priority value is served first.

set traffic control QDISC type htb class 1 priority 1
set traffic control QDISC type htb class 2 priority 4

Burst

The burst parameter controls how much data a class can send at once before the rate limiter kicks in. A larger burst allows short traffic spikes above the configured rate, while a smaller burst enforces a smoother output.

It can be set at qdisc level (applies as default to all classes) or per class. Values are specified in milliseconds (ms) or megabits (mbit). The default value is 125ms.

set traffic control QDISC type htb burst 125ms
set traffic control QDISC type htb class 1 burst 50ms

Traffic classification

Filters (match) determine which class receives each packet. Each match has a numeric identifier; with the lowest number being reviewed first. The first matching rule wins.

set traffic control QDISC type htb match 1 class 2
set traffic control QDISC type htb match 1 ip destination port 8080
set traffic control QDISC type htb match 1 ip protocol tcp
set traffic control QDISC type htb match 2 class 3
set traffic control QDISC type htb match 2 ip protocol tcp

In this example, TCP traffic on port 8080 goes to class 2, all other TCP traffic goes to class 3, and everything else falls to the default class.

Matches can filter on the basis of a wide range of criteria:

  • IP/IPv6 headers: source/destination address, source/destination port, protocol, DSCP, minimum/maximum packet length, TCP flags (SYN, ACK).

    Note

    For IPv4, min-length and max-length apply to the total packet length (IP header included). For IPv6, they apply to the payload length field of the IPv6 header (the 40-byte fixed header is not counted).

  • Ethernet headers: source/destination MAC address, Ethernet protocol.

  • Marks: firewall mark (mark) or VRF mark (vrf-mark).

  • Inbound interface: match traffic arriving from a specific interface.

Note

A match filter cannot target a class that is not a leaf node, because non-leaf classes do not own a qdisc queue. Attempting to assign a match to a non-leaf class yields a CLI error.

A match can also modify outgoing packet fields using set:

  • set tos or set dscp: change the DiffServ field in the IP header.

  • set cos-mark: sets the Class of Service for VLAN tagging.

For debugging purposes, a match can log packets that hit the rule:

set traffic control QDISC type htb match 1 log <prefix>

Matching packets will be logged in the system journal and can be reviewed with the system journal show command.

Alternatively, traffic policies can be used for classification purposes, instead of (or in addition to) matches. See here for more details.

VLAN Traffic Classification

The Hierarchy Token Bucket discipline also supports matches that classify packets based on VLAN headers, including single-tagged 802.1Q frames and double-tagged QinQ frames (802.1ad outer + 802.1Q inner). VLAN matching is enabled by setting the Ethernet protocol of the match and then narrowing the rule with the vlan subtree:

  • ether protocol: must be set to 802.1Q (single-tagged C-VLAN) or 802.1ad (S-VLAN / QinQ outer) to enable any VLAN match.

  • Outer tag (vlan subtree):

    • vlan id: outer VLAN identifier (0-4094).

    • vlan pcp: outer Priority Code Point (0-7).

    • vlan protocol: encapsulated protocol; for QinQ it must be set to 802.1Q to allow any inner-* match.

  • Inner tag (QinQ, vlan inner-*): require ether protocol 802.1ad and vlan protocol 802.1Q.

    • vlan inner-id: inner VLAN identifier (0-4094).

    • vlan inner-pcp: inner Priority Code Point (0-7).

    • vlan inner-protocol: protocol encapsulated inside the inner tag.

Classifying single-tagged 802.1Q traffic on VLAN 100 into class 1:

set traffic control QDISC type htb match 1 class 1
set traffic control QDISC type htb match 1 ether protocol 802.1Q
set traffic control QDISC type htb match 1 vlan id 100

Classifying a QinQ flow with outer 802.1ad VLAN 100 and inner 802.1Q VLAN 200 into class 1:

set traffic control QDISC type htb match 1 class 1
set traffic control QDISC type htb match 1 ether protocol 802.1ad
set traffic control QDISC type htb match 1 vlan id 100
set traffic control QDISC type htb match 1 vlan protocol 802.1Q
set traffic control QDISC type htb match 1 vlan inner-id 200

The vlan pcp and vlan inner-pcp fields match the 3-bit Priority Code Point of the outer and inner VLAN tags respectively. This is the 802.1Q field used to signal Class of Service, so it can be used to classify traffic by its CoS marking:

set traffic control QDISC type htb match 1 vlan pcp 3
set traffic control QDISC type htb match 1 vlan inner-pcp 5

VLAN matches can also be combined with IP/IPv6 matches in the same rule, provided the encapsulated protocol points to IP. For a single-tagged frame, set vlan protocol to ip or ipv6; for a QinQ frame, set vlan inner-protocol instead:

set traffic control QDISC type htb match 1 ether protocol 802.1Q
set traffic control QDISC type htb match 1 vlan id 100
set traffic control QDISC type htb match 1 vlan protocol ip
set traffic control QDISC type htb match 1 ip destination port 8080

Note

The CLI rejects inconsistent combinations at commit time. Using vlan fields without the appropriate ether protocol, mixing inner-* fields without vlan protocol 802.1Q, or adding IP/IPv6 matches without pointing the encapsulated protocol to ip or ipv6 yields a validation error.

Child qdiscs

Each HTB class uses a simple FIFO queue by default. A different traffic control discipline can be attached as a child to apply additional scheduling within a class, such as fair queuing:

set traffic control FQ_LEAF type fq-codel
set traffic control QDISC type htb class 1 child-control FQ_LEAF

Note

The child traffic control must be created before it can be referenced. A traffic control discipline cannot be used as a child of its own class, and wfq disciplines cannot be used as children.

Here you can find a basic HTB configuration example. For more advanced scenarios, see the examples on class hierarchy, traffic classification, ceiling and borrowing, policy-based classification, match filters, and validation.

LWDRR (Lockless Weighted Deficit Round Robin)

LWDRR is a classful qdisc that schedules classes with weighted deficit round robin. A token bucket shaper at the root enforces the overall link rate, and optional per-class shapers allow rate limiting individual classes.

Class hierarchy

An LWDRR qdisc requires defining the total available bandwidth for the link and a default-class where unmatched traffic is sent.

Classes are either leaf or inner:

  • A leaf class holds packets. It is identified by having a limit parameter (maximum queue depth in packets).

  • An inner class does not hold packets — it schedules its children using DRR. Inner classes are created by omitting the limit parameter.

Classes can be organized in a tree (up to 8 levels deep) by assigning a parent to each class. Classes without an explicit parent are direct children of the root qdisc.

root (bandwidth 1000 Mbps)
├── class 1 (inner)                weight 3, priority 0
│   ├── class 10 (leaf, default)   parent 1, weight 1, limit 1024
│   └── class 11 (leaf)            parent 1, weight 2, limit 2048
└── class 2 (leaf)                 weight 1, limit 10240

set traffic control QDISC type lwdrr bandwidth 1000
set traffic control QDISC type lwdrr default-class 10

set traffic control QDISC type lwdrr class 1 weight 3
set traffic control QDISC type lwdrr class 1 priority 0

set traffic control QDISC type lwdrr class 10 weight 1
set traffic control QDISC type lwdrr class 10 limit 1024
set traffic control QDISC type lwdrr class 10 parent 1

set traffic control QDISC type lwdrr class 11 weight 2
set traffic control QDISC type lwdrr class 11 limit 2048
set traffic control QDISC type lwdrr class 11 parent 1

set traffic control QDISC type lwdrr class 2 weight 1
set traffic control QDISC type lwdrr class 2 limit 10240

The default class must be a leaf.

Weight and quantum

Each class has a weight parameter (minimum 1) that determines its share of available bandwidth relative to its siblings. A class with weight 2 receives twice the throughput of a sibling with weight 1 when both are backlogged.

The quantum parameter at the root qdisc level controls the base DRR quantum in bytes (default: 1514). Each class’s effective quantum is weight × quantum. A larger quantum reduces scheduling overhead but increases burstiness.

set traffic control QDISC type lwdrr quantum 32768
set traffic control QDISC type lwdrr class 1 weight 3
set traffic control QDISC type lwdrr class 2 weight 1

Per-class rate limiting

Any class (leaf or inner) can have an optional token bucket rate limit. This is configured with bandwidth rate (absolute Mbps) or bandwidth percentage (percentage of the parent’s rate). An optional burst parameter controls the token bucket burst size.

set traffic control QDISC type lwdrr class 1 bandwidth rate 500
set traffic control QDISC type lwdrr class 1 burst 50ms

set traffic control QDISC type lwdrr class 2 bandwidth percentage 25

Without a per-class rate limit, the class is constrained only by the root bandwidth and its DRR weight share.

Priority

The priority parameter (0–7) controls strict priority scheduling among sibling classes. A lower value means higher priority. Classes at the same priority level are scheduled using weighted DRR among themselves.

set traffic control QDISC type lwdrr class 1 priority 0
set traffic control QDISC type lwdrr class 2 priority 4

Burst

The burst parameter at the root level controls the token bucket burst for the overall rate limiter. It can be specified as a time (e.g., 125ms) or size (e.g., 64mbit). The default is 125ms.

set traffic control QDISC type lwdrr burst 125ms

Traffic classification

Traffic classification works the same way as in HTB. Filters (match) determine which class receives each packet. Each match has a numeric identifier, with the lowest number being reviewed first. The first matching rule wins.

set traffic control QDISC type lwdrr match 100 class 2
set traffic control QDISC type lwdrr match 100 ip protocol udp
set traffic control QDISC type lwdrr match 100 ip destination port 5060

set traffic control QDISC type lwdrr match 200 class 1
set traffic control QDISC type lwdrr match 200 ip dscp 46

Matches support the same filter criteria as HTB: IP/IPv6 headers, Ethernet headers, marks, VRF marks, and inbound interface. They can also modify outgoing fields (set tos, set dscp, set cos) and log matching packets.

Alternatively, traffic policies can be used for classification. See here for more details.

Examples

Prioritizing traffic

Let’s suppose we want to set up a traffic control discipline in the egress hook of our eth0.

We want the following features:

  • Device bandwidth limited to 10 Mbps.

  • Three priority levels that will be mapped to three different queues: gold queue (high priority, traffic with mark 99), silver queue (medium priority, vrfs SRV1 or SRV2) and bronze queue (low priority, unmatched traffic).

  • The gold queue will have an assured bandwidth of 50%; the silver queue 30%; and the bronze one 20%.

  • Any remaining bandwidth will be distributed amongst all queues.

In OSDx, we can achieve this by configuring the following commands:

set traffic control QoS type htb bandwidth 10
set traffic control QoS type htb class 1 bandwidth percentage 50
set traffic control QoS type htb class 1 priority 1
set traffic control QoS type htb class 2 bandwidth percentage 30
set traffic control QoS type htb class 2 priority 4
set traffic control QoS type htb class 3 bandwidth percentage 20
set traffic control QoS type htb class 3 priority 7
set traffic control QoS type htb match 1 class 1
set traffic control QoS type htb match 1 mark 99
set traffic control QoS type htb match 1 set tos 8
set traffic control QoS type htb match 2 class 2
set traffic control QoS type htb match 2 vrf-mark SRV1
set traffic control QoS type htb match 3 class 2
set traffic control QoS type htb match 3 vrf-mark SRV2
set traffic control QoS type htb default-class 3

Finally, to attach this traffic control to eth0, use this command:

set interfaces ethernet eth0 traffic control out QoS

Limiting interface throughput

Now, let’s suppose we want to limit inbound traffic in eth1 to 512 Kbps. This can be easily done using a TBF traffic control.

In this case, we will configure a bucket size of 1875 bytes (0.015 mbit), this parameter should always be higher than the device mtu.

To create the traffic control discipline, type the following commands:

set traffic control LIMITER type tbf bandwidth 0.512
set traffic control LIMITER type tbf burst 0.015
set traffic control LIMITER type tbf latency 1
set interfaces ethernet eth1 traffic control in LIMITER

In the case of TBF traffic control, we could also specify a child qdisc.

Example:

set traffic control FQ_LEAF type fq-codel
set traffic control LIMITER type tbf child-control FQ_LEAF

Hardware-offloaded multiqueue scheduling

The mqprio qdisc provides hardware-offloaded traffic management for interfaces that support multiple transmit queues. Unlike software-based qdiscs, mqprio delegates traffic scheduling to the network interface hardware.

Let’s configure eth0 with four traffic classes for different service levels: voice (high priority), gold (premium data), silver (standard data), and a default class for best-effort traffic.

We’ll allocate a maximum bandwidth of 2.5 Gbps to each service class:

set traffic control ETHWAN_QOS type mqprio class 1 bandwidth rate 2500
set traffic control ETHWAN_QOS type mqprio class 1 description "VOICE"
set traffic control ETHWAN_QOS type mqprio class 2 bandwidth rate 2500
set traffic control ETHWAN_QOS type mqprio class 2 description "GOLD"
set traffic control ETHWAN_QOS type mqprio class 3 bandwidth rate 2500
set traffic control ETHWAN_QOS type mqprio class 3 description "SILVER"
set traffic control ETHWAN_QOS type mqprio class 4 description "DEFAULT"
set traffic control ETHWAN_QOS type mqprio default-class 4

Note

Bandwidth rate in mqprio qdiscs is configured internally in 50 Mbps steps. This means that values that are not multiple of 50 Mbps will be rounded down to the nearest multiple of 50 Mbps.

Attach the qdisc to the interface:

set interfaces ethernet eth0 traffic control out ETHWAN_QOS

Since mqprio does not support match filters, we must use traffic policies to classify packets into their appropriate class. Create a policy that assigns voice traffic (UDP port 5060) to class 1, and premium customers (specific IP range) to class 2:

set traffic policy CLASSIFIER rule 10 selector VOICE
set traffic policy CLASSIFIER rule 10 set class 1
set traffic policy CLASSIFIER rule 20 selector GOLD
set traffic policy CLASSIFIER rule 20 set class 2
set traffic selector VOICE rule 10 protocol udp
set traffic selector VOICE rule 10 destination port 5060
set traffic selector GOLD rule 10 source address 10.100.0.0/24

Apply the traffic policies to the interface:

set interfaces ethernet eth0 traffic policy link-out CLASSIFIER

All unclassified traffic will be handled by the default class (class 4).

Note

mqprio is only available on interfaces with hardware multiqueue support and can only be configured in egress (out) mode.

Hierarchical QoS with LWDRR

Let’s configure a two-level LWDRR hierarchy on eth0 with three service tiers: a high-priority control class, a data inner class with two weighted leaf classes, and rate limiting on the data tier.

set traffic control WAN_QOS type lwdrr bandwidth 1000
set traffic control WAN_QOS type lwdrr default-class 20

set traffic control WAN_QOS type lwdrr class 1 weight 1
set traffic control WAN_QOS type lwdrr class 1 limit 512
set traffic control WAN_QOS type lwdrr class 1 priority 0
set traffic control WAN_QOS type lwdrr class 1 description "Control"

set traffic control WAN_QOS type lwdrr class 2 weight 3
set traffic control WAN_QOS type lwdrr class 2 priority 4
set traffic control WAN_QOS type lwdrr class 2 bandwidth rate 800

set traffic control WAN_QOS type lwdrr class 20 weight 1
set traffic control WAN_QOS type lwdrr class 20 limit 4096
set traffic control WAN_QOS type lwdrr class 20 parent 2
set traffic control WAN_QOS type lwdrr class 20 description "Default data"

set traffic control WAN_QOS type lwdrr class 21 weight 2
set traffic control WAN_QOS type lwdrr class 21 limit 4096
set traffic control WAN_QOS type lwdrr class 21 parent 2
set traffic control WAN_QOS type lwdrr class 21 description "Premium data"

set traffic control WAN_QOS type lwdrr match 1 class 1
set traffic control WAN_QOS type lwdrr match 1 mark 50

set traffic control WAN_QOS type lwdrr match 2 class 21
set traffic control WAN_QOS type lwdrr match 2 ip dscp 46

Attach the qdisc to the interface:

set interfaces ethernet eth0 traffic control out WAN_QOS

In this example, packets with mark 50 are sent to the strict high-priority class 1. DSCP 46 traffic goes to class 21 under the data tier. All other traffic falls to the default class 20. The data tier (class 2) is rate limited to 800 Mbps and its two leaf classes share that bandwidth in a 2:1 ratio.

Here, you can find more examples related to traffic control disciplines.

Advanced features

Combining traffic policies and traffic control disciplines could prove very useful in some situations. Although there are many filters that can be used in a traffic control to classify network traffic (e.g., mark, TOS, etc), in some situations you may need to use more specialized ACLs (or traffic selectors).

Network packets can be marked using traffic policies and, depending on this value, queue packets in a specific traffic control class. For mqprio this is mandatory, since it does not support match.

In addition to this, if we just need to limit traffic, we can use traffic policy action rate-limit.

Example:

set traffic policy LIMITER rule 1 action rate-limit <rate> [burst <burst>]

The traffic policy will drop any packets that exceed the rate-limit.

Here you can find more information about traffic policies.

Monitoring

The traffic control show operational command can be used to display statistics related to traffic control disciplines.

Example:

admin@osdx$ traffic control show
Traffic control for interface 'eth1' - 'ingress' mode

---------------------------------------------------------------------------
ID   traffic control    type    parent  bytes sent  pkts sent  pkts dropped
---------------------------------------------------------------------------
1:0  LIMITER          tbf       root        647202       9805          2369
2:0  FQ_LEAF          fq_codel  1:1         647202       9805          2369

Traffic control for interface 'eth0' - 'egress' mode

------------------------------------------------------------------------------------
ID   traffic control        type         parent  bytes sent  pkts sent  pkts dropped
------------------------------------------------------------------------------------
1:0  QoS              htb                root          1320         13             0
1:1  QoS              class 1            1:0            220          2             0
1:2  QoS              class 2            1:0            430          3             0
1:3  QoS              class 3 (default)  1:0            670          8             0

Command Summary

Configuration commands

Operational commands