TD(07)037. The 10th COST 290 MC Meeting. Technische Universität Wien Wien, Austria October 1 2, 2007

Transcription

1 TD(07)037 The 10th COST 290 MC Meeting Technische Universität Wien Wien, Austria October 1 2, 2007 Performance of Wireless IEEE e-Based Devices with Multiple Hardware Queues Gabriel Lazar, Virgil Dobrota, Tudor Blaga Technical University of Cluj-Napoca, Communications Department Baritiu Street, Cluj-Napoca, Romania Tel/Fax: virgil.dobrota@com.utcluj.ro Abstract The network subsystem of the Linux kernel connects to the underlying device drivers through a single entry point. With the advent of wireless adapters with multiple hardware queues, this method of communication poses difficulties in enabling efficient usage of the multiqueue features and assuring cross-layer QoS interoperation. The paper also describes some of the problems that can appear when Linux traffic control is employed for devices that operate over links with variable output rates. A popular device driver for wireless cards based on the Atheros chipsets is analyzed with respect to its support for multiqueue features, in conjunction with a recent version of the Linux kernel. Keywords Linux traffic control, cross-layer quality of service, IEEE e, multiple hardware queues COST 290 :: Wi-QoST :: Traffic and QoS Management in Wireless Multimedia Networks

2 Performance of Wireless IEEE e-Based Devices with Multiple Hardware Queues Gabriel Lazar, Virgil Dobrota, Tudor Blaga Abstract The network subsystem of the Linux kernel connects to the underlying device drivers through a single entry point. With the advent of wireless adapters with multiple hardware queues, this method of communication poses difficulties in enabling efficient usage of the multiqueue features and assuring cross-layer QoS interoperation. The paper also describes some of the problems that can appear when Linux traffic control is employed for devices that operate over links with variable output rates. A popular device driver for wireless cards based on the Atheros chipsets is analyzed with respect to its support for multiqueue features, in conjunction with a recent version of the Linux kernel. 1. LINUX NETWORK SUBSYSTEM One of the important reasons responsible for the efficiency and flexibility of the Linux network subsystem is the architecture of the data structures that manages network packets. When a packet is passed from the application layer into the kernel space using sockets, a socket kernel buffer is created (called skb for short). The socket buffer is passed through each layer of the network stack before reaching the device driver in order to be transmitted. This data structure (struct sk_buff) plays a central role in the network implementation of the Linux kernel, and it represents a packet during its entire processing lifetime inside the kernel space. A socket kernel buffer matches a sending or received packet, depending on traffic direction. In the network architecture of the Linux kernel, the interface between protocols implemented in software and the network adapters is accomplished by the concept of network devices. A network device provides an abstraction from the technical properties of the network adapter and also a uniform interface to be used by the protocols from the upper layers. The net_device structure represents each network device inside the Linux kernel. All network devices are connected into a doubly linked list. The entry point to the list of registered devices is obtained by accessing the list head, which is the kernel variable dev_base. Each element in the list is a net_device instance that contains all information and function pointers for that device. One of the pointers stored in this structure is the queuing discipline associated with the packets transmitted through this network adapter (the qdisc item). The process of transmitting a packet through the kernel can be handled in several ways. In the normal transmission process, the protocol instances of the higher network layers will use the function dev_queue_xmit(skb) to send a packet. The network device is specified in the socket buffer structure by the skb->dev parameter. If there is no queue management defined for that device (the.enqueue operation is NULL), the packet is put immediately into the hardware queue through a call to the dev->hard_xmit_method() method, if the driver s 1

3 hardware Tx queue is not full (Fig. 1). Otherwise, without any queues defined above the hardware layer, the packet is lost before the reaching the medium. Fig. 1. skb transmission using dev_queue_xmit It is the hard_start_xmit method s responsibility to check the hardware queues status. When the hardware queue is full, netif_stop_queue should be called, the skb has to be refused for transmission and the return code is NETDEV_TX_BUSY [2]. 2. LINUX TRAFFIC CONTROL In normal conditions, each network device has associated a queue management implementation, in order not to rely solely on the device hardware queues. This offers not only an additional buffer space in the form of software queues, but an entire QoS implementation inside the Linux kernel through the use of custom schedulers, configurable queues and detailed packet classification methods. This traffic control subsystem is placed between the upper network protocols and the net_device interface. When the higher protocols send a packet for transmission, by calling dev_queue_xmit(skb), the socket buffer is placed in one of the output queues of the network device. This is accomplished by use of the method.enqueue of the qdisc item associated with the device. This method is responsible with selecting the queue (if more than one) and the position of the packet inside the selected queue. Handling of packets inside queues is done by qdisc_run and depends on the type of the packet scheduler associated with the queue discipline. In the case of a work-conserving-scheduler, qdisc_run will call qdisc_restart until either there are no more packets in the queue(s), or the network packets does not accept any more packets (through a netif_stop_queue call). Unless the device is stopped, qdisc_restart will use the.dequeue operation to extract a packet from the traffic control queue discipline(s) and call hard_start_xmit for the device [1]. 2

4 If a non-work-conserving-scheduler is used (e.g. a traffic shaper), packets are serviced using a special Tx Soft IRQ triggered by a scheduling timer (Fig.2). Fig. 2. skb transmission with traffic control The basic building block of Linux traffic control is the queue discipline. While some queue disciplines are simple and consist of just one queue with a simple FIFO scheduling algorithm, other are much more complicated and of a different nature. A classful qdisc such as PRIO contains classes in which traffic is sent according to the priority encoded in the packet or using traffic control filters. Each class of such a queue discipline can contain another class or a qdisc. As these internal disciplines can be also classful, a powerful hierarchy of traffic queues, schedulers and classifiers can be built (Fig.3). Fig. 3. Traffic control hierarchy To configure traffic control elements from user space, the tc program is used. This command-line configuration application is part of the iproute2 package. The necessary information is passed to and from the kernel using the NETLINK sockets interface [3]. 3

5 3. LINK CHARACTERISTICS AND TRAFFIC CONTROL EFFICIENCY An important component of any QoS architecture is the resource reservation concept. In the core nodes of a DiffServ network for example, per-hop behaviors (PHBs) are implemented in order to decide the class of traffic to which a packet belongs and its corresponding treatment in that router. For some type of traffic such as voice or video data, the end-to-end delay must be kept to a minimum each node will try to reserve handling speed for these flows, usually by scheduling multimedia packets to high-priority, separate and short queues. In addition, a minimum, guaranteed rate has to be reserved in order to keep packet loss to an accepted level for certain data flows. In addition, when priorities are employed, important traffic has to be limited to a maximum service rate, or the lower priority packets have big chances to remain in their queues without being serviced for a long time. Minimum and maximum rate guarantees for different flows or flow aggregates form the basis of the link sharing concept. But in order to accomplish link sharing through resource reservation, the service rate of the outgoing link must be not only known, but of a relatively constant value. For example, consider a link being shared by traffic belonging to three different flows of different priorities. The QoS requirements state that each flow must have a guaranteed rate of one third of the link capacity. If the output rate is known and of constant value, a classful Hierarchical Token Bucket (HTB) queue discipline with 3 classes can be employed at a Linux router, with a total shaping rate slightly lower than the outgoing link rate. This limitation ensures that, when the arrival rate is bigger than the service rate, the queues of the HTB will fill up, and not the hardware queues (as device buffers cannot be configured). Each class of traffic has a different priority and a service rate (accomplished through usage of tokens) of one third of the total HTB service rate. This is the common usage in the case of wired links, for which the outgoing rate is of an approximate constant value. In the case of links with variable capacity, the situation is more complex. If the HTB total service rate is set to a value close to the maximum link capacity, when the link rate is lower, the packets from the high priority class can starve the lower priority flows, as they use a third of the maximum link capacity, and not one third of the actual value. In addition, there will be packets stationed into the hardware queue(s); depending on the hardware type, this may conflict with the QoS policy set in the traffic control subsystem. On the other hand, if the HTB total service rate is set to a smaller value, when the link rate goes higher than this value, some part of the link capacity is wasted as traffic is shaped to the HTB rate. This is often the case of wireless links where Linux traffic control is not so efficient, as it implements static QoS rules. Using fixed rate QoS rules for non-work-conserving schedulers in the presence of variable rate links is not an effective solution. A QoS mechanism that works well even when the quality of the outgoing link is variable is traffic prioritization. Employment of a classful queue discipline such as PRIO, without any traffic shaping, can ensure that important packets reach the hardware queue(s) before low priority packets leave the PRIO queues. In this case, there can not be no fixed service rate guarantees. 4

6 4. DEVICES WITH MULTIPLE HARDWARE QUEUES Traditionally, network adapters used only one Tx hardware queue, associated with a FIFO scheduling policy. In order to implement more grained QoS policies, a different set of queues and schedulers is created and managed above the device driver by means of traffic control mechanisms. The intention is to keep the hardware queue empty as much as possible and rely on the queue disciplines associated with the device by the router administrator. With the advent of network devices with multiple hardware queues, two interrelated problems arise. The first one is caused by a design limitation in the Linux network subsystem, while the second is more conceptual in nature and involves cross-layer QoS issues. A. Linux Network Design Limitation The general rules to be implemented by a normal network device driver in order to place packets on the hardware Tx queue are: When hardware queue is full, the network driver calls netif_stop_queue; The qdisc scheduler will stop sending packets to the network card; If dev->hard_start_xmit is called in these conditions, the dequeued packet is is not accepted, a NETDEV_TX_BUSY code is returned and packet must be requeued at the above layer; When actual transmission takes place and there is room for packets in the hardware queue, the driver calls netif_start_queue. If a driver does not call netif_stop_queue when the internal queue is full, CPU usage will rise drastically as the queue discipline set up for the device will dequeue and requeue packets continuously without any knowledge of the state of the hardware queue. A well-behaved device driver uses netif_stop_queue and netif_start_queue to provide feedback to the network scheduler. When the network adapter has multiple hardware queues, with QoS support at the hardware level, the method of providing feedback presented above is not sufficient. For example, consider a device with two Tx hardware queues, intended for packets with different priorities (Fig.4). In the situation when the low priority hardware queue is full, if the driver calls the netif_stop_queue method, the high priority packets will also be blocked, as the Tx queue is stopped. If the driver does not call netif_stop_queue, each time a low priority packet arrives, the driver has to refuse the packet, returning it to the qdisc where it will be requeued only to start over again and thus consuming CPU resources. 5

7 Fig. 4. Network adapter with two Tx hardware queues B. Cross-Layer QoS Issues If the network adapter implements any form of QoS at the hardware level, traditional traffic control disciplines must take this into account, or risk loosing some important benefits. For example, in the wireless case, the IEEE e offers higher chances of transmission for packets placed in the Voice Access Category hardware queue [4]. Lack of correlation between QoS policies implemented at different layers on the transmission path can nullify its effects. In the current Linux network subsystem implementation, the qdisc interface is not aware of multiple hardware queues. A.dequeue operation does not have enough information coming from the device driver, in order to know what skb to dequeue from the configured queue discipline. In [2], the authors propose a generic solution which involves adding support for multi-queue features to the Linux network schedulers implemented in various queue disciplines and also to the multiqueue devices drivers. One additional requirement is that the stack will have to handle traffic from multiqueue and non-multiqueue devices in a transparent manner, with little or no additional overhead. The first part of the implementation exposes the number of hardware queues present in the device to the net_device structure, and enables manipulation of each queue s state by using a new set of methods netif_{start stop wake}_subqueue. The crosslayer QoS coordination was implemented by the authors in the PRIO queue discipline, where the prio_dequeue operation checks that the hardware queue mapped to the skb priority is running or full, each time a packet has to be dequeued from the qdisc. Because this solution modifies core parts of the kernel network stack, it is still discussed on various kernel related mailing lists and it is not currently integrated into the standard kernel versions. In order to preserve the current network stack software unmodified, an alternative solution involves the creation of a new set of queue disciplines, specifically designed and tied to a particular multiqueue device driver. 6

8 5. IMPLEMENTATION AND PRELIMINARY RESULTS Based on the considerations presented in the previous sections, a testbed was set up in a Linux environment, in order to measure the performance of a multiqueue enabled wireless adapter with hardware QoS extensions. The network adapter tested was an SMC wireless card based on the Atheros chipset AR5212, with four internal hardware queues, one for each access class defined in the IEEE802.11e standard. This chipset is supported in Linux by the MadWifi WLAN driver [5]. The Linux distribution installed on the test client machine was Fedora Core 6, with a kernel based on version and the wireless card device driver (MadWifi version ) implemented as a kernel module. The client machine was configured as a WLAN station connected to a Linksys Access Point, at a statically configured rate of 1Mbps (Fig. 5). Fig. 5. Testbed Multiqueue Wireless Client sending traffic at 1Mbps As the network stack of standard kernels is not multi-queue aware, one of the first goals was to find out what is the device driver behavior when one of its queues is full. The first possibility was that the driver would call netif_stop_queue, although the other queues are still able to receive packets, while the other possibility was that the driver would not stop the device from receiving the packets, even if some (but not all) of its queues are full. The source code of the driver was modified in order to log its behavior at congestion. Traffic with different priorities was generated using an UPD command line application, in order to provoke link congestion. As a first result, to our surprise, the driver did not answer to congestion in a proper way: even if a queue was full, the returned status in its.hard_start_xmit method was NETDEV_TX_OK instead of NETDEV_TX_BUSY, and the packet was simply dropped without being returned to the qdisc buffers. This solution completely bypassed any level 3 nonshaping QoS schedulers, because packets were serviced to the hardware queues at a very high speed rate, not correlated with the actual transmission rate. For example, a FIFO or a PRIO queue discipline attached to the device would have a null effect in this conditions, as it will be always empty. This issue was addressed in the MadWifi 1428 Ticket [6], but was not included by default in our tested device driver. All the subsequent tests were done after applying this patch to the MadWifi kernel module. The UDP traffic generator was configured to send datagrams with two different priorities. In the first test, the queue discipline attached to the device was FIFO with a maximum queue length of 7

9 20 packets, while in the second test, the queue discipline was changed to PRIO with four classes of traffic, each class containing a FIFO queue with a maximum length of 10 packets. Preliminary test results show that in both cases the Madwifi device driver does not call netif_stop_queue when a queue is full. In these conditions the packet is refused and has to be returned at the configured queue discipline. Because the qdisc does not know some of the queues are full, it will try to dequeue the packet again, causing an excessive number of.requeue operations and thus consuming most of the CPU resources (Fig. 6, 7). As the UDP traffic generator uses the blocking sendto library function, even at congestion, only the low priority hardware queue was filled. Thus, the high priority packets were not requeued. Requeue Calls Per Packet Requeue Operations Time [ ms ] Fig. 6..requeue operations with FIFO qdisc, LOW priority traffic Requeue Calls Per Packet Requeue Operations Time [ ms ] Fig. 7..requeue operations with PRIO qdisc, LOW priority traffic 8

10 Mean Values Inter-Layer Delay [ms].requeue operations Queue occupation [skb] PFIFO PRIO qdisc qdisc LOW HIGH LOW HIGH Table 1. Mean values 6. CONCLUSIONS The paper described some of the shortcomings in the implementation of the network subsystem in the recent Linux kernels and device drivers, in conjunction with wireless network adapters with multiple hardware queues. Measurements performed showed that, although the high priority packets were delivered faster than the low priority packets, the number of.requeue operations is excessive. As a next step, subsequent tests will be performed with the driver modified to call netif_stop_queue when one of its queues is full. This would eliminate the.requeue problem, but will introduce head-of-line blocking for packets of different priorities than the priority mapped to the full hardware queue that is blocking communication. A further goal involves measuring rate, loss and delay performance of packets with different priorities, for a multiqueue-enabled kernel and driver, in order to enable cross-layer QoS interoperation. REFERENCES [1] K. Wehrle, F. Pählke, H. Ritter, D. Müller and M. Bechler, The Linux Networking Architecture: Design and Implementation of Network Protocols in the Linux Kernel, August [2] Z.Yi and P.P.Waskiewicz, Jr., Enabling Linux Network Support of Hardware Multiqueue Devices, Proceedings of the Linux Symposium, pp , June [3] A. Kuznetsov and S.Hemminger, Iproute 2: a collection of utilities for controlling TCP/IP networking and Traffic Control in Linux, [4] IEEE e, Wireless LAN Medium Access Control (MAC) Enhancements for Quality of Service (QoS), e Standard, 2005 [5] Multiband Atheros Driver for Wireless Fidelity, [6] MadWifi Ticket #1428, ath_hardstart returns 0 even if queue is full, [7] Linux kernel mailing list archive, [8] Linux NetDev, 9