Performance of Wireless IEEE e-Based Devices with Multiple Hardware Queues

Transcription

1 Performance of Wireless IEEE e-Based Devices with Multiple Hardware Queues Gabriel Lazăr, Virgil Dobrotă, Member, IEEE, Tudor Blaga, Member, IEEE 1

2 Agenda I. Linux Network Subsystem II. Linux Traffic Control III. Link Characteristics and Traffic Control Efficiency IV. Devices with Multiple Hardware Queues V. Implementation and Preliminary Results VI. Conclusion and Further Work 2

3 I. Linux Network Subsystem Efficient and flexible network subsystem implementation in the Linux kernel, due to the design of its data structures A socket kernel buffer skb wraps a packet during its processing lifetime at each layer inside the kernel space Network device (struct net_device) uniform interface between software-based protocols and network adapters All registered devices are connected into a doubly linked list One of the pointers stored in net_device -the queuing discipline (qdisc) associated with packets transmitted through this network card 3

4 I. Linux Network Subsystem (cont.) Packet transmission for devices with no qdisc attached (.enqueue pointer is NULL) Protocol instances of higher network layers call dev_queue_xmit(skb) Network device to be used is specified in skb->dev parameter The packet is put immediately on the hardware queue through method hard_start_xmit() 4

5 II. Linux Traffic Control Each device has normally associated a qdisc Advantages: additional buffer space (software queues) an entire kernel QoS implementation: custom schedulers, queue managers and detailed classification 5

6 II. Linux Traffic Control (cont.) qdisc the basic building block of Linux traffic control subsystem Simple one queue with FIFO scheduling Complex a classful qdisc contains classes where traffic is sent according to custom filters; each class can contain another class or qdisc => powerful hierarchy of queues and schedulers User space configuration: tc tool from the iproute2 package. 6

7 III. Link Characteristics and Traffic Control Efficiency Resource reservation important component in QoS architectures Minimum and maximum rate guarantees for flows basis of link sharing Link sharing example: HTB qdisc for three flows of different priorities A. Constant rate link Keep packets in software (configurable) queues through total rate limitation (shaping) Packets from each flow have tokens for 1/3 of the total rate B. Variable rate link Important flows can starve low priority flows Packets are not confined to software queues may conflict with configured QoS Linux traffic control limitation: static QoS rules do not work well for links with variable capacity Shaping is hard to obtain with statically configured rates Device hardware queue not empty 7

8 CW max IV. Devices with Multiple HW Queues Traditionally, network adapters use one Tx hardware queue with FIFO scheduling: QoS policies are configured at higher layers e.g. Linux traffic control components Hardware queues empty rely on queue disciplines New problems if devices have multiple HW queues: Linux network design limitation Cross-layer QoS issues 8

9 IV. Devices with Multiple HW Queues (cont.) Linux Network Design Limitation General network driver rules: When HW queue is full, driver calls netif_stop_queue Scheduler of associated qdisc stops sending packets to the adapter If dev->hard_start_xmit is still called, the packet is refused by the adapter and has to be requeued at the above layer; returned status code is NETDEV_TX_BUSY After transmission takes place, HW queue not full anymore driver calls netif_start_queue. Design Limitation: if driver has multiple HW queues, feedback is not sufficient Example: Device with two HW queues, for packets of different priorities; if one of the queues is full, should the driver call netif_stop_queue? 9

10 IV. Devices with Multiple HW Queues (cont.) Cross-Layer QoS Issues If hardware QoS is present on device, it should be used. E.g. Wireless devices with IEEE e support important packets have higher chances of transmission Lack of correlation between QoS policies implemented at different layers can nullify its effects Linux traffic control is not aware of multiple hardware queues and Layer-2 QoS: the qdisc.dequeue operation does not have enough information from the device driver, in order to find out what skb to extract from its buffers Possible solutions Generic: add multiqueue support to the Linux network subsystem and to the device drivers: Expose the hardware queues to the net_device structure Enable manipulation of each queue s state e.g. netif_stop_subqueue Local: creation of queue disciplines designed and tied to a particular multiqueue device driver 10

11 V. Implementation and Preliminary Results Testbed Linux Fedora Core 6, kernel version SMC wireless network adapter based on Atheros chipset (AR5212) with 4 HW queues for each IEEE AC MadWifi WLAN driver version Linksys Access Point Transmission rate statically configured at 1Mbps 11

12 V. Implementation and Preliminary Results (cont.) Goal device driver behavior when one of the queues is full (is netif_stop_queue called or not) Modified driver - log internal state UDP traffic generator packets with 2 priorities, link congestion First result: driver will not refuse packets even if queue is full (return code: NETDEV_TX_OK) => Linux qdisc (e.g. FIFO, PRIO) completely bypassed; if queue is full, packet is simply dropped before leaving the source node Correct MadWifi patch was issued 3 months ago, but not included in standard version (Ticket #1428) Driver was modified to return NETDEV_TX_BUSY if queue full, for further experiments Packets are now refused when HW queue full, and have to be requeued by attached qdisc 12

13 V. Implementation and Preliminary Results (cont.) Preliminary test results Driver does not call netif_stop_queue when a HW queue is full => excessive number of.requeue operations, very high CPU usage Only low priority packets were sent back at Network Layer First test: PFIFO qdisc attached Second test: PRIO qdisc attached Mean Values PFIFO qdisc PRIO qdisc LOW HIGH LOW HIGH Inter-Layer Delay [ms] requeue operations Queue occupation [skb]

14 V. Implementation and Preliminary Results (cont.) PFIFO qdisc Requeue Calls Per Packet Requeue Operations Time [ ms ] 14

15 V. Implementation and Preliminary Results (cont.) PRIO qdisc Requeue Calls Per Packet Requeue Operations Time [ ms ] 15

16 VI. Conclusion and Further Work The paper describes several shortcomings of the Linux network subsystem for network adapters with multiple HW queues Design limitation Cross-layer QoS issues Measurements show that, for a popular wireless driver (MadWifi), behavior is not entirely correct excessive number of.requeue operations. Correct solution may not be possible without a generic kernel modification in the network subsystem. Next steps: Driver will be further modified in order to call netif_stop_queue if a HW queue is full. This would eliminate the.requeue problem, but will introduce head-of-line blocking. Measurements of rate, loss and delay for packets of different priorities, for a multiqueue enabled kernel and driver Solutions for cross-layer QoS interoperation 16

17 Thank you! 17