A Network-centric TCP for Interactive Video Delivery Networks (VDN)

Transcription

1 A Network-centric TCP for Interactive Video Delivery Networks (VDN) MD Iftakharul Islam, Javed I Khan Department of Computer Science Kent State University Kent, OH 1 / 44

2 Outline 1 Interactive Video Network 2 Why TCP? 3 Delay-based TCP 4 Network-centric TCP 5 Congestion Control Stability Analysis Encoder Rate Control 6 Implementation 7 Experimental Setup 8 Evaluation Single-bottleneck Topology Self-fairness RTT-fairness Multiple bottlenecks topology Differential Fairness Visual Interruptions 9 Data Plane Implementation Software Defined Network 2 / 44

3 Interactive Video Network Interactive video streaming in the Internet is expected to be TCP-friendly. Loss-based TCP flows results long queuing delay on the router (Bufferebloat problem) Solution: Differentiated service enable us to create exclusive video network where interactive video flows competes among themselves. We have designed a TCP for Interactive Video Network. 3 / 44

4 Why TCP? Video Streaming traditionally uses UDP. But UDP has many disadvantages: It does not have any congestion control. Application developers needs to implement their own congestion control. Different implementation of congestion control interacts poorly with each other and stability of the network is compromised. UDP is also not NAT and firewall friendly which are highly desirable attributes in today s Internet 4 / 44

5 Why TCP? TCP solves these problems, however TCP has some problems regarding interactive video streaming: The ordered delivery of TCP delays some segment delivery if a prior segment is missing (head-of-line blocking problem ). Retransmission is also not required in interactive video streaming These problems can be solved by implementing different TCP implementation. For example: TCP Hollywood allows unordered segment delivery. Retransmission can be avoided by extensive packet caching in the router. 5 / 44

6 Delay-based TCP Video streaming traditionally uses delay based congestion control. It modifies the congestion window based on RTT. TCP Vegas, TCP FAST RTT based TCP loses throughput in the presence of reverse-path congestion. To solve this problem, LEDBAT uses one-way delay based congestion control. LEDBAT however suffers latecomer effect: second flow may starve first flow. One-way delay gradient has been used to overcome the late comers effect. TCP CDG, TCP Inigo Google Hangout (doesnt use TCP though) 6 / 44

7 Delay-based TCP Delay-based TCP controls the congestion window based on one-way delay observed in the receiver. It has no way of knowing about the actual queuing delay in the network. This is why, it faces several problems: It cannot result near-zero queuing delay in the network. It performs poorly in fairness. This is why some flows experience poor throughput which results poor video quality in video streaming. Recent advances in Software Defined Networks (SDN) can solve this problem 7 / 44

8 Network-centric TCP Figure: Work flow of NC-TCP 8 / 44

9 Network-centric TCP Routers play an active role in allocating throughput. NC-TCP uses rate-based congestion control rather than window based congestion control. Window-based TCP produces bursty traffic. Rate-based congestion control shape the traffic before sending It also ensures that the network does not have more traffic than it is able to handle. We used TCP-pacing NC-TCP has been implemented as a new TCP option. Figure: NC option 9 / 44

10 NC-TCP Workflow Senders specify the required throughput in SYN and DATA segment. Routers along the path inspect the option header and allocate the feedback throughput. A router will set the feedback throughput only when the feedback throughput is smaller than the feedback throughput allocated by a previous router. It ensures that feedback throughput is calculated based on the most congested link along the path Receiver copies the feedback throughput in the ACK segment. Sender sets the sending rate based on the ACK segment. 10 / 44

11 Feedback Throughput Calculation We have divided the feedback controller into two parts: Delay Controller: Proportional Integral (PI) controller Fairness Controller: Min-max fairness Delay Controller ri (t) = α[c(t) q(t)] (1) where the r i (t) is the aggregate feedback throughput, c(t) is the bottleneck link capacity, q(t) is the queue length in bytes and α is the coefficient. Fairness Controller r i (t) = α[c(t) q(t)] N(t) where r i (t) is the feedback throughput for flow i and N(t) is the number of flows (2) 11 / 44

12 Stability Analysis Let us assume that the feedback delay is t f As the feedback rate becomes the sending rate after t f,the sending rate at time t can be written as x i (t) = α[c q(t t f )] N The queuing-delay gradient can be represented as (3) q(t) = x i (t) C (4) As flows adapt their sending rate x i (t) at the same rate, xi (t) = Nx i (t) for all i. q(t) = Nx i (t) C (5) q(t) = αq(t t f ) + C(α 1) (6) 12 / 44

13 Stability Analysis q(t) = αq(t t f ) + C(α 1) is an autonomous differential equation The autonomous system is stable if q(t ) < 0 where q(t ) represents the queuing delay at the equilibrium point This is why the system is stable if α > 0. If the system is perturbed, the queue will be drained at αq(t t f ) + C(α 1) rate and eventually reaches the equilibrium point At the equilibrium point, q(t) = 0. So we get q(t t f ) = C(α 1) α (7) C(α 1) α is the queuing delay at the equilibrium point. Here we want the queuing delay to be be zero while maximizing the throughput. That is why we set α = / 44

14 Encoder Rate Control The encoding rate of video cannot be changed quickly in order to avoid congestion (takes more than 500ms) The actual encoder output rate also fluctuates randomly around the input target rate. NC-TCP based application sets the target rate based on the feedback rate of the network. As NC-TCP uses TCP-pacing, packets wait in the TCP s sending buffer (congestion window) to be scheduled to sent out RTT produced by the NC-TCP is not exactly same as the propagation delay RTT /RTT min indicates how much encoder has overshot with respect to the network throughput. Here RTT min is the minimum RTT which is considered as the propagation delay We set the encoder target rate as R t (t) = feedback throughput r i (t) RTT RTT min where r i (t) is the 14 / 44

15 NC-TCP Implementation We have implemented NC-TCP in Linux kernel. Source: NC-TCP implementation has two parts: NC-TCP host and NC-TCP router We have modified the Linux TCP stack to implement the NC-TCP host. We have implemented the NC-TCP router as a Qdisc kernel module in Linux router (ip forward=1). We also modified the kernel to trace system variables such as throughput, queue length, and so on. 15 / 44

16 Video Streaming Application We also developed a TCP based video streaming application based on GStreamer. (350 lines of C code) The application reads a video file, encode in H.264 constant bit rate. The bitrate is set by the feedback rate received from the TCP stack. It uses It uses getsockopt system call. The application stream the video frames to the client. Client displays the frames as soon as it receives it. 16 / 44

17 Experimental Setup Figure: Topology The topology is created in Mininet. All the nodes in the picture is a Linux container (light-weight VM) All the links are virtual ethernet (veth) pair The delay and throughput on the links are set by NetEm and htb (hierarchical token bucket) Qdisc. The setup takes 180 lines of Python code. 17 / 44

18 Experiment Figure: Topology We streamed a clip from Big Buck Bunny in our experiment. The sender produces 512X340 video at 30 fps in H.264 format The three senders stream video simultaneously to the three receivers The experiment is conducted for 80 seconds. 18 / 44

19 Evaluation Loss based TCP such as TCP CUBIC results long queuing delay. Recently proposed Google BBR also results in long queuing delay. We compared NC-TCP to TCP Inigo and XCP We found TCP Inigo is a representative of one-way delay gradient based TCP. The workflow of XCP is similar to NC-TCP. But it has been designed for short-lived flows. We have found the implementation TCP CUBIC, TCP BBR and TCP CDG in Linux kernel. The Linux kernel implementation of TCP Inigo is also shared by it s author. We also have implemented the XCP in Linux kernel. We compared the queuing delay on the bottleneck router for different protocols. We also have compared the fairness for each protocol. Note that higher fairness indicates that all the flows get higher throughput from the bottleneck link 19 / 44

20 Bottleneck Queuing Delay Single-bottleneck Topology The queuing delay is q(t) B where B is the bottleneck throughput. 20 / 44

21 Throughput Single-bottleneck Topology Figure: Throughput 21 / 44

22 Fairness Index Single-bottleneck Topology The fairness index has been calculated using the formula ( n i=1 x i ) 2 n n i=1 x i 2 where x i is the throughput of flow i and n is the total number of flows 22 / 44

23 Round-trip time Single-bottleneck Topology Figure: RTT 23 / 44

24 Self-fairness Three TCP Inigo flows 15 seconds apart Figure: Three TCP Inigo flows 15 seconds apart 24 / 44

25 Self-fairness Three NC-TCP flows 15 seconds apart Figure: Three NC-TCP flows 15 seconds apart 25 / 44

26 RTT-fairness Figure: Topology We modified the topology such that each flow has different RTTs: 80ms, 100ms and 120ms 26 / 44

27 Fairness Index RTT-fairness Figure: Fairness Index 27 / 44

28 Queuing delay RTT-fairness Figure: Queuing delay in the bottleneck router 28 / 44

29 Multiple bottlenecks topology Figure: The throughput of bottleneck and access links are 660kbps and 1024kbps respectively 29 / 44

30 Queuing delay Multiple bottlenecks topology 30 / 44

31 Fairness Index Multiple bottlenecks topology 31 / 44

32 Differential Fairness Interactive VDN needs to support heterogeneous endpoints such as smart phones, laptops, TV and VR headset at the same time The videos used in different endpoints often differ in encodings, resolutions and frame rates. Network-centric congestion control has a major advantage over host-centric approach in this regard As a router is aware of the throughput requirement of each flow passing through it, it can apply differential or weighted fairness in throughput allocation 32 / 44

33 Visual Interruptions Table: Number of visual interruptions Single bottleneck Multiple bottleneck topology topology TCP Inigo 2 20 XCP 0 0 NC-TCP / 44

34 Data Plane Implementation Currently NC-TCP has been implemented in Linux Kernel to be executed on a CPU. Implementation of NC-TCP on a programmable switch will be needed in order for real world deployment. Software Defined Network (SDN) enables to implement NC-TCP on the router The main idea behind SDN is programmable switching data-plane. Let s take a closer look into SDN! 34 / 44

35 Software Router CPU can forward packets at 40Gbps. SDN router uses DPDK for packet processing and forwarding. DPDK stands for Data Plane Development Kit. DPDK is a technology that bypasses the kernel and enables to process packet in application layer. DPDK application can be programed using BPF framework. BPF stands for Binary packet Filter BPF program can be written in C or P4 Although software router provides 40Gbps throughput, its very small compared to hardware switch which provides several terabytes per second. 35 / 44

36 Is Software Forwarding Scalable? 100Gb NIC results in 8.4 Mpps packets /(8 1500) 119ns per packet 238 cpu cycles for a 2GHz Xeon core But packet forwarding is IO intensive, not CPU intensive. For example, IPv4 forwarding requires at least 2 memory access. Memory access requires cpu cycles This problem gets much worse for smaller packets. Software Router cannot match the speed of hardware (40Gb/s vs several Tb/s). Solution? Software Processing 36 / 44

37 Software Packet processing Do the packet forwarding in hardware. CPU cannot simply beat it. Do the packet processing such as header manipulation on CPU. This is why we need a programmable switch that has both a switch fabric and CPU. The switch fabric is responsible for packet forwarding CPU is responsible for packet processing. 37 / 44

38 Programmable Switch Forwarding is done by a switch fabric in the switch. IP lookup and forwarding is done by a TCAM. TCAM offers O(1) lookup time. Each port is connected to the switch fabric through a BPF CPU. The BPF CPU differs from the control plane CPU. It only supports limited instruction set (often BPF instruction set). The BPF CPU is exposed to control plane CPU via memory mapped IO (MMIO). Control plane CPU can load a BPF program on to the 38 / 44

39 Data-plane Programming The programming environment depends on switch manufacturer. Switch manufacturer provides a proprietary SDK that provides very limited programming capability to others. Recently P4 and BPF programming framework have been developed that provides a standardized way to program the data-plane. NC-TCP data-plane needs to modify packets based on queue length and number of flows. P4 does not have those information. ebpf (extened Binary packet Filter) is a framework where queue length and number of flows are accessible through meta-data. This is why we are planning to use ebpf to program NC-TCP router. Currently there is no ebpf supported hardware switch to the best of our knowledge. It s mostly used in virtual switches. 39 / 44

40 SDN Vendors Switch Vendor Intel, Mellanox, Brodcom, Cavium, Barefoot, Marvell Appliance Vendor Cisco, Juniper, Arista, Brocade, 6Wind, Huawei, Sophos Cloud providers also developing their own appliance. This include RedHat, SUSE, Intel, Facebook, Google, IBM, VMWare, Cumulus, Amazon, Microsoft, etc The switch manufactures provides a proprietary SDK to utilize the switch. But closed source nature of SDK refrains appliance vendor and individual developers to program the switch. 40 / 44

41 Open Source SDN Recently people from Mellanox and Intel came forward to solve this problem. A switch driver model switchdev has been proposed to replace the proprietary SDK. Mellanox have few hardware switches based on switchdev. However currently they do not support ebpf program Mellanox however expressed interest to support ebpf offloading in the future. We however can use virtual switch to develop and analyze ebpf program 41 / 44

42 Packet Processing at Line-rate It has shown that a programmable switch can forward packets at line-rate if each port can process packets at line-rate Most of the switches come with 10Gb/s and 40Gb/s port This is why ebpf program needs to process packets at 40Gb/s. 42 / 44

43 Line-rate NC-TCP We are planning to develop a ebpf program to implement NC-TCP. Analyzing the Instructing set of the ebpf program, we can find out how powerful CPU is needed to achieve line-rate. 43 / 44

44 Thank You 44 / 44