End-to-End Delay Modeling for Embedded VNF Chains in 5G Core Networks

Transcription

1 End-to-End Delay Modeling for Embedded VNF Chains in 5G Core Networks Qiang Ye, Member, IEEE, Weihua Zhuang, Fellow, IEEE, Xu Li, and Jaya Rao Abstract In this paper, an analytical end-to-end E2E packet delay modeling is established for multiple traffic flows traversing an embedded virtual network function VNF chain in fifth generation 5G communication networks. The dominantresource generalized processing sharing DR-GPS is employed to allocate both computing and transmission resources among flows at each network function virtualization NFV node to achieve dominant-resource fair allocation and high resource utilization. A tandem queueing model is developed to characterize packets of multiple flows passing through an NFV node and its outgoing transmission link. For analysis tractability, we decouple packet processsing and transmission of different flows in the modeling and determine average packet processing and transmission rates of each flow as approximated service rates. An M/D/ queueing model is developed to calculate packet delay for each flow at the first NFV node. Based on the analysis of packet inter-arrival time at the subsequent NFV node, we adopt an M/D/ queueing model as an approximation to evaluate the average packet delay for each flow at each subsequent NFV node. The queueing model is proved to achieve more accurate delay evaluation than that using a G/D/ queueing model. Packet transmission delay on each embedded virtual link between consecutive NFV nodes is also derived for E2E delay calculation. Extensive simulation results demonstrate the accuracy of our proposed E2E packet delay modeling, upon which delay-aware VNF chain embedding can be achieved. Index Terms NFV, SDN, embedded VNF chains, E2E delay modeling, tandem queueing model, CPU and bandwidth resources, bi-resource allocation, DR-GPS, rate decoupling. I. INTRODUCTION Future communication networks are expected to provide customized delay-sensitive end-to-end E2E service deliveries [] e.g., video streaming, machine-to-machine communications with fine-grained quality-of-service QoS for Internet-of-Things IoT [2] [6]. Typical IoT application scenarios include remote control for smart homing, smart sensing [7], large-scale mobile social networking [8], industrial automation [9], high-definition video conferencing, and intelligent transportation systems [0], []. To support diversified applications and use cases, network servers providing different functions e.g., classifiers, firewalls, and proxies are required to be augmented in a large scale to fulfill customized service requirements in both wireless and core network domains. However, the increasingly densified network Qiang Ye and Weihua Zhuang are with the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada, N2L 3G s: {q6ye, wzhuang}@uwaterloo.ca. Xu Li and Jaya Rao are with Huawei Technologies Canada Inc., Ottawa, ON, Canada, K2K 3J s: {Xu.LiCA, jaya.rao}@huawei.com. Copyright c 202 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieee.org. deployment significantly expands installation and operational cost of the infrastructure. Network function virtualization NFV [2] [5] provides a promising solution to reduce the deployment cost and realize flexible function placement and service customization. With NFV, network functions are decoupled from function-specific servers, softwarized as virtual network functions VNFs, and placed on general-purpose programmable servers also called NFV nodes [3], [6]. Specifically, through a resource virtualization platform [7], computing resources i.e., CPU cores for task processing on each NFV node are virtualized as virtual CPU vcpu cores, upon which virtual machines VMs are installed. Then, VNFs are programmed on VMs of different NFV nodes at different network locations to achieve high resource utilization. For the evolving network paradigm, the backbone core network consists of a combination of network switches and NFV nodes interconnected via high-speed wired transmission links. NFV nodes hosting and operating VNFs are introduced to improve service provisioning and resource utilization. A set of VNFs and the virtual links connecting them constitute a logic VNF chain, referred to as service function chain SFC [8], representing a specific sequence of network functions that a traffic flow requires to traverse for E2E service provisioning. All VNF chains are managed by a virtualization controller and are placed onto the physical substrate network, with each VNF embedded on an NFV node and virtual links represented by transmission links and network switches. This process is known as VNF chain embedding [3], [4], [9], [20]. Note that the virtualization controller is software-defined networking SDN enabled [2], with all control functions decoupled from the underlying physical network [22], [23]. Therefore, the controller has direct control programmability on all VNFs to enhance resource utilization via traffic balancing and VM migration [24], [25]. E2E packet delay of a delay-sensitive service flow traversing an embedded VNF chain is a main metric indicating the embedding performance. Exsiting research works investigate how to achieve optimal embedding of VNF chains on the core network to minimize the deployment, operational, and delay violation cost under physical resource constraints and flow conservation constraints. E2E packet delay of each traffic flow is calculated as a summation of packet transmission delays on each physical link, without considering packet processing delay due to CPU processing on NFV nodes [3], [4], [26]. However, each packet from different traffic flows passing A traffic service flow refers to an aggregation of packets belonging to the same service type for the same source and destination node pair in the backbone network.

2 2 through an NFV node usually requires different amounts of CPU processing time on the NFV node and packet transmission time on the outgoing link in sequence [27], [28]. Depending on the type of VNF each flow traverses, some flows packets with large headers bottleneck on CPU processing, whereas packets of other flows having large packet payload sizes demand more transmission time. In addition, the packet arrival process of a flow at an NFV node correlates with packet processing and transmission at its preceding NFV nodes, which makes E2E delay analysis difficult. Therefore, how to develop an analytical model to evaluate the delay that each packet of a flow experiences when passing through an embedded VNF chain, including packet queueing delay, packet processing delay on NFV nodes, and packet transmission delay on links, is a challenging research issue. For VNF chain embedding, there is a tradeoff between E2E delay satisfaction and reducing the network cost. Different VNF chains are often embedded on a common network path with multiple VNF instances operated on an NFV node to improve resource utilization and reduce the deployment and operational cost of network functions and links. On the other hand, sharing a set of physical resources with other flows on NFV nodes and links may degrade the delay of one individual flow. Thus, modeling E2E packet delay of each flow is essential to achieve delayaware VNF chain embedding. Since traffic flows traversing each NFV node demonstrate discrepant dominant resource consumption on either CPU or link bandwidth, how to allocate the two resources among traffic flows to guarantee allocation fairness and achieve high resource utilization, referred to as bi-resource allocation, needs investigation and is required for E2E delay modeling. In this paper, we employ dominant-resource generalized processor sharing DR-GPS [28] as the bi-resource allocation scheme among traffic flows sharing resources at each NFV node. The DR-GPS balances the tradeoff between service isolation and high resource utilization, and maintains dominantresource fairness [29] among flows. Then, we model packet delay of a flow passing through NFV nodes and physical links and switches of an embedded VNF chain. The contributions of this paper are two-folded: With DR-GPS, we establish a tandem queueing model to extract the process of each flow going through CPU processing and link transmission at the first NFV node. To remove the rate coupling effect among flows, we derive average processing rate as an approximation on instantaneous processing rate for each flow with the consideration of resource multiplexing. An M/D/ queueing model is then developed for each decoupled processing queue to determine the average packet processing delay. Based on the analysis of packet departure process from each decoupled processing queue, the decoupled transmission rate is derived for each flow. 2 We analyze packet arrival process and then remove rate coupling effect of each flow traversing the subsequent NFV node. To eliminate the dependence of packet processing and packet transmission between consecutive NFV nodes, the arrival process of each flow at the subsequent NFV node is approximated as a Poisson process and an M/D/ queueing model is employed to calculate the average delay for packet processing. It is proved that the average packet queueing delay for CPU processing based on the approximated M/D/ queueing model is an improved upper bound over that calculated upon a G/D/ queueing model. The rest of the paper is organized as following sections. Existing studies on E2E packet delay modeling for embedded VNF chains are reviewed in Section II. The system model under consideration is described in Section III. In Section IV, we present a bi-resource allocation scheme employed for flows traversing an NFV node. The end-to-end delay modeling for packet flows going through an embedded VNF chain is established in Section V. In Section VI, numerical results are presented to demonstrate the accuracy of the proposed delay analytical framework and its effectiveness for achieving delayaware VNF chain embedding. Lastly, we draw the conclusions in Section VII. Main parameters and symbols throughout this paper are summarized in Table I. II. RELATED WORK In most existing studies, E2E packet delay for an embedded VNF chain is modeled as a summation of transmission delays when every packet of a flow traverses each embedded physical link, without considering packet processing delay associated with VNFs. In [3] and [4], delay violation penalty is expressed as a function of E2E packet transmission delay for each traffic flow, which indicates the cost if the delay requirement for packets traversing an embedded VNF chain is violated. A delay violation cost minimization problem is then formulated to determine an optimal embedded physical network path that achieves minimum E2E packet transmsission delay for each flow. In [26], the total delay when a packet is routed between consecutive virtual nodes consists of packet processing delay, packet queueing delay, and packet transmission delay, and is determined using network traffic measuring tools instead of analytical modeling. The E2E delay for packets going through each source-destination S-D node pair in an embedded virtual network is then calculated to achieve QoS-aware multicast virtual network embedding. For embedded VNF chains, how CPU and bandwidth resources are shared among multiple flows traversing each NFV node determines the fractions of processing and transmission rates allocated to each flow, and thus affects E2E packet delay calculation. Multi-resource sharing is studied in a data center DC environment [27] [29], where dominant-resource fair allocation is adopted to equalize the dominant resource shares of multiple flows, while achieving high resource utilization. However, how to model E2E packet delay for each flow traversing a set of DC nodes needs further investigation [28]. An analytical E2E delay model for packets passing through each embedded VNF chain is established in [30], where an independent M/M//K queueing model is employed to evaluate packet delay at each VNF. However, packet inter-arrival time at a subsequent VNF correlates with the processing and transmission rates at its preceding VNF, making packet delay

3 3 TABLE I: Main parameters and symbols Symbol Definition I Set of traffic flows traversing a common embedded physical network path N z z th NFV node along one embedded VNF chain n z Number of network switches and physical links between N z and N z+ λ i Packet arrival rate of traffic flow i at N τ i, /τ i,2 Time consumption of each packet of flow i for CPU processing/link transmission at N C i, /C i,2 Maximum packet processing/transmission rate allocated to flow i at N c i, /c i,2 Allocated packet processing/transmission rate out of C i, /C i,2 to flow i h i, /h i,2 Fraction of CPU/link bandwidth resources allocated to flow i at N B One of the flow combinations for a backlogged flow set out of I µ i, /µ i,2 Decoupled packet processing/transmission rate for flow i at N ϱ i, Non-empty probability of flow i s processing queue at N µ i, /µ i,2 Decoupled packet processing/transmission rate for flow i at N 2 ϱ i, Non-empty probability of flow i s processing queue at N 2 Y i Inter-departure time of successive packets of flow i from the decoupled processing at N Z i Packet inter-departure time for flow i passing though link transmission at N D i,z Average packet delay for flow i passing through z th NFV node D f i,z Total packet transmission delay for flow i traversing n z switches and links calculation unique at each VNF. In addition, the assumption of exponential distribution on packet processing time needs justification. Existing studies establish analytical delay models for packets of a traffic flow going through a set of OpenFlow network switches in SDN [3], [32]. Since control functions are migrated from each network switch to the SDN controller where processing resources are consumed for making routing decisions, the network switches are simplified with only packet forwarding functions. E2E packet delay for a flow passing through a sequence of network switches is determined based on an M/M/ queueing network modeling. Overall, developing an accurate analytical E2E delay model for each traffic flow traversing an embedded VNF chain is challenging and of importance for achieving delay-aware VNF chain embedding. A. Embedded VNF Chains III. SYSTEM MODEL An aggregated data traffic flow from the wireless network domain, belonging to an identical service type, is required to traverse a sequence of VNFs in the core network to fulfill certain service requirements. Each VNF is embedded and operated on an NFV node, and each virtual link represents a set of transmission links and network switches. Multiple VNF chains can be embedded on a common network path to improve resource utilization and reduce network deployment and operational cost. We consider a set, I, of traffic flows traversing different logic VNF chains over a common embedded physical path. In Fig., two flows i and j I, representing two logic VNF chains f f 3 and f f 2, respectively, traverse one embedded network path and share the same physical resources. At the service level, we have flow i traversing a firewall function and a domain name system DNS function sequentially to fulfill a secured DNS service request, and flow j traversing a firewall function and an intrusion detection system IDS for secured end-to-end data streaming. At the network level, flow i goes through the first NFV node N operating VNF f and transmission link L 0, and are then forwarded by n network switches {R,..., R k,..., R n } and n transmission links {L,..., L k,..., L n } in between before reaching the second NFV node, N 2, operating VNF f 3 ; Flow j traverses the same physical path but passes through VNF f 2 at the second NFV node N 2. After passing through N 2, traffic flows i and j are forwarded by a sequence of n 2 switches and n 2 links not depicted in Fig. for brevity to reach the destination node in the core network. For a general case, we have the set I of flows traversing and sharing an embedded physical path, with m NFV nodes denoted by N z z =, 2,..., m and n z pairs of network switches and physical links forwarding traffic between NFV nodes N z and N z+ before reaching the destination node. With NFV, different VNFs can be flexibly orchestrated and installed at appropriate NFV nodes to enhance traffic balancing and reduce deployment cost of network infrastructure. When a traffic flow passes through an NFV node, each packet of the flow first requires a CPU time for packet processing, after which the processed packet is allocated link bandwidth resources for transmission [28]. The total amount of CPU time is assumed infinitely divisible [27], [28] on each NFV node and needs to be properly shared among traffic flows passing through, and the bandwidth resources on transmission links are also shared among traffic flows. B. Traffic Model Packet arrivals of flow i at the first NFV node, N, are modeled as a Poisson process with arrival rate λ i. For resource requirements of a flow, we define time profile for flow i passing through N as a two dimensional time vector [τ i,, τ i,2 ], indicating that every packet of flow i requires τ i, time for CPU processing and τ i,2 time for transmission if all CPU time on N and link bandwidth resources on L 0 are allocated to flow i [28]. Correspondingly, the rate vector [C i,, C i,2 ] in the unit of packet per second for flow i is the reciprocal of the time profile, C i, = τ i, and C i,2 = τ i,2. Service flows with different packet structures often have discrepant time profile when passing through an NFV node. For example, when going through a firewall function, service flows carrying small packets with a large header size, such as DNS request

4 4 Flow i Flow j Flow i Flow j Switch NFV node Firewall IDS DNS Transmission link Programmable connection Traffic flow direction Fig. : Two embedded VNF chains sharing a common physical network path. packets, are more CPU time demanding, whereas other data traffic flows, e.g., video traffic, with a large packet size will require more time for packet transmissions but less time on packet processing. Therefore, service flows always have a more critical resource consumption on either CPU time or link bandwidth, which is referred to as dominant resource. With a given set of maximum available CPU resources and link bandwidth resources on an NFV node and its outgoing link, time profile for different traffic flows sharing one NFV node can be very different. The maximum available CPU time on N and maximum link bandwidth on L 0 are shared among flows. Suppose flow i is allocated packet processing rate c i, out of the maximum processing rate C i,, and packet transmission rate c i,2 out of the maximum transmission rate C i,2. Note that when the CPU time is shared among multiple flows, there can be some overhead time as the total CPU resources switch among flows for different processing tasks. Existing studies show that this CPU switching overhead only becomes obvious when the traffic of each flow is saturated with a high percentage of CPU utilization i.e., CPU cores are frequently interrupted for switching tasks [8], [33]. Considering only non-saturation traffic, we assume that the allocated processing rate c i, for flow i I varies linearly with its occupied fraction of CPU time i.e., the useful fraction of CPU usage. Thus, we denote the fraction of CPU resources allocated to flow i by h i, bandwidth resources by h i,2 = ci,2 = ci, C i, C i,2. and the fraction of link IV. BI-RESOURCE ALLOCATION DISCIPLINE A. Dominant-Resource Generalized Processor Sharing When different service flows multiplex at a common NFV node, we want to determine how CPU and link bandwidth resources should be shared among the traffic flows to achieve high utilization on each resource type and, at the same time, maintain a weighted fair allocation among the services. Since service flows can have different dominant resources, the biresource sharing becomes more challenging than single resource allocation, to balance between high resource utilization and fair allocation for both resource types. The generalized processor sharing GPS discipline is a benchmark fluid-flow i.e., with resources being infinitely divisible based single resource allocation model for integrated service networks in the traditional communication networks [34], [35]. Each service flow, say flow i, at a common GPS server e.g., a network switch is assigned a positive value, ϕ i, indicating its priority in bandwidth allocation. The GPS server guarantees that the allocated transmission rate g i for flow i satisfies g i ϕ i G ϕ i i I where G is the maximum service rate of the GPS server. Note that the inequality sign in holds when some flows in I do not have packets to transmit, and thus more resources can be allocated among any backlogged flows. Therefore, GPS has the properties of achieving both service isolation and a high resource multiplexing gain among flows. However, if GPS is directly applied in the bi-resource context i.e., bi-resource GPS, it is difficult to simultaneously maintain fair allocation for both CPU time and link bandwidth and achieve high system performance. Consider flow i and flow j, with time profiles [τ i,, τ i,2 ] and [τ j,, τ j,2 ] respectively, traversing NFV node N, with the same service priority for fair resource sharing. Assume τ i, > τ i,2 and τ j, < τ j,2. If we apply the bi-resource GPS, both the maximum processing and transmission rates are equally divided for the two flows. Consequently, the performance of both flows traversing N is not maximized: For flow i, due to unbalanced time profiles, the allocated link transmission rate c i,2 is larger than the processing rate c i,, leading to resource wastage on link transmission; The situation reverses for flow j, where packets are accumulated for transmissions, causing an increase of queueing delay. Therefore, to improve the system performance, a basic principle [28] is that the fractions, h i, and h i,2, of CPU and bandwidth resources allocated to any flow i I should be in the same proportion as its time profile, i.e., hi, h i,2 = τi, τ i,2, to guarantee the allocated processing rate be equalized with the transmission rate, i.e., c i, = c i,2. In such a way, queueing delay before packet transmissions of each flow can be eliminated. However, with this basic principle, if we apply GPS on one of the two resources i.e., single-resource GPS with equalized processing and transmission rates, the allocation of the other type of resources among traffic flows continues to be unbalanced due to the discrepancy of time profiles among different flows. To maximize the tradeoff between high performance and fair resource allocation for each traffic flow, we employ a

5 5 dominant-resource generalized processor sharing scheme [28] for the bi-resource allocation. The DR-GPS combines the concepts of dominant resource fairness DRF [29] and GPS, in which the fractions of allocated dominant resources among different backlogged flows are equalized based on service priority, and the other type of resources are allocated to ensure the processing rate equal transmission rate for each flow the basic principle applies. When some flows do not have backlogged packets for processing, their allocated resources are redistributed among other backlogged flows if any. Since there are a finite number of flow combinations to form a backlogged flow set out of I, we denote B I as one of the flow combinations for a backlogged flow set. Each backlogged flow has a dominant resource consumption in either packet processing or packet transmission. We mathematically formulate DR-GPS in P when the set, I I, of traffic flows traverse N, where is the set cardinality, as follows: P : max{h,,..., h i,,..., h j,,..., h B, } h i, i B h i,2 i B s.t. h i, = τ i, h i,2 τ i,2 h i,d 2a 2b 2c = h j,d, i, j B 2d w i w j h i,, h i,2, w i, w j [0, ]. 2e In P, w i and w j are weights of resource allocation to represent service priority for flow i and flow j, respectively, and h i,d is the fraction of occupied dominant resources of flow i, which is either h i, or h i,2. Constraint 2c guarantees c i, = c i,2 ; Constraint 2d equalizes the fractions of allocated dominant resources among the backlogged flows. Problem P is a linear programming problem and can be solved efficiently to obtain the optimal solutions of h i, and h i,2 for any flow i. The DR-GPS has properties of i service isolation by guaranteeing a service rate in to each flow, and ii work conservation by fully utilizing at least one of the two types of resources in serving the backlogged flows [28], [36], [37]. Although the queueing delay for packet transmissions is reduced by employing the DR-GPS scheme, the total packet delay 2 for each flow traversing the same NFV node should be evaluated. With GPS [34], [35], the process of multiple traffic flows passing through common NFV node N is extracted as a tandem queueing model, shown in Fig. 2. The total packet delay is the summation of packet queueing delay before processing, packet processing delay and packet transmission delay. In the following, we develop an analytical model to evaluate the total packet delay for each traffic flow traversing N. Based on the delay modeling for flows traversing the first NFV node, the end-to-end packet delay for traffic flows passing through the embedded VNF chains can be analyzed. 2 Total packet delay refers to the duration from the instant that a packet of one traffic flow reaches to the processing queue of the NFV node to the instant when it is transmitted out of the NFV node over a physical link. Flowi... Flowj l i l j Processing queue c i, c i, c c j, j,2 CPU processing Link transmission Fig. 2: A queueing model for multiple traffic flows traversing N. V. END-TO-END DELAY ANALYSIS In this section, we first analyze the total packet delay for each traffic flow traversing the first NFV node, and then extend the delay model to evaluate the end-to-end delay for traffic lows passing through a sequence of NFV nodes of an embedded VNF chain. A. Rate Decoupling The main difficulty in analyzing the packet delay for each flow is that both the processing and transmission rates of each flow depend on the backlog status of other flows at the same NFV node. For the case of two flows, when one of the flows has an empty processing queue, its processing and transmission resources are given to the other backlogged flow to exploit the traffic multiplexing gain. Therefore, the processing transmission rate of each flow switches between two deterministic rate values, depending on the status of the other flow. For a general case where we have the set, I, of multiplexing flows which includes a set, B, of backlogged flows excluding the tagged flow i, the processing rate c i, of flow i can be determined by solving P, based on set B of backlogged flows. We further denote B as B r where r =, 2,..., I B, representing one of I B combinations of B backlogged flows. Thus, c i, changes with B r. This correlation of queue status among flows leads to the processing rate of each flow jumping over discrete deterministic values, making the total packet delay analysis complex. To remove the coupling effect of instantaneous processing rate of one flow fluctuating with the backlog status of other flows, we first determine the average processing rate µ i, for flow i, with the consideration of resource multiplexing among different flows, i.e., non-empty probabilities of processing queues from all other flows, through a set of high-order nonlinear equations given by µ i, = I M ϱ l, B =0 r= l B r ϱ i, = λ i, i I. µ i, In 3, M = I B, Br = I\{i B r }, and ϱ i, is the nonempty probability of processing queue for flow i at N. Given packet arrival rate for any flow in I, 3 has 2 I equations with 2 I variables and can be solved numerically for the set of average processing rates of each flow. For analysis tractability, we use the average processing rate, µ i,, as an approximation for the instantaneous processing rate c i, to transform the original correlated processing system to a number of I k B r ϱ k, c i, 3

6 6 uncorrelated processing queues. A case of two traffic flows at an NFV node is demonstrated in Fig. 3. With the decoupled Flowi Flowj Flowi Flowj Processing queue l i l j Processing queue l i l j Decoupled processing m i, c i,2 m c,2 j, j CPU processing Link transmission Decoupled processing m i, m j, a b m i,2 m j,2 Decoupled transmission Fig. 3: A queueing model for a decoupled packet processing and b decoupled packet processing and transmission. deterministic processing rates, packet processing for each flow can be modeled as an M/D/ queueing process, upon which we can further calculate both the decoupled packet processing delay and the total packet delay for processing. The accuracy of processing rate decoupling is verified through extensive simulations presented in Section VI. In Fig. 3a, the instantaneous link transmission rate for each flow is also correlated with the backlog status of other flows. To remove the transmission rate correlation for packet delay analysis at the first NFV node, we analyze the packet arrival process of each flow at link transmission i.e., the departure process from the preceding packet processing in the following. B. Queue Modeling at the First NFV Node We study the packet departure process from each decoupled processing at N. Taking flow i in Fig. 3 as an example, we focus on the inter-departure time between two successively departed packets from the decoupled processing. As indicated in [38] and [39], for an M/D/ queueing system, the queue occupancy distribution in a steady state seen by a departing packet is the same as that seen by an arriving packet due to the Poisson characteristic of the arriving process. Therefore, a departing packet from the decoupled processing sees the same empty probability of the processing queue as an arriving packet. Let random variable Y i be the inter-departure time of successive packets of flow i departing from the decoupled processing at N. If the l th departing packet sees a nonempty queue, then Y i = T i, where T i = µ i, is the decoupled processing time for a packet of flow i; If the departing packet sees an empty queue upon its departure, Y i = X i + T i, where random variable X i denotes the duration from the time of the l th packet departure of flow i to the time of l + th packet arrival. Because of the memoryless property of a Poisson arrival process, X i has the same exponential distribution as packet inter-arrival time with parameter λ i. Therefore, the probability density function PDF of Y i, ξ Yi t, can be calculated as ξ Yi t = ϱ i, ξ Xi+T it + ϱ i, ξ Ti t 4 where ϱ i, = λi µ i,. Since X i and T i are independent random variables, the PDF of X i + T i is the convolution of the PDFs of X i and T i. Thus, 4 is further derived as ξ Yi t = λ i [ξ Xi t ξ Ti t] + λ i ξ Ti t µ i, µ i, = λ i [λi e λit ut δt T i ] µ i, + λ i µ i, δt T i = λ i µ i, λ i µ i, e λit Ti ut T i + λ i µ i, δt T i where ut is the unit step function, δt is the Dirac delta function, and is the convolution operator. From 5, the cumulative distribution function CDF of Y i is given by F Yi t = [ λ i µ i, 5 e λit Ti ] ut T i. 6 Based on 5 and 6, both mean and variance of Y i can be calculated as E[Y i ] = and D[Y i ] = λ i λ 2 i µ 2. 7 i, From 6 and 7, we observe that, when λ i is small, the departure process, delayed by the service time T i, approaches the Poisson arrival process with parameter λ i ; when λ i is increased to approach µ i,, the departure process approaches the deterministic process with rate µ i,. Since the packet departure rate from each decoupled processing is the same as packet arrival rate at each processing queue, the decoupled transmission rate µ i,2 for flow i is the same as µ i,, by solving the set of equations in 3. At this point, we have a complete decoupled queueing model of both packet processing and packet transmission for flow i traversing the first NFV node N, shown in Fig. 3b. The average total packet delay for flow i is determined by D i, = λ i + µ i, 2µ i,2 ϱ i, +. 8 µ i,2 Before modeling the delay for flows going through the second NFV node, N 2, we analyze the departure process for packet transmissions of each flow at N. Similar to the analysis of packet departure process from the decoupled processing for flow i at N, we set time 0 as the instant when the l th packet departs from the processing queue and reaches the transmitting queue for immediate packet transmission. Since we have µ i, = µ i,2, T i also indicates packet transmission time, i.e., T i = µ i,2. Let Z i denote packet inter-departure time for flow i passing though link transmission. If the l th departing packet from the processing queue sees a nonempty queue, we have Z i = T i ; otherwise, the following two cases apply, as illustrated in Fig. 4:

7 7 Case If the l + th packet s arrival time at the processing queue is greater than the l th packet s transmission time at the transmitting queue, i.e, X i > T i, we have Z i = ζ + 2ζ 2 = X i T i + 2T i = X i + T i 9 where ζ indicates the duration from the instant that the l th packet departs from the transmission queue till the instant that the l + th packet arrives at the processing queue, and ζ 2 = T i ; Case 2 If the l + th packet arrives at the processing queue while the l th packet is still at the transmission queue, i.e., X i T i, we have Z i = ζ + ζ 2 = [T i T i X i ] + T i = X i + T i 0 where ζ denotes the remaining processing time on the l + th packet at the processing queue after the l th packet departs from transmission queue, and ζ 2 = T i. l th departure from transmitting queue l+ th departure Z i z z2 z2 0 Ti Xi Xi+ Ti Xi+ 2Ti Time a Xi > Ti l th departure from transmitting queue l+ th departure Z i ' ' z z 2 0 Xi T i Xi+ Ti Xi+ 2Ti Time b Xi Ti Fig. 4: A composition of Z i under different cases. As a result, the PDF of Z i is given by ξ Zi t = ϱ i, [ P {X i T i }ξ Xi+T it +P {X i > T i }ξ Xi+T it ] + ϱ i, ξ Ti t = ϱ i, ξ Xi+T it + ϱ i, ξ Ti t. Comparing with 4 and, we conclude that Z i and Y i have exactly the same probability distribution, and thus any order of statistics e.g. expectation, variance. C. Delay over the Virtual Link between NFV Nodes N and N 2 So far, we derive the packet departure process of each flow from the link transmission at the first NFV node N. Before reaching to the second NFV node N 2, the flows may go through a sequence of network switches and physical links forwarding the traffic. The transmission rates allocated to flow i from these switches and links are the same as the transmission rates µ i,2 from N to maximize the bandwidth utilization [40]. Therefore, queueing delays on switches and links are not considered for each flow. The total packet transmission delay for flow i traversing n switches and n links before reaching N 2 is given by D f i, = 2n µ i,2. 2 D. Delay at the Second NFV Node Proposition. A Poisson packet flow traverses a tandem of k k =, 2, 3,... servers, each with deterministic service capacity Y k. If we have Y q Y q, q [2, k], the departure process of the traffic flow coming out of the k th k 2 server remains the same as the departure process from the first server. The proof of Proposition is given in Appendix A. According to Proposition, the arrival process at the second NFV node N 2 is the same as the traffic departure process from N. Based on 7 and, the arrival rate for flow i at N 2 is λ i. Thus, the same method can be used as in 3 to get a set of decoupled processing and transmission rates µ i, and µ i,2 for flow i at N 2, as shown in Fig. 5, by taking into consideration the time profiles of traffic flows going through the new VNFs at N 2 and instantaneous processing and transmission rates c i, and c i,2 allocated to flow i. The main difference in packet delay modeling for flow i at N 2 from that at N is that the packet arrival process for flow i is a general process with average arrival rate λ i. The process has the inter-arrival time Z i with the same CDF, expectation and variance as those of Y i in 6 and 7. Thus, we can model packet processing at N 2 as a G/D/ queueing process 3, where the average packet queueing delay before processing for flow i at N 2 is given by [38] λ i λ W i,2 = 2 µ 2 i i, σi 2 2 λ i λ ϱ 2 µ 2 i i, i, 2 ϱ. 3 i, In 3, ρ i, = λi µ, T e is the idle duration within inter-arrival i, time of successive packets of flow i at N 2, with variance σi 2. Since the arrival process at N 2 for each flow correlates with the preceding decoupled processing rates at N, as indicated in 6, the G/D/ queueing model is not accurate especially when λ i becomes large [38]. Also, it is difficult to obtain the distribution of T e to calculate σi 2 in 3. Using the upper bound in 3 to approximate W i,2 is not accurate when λ i is small the queueing system is lightly loaded, since the probability of an arriving packet at the processing queue of N 2 seeing an empty queue increases, and σi 2 becomes large. Processing queue Decoupled processing Decoupled transmission Flow i... Flow j Node Node Fig. 5: A decoupled queueing model for traffic flows traversing the first and second NFV nodes in sequence. From 6 and 7, in the case of µ i, < µ i,2, Z i is more likely to approach an exponentially distributed random variable than a deterministic value with varying λ i under the 3 Note that we consider the case where µ i, < µ i,2, i I ; For the case of µ i, µ i,2, there is no queueing delay for processing at N 2.

8 8 condition of ρ i, <. Therefore, to make the arrival process of each flow at the processing queue of N 2 independent of processing and transmission rates at N, we approximate packet arrival process of flow i at N 2 as a Poisson process with rate parameter λ i, and establish an M/D/ queueing model to represent the packet processing for flow i. Proposition 2 indicates that the average packet queueing delay Q i,2 in the M/D/ queueing model is an improved upper bound over that in the G/D/ system in 3, especially when the input traffic is lightly loaded. Proposition 2. Given µ i, < µ i,2, Q i,2 is an upper bound of W i,2 when the processing queue for flow i at N 2 is both lightly-loaded and heavily-loaded. The proof of Proposition 2 is given in Appendix B. According to the approximation on packet arrival process for flow i at N 2, average total packet delay at N 2 is calculated, independently of the processing and transmission rates at N, given by λ i + +, µ i, < µ i,2 D i,2 = µ i, µ i, E. Average E2E Delay 2µ i,2 ρ i, + µ, µ i, µ i,2. i,2 µ i,2 4 Based on the same methodology of delay modeling for packets traversing N 2, the average total packet delay for flow i traversing each subsequent NFV node if any can be derived independently. Under the condition that the decoupled packet processing rate of flow i at one subsequent NFV node N z z > 2 is smaller than the decoupled packet transmission rate at its preceding NFV node N z, using an approximated M/D/ queueing model to represent packet processing at N z is valid since packet arrival process of flow i at N z is more likely to approach a Poisson process with varying λ i. In general, the average E2E delay for a packet of flow i passing through an embedded VNF chain, consisting of m NFV nodes, is the summation of average total delay for a packet passing through all NFV nodes and the total transmission delay on switches and links along the path forwarding the packet, given by m m D i = D i,z + D f i,z. 5 z= z= In 5, D i,z is the average packet delay for flow i passing through z th NFV node of the embedded VNF chain, D f i,z is the total packet transmission delay for flow i traversing n z switches and n z links before reaching NFV node N z+, and is determined in the same way as in 2. VI. SIMULATION RESULTS In this section, simulation results are presented to verify the accuracy of the proposed packet delay modeling of each flow passing through an embedded VNF chain. All simulations are carried out using the network simulator OMNeT++ [4]. We consider the network scenario where two equally weighted flows, i and j, traversing logic VNF chains firewall f DNS f 3 and firewall f IDS f 2 respectively, are embedded on a common physical path and share a set of processing and transmission resources, as shown in Fig.. Flow i represents DNS request traffic, whereas flow j indicates a video-conferencing data streaming. The packet arrival rate λ i for flow i is set to 50 packet/s with packet size of 4000 bits [42]. The packet size for flow j is set to 6000 bits, and we vary its arrival rate, λ j, from 75 packet/s to 350 packet/s to reflect different traffic load conditions. The rate vector for each flow traversing an NFV node is tested over OpenStack [43], which is a resource virtualization platform installed on each NFV node. By injecting traffic flows with different packet sizes into different VNFs programmed on the OpenStack, we test maximum available packet processing and transmission rates for different flows. With DR-GPS, each flow is allocated a fraction of the maximum available processing and transmission rates, upon which packet-level simulation is conducted to evaluate packet delay of each flow traversing each NFV node. Table II summarizes the rate vectors for flows i and j traversing different VNFs. Other important simulation settings are also included. A. Packet Delay at the First NFV Node N We first compare packet processing delay and packet queueing delay for each flow traversing N. In Fig. 6, it is demonstrated that both packet processing delay and packet queueing delay derived using rate decoupling between flows i and j are close to the simulation results with rate coupling. We can see from Fig. 6a that the decoupled processing rate for flow i decreases with λ j, since the processing queue nonempty probability for flow j increases statistically, shrinking the decoupled processing rates for the other flow. Packet transmission delay and queueing delay before transmission are evaluated for both flows at N in Fig. 7. For packet transmission delay, the analytical results match the simulation results well, and there is almost no queueing delay before packet transmissions, which indicates the accuracy of the proposed transmission rate decoupling. In Fig. 8, we compare queueing delays before packet transmissions for flows i and j at N by employing DR-GPS and bi-resource GPS schemes. Although the bi-resource GPS achieves fair allocation on both CPU and bandwidth resources between the two flows, the amount of allocated bandwidth resources is overly provisioned for flow i and is underestimated for flow j, due to the discrepancy of time profiles for different flows. Thus, packet queueing delay before link transmission for flow j becomes much larger than that with the DR-GPS where packet queueing delays for both flows at the transmission link of N are minimized. B. Packet Delay at the Second NFV Node N 2 After traversing the first NFV node, N, packet processing delay for flow i and flow j at the second NFV node, N 2, is evaluated in Fig. 9. With a close match between analytical and simulation results, it is verified that the processing rate decoupling at N 2 is accurate. Packet queueing delay before processing for both flows at N 2 are demonstrated in Fig.

9 9 TABLE II: Simulation parameters Traffic flows Parameters Flow i Flow j Rate vector Firewall [000, 2000] packet/s [750, 500] packet/s Rate vector DNS [000, 250] packet/s Rate vector IDS [800, 32.5] packet/s n n Simulation time 000 s 000 s Packet Processing Delay at N ms Packet Queueing Delay for Processing ms Flow i decoupled Flow i simulation Flow j decoupled Flow j simulation Packet Arrival Rate of Flow j λ j a Flow i decoupled Flow i simulation Flow j decoupled Flow j simulation Packet Arrival Rate of Flow j λ j b Fig. 6: Packet delay for processing at N ; a Average packet processing delay b Average packet queueing delay. Packet Delay ms Queueing Delay for Packet Transmission ms Transmission delay flow i, decoupled Transmission delay, flow i, simulation Transmission delay flow j, decoupled Transmission delay, flow j, simulation Queueing delay flow i, simulation Queueing delay flow j, simulation Packet Arrival Rate of Flow j λ j Fig. 7: Delay for packet transmissions at N. DR-GPS, flow i simulation DR-GPS, flow j simulation Bi-resource GPS, flow i simulation Bi-resource GPS, flow j simulation Packet Arrival Rate of Flow j λ j Fig. 8: Queueing delay for packet transmissions under different resource allocation disciplines. 0 and Fig.. As λ j increases, the packet queueing delay for flow i increases slightly since the resource multiplexing gain obtained by flow i from flow j becomes small. We can see from Fig. 0 that the M/D/ queueing model for packet processing at N 2 provides a tighter upper bound than the G/D/ queueing delay upper bound for flow i. In comparison with the results for flow j in Fig., we observe that the G/D/ upper bound of packet queueing delay is looser for flow i, since the processing queue for flow i is lightly loaded with low queue nonempty probability shown in Fig. 2, whereas the G/D/ queueing delay upper bound becomes tighter for flow j with the increase of λ j but less accurate in a heavy traffic load condition. For both flows, the proposed M/D/ queueing model is an improved upper bound to approximate the packet queueing delay before processing at N 2. Packet transmission delay for both flows at N 2 is demonstrated in Fig. 3. Similar to Fig. 7, we can see the decoupled transmission rates for each flow are close to the simulation

10 0 Packet Processing Delay at N2 ms Flow i decoupled Flow i simulation Flow j decoupled Flow j simulation Packet Arrival Rate of Flow j λ j Fig. 9: Packet processing delay at N 2. Nonempty Probability for Processing Queues at N Flow i Flow j Packet Arrival Rate of Flow j λ j Fig. 2: Nonempty probability for processing queues at N Packet Queueing Delay for Flow i ms M/D/ approximation Simulation G/D/ upper bound Packet Delay at N2 ms Transmission delay flow i, decoupled Transmission delay, flow i, simulation Transmission delay flow j, decoupled Transmission delay, flow j, simulation Queueing delay flow i, simulation Queueing delay flow j, simulation Packet Arrival Rate of Flow j λ j Average Packet Arrival Rate of Flow j λ j Fig. 0: Packet queueing delay for flow i at N 2. Fig. 3: Delay for packet transmissions at N Packet Queueing Delay for Flow j ms M/D/ approximation Simulation G/D/ upper bound End-to-End Packet Delay ms Flow i analytical Flow i simulation Flow j analytical Flow j simulation Packet Arrival Rate of Flow j λ j Packet Arrival Rate of Flow j λ j Fig. : Packet queueing delay for flow j at N 2. Fig. 4: End-to-End packet delay for both flows.

11 results, and queueing delay before packet transmissions is negligible. Lastly, we evaluate in Fig. 4 the end-to-end delay for packets of each flow going through the whole embedded physical network path, which is a summation of the total packet delay for traversing all NFV nodes and packet transmission delay over all physical links and network switches. It is demonstrated that the proposed analytical modeling provides an accurate delay evaluation on end-to-end packet processing and transmission for multiple flows traversing VNF chains embedded on a common physical network path. The proposed analytical modeling can help to support E2E delay-aware VNF chain embedding. VII. CONCLUSION In this paper, we present an analytical model to evaluate E2E packet delay for multiple traffic flows traversing a common embedded VNF chain. With DR-GPS, both CPU and bandwidth resources are allocated among different flows at each NFV node to achieve dominant-resource fair allocation with high resource utilization. A tandem queueing model is established to describe packets of each flow passing through an NFV node and its outgoing link. By removing the coupling effect on instantaneous packet processing rates among multiple flows, an M/D/ queueing model is used to determine average packet delay at each decoupled processing queue of the first NFV node. The correlation of packet transmission rates is also removed in the modeling based on the analysis of packet departure process from each decoupled processing queue. We further analyze the packet arrival process of each flow at the subsequent NFV node, and establish an approximated M/D/ queueing model to determine the average packet delay of a flow at a decoupled processing queue of the NFV node, which is proved to be a more accurate upper bound than that using a G/D/ queueing model in both lightly- and heavilyloaded traffic conditions. Packet transmission delay over each embedded virtual link between consecutive NFV nodes is also derived for E2E delay calculation. Simulation results demonstrate the accuracy and effectiveness of the proposed analytical E2E packet delay modeling for achieving delayaware VNF chain embedding. ACKNOWLEDGMENT The authors would like to thank Yushi Alfred Cao an Undergraduate Research Assistant for his help in testing over OpenStack the time profiles for traffic flows traversing an NFV node. APPENDIX A. PROOF OF PROPOSITION Proof: Define Z k as packet inter-departure time at the k th server, and define ρ as the probability that the l th departing packet sees a nonempty queue at the first server. If the l th departing packet sees an empty queue at the first server, we let X be the duration from time 0 till the instant that l + th packet arrives at the server. With Y q Y q, there is no queueing delay at each of the servers following the first server. Similar to the description in Fig. 4, if the l th departing packet sees a nonempty queue, we have Z k = Y ; Otherwise, two cases are considered: Case If X > Z k = X q=2 Case 2 If X Z k = k q=2 Y q, k k Y q + Y q = X + Y ; 6 k q=2 Y q, q= k k Y q Y q X = Y + X. 7 q= q=2 Hence, Z k has the same PDF as Y i derived in 4, which ends the proof. APPENDIX B. PROOF OF PROPOSITION 2 Proof: When λ i is small, σi 2 in 3 is close to that for an M/D/ queueing system, given by [38] σ 2 i λ 2 i µ 2. 8 i, Hence, W i,2 is further derived as [ ] λ i λ W i,2 2 µ 2 i i, λ 2 i µ i, 2 2 ϱ i, λ i µ = i, 2 µ 2 i, 2 ϱ. i, 9 When λ i becomes large, the idle duration within inter-arrival time of successive packets of flow i at N 2 is small, making σi 2 negligible. Thus, we have λ i λ W i,2 2 µ 2 λ i i i, 2 µ ϱ i, 2 µ 2 i, i, 2 ϱ. 20 i, On the other hand, under both traffic load cases, Q i,2 in the approximated M/D/ queueing system is derived as λ i Q i,2 = 2µ i,2 W ρ i,2. 2 i, Thus, we prove that Q i,2 is an upper bound of W i,2 in both lightly-loaded and heavily-loaded traffic conditions, and becomes a tighter upper bound than that in the G/D/ system in 3 when λ i is small. REFERENCES [] Q. Ye, J. Li, K. Qu, W. Zhuang, X. Shen, and X. Li, End-to-end quality of service in 5G networks Examining the effectiveness of a network slicing framework, IEEE Veh. Technol. Mag., vol. 3, no. 2, pp , Jun [2] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, Internet of Things: A survey on enabling technologies, protocols, and applications, IEEE Commun. Surv. Tutor., vol. 7, no. 4, pp , Fourth Quarter 205. [3] Q. Ye and W. Zhuang, Distributed and adaptive medium access control for Internet-of-Things-enabled mobile networks, IEEE Internet Things J., vol. 4, no. 2, pp , Apr. 207.

12 2 [4] Q. Ye and W. Zhuang, Token-based adaptive MAC for a two-hop Internet-of-Things enabled MANET, IEEE Internet Things J., vol. 4, no. 5, pp , Oct [5] H. Li, M. Dong, and K. Ota, Radio access network virtualization for the social Internet of Things, IEEE Cloud Comput., vol. 2, no. 6, pp , Nov [6] W. Wu, Q. Shen, K. Aldubaikhy, N. Cheng, N. Zhang, and X. Shen, Enhance the edge with beamforming: Performance analysis of beamforming-enabled WLAN, in Proc. IEEE WiOpt 8, May 208, pp. 6. [7] X. Duan, C. Zhao, S. He, P. Cheng, and J. Zhang, Distributed algorithms to compute Walrasian equilibrium in mobile crowdsensing, IEEE Trans. Ind. Electron., vol. 64, no. 5, pp , May 207. [8] G. Yang, S. He, and Z. Shi, Leveraging crowdsourcing for efficient malicious users detection in large-scale social networks, IEEE Internet Things J., vol. 4, no. 2, pp , Apr [9] L. Lyu, C. Chen, Z. Shanying, and X. Guan, 5G enabled co-design of energy-efficient transmission and estimation for industrial IoT systems, IEEE Trans. Ind. Informat., to appear, doi: 0.09/TII [0] F. Lyu, H. Zhu, H. Zhou, W. Xu, N. Zhang, M. Li, and X. Shen, SS-MAC: A novel time slot-sharing MAC for safety messages broadcasting in VANETs, IEEE Trans. Veh. Technol., to appear, doi: 0.09/TVT [] W. Xu, H. A. Omar, W. Zhuang, and X. Shen, Delay analysis of invehicle Internet access via on-road WiFi access points, IEEE Access, vol. 5, pp , Feb [2] R. Mijumbi, J. Serrat, J. L. Gorricho, N. Bouten, F. D. Turck, and R. Boutaba, Network function virtualization: State-of-the-art and research challenges, IEEE Commun. Surv. Tutor., vol. 8, no., pp , First Quarter 206. [3] L. Wang, Z. Lu, X. Wen, R. Knopp, and R. Gupta, Joint optimization of service function chaining and resource allocation in network function virtualization, IEEE Access, vol. 4, pp , Nov [4] F. Bari, S. R. Chowdhury, R. Ahmed, R. Boutaba, and O. C. M. B. Duarte, Orchestrating virtualized network functions, IEEE Trans. Netw. Serv. Manage., vol. 3, no. 4, pp , Dec [5] N. Zhang, P. Yang, S. Zhang, D. Chen, W. Zhuang, B. Liang, and X. Shen, Software defined networking enabled wireless network virtualization: Challenges and solutions, IEEE Netw., vol. 3, no. 5, pp , May 207. [6] S. Q. Zhang, Q. Zhang, H. Bannazadeh, and A. Leon-Garcia, Routing algorithms for network function virtualization enabled multicast topology on SDN, IEEE Trans. Netw. Serv. Manage., vol. 2, no. 4, pp , Dec [7] J. Ordonez-Lucena, P. Ameigeiras, D. Lopez, J. J. Ramos-Munoz, J. Lorca, and J. Folgueira, Network slicing for 5G with SDN/NFV: Concepts, architectures, and challenges, IEEE Commun. Mag., vol. 55, no. 5, pp , May 207. [8] X. Li, J. Rao, H. Zhang, and A. Callard, Network slicing with elastic SFC, in Proc. IEEE VTC 7, Sept. 207, pp. 5. [9] J. Li, N. Zhang, Q. Ye, W. Shi, W. Zhuang, and X. Shen, Joint resource allocation and online virtual network embedding for 5G networks, in Proc. IEEE GLOBECOM 7, Dec. 207, pp. 6. [20] M. Mechtri, C. Ghribi, and D. Zeghlache, A scalable algorithm for the placement of service function chains, IEEE Trans. Netw. Serv. Manage., vol. 3, no. 3, pp , Sept [2] B. A. A. Nunes, M. Mendonca, X. N. Nguyen, K. Obraczka, and T. Turletti, A survey of software-defined networking: Past, present, and future of programmable networks, IEEE Commun. Surv. Tutor., vol. 6, no. 3, pp , Third Quarter 204. [22] H. Li, K. Ota, M. Dong, and M. Guo, Mobile crowdsensing in software defined opportunistic networks, IEEE Commun. Mag., vol. 55, no. 6, pp , Jun [23] H. Li, M. Dong, and K. Ota, Control plane optimization in softwaredefined vehicular ad hoc networks, IEEE Trans. Veh. Technol., vol. 65, no. 0, pp , Oct [24] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, Live migration of virtual machines, in Proc. ACM USENIX 05, May 2005, pp [25] S. Akoush, R. Sohan, A. Rice, A. W. Moore, and A. Hopper, Predicting the performance of virtual machine migration, in Proc. IEEE MASCOTS 0, Aug. 200, pp [26] S. Ayoubi, C. Assi, K. Shaban, and L. Narayanan, MINTED: Multicast virtual network embedding in cloud data centers with delay constraints, IEEE Trans. Commun., vol. 63, no. 4, pp , Apr [27] A. Ghodsi, V. Sekar, M. Zaharia, and I. Stoica, Multi-resource fair queueing for packet processing, ACM SIGCOMM Comput. Commun. Rev., vol. 42, no. 4, pp. 2, Oct [28] W. Wang, B. Liang, and B. Li, Multi-resource generalized processor sharing for packet processing, in Proc. ACM IWQoS 3, Jun. 203, pp. 0. [29] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica, Dominant resource fairness: Fair allocation of multiple resource types, in Proc. ACM NSDI, Apr. 20, pp [30] F. C. Chua, J. Ward, Y. Zhang, P. Sharma, and B. A. Huberman, Stringer: Balancing latency and resource usage in service function chain provisioning, IEEE Internet Comput., vol. 20, no. 6, pp. 22 3, Nov [3] B. Xiong, K. Yang, J. Zhao, W. Li, and K. Li, Performance evaluation of OpenFlow-based software-defined networks based on queueing model, Comput. Netw., vol. 02, pp , Jun [32] M. Jarschel, S. Oechsner, D. Schlosser, R. Pries, S. Goll, and P. Tran-Gia, Modeling and performance evaluation of an openflow architecture, in Proc. 23rd ITC, Sept. 20, pp. 7. [33] N. Egi et al., Understanding the packet processing capability of multicore servers, Intel Tech. Rep., [34] A. K. Parekh and R. G. Gallager, A generalized processor sharing approach to flow control in integrated services networks: The singlenode case, IEEE/ACM Trans. Netw., vol., no. 3, pp , Jun [35] Z.-L. Zhang, D. Towsley, and J. Kurose, Statistical analysis of the generalized processor sharing scheduling discipline, IEEE J. Sel. Areas Commun., vol. 3, no. 6, pp , Aug [36] D. C. Parkes, A. D. Procaccia, and N. Shah, Beyond dominant resource fairness: Extensions, limitations, and indivisibilities, ACM Trans. Econ. Comput., vol. 3, no., p. 3, Mar [37] A. Gutman and N. Nisan, Fair allocation without trade, in Proc. ACM AAMAS 2, Jun. 202, pp [38] D. P. Bertsekas, R. G. Gallager, and P. Humblet, Data networks. Englewood Cliffs, NJ, USA: Prentice-hall, 987, vol. 2. [39] L. Kleinrock, Queueing systems, volume : Theory. New York: Wiley, 976. [40] S. Larsen, P. Sarangam, R. Huggahalli, and S. Kulkarni, Architectural breakdown of end-to-end latency in a TCP/IP network, Int. J. Parallel Program., vol. 37, no. 6, pp , Dec [4] OMNeT++ 5.0, [Online]. Available: omnetpp. [42] A. Liska and G. Stowe, DNS security: Defending the domain name system. Syngress, 206. [43] Openstack Release Pike, [Online]. Available: org. Qiang Ye S 6-M 7 received the B.S. degree in network engineering and M.S. degree in communication and information system from the Nanjing University of Posts and Telecommunications, Nanjing, China, in 2009 and 202, respectively, and the Ph.D. degree in electrical and computer engineering from the University of Waterloo, Waterloo, ON, Canada, in 206. He has been a Post-Doctoral Fellow with the Department of Electrical and Computer Engineering, University of Waterloo, since 206. His current research interests include SDN and NFV, network slicing for 5G networks, VNF chain embedding and end-to-end performance analysis, medium access control and performance optimization for mobile ad hoc networks and Internet of Things.

13 3 Weihua Zhuang M 93-SM 0-F 08 has been with the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada, since 993, where she is a Professor and a Tier I Canada Research Chair in Wireless Communication Networks. She is the recipient of 207 Technical Recognition Award from IEEE Communications Society Ad Hoc & Sensor Networks Technical Committee, one of 207 ten N2Women Stars in Computer Networking and Communications, and a co-recipient of several best paper awards from IEEE conferences. Dr. Zhuang was the Editor-in-Chief of IEEE Transactions on Vehicular Technology , Technical Program Chair/Co-Chair of IEEE VTC Fall 207 and Fall 206, and the Technical Program Symposia Chair of the IEEE GLOBECOM 20. She is a Fellow of the IEEE, the Royal Society of Canada, the Canadian Academy of Engineering, and the Engineering Institute of Canada. Dr. Zhuang is an elected member in the Board of Governors and VP Publications of the IEEE Vehicular Technology Society. She was an IEEE Communications Society Distinguished Lecturer Xu Li is a staff researcher at Huawei Technologies Inc., Canada. He received a Ph.D degree from Carleton University, an M.Sc degree from the University of Ottawa, and a B.Sc. 998 degree from Jilin University, China, all in computer science. Prior to joining Huawei, he worked as a research scientist with tenure at Inria, France. His current research interests are focused in 5G system design and standardization, along with 90+ refereed scientific publications, 40+ 3GPP standard proposals and 50+ patents and patent filings. He is/was on the editorial boards of the IEEE Communications Magazine, the IEEE Transactions on Parallel and Distributed Systems, among others. He was a TPC co-chair of IEEE VTC 207 fall LTE, 5G and Wireless Networks Track, IEEE Globecom 203 Ad Hoc and Sensor Networking Symposium. He was a recipient of NSERC PDF awards, IEEE ICNC 205 best paper award, and a number of other awards. Jaya Rao M 4 received his B.S. and M.S. degrees in Electrical Engineering from the University of Buffalo, New York, in 200 and 2004, respectively, and his Ph.D. degree from the University of Calgary, Canada, in 204. He is currently a Senior Research Engineer at Huawei Technologies Canada, Ottawa. Since joining Huawei in 204, he has worked on research and design of CIoT, URLLC and V2X based solutions in 5G New Radio. He has contributed for Huawei at 3GPP RAN WG2, RAN WG3, and SA2 meetings on topics related to URLLC, network slicing, mobility management, and session management. From 2004 to 200, he was a Research Engineer at Motorola Inc. He was a recipient of the Best Paper Award at IEEE WCNC 204.