MOBILE data is continuing to grow exponentially, and

Transcription

1 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE Enhancing Mobile Networks With Software Define Networking an Clou Computing Zizhong Cao, Stuent Member, IEEE, Shivenra S. Panwar, Fellow, IEEE, Murali Koialam, Member, IEEE, an T. V. Lakshman, Fellow, IEEE, ACM Abstract In the past ecae, mobile evices an applications have experience an explosive growth, an users are expecting higher ata rates an better quality services every year. In this paper, we propose several ieas to increase the functionality an capacity of wireless networks using software-efine networking (SDN an clou computing technologies. Connections between users an services in mobile networks typically have to pass through a require set of mileboxes. The complex routing is one of the major impetus for the SDN paraigm, which enables flexible policy-aware routing in the next generation mobile networks. In aition, the high costs of mileboxes an limite capabilities of mobile evices call for revolutionary virtualization technologies enable by clou computing. Base on these, we consier an online routing problem for mobile networks with SDN an clou computing. In this problem, connection requests are given one at a time (as in a real mobile system, an the objective is to steer traffic flows to maximize the total amount of traffic accepte over time, subject to capacity, buget, policy, an quality of service constraints. A fast log-competitive approximation algorithm is evelope base on time-epenent uals. Inex Terms Mobile network, routing, software-efine network, clou computing, network function virtualization. I. INTRODUCTION MOBILE ata is continuing to grow exponentially, an is taking up a larger portion of total Internet traffic in recent years. As various portable mobile evices like smartphones, tablets, an netbooks become more prevalent, people are using mobile Internet more frequently an have shifte a large portion of their online activities, incluing web surfing, online banking, vieo streaming, social networking, an online gaming, from traitional cable Internet services to their mobile counterparts. Such an exponential growth of mobile ata, as well as the vast variety of online activities, imposes a heavy buren on mobile network service proviers. Not only o these carriers nee to upgrae their network capacities frequently, but they also nee to provie ifferentiate services an meet the varie QoS requirements impose by ifferent mobile Internet applications. Mobile networks significantly iffer from fixe Manuscript receive January 24, 2016; revise August 16, 2016; accepte November 14, 2016; approve by IEEE/ACM TRANSACTIONS ON NETWORKING Eitor S. Puthenpura. Date of publication January 16, 2017; ate of current version June 14, Z. Cao an S. S. Panwar are with the Tanon School of Engineering, New York University, Brooklyn, NY USA ( caozizhong@ gmail.com; panwar@catt.poly.eu. M. Koialam an T. V. Lakshman are with Bell Laboratories, Nokia, Crawfor Hill, NJ USA ( muralik@alcatel-lucent.com; t.v.lakshman@alcatel-lucent.com. Digital Object Ientifier /TNET Fig. 1. Routing in 4G LTE mobile network. Internet service provier (ISP networks in that they aopt a ifferent set of protocols for aressing/routing, an special network elements (NE are use for various control an ata plane tasks. The latest 4G LTE network uses an all-ip evolve packet core (EPC [1], but the network structure is still hien from the public Internet, an mobile traffic from/to a mobile user equipment (UE must pass through special NE incluing enoeb, serving gateway (SGW, packet ata network gateway (PGW, an certain mileboxes in a specific manner. Depening on the specific applications, typical mileboxes [2], [3] may inclue intrusion etection an prevention for enterprise customers, link monitoring an rate aaptation for vieo streaming, echo cancellation for voice-over-ip calls, ata volume measurement for billing an charging, etc. At present most NE are statically eploye in the network, an routing paths among these noes are often pre-etermine. Accoring to 3GPP specifications [4], mobile ata traffic in current 4G LTE networks is route in a hop-by-hop manner, either accoring to fixe routing tables or with limite loa balancing capabilities, as shown in Fig. 1. More specifically, a UE selects an enoeb accoring to channel measurements, the enoeb selects a Mobility Management Entity (MME accoring to tracking area ientification (TAI, an the MME selects a PGW accoring to access point name (APN an an SGW accoring to TAI. The PGW is solely responsible for QoS. It maps traffic flows into ifferent QoS classes, which are then translate into DiffServ priorities enforce in a per-hop manner. This strategy cannot fully utilize the networks to meet customers nees ue to its poor flexibility when facing asymmetric traffic fluctuations, an lacks finegraine optimization of network resources an en-to-en traffic steering. The presence of various NE an multiple instances is one of the significant reasons for the emergence of the software efine networking (SDN paraigm. SDN [7] ecouples the ata an control planes, an routing ecisions are mae in a IEEE. Personal use is permitte, but republication/reistribution requires IEEE permission. See for more information.

2 1432 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 Fig. 2. Routing in a software-efine mobile network. centralize manner rather than hop-by-hop. For each traffic flow, the controller not only selects the best NE, but can also specify the best route to take, as shown in Fig. 2. This gives SDN the ability to steer traffic in fine granularity to enforce policy-base routing, meet QoS requirements of various applications, an perform network-wie resource optimizations as well. Therefore, SDN becomes a promising caniate to enhance mobile networks [8]. Another emerging technology to enhance the performance of mobile networks is clou computing. Instea of the scale-up approach aopte by traitional IT inustry in builing eicate, stan-alone, high performance software an harware systems, clou computing follows a scale-out approach that utilizes a large pool of commoity servers as general-purpose computing resources an eliver configurable IT services on eman to various users. Clou computing can be applie to several ifferent aspects of mobile networking, such as network function virtualization (NFV [9] [11] an mobile clou computing [12] [14], an it is a promising complement to SDN. In this paper, we aress the unique challenges of traffic steering an service eployment in mobile networks, an aopt SDN an clou computing as key enabling technologies to esign an optimize next generation mobile networks. Part of this work is base on our previous workshop paper [15]. Our main contributions are: 1 We utilize an SDN framework to enable flexible policyaware routing an centralize network optimization, an incorporate clou computing to rive next generation mobile networks. 2 We formulate a mixe integer programming to maximize the total traffic accepte over time uner capacity/ buget constraints, an propose a competitive online routing algorithm to achieve policy-awareness an QoS guarantee. 3 We perform extensive numerical experiments to investigate the performance of our propose routing algorithm uner various network settings, an emonstrate its avantages over other prevalent heuristic algorithms. II. RELATED WORK As key supporting components towar next generation mobile networks, routing an provisioning for mobile ata traffic have attracte much attention in recent years. In [3] an [8], the authors call for rethinking of cellular core networks, an propose to use an SDN framework. It is envisione that with SDN, mobile networks can allow for more flexible traffic steering, fine-graine monitoring, seamless mobility, istribute QoS an access control, etc. A SoftCell prototype has also been built to support these arguments. In [10], NFV an SDN are applie to the 4G LTE core gateways, an two reesigns of the these gateways are propose: a virtualize one provisione through NFV, an a ecompose one controlle by SDN. The authors also perform extensive measurements an solve function placement problems in the new framework. These works have greatly influence this paper, but note that they mostly focus on the system architecture aspect but not the algorithmic aspect an thus have not consiere flow scheuling an routing algorithms that work specifically with SDN an NFV. In the meantime, milebox implementation issues have also receive increasing attention [2], [16] [18]. It has been recognize [2], [16] that network planning has become more complicate with policy enforcing, an can be a potential bottleneck. Therefore, various loa balancing strategies have been evelope to istribute milebox traffic across the networks. In [19], routing policies are enforce in an SDN framework using a non-eterministic finite automation with graph encoing. However, the authors o not provie a fast approximation scheme to tackle the NP-completeness, an traffic steering is performe offline. In [20], the authors stuy the problem of policy-aware application clou embeing. The paper shows that several embeing problems are NP-complete, an eals with online embeing of flow security graphs (FSG for each request. One caveat with this work is that the number of caniate paths must be polynomial, an it is assume that once a request enters the system it is there permanently. In contrast to these papers, we propose to incorporate SDN an clou computing into mobile networks, an tackle the problem with both system architecture an algorithm esigns. In aition, we consier more realistic mobile networks in which each flow can have an exponential number of caniate paths, stays for a finite uration, an is not available to the scheuler ahea of time. As will be shown later, we solve this more challenging problem by eveloping a fast online routing algorithm that is aware of the latency/congestion/buget an guarantees log-competitiveness with respect to the optimal offline solution. III. DESIGN MOTIVATIONS A. Expectations of the Next Generation 5G Network Over the past few years, the next generation 5G mobile network has been gaining interest in both acaemia an inustry. While the stanarization of 5G an commercial eployments are still a few years away [5], there have been clearer expectations on what 5G shoul bring to users [6] with several proposals on air link, fronthaul, an backhaul technologies. Among various expectations on ifferent aspects, we pick the two that are most perceive by users: 1 Data Rate: 1Gbps to tens of Gbps peak ata rate; 2 Latency: 1ms for extremely low latency use cases or at most 10ms for others.

3 CAO et al.: ENHANCING MOBILE NETWORKS WITH SDN AND CLOUD COMPUTING 1433 Compare with the latest 4G LTE network, the next generation 5G network nees to sustain orers of magnitue higher ata rate at orers of magnitue lower latency, which is very emaning an requires rethinking of the network architecture. On the latency issue, we shoul first realize that a low latency in the orer of 1ms is not always achievable ue to physical constraints. For example, if two users are connecting irectly between New York City an San Francisco, then the 2700 miles istance woul cause no less than 14ms network latency at the spee of light. Fortunately, such an extreme case is not common, an in many cases a higher replication factor can be trae for a lower latency. For static content like images an vieos, Content Delivery Networks (CDN can replicate content to the ege of the network, which then are able to serve users in close network proximity at much lower latencies. For other ynamic web services, we can also create multiple web server instances an atabase replicas, an reirect users to the closest one. In the following, we shall focus on the throughput-latency traeoff in resource replication, while in network planning there is also a latency-expense traeoff since replication often results in higher migration an storage costs. Consier a simplifie scenario in which many users an r resources are uniformly place on a planar area, then the average istance between each user an its closest resource is proportional to 1/ r. Following this iea, in orer to serve all users across the coasts of the Unite States within the 1ms latency boun, we woul nee at least 8 resource replicas along the east-west span. This number nees to be further multiplie by at least 5 if we take the south-north span into account as well, resulting in a total of 40 replicas. Even more woul be neee if we consier other factors like network topologies an processing latencies. The same argument also applies to the gateways an mileboxes that nee to be traverse by user traffic flows, because otherwise the scarcity of these NE may result in triangular routing an voi the efforts we have mae in replicating the resources. In aition to the strict latency requirement, the high ata rate requirement is also an important concern. The ata rate of mobile traffic has experience exponential growth over the past few years, an this tren shall continue to a more burens to the infrastructure of mobile networks. Great efforts have been mae an new technologies have been propose to increase the network capacity commensurately. However, because network infrastructures are often statically eploye an cannot easily aapt to varying emans, traffic engineering an network scheuling are still key to high performance. On the other han, when we replicate resources an eploy more NE to achieve a lower latency, the throughput (the amount of traffic elivere from source to estination in unit time of the network can be booste at the same time. As the user requests get serve in closer proximity, the traffic flows will traverse through a smaller portion of routers an links, which effectively reuces the buren on the network loa an increases the amount of traffic that can be sustaine. For a given network with fixe topology an capacity, if we assume that all link utilizations are always kept unchange, then the average throughput shoul be inversely proportional to the average latency, an thus is proportional to r. B. A Software Define Mobile Network With Clou Computing As iscusse above, the stringent latency constraints on 5G mobile networks call for extensive replications of resources an NE, which in turn a more burens to network capacity an make traffic routing more complicate. Complex routing is one of the major impetus for the SDN paraigm, an the high costs of flexible resource/ne replications greatly benefit from clou computing. Therefore, we propose an SDN framework with clou computing to aress the challenges of next generation mobile networking. 1 Fine-Graine Traffic Steering in a Software Define Network: In an SDN, the control plane an the ata plane are ecouple, which allows for centralize routing ecisions an istribute packet forwaring. The control plane of the network is aggregate in an SDN controller which communicates with the routers using a stanar interface like OpenFlow. Typically, a special request or the first packet is sent to the SDN controller for each new flow, an the following operations are one by the SDN controller: 1 Policy Lookup: The policy table at the controller etermines the logical sequence of NE that the packets in the flow have to pass to satisfy the policy requirement. 2 Flow Steering: The SDN controller maps this logical sequence of NE into the physical network accoring to certain QoS requirements an optimization criteria. 3 Route Installation: Once this logical path is mappe into the physical path, the SDN controller makes changes in the forwaring table on all the routers in the path so that all packets in the flow are route along the esire path. A oubt may arise here that the scheuling time at the SDN controller an roun-trip time to set up the routing path can become large as the network grows. This is true but we o not nee to be overly pessimistic about this. On one han, the control-plane latency oes not necessarily a to the ata-plane latency perceive by the en user. For those latency-sensitive applications like virtual reality, automate riving, an remote surgery, routing is performe at most once uring connection initialization but oes not affect the connection quality later on. For other less emaning applications like web browsing, a long-live connection can be set up beforehan rather than ynamically allocate on eman. This iea is similar to the efault vs. eicate bearers in the current 4G LTE networks. On the other han, the routing an scheuling tasks can be istribute to multiple SDN controllers place at ifferent locations, an thus come closer to the en users. This may significantly reuce the control-plane latency at the cost of looser consistency in network states an/or more sophisticate synchronization. SDN brings about many opportunities to optimize current mobile networks [3], [21], [22]. More importantly, it gives us more freeom to rethink what future network architectures shoul look like an how routing shoul be performe to meet the increasing emans. 2 More Flexible an On-Deman Service Deployment With Clou Computing: Network function virtualization (NFV is a network architecture concept that proposes to virtualize

4 1434 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 Fig. 3. Network function virtualization an mobile clou computing. application-specific NE functions on top of commoity servers an can leverage the clou computing infrastructure, as shown in Fig. 3. It can boost the carrier capabilities an reuce the high capital an operational costs incurre by special-purpose NE harware. Instea of spening large capital to purchase powerful proprietary harware in prior, mobile network service proviers can now shift to NFV with clou computing an pay operational expenses over time. With clou-base NFV, these special NE can be eploye on eman an sprea throughout the network, which grants more flexibilities to mobile traffic scheuling. All these benefits make NFV a scalable an profitable solution for mobile networks. Besies NFV, clou computing has been applie to mobile UE as well, also known as mobile clou computing (MCC. MCC can host proxies or clones of UE in large ata centers or regional cloulets, an allow mobile users to offloa computation-intensive applications to the clou. When carefully one, MCC coul significantly enhance the user experience by boosting evice capabilities an extening battery life at the same time without affecting users with network latencies. Full-feature MCC can also serve as a proxy for any UE to terminate connections, serve requests, or participate in ata collections in the capacity of the UE. Furthermore, clou computing can also be use to scale the central controller in the SDN framework, which is computation-intensive an coul be a bottleneck as well. Instea of using a single super-scheuler to perform all the resource scheuling an traffic engineering, we can istribute these tasks to many ifferent servers if parallelism coul be exploite from the routing algorithm. However, note that resource virtualization through ata centers shoul be taken into account when making allocation ecisions, as virtualize resources have ifferent costs an limitations from the eicate ones we are familiar with. 3 Design Goals: Viewing the limitations of the latest 4G LTE network versus the requirements of the 5G mobile network, we propose an SDN architecture with clou computing to enhance the network capabilities, as shown in Fig. 4. The main esign goals are as follows: 1 Integrate resource selection an content routing. As content/computing resources become more scattere an ynamic, the traitional way of performing selection separately from routing will no longer suffice. 2 Flexible policy enforcement through SDN. Fine-graine control shoul be exercise to steer ata flows through NE eploye throughout the network. 3 Proviing truly en-to-en QoS guarantees. QoS optimization shoul be pushe to both ens of ata flows rather than on a hop-by-hop basis. Fig. 4. Policy-aware routing in a software-efine mobile network. 4 Scattering of network functions. Light-weight NE shoul be wiely eploye to avoi triangular routing, probably on-eman through NFV. 5 Incorporate MCC to enhance the UE capabilities using clou infrastructures. Special consierations nee to be mae for allocation an routing. 6 Competitive network resource optimization. This has often been carrie out in a istribute way an settle at sub-optimal heuristic solutions, yet it can be further explore in a centralize manner. 7 Coorination across mobile networks, either for ata roaming across ifferent carriers or when multiple raio access technologies (RAT co-exist for the same carrier. IV. PROBLEM FORMULATION We consier a generic mobile network represente by a irecte graph G =(V, E, wherev enotes a set of n noes making up the network, an E enotes a set of m irecte links connecting the noes, as shown in Fig. 4. The graph G is assume to be mesh-connecte, having multiple paths connecting each pair of noes. Every link e is associate with a fixe capacity c e an a unit operational cost b et that may change over slotte time t. The noe set V contains stanar routers an special NE attache to routers, incluing gateways, mileboxes, ata centers, etc. Note that we only care about those NE that are hoste insie the mobile network or within the reach of the SDN controller, an there may be multiple instances of each type of NE in the network. These NE can be either eicate equipment with a fixe processing capacity, or ynamically provisione through NFV from ata centers at certain operational costs. Such noe capacities c v an unit costs b vt are assume to be pre-processe an transforme into link parameters mentione above. This can be one by splitting each noe v into two shaow noes v an v,an connecting them with a single irecte link e = vv that bins with capacity c e = c v an unit cost b et = b vt. Also assume that we have a buget B t that restricts the total costs we coul spen uring any time slot t. This buget may change over time in response to varying traffic volume, electricity price, company strategy, an many other factors. Each mobile traffic flow enters the network at its ingress noe an has to visit a set of NE in a specifie orer before

5 CAO et al.: ENHANCING MOBILE NETWORKS WITH SDN AND CLOUD COMPUTING 1435 leaving the network at its egress noe. Let s enote the ingress noe for flow, t enote the egress noe, an k represent the number of NE that flow nees to traverse. The policy list for flow is efine as an orere list {M [1],M [2],...,M [k ]}, with each M [i] representing the i-th type of NE that flow nees to visit in a sequential orer. When MCC is enable for a certain flow, its traffic shoul always route through the hosting ata center, so this shoul also be inclue in the policy list. For each flow, policyaware routing aims at fining paths p that conform to its policy list, an we enote the set of all such paths by P. A policyaware path p is mae up of k +1 route segments p (i,where p (1 routes from s to M [1], p (i for 2 i k routes from M [i 1] to M [i], anp (k+1 routes from M [k ] to t. Also, each traffic flow is associate with a certain traffic eman. For mobile networks, we assume that each traffic eman can be specifie using a fixe (or boune banwith h an a certain time uration τ τ f τ s +1, where τ s represents the starting time slot, an τ f represents the finishing time slot. For MCC-enable flows, a single h may not accurately escribe the traffic eman, because the banwith before an after passing the host can be ifferent. This can happen if the MCC acts as a file server for external request, performs aaptive transcoing on incoming vieos, or simply caches uplink/ownlink ata when the connection is unstable. Things can become even more complicate uring mobile-to-mobile communications. Therefore, we a another set of weight parameters γ (i for 1 i k +1 such that γ (i h enotes the banwith on route segment p (i. Note that h is always chosen such that γ (1 =1. In aition to routing policies an traffic emans, mobile traffic is also subject to certain QoS constraints. Typical QoS constraints [23] inclue boune packet latency, limite packet loss, etc. In this paper, we mainly focus on the en-toen packet latency constraints, while the packet loss limit can be enforce in a similar way. The en-to-en latency constraint W for each mobile traffic flow can be ifferent. In this paper, we assume that the network latencies are mainly cause by processing at NE noes an transmission through physical links, an each noe/link contributes a fixe amount of latency w e regarless of the traffic loa. Such an assumption is reasonable if we set asie a cushion banwith an balance/restrict the traffic loa on each link to be strictly lower than a certain threshol, which is part of the goal of finegranularity scheuling enable by SDN. The more challenging problem with congestion elays will be stuie in future work. Therefore, the sum of latencies on any QoS-guarantee path p serving flow cannot excee W. In aition to the ento-en latency constraint, sometimes we may nee to enforce some other constraints as well. One example is the latency between a UE an its host when MCC is supporte because the user experience of MCC is often latency-sensitive. In this case, the sum of w e on the UE-to-MCC segment p mcc of any QoS-guarantee path p serving flow cannot excee W mcc. Another example of this type is bouning the UE-to-SGW latency for more precise locality or less own time uring hanover, which can be moele in a similar way. See Table I for a complete list of notations. TABLE I LIST OF NOTATIONS V. ONLINE TRAFFIC STEERING FOR MOBILE NETWORKS A. Maximizing Total Accepte Traffic Over Time A major objective of mobile traffic steering is to maximize the total amount of traffic accepte over time in a policy-aware manner without violating any noe/link capacity, operational buget, or QoS constraints. In this section, we formulate a variant of the well-known maximum multi-commoity flow problem [24], an incorporate other techniques to meet policy awareness an QoS constraints. We now formulate the problem as a mixe integer programming problem. We use a binary variable x p to inicate whether path p is chosen for flow. When a request enters the system, it is either accepte for the entire uration an carrie on a single path or is rejecte. Maximize α h τ x p (1a p P Subject to x p 1, (1b p P γ (i h x p c e, e, t t [τ s,τ f ] p P e p i e p (i (1c b et γ (i h x p B t, t t [τ s,τ f ] p P i e p (i (1 x p w e x p W,, p P (1e e p x p {0, 1},, p (1f Objective (1a seeks to maximize the total accepte traffic over time. Note that the banwith of a single flow may change across ifferent route segments i ue to MCC or other NE, while we are only intereste in the component h that is irectly perceivable from the UE, because users are often charge base on this portion of mobile ata. Constraint (1b ensures that each flow is allocate at most once. Constraint (1c enforces the link capacity constraints at

6 1436 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 all times. Since each link can be traverse by multiple route segments of multiple flows with ifferent banwith requirements, we must aggregate all of them together for the overall link utilizations. Constraint (1 restricts the operational costs to be lower than the buget we have at any time. Constraint (1e enforces the en-to-en latency constraints on all selecte paths. For paths that are not selecte, these constraints are satisfie trivially ue to x p =0. Other QoS constraints can be ae in a similar way on eman. Constraint (1f efines the allocation to be unsplittable (all-or-nothing, which makes the formulation a mixe integer programming (MIP. Solving the above MIP formulation irectly woul incur exponential complexity ue to the large number of policyaware caniate paths in P, which is unesirable. Thus we follow a primal-ual approach to reuce the complexity. In orer to write the ual, we efine three sets of ual variables: z correspons with every eman fulfillment constraint (1b, l et correspons with every link capacity constraint (1c; an φ t correspons with every buget constraint (1. Enforcing QoS constraints (1e an fining policy-aware paths P will be hanle separately. Minimize β z + c e l et + B t φ t (2a t e t (i γ Subject to (l et + b et φ t + z 1, τ t [τ s,τ f ] i h τ e p (i, p P (2b l et 0, e, t (2c z 0, (2 φ t 0, t (2e B. A Generic Competitive Online Routing Algorithm As both primal an ual formulations of the problem have been set up, we can now move forwar to solve them. There is abunant work in the literature giving optimal or heuristic offline solutions [25] to primal-ual problems of this kin. One major assumption of these offline solutions is that all traffic flows are known ahea of time. However, ue to the continuous an first-come-first-serve (FCFS nature of mobile ata traffic, it is usually infeasible to preict the flows in avance an solve the routing problem offline or through batch processing. Therefore we shall focus on online algorithms that hanle flow allocations in real-time. We shall assume for now that the algorithm is execute by a stanalone server, an then she light on practical constraints an istribute execution later. Our objective is to fin an online algorithm with a guarantee competitive ratio. The competitive ratio of an online algorithm is the worst case ratio of the objective function achieve by the online algorithm to that achieve by the optimal offline algorithm. We first escribe a generic competitive online routing algorithm [25], an later exten it to aress the unique challenges in mobile traffic steering. As we shall see later, the primal-ual algorithm uses the a combination of the ual variables l et + b et φ t as the link lengths, an upates them accoring to the primal allocations. 1 Initially, all link lengths are set to 0; 2 Whenever a new traffic flow is requeste: a if the shortest path to route this flow is no longer than a certain threshol, then the flow is accepte an allocate onto its shortest path, an the length of all links on this path are incremente accoring to their utilizations; b if the shortest path to route this flow is longer than the threshol, the flow is rejecte. The intuition behin this generic algorithm is that highly congeste links will accumulate larger lengths an thus are less likely to be further utilize when performing shortest path routing for future traffic flows. In this way, mobile traffic can be steere throughout the network rather than limite to ranom hot-spot areas. Meanwhile, each iniviual flow is still allocate onto a short path, consuming as few resources as possible while avoiing congestion in the network. Furthermore, when the network is alreay congeste, those flows that require too many hops or must go through certain hot-spot areas will be roppe in orer to leave space for less-emaning future requests. With all these heuristics working together, a logcompetitive online algorithm can be esigne with a careful selection of parameters an formulas. Using this metho, the network-wie multi-commoity flow problem is essentially transforme into a series of less complicate shortest path routing problems. Compare with other alternatives that approaches the MIP formulation irectly, this algorithm retains the inepenence an integrity of each flow an is especially extensible for more complicate routing problems. As we shall see later, special routing policies an various QoS constraints for each flow can be easily incorporate into this algorithm. C. Enforcing Routing Policies With Graph Layering A key step in the generic online routing algorithm involves computation of the shortest path from s to the t for each flow. When applying this generic algorithm to policy-aware routing, the simple shortest path will nee to be replace by a policy-aware shortest path visiting multiple NE along the way, which is much more ifficult to compute. Since there are multiple instances of each NE, part of the optimization is to etermine which NE to route through. Our approach to enforce routing policies is to construct a layere graph G, which is constructe as follows. For any flow, wemakek +1 copies of the original graph G an turn each copy G (i =(V (i, E (i into one layer of G.As we shall see later, layer 1 at the bottom is use for routing from the source s (1 to M (1 [1], layer 2 i k is use for routing from M (i (i [i 1] to M [i], an the top layer k +1 is use for routing from M (k +1 [k ] to the estination t (k +1. The length of link e (i in layer i is set to γ (i times the its mean length over the uration of flow, i.e., l e (i (l et + b et φ t /τ, i, e (i. Because there are τ s t τ f γ (i no negative links in the graph, the shortest path in each layer must be a simple path visiting each noe/link at most once. In orer to enforce the routing policies, we then introuce (0, 0, 0, links to stitch the layers together,

7 CAO et al.: ENHANCING MOBILE NETWORKS WITH SDN AND CLOUD COMPUTING 1437 Fig. 5. Enforcing routing policies in a layere graph G. where (l et,b et,w e,c e enotes the link parameters. Each instance of M (i [i] in layer 1 i k is connecte to its corresponent M (i+1 [i] in layer i +1.Also,weaa unique ingress noe ṡ to connect with s (1 in the bottom layer through a (δ, 0, 0, link of length 0 <δ 1, an a unique egress noe ṫ to connect with t (k +1 in the top layer through a (0, 0, 0, link. The reason why we nee this small δ here will be explaine later. In this way, flow must start from the bottom layer, climb upwars through the (0, 0, 0, links by visiting the NE in a specific orer, an finally reach the estination in the top layer. The policy-aware shortest path ṗ can be foun by running a simple shortest path routing algorithm in the layere graph Ġ from ingress noe ṡ to egress noe ṫ, as shown in Fig. 5. It is important to note that the parameters of all auxiliary links introuce in this paragraph are fixe an will never change throughout the optimization process regarless of the path upates. Finally, the shortest path ṗ we have just foun in the layere graph G can be mappe back to p in the original graph G, by suppressing all copies into one single layer an eliminating auxiliary links connecting the layers. D. Meeting QoS Requirements Recall that when writing the ual formulation in section V-A, we have intentionally set asie the QoS constraints x p e p w e x p W. These constraints are ifficult to transform into the ual formulation, hence they are enforce as aitional constraints to the caniate paths. Similar to enforcing routing policies, we coul substitute the simple shortest path in the generic online routing algorithm with a constraine shortest path to guarantee the QoS. There have been much work in literature aressing the constraine shortest path problem since 1980 s, an many ifferent algorithms have been propose. Therefore, we coul reuse any existing constraine shortest path algorithm as is, an plug it into the generic framework as a subroutine of our online routing algorithm. It is also important to note that QoS-guarantee can be achieve simultaneously with policy-awareness. This is because the key step of policy enforcement in section V-C is still a simple shortest path routing, though in a much larger graph with multiple layers. If we set w e (i = w e for all links e (i in all layers i an replace the simple shortest path in the layere graph G with a constraine shortest path, then QoS enforcement can be integrate with policy-aware routing. Unlike the simple shortest path problem that can be solve efficiently within O(n log n+m time in a generic graph with n noes an m links, the constraine shortest path problem is generally NP-har, so we have to settle for a certain level of approximation. Many propose algorithms only work for the single constraint scenario, while only a few can be extene to support multiple constraints as well. For example, in aition to meeting en-to-en latency requirements, sometimes we may want to ensure that the selecte SGW an MCC hosts are close enough to the users as well, so as to improve the user experience uring cellular hanover or computation hanoff. In the following we show how ifferent (1 + ɛ approximations can be achieve in various scenarios. 1 Single-Constraint Scenario: Fast fully polynomial time approximation schemes (FPTAS, like Hassin s algorithm [26] achieving (1 + ɛ path length approximation within O ( log log UB LB (ɛ 1 mn +loglog UB LB time, can be use to compute the constraine shortest path efficiently. When computing the policy-aware shortest path in G with en-to-en latency constraint, the upper boun of the shortest path length coul be set as UB = k max n l max, the lower boun coul be set as LB = δ, an n, m shoul increase by k max times. We shall prove later that lmax = O(1, while δ in the lower boun comes from the short link connecting ṡ to s (1 an is intentionally ae to avoi the infinite loop to achieve (1 + ɛ path-length approximation aroun L(ṗ =0. Here we use a max subscript to inicate the maximum value of the parameter, an a min subscript to inicate the minimum positive value of( the parameter. Therefore, the final complexity is Tṗ = O log log(k max n/δ ( ɛ 1 k 2 max mn + log log(k max n/δ. 2 Relaxe Multi-Constraint Scenario: Computing the shortest path uner multiple sie constraints is much more ifficult. Existing literature often gives solutions to the problem after certain relaxations. In [27, Sec. 3.3], the authors relax the original problem by ropping the path integrality constraint, so that traffic flow can be split into multiple paths, an the constraints are only satisfie by the average values of these paths. Then they use a hull approach to obtain the solution to the relaxation, which is a lower boun of the multiconstraine shortest path but without a constant approximation guarantee. As a result, this metho is only suitable for splittable traffic an QoS guarantee in an average sense. When enforcing several ifferent latency constraints (en-to-en, MCC, UE-to-SGW in a generic graph mae up of n noes an m links with integral lengths an latencies, this algorithm can erive an exact solution to the relaxation with a conjecture

8 1438 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 time complexity of O ( log(nl max w max (n log n + m.but for a layere graph G with real-value parameters, if we terminate the hull approach as soon as the gap between the upper boun an lower boun of L(ṗ becomes smaller than ɛδ ɛl(ṗ, then the final time complexity is Tṗ = O(log(nk max lmax w max /ɛδw min (nk max log(nk max + mk max. 3 Strict Multi-Constraint Scenario: Designing an online algorithm for the strict multi-constraint shortest path is even more challenging, an fast solutions with a constant path-length approximation guarantee might not be available. Therefore, we coul only fall back to classical ynamic programming approaches [27]. E. A Log-Competitive Online Algorithm for Maximizing Total Accepte Mobile Traffic Over Time Summing up all the aforementione techniques, a logcompetitive online solution is esigne as follows. Denote the largest values of k an τ for all flows as k max an τ max. Also, because the network is connecte, we have n = O(m. Hence the time complexity for constructing G is T G = O(mk max, the time complexity for efining the length system L is T L = O(mk maxτ max, the time complexity for fining the policy-aware an QoS-guarantee shortest path ṗ in G is Tṗ as in section V-D, an the time complexity for upating the path lengths along p in G is T p = O(nk max τ max. Therefore, the overall time complexity for each flow allocation is T = O(mk max τ max + Tṗ. Such a complexity can become high as the network grows larger an more traffic flows emerge, an even worse if we want to fin close approximations of QoS-constraine shortest paths. In orer to scale the system, we nee to exploit the parallelism of the algorithm an istribute the computationintensive tasks to multiple scheulers. Fortunately, the algorithm above is inee parallelizable uner reasonable assumptions. To be more specific, if the cost of each single traffic flow is much smaller than the link capacities an buget constraints, then the allocation of a set of traffic flows can be elegate to multiple ifferent scheulers an performe simultaneously using their local versions of the network states (which may be slightly outate. Allocations performe by each single scheuler nee to be eventually propagate to all other scheulers to upate their local versions of network states, but strong consistency among ifferent scheulers is not a must, because the allocation of each single traffic flow will not significantly affect the network an buget utilizations. While this breaks the serializable consistency, the loss of accuracy is minor an the whole system can tolerate out-of-sync network states to some reasonable extent. F. Competitiveness Analysis of the Online Algorithm Denote by Γ max the maximum value of Γ p, Ψ max the maximum value of Ψ p, γ min the minimum positive value of γ (i, b min the minimum positive value of b et,anτ max the maximum value of τ. For those γ (i =0(or b et =0, l et (or φ t is meaningless an therefore ignore. The competitiveness of Algorithm 1 can be prove as follows: Algorithm 1 A Log-Competitive Online Algorithm for Maximizing Total Accepte Mobile Traffic Over Time Data: GraphG =(V, E; link capacities c e an unit costs b et ; a sequence of flows, each with source s, sink t, banwith requirement h, policy list {M [i]} together with banwith variation {γ (i } for 1 i k, starting time τ s, an finishing time τ f ; operational bugets B t;asmallδ; Result: Sequential ecision of flow allocation x p ; Initialize x p := 0,, p; l et := 0, e, t; z := 0, ; φ t := 0, t; foreach flow o Construct a layere graph G with k +1copies of the original graph G; Stitch ajacent layers G (i in the layere graph G together by connecting each instance of M (i [i] in layer 1 i k to its corresponent M (i+1 [i] in layer i +1 through (0, 0, 0, links; Define a length system L on each link e (i such that l e (i t [τ s,τ f ] γ(i (l et + b et φ t /τ, i, e (i ; Connect ṡ to s (1 through a (δ, 0, 0, link, an ṫ to t (k +1 through a (0, 0, 0, link; Fin the policy-aware an QoS-guarantee shortest path ṗ from ṡ to ṫ in the layere graph G uner the length system L; Map ṗ from the layere graph G to its realization p in the original graph G; if L(p L(ṗ i x p := 1; z := h τ ; Γ p i e γ(i 1 ; e p (i l et := l et (1+ i e p (i γ (i h e (i ṗ (i γ (i h c e + l e (i < 1 then i e p (i Γ p c e, e p, t [τ s,τf ]; Ψ p t [τ i s,τ f ] b e p (i et γ (i /τ ; φ t := φ t (1+ b etγ (i i e p (i h B t + i e p (i b etγ (i h Ψ p B t, t [τ s,τf ]; t [τ s,τ f ] i e p (i γ (i 1 The algorithm prouces a feasible ual solution. Whenever a new flow is requeste, either it is accepte an we assign z = h τ, or it is rejecte because its shortest path length L(p =L(ṗ = i l e (i ṗ (i e (i = (l et + b et φ t /τ 1, so inequality (2b always hols for any flow. Obviously, inequalities (2c an (2 are also vali, hence the ual solution is feasible. 2 In each iteration, the change of the ual objective is boune by that of the primal objective Δβ 4Δα. If L(p 1, nothing changes, an Δα =Δβ =0.

9 CAO et al.: ENHANCING MOBILE NETWORKS WITH SDN AND CLOUD COMPUTING 1439 If L(p < 1, wesetx p := 1, thus Δα = h τ, whereas Δβ =Δz + c e Δl et + B t Δφ t (3 t e t = h τ + ( γ (i c h l et e + γ(i h c t [τ s,τ f ] e p e Γ p c e i e p (i + ( b et γ (i B h φ t t + b etγ (i h B t [τ s,τ f ] i t Ψ p B t e p (i =3h τ + γ (i h (l et + b et φ t t [τ s,τ f ] i e p (i =3h τ + L(p h τ 4h τ =4Δα (4 Therefore, Δβ 4Δα always hols. 3 The link capacity constraints in the ual problem is violate by at most O ( log(γ max τ max /γ min,ifc e 1 an c e i e p (i γ(i h,, p P, e. We shall prove through mathematical euction that the value of l et just before the arrival of flow can be boune by the following expression: l et ( (1+ 1 c e P P < t [τ s,τf ] p P e p Pi e p (i γ (i x p h 1. Γ max (5 Initially l et (1 = 0, anallx p =0, so the inequality hols. Assume that the inequality hols before flow arrives. Then after this iteration, either flow is rejecte an x p =0oes not affect the valiity of l et ( +1,orflow is accepte an l et ( +1 (6 = l et ( 1+ γ (i h + γ (i h c e Γ p c e i e p (i l et ( 1+ = (1 + 1 c e 1+ (1+ 1 c e i e p (i γ (i h + c e i e p (i i e p (i P P < t [τ s,τf ] p P e p Γ max γ (i h 1 c e Γ max i e p (i P P <+1 t [τ s,τf ] p P e p γ (i h Γ max c e Pi e p (i γ (i x p h Pi e p (i γ (i x p h 1. Γ max (7 The last approximation hols if c e 1 an c e i e p (i γ(i h,, p P, e. Therefore, the inequality still hols after serving flow, an the mathematicaleuction is vali. On the other han, ue to the acceptance conition L(p < 1, an the assumption c e γ (i i e p (i h,, p P, e, wehavel et ( < (τ max /γ min ( = O(τ max /γ min,an l max < 1 ( = 3. Combining this with inequality (5, the link utilization can be boune by u et t [τ s,τ f ] p P e p i e p (i γ(i h x p c e O( log(γ max τ max /γ min c e log(1 + 1 c e O ( log(γ max τ max /γ min (8 The last approximation hols if c e 1, e. 4 The operational buget constraints in the ual problem is violate by at most O ( log(ψ max τ max /γ min,ifb t 1 an B t i e p b etγ (i (i h,, p P, t. Similar to the proof in step 3, we show through mathematical euction that the value of φ t just before the arrival of flow is boune by the following expression (9, as shown at the bottom of this page. On the other han, ue to the acceptance conition L(p < 1, an the assumption B t i e p (i b et γ (i h,, p P, t, we have φ t ( (τ max /b min γ min ( = O(τ max /b min γ min. Combining this with inequality (5, the buget utilization can be boune by η t t [τ s,τ f ] p P i e p b (i et γ (i h x p B t O(log ( Ψ max τ max /b min γ min B t log(1 + 1 B t O ( log(ψ max τ max /b min γ min (10 The last approximation hols if B t 1, t. Rather than viewing u et in steps 3 an η t in 4 as violations to the capacity an buget constraints, we can scale c e an B t instea, an absorb the logarithmic factors in the competitive ratio so as to avoi violations. It is possible to show that (within constants this is the best possible competitive ratio. VI. MORE DISCUSSION A. Support for Multi-RAT Scenarios Latest UE may support multiple raio access technologies an connect to ifferent mobile networks that iffer significantly in performance, availability, an routing policies. In aition to the ifference in their inherent capabilities, the network performances are also affecte by fluctuations in φ t ( (1 + 1 B t P P P < t [τ s,τf ] p P e p i Pe p (i b etγ (i x p h 1 Ψ max. (9

10 1440 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 Fig. 6. Extension on graph layering to support multi-rat. Fig. 7. Merging layers representing common route segments in multi-rat. traffic emans an change of user locations. Therefore, when routing mobile traffic flows, these ifferences must also be taken into account in orer to optimize the system performance an the user experience. In multi-rat scenarios, throughput maximization can be one in the same way as in a single-rat scenario, except that we shoul merge multiple networks into one large graph an enforce ifferent routing policies for each possible routing option that the flow can be serve with. A straightforwar solution is to reuse the graph layering technique to set up separate branches for each routing option. For example, assume that the UE can access either 2G or 4G LTE at this point, then it has at least three routing options: 1 UE BTS SGSN GGSN Server over 2G GERAN an GPRS core network. 2 UE BTS SGSN SGW PGW Server over GERAN an GPRS interworking with EPC. 3 UE enoeb SGW PGW Server over 4G LTE E-UTRAN an EPC. If we merge both 2G an 4G LTE networks an set up a branch for each routing option, then the layere graph Ġ shoul look like the following Fig. 6, which requires 3 branches an a total of 11 copies of G. Note that ifferent copies of the estination noe t in each branch shoul be aggregate at the top because we only nee one feasible path from s to t across all 3 branches. The straightforwar solution meets our expectations but can be further improve. Because option 1 shares route segment UE BTS SGSN with option 2, an option 2 shares segment SGW PGW Server with option 3, it is possible to use less copies of G to cover all routing options in the multi-rat scenario by merging those layers associate with common route segments. As shown in Fig. 7, only 7 copies of G is sufficient to get the same result. B. Routing Across Network Bounaries In many countries, mobile networks are operate by multiple mobile Internet service proviers across ifferent regions. In many cases, a UE may not receive aequate cellular coverage from its home network an thus requires roaming to a visite network. However, charges mae by ifferent mobile networks an QoS offere by ifferent routing options may vary wiely. In such scenarios, coorination across ifferent mobile networks an communications between their SDN controllers are necessary. We aopt a master-slave moel for network cooperations. Since it is often the home network that etermines a user s privileges an charges, we let the visite network serve as the slave to provie a list of routing options to the master, while the home network shoul act as the master to make the final routing ecisions base on the given options. In 4G LTE networks, home-route roaming traffic utilizes a visite SGW (V-SGW an a home PGW (H-PGW, an crosses the bounary of home an visite mobile networks via an IP exchange (IPX network. We assume that the visite network etermines the route segment from the UE to the IPX (enote by noe v an makes a competitor charge epening on the resources use, while the home network takes over the control thereafter. In this process, the visite network ecies how to moel the link lengths/costs an how to route the roaming traffic till the IPX noe v, e.g., accoring to the propose primal-ual algorithm, an then informs the home network of the competitor charge g v an the actual network latency w v. On the other han, the home network effectively sees a single (0,g v,w v, link between the UE an the IPX noe v, an the total cost of routing is the sum of noe/link costs from noe v to the server plus the competitor charge g v, as shown in Fig. 8. In case there are multiple IPXs connecting these two networks, the visite network nees to provie routing options from the UE to all possible IPXs, while the home network nees to select the best route from all possible IPXs to the server. The rest of the problem is exactly the same as before. The metho above applies to two selfish network operators with conflict of interest an limite cooperation. Another version of the problem is for two network operators with a share objective or two SDN controllers in the same network

11 CAO et al.: ENHANCING MOBILE NETWORKS WITH SDN AND CLOUD COMPUTING 1441 TABLE II SIMULATION PARAMETERS FOR EXPERIMENT I Fig. 8. Routing across two mobile network operators. aiming for full cooperation. In this case, it is tricky to ivie the en-to-en latency quota across the two networks, but with share network state information each SDN controller is able to make the routing ecision alone an the scheuling tasks can be istribute to both controllers. VII. SIMULATION RESULTS In this section, we perform numerical simulations to compare the effectiveness of the propose algorithm against other prevalent heuristics, an emonstrate how our algorithm outperforms these alternatives. First, we list the routing algorithms to be compare: 1 PhSp: per-hop shortest-path routing. This is the baseline algorithm in which all links are given a uniform unit length, an the scheuler iteratively fins the closest (in terms of the number of links next-hop NE in a hop-by-hop manner an connects these segments up to construct an en-to-en path. 2 PhMl: per-hop minimum-latency routing. This works in a similar way to PhSp except that the network latency is minimize instea of the length of each route segment. 3 Sp: en-to-en shortest-path routing. This algorithm is an improve version of PhSp that utilizes the graph layering technique to fin the en-to-en shortest (in terms of the number of links path instea of connecting up shortest route segments hop by hop. 4 Ml: en-to-en minimum-latency routing. This works in a similar way to Sp except that we are now trying to fin an en-to-en path with the minimum network latency rather than the number of links that compose the path. 5 Csp: en-to-en constraine shortest-path routing. In this algorithm, both network latencies an path lengthes are consiere, an our objective is to minimize the en-toen path length uner network latency constraints. 6 PCsp: en-to-en primal-ual constraine shortest-path routing. This is the propose competitive online algorithm that is aware of banwith variations an follows the primal-ual approach to assign link lengths an perform en-to-en latency-constraine shortest path routing. The next step is to construct a mobile network an some traffic flows for simulation. In orer to reflect real-worl latencies, we abstract the contiguous Unite States territory as a 2800mi 1500mi rectangular plane, an eploy network noes an links on this plane. We first construct a backbone network with 20 backbone routers an 30 backbone links. These backbone noes are ranomly place an linke but we ensure that the whole network is connecte. The backbone links are assigne a uniform capacity of 40Gbps, which correspons with the OC-768 fiber links that were eploye by AT&T aroun Then we a 80 more routers an some more links to simulate the wie area networks (WANs operate by local ISPs. These WAN noes are eploye sequentially an they only connect to one or two nearest alreay eploye noes. These WAN links are assume to have a uniform capacity of 10Gbps. Regarless of backbone network or WAN, the network latency of a link is approximate as the geographical istance between the two en noes ivie by the spee of light in fiber. After that, 2000 traffic flows are generate with ranom starting/ening time slots, UE/resource noes, an banwiths between 1Mbps an 1Gbps. Each traffic flow has only one UE noe, but may have multiple resource noes to account for CDN or geographical istribution of resources. In Experiment I, we assume that all UE-generate traffic flows nee to traverse through a SGW an a PGW before reaching the server. These UE, SGW, PGW, an server noes are ranomly attache to backbone an WAN noes. A list of all parameters use in the simulation is summarize as in Table II, where ranint(a, b represents a ranom integer value uniformly chosen from range [a, b]. We o not impose any operational costs, capacity limits, or banwith variations on NFV an MCC elements in this experiment, so that the avantages of the propose algorithm in terms of routing uner capacity an latency constraints as well as the impact of ifferent NE ensities can be clearly emonstrate. Also note that all topologies an capacity/eman values are arbitrarily chosen here for emonstration purposes only; the actual values may change significantly over time. When applying the algorithms to real networks, up-to-ate link capacities, accurate user eman estimations, an realistic buget/cost assessment are key consierations. We first comparethe performanceof all 6 routing algorithms uner ifferent NE ensities. The metric to be use for

12 1442 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 Fig. 9. Total amount of traffic accepte uner ifferent NE ensity levels. Fig. 11. Impact of SGW/PGW/resource ensity levels on the total amount of traffic accepte over time uner 10ms latency constraints. Fig. 10. Total amount of traffic accepte uner ifferent latency bouns. evaluation is the total amount of traffic accepte over time, which is also the objective of the problem aresse in this paper. The simulation results are shown in Fig. 9. As we can see, the performance of the baseline PhSp algorithm is always lower than others in this simulation. PhMl performs better in this simulation because it seeks the minimize the latency for each flow so that more flows can meet the latency constraints. Sp an Ml are superior to their per-hop counterparts in this simulation because the routing path is optimize from en to en. Csp achieves even higher performance in this simulation by balancing the traeoff between path lengths an network latencies. Finally, our propose PCsp algorithm is able to accept the highest amount of traffic over time, which is more than 100% higher than the baseline algorithm in many cases. The significant improvement shoul be attribute to 1 ento-en path optimization, 2 congestion-aware traffic steering, an 3 the primal-ual approximation. Next, we graually ecrease the latency boun from 10ms to 2ms, an see how ifferent algorithms react to such changes. The results are plotte in Fig. 10, which has a similar look to Fig. 9. Again, the propose PCsp algorithm significantly outperforms other heuristics. Also, we can see from both figures that the performances of Ml, Csp,anPCsp are very close when the latency constraint is stringent given the NE ensity. This is ue to the fact that 1 there are few alternative routes that can meet the low latency boun, an 2 the network is generally uner-utilize so congestion avoiance oes not take effect. Fig. 12. Impact of SGW/PGW/resource ensity levels on the network latency. Then we set out to investigate the impact of separate SGW, PGW, an resource ensities on the network performance. In aition to the All scenario in which the ensities of all NE are change simultaneously, we also test the network performances when only one kin of NE is affecte at a time (e.g., n_sgw =4, 8, 12, 16, 20 while all others are kept the same (e.g., n_pgw = n_resources =20, as shown in Fig. 11. Note that all bars at n_sgw = n_pgw = n_resources =20are the same because they are essentially using the same settings. From the plot we can see that the impact of SGW ensity is more significant than that of PGW ensity, which in turn is more significant than that of resource ensity. The reason why SGW ensity plays a more important role is ue to the fact that there is only one UE involve in each flow, whereas multiple caniate servers can participate in the same route selection process. Therefore, the triangular routing problem is more sensitive to the SGW ensity. On the other han, resource ensity has a much weaker influence than SGW an PGW ensities because flows terminate at resource noes an the choice of resource noes o not cause any further routing inefficiencies. Network latency is another important metric to be consiere. In all previous simulations, we have set a latency boun, which not only limits the amount of traffic that can be accepte

13 CAO et al.: ENHANCING MOBILE NETWORKS WITH SDN AND CLOUD COMPUTING 1443 TABLE III SIMULATION PARAMETERS FOR EXPERIMENT II TABLE IV COMPARISON OF ALGORITHMS Fig. 13. Total amount of traffic accepte over time uner ifferent bugets. but also biases the average network latency observe by all flows. In Fig. 12, we remove the latency constraints from all flows, an use the propose PCsp algorithm to show how ifferent NE ensities affect the average network latency. Note that after we remove the latency constraints, almost all traffic emans can be accepte in most cases, but the average network latency increases significantly. This is becausepcsp seeks to maximize the total amount of traffic accepte over time by balancing the link utilizations throughout the network, an thus the paths chosen may not have low latencies after the latency constraints are remove. Again, we coul see that SGW ensity most significantly affects the average network latency, followe by PGW an resource ensities. Till now we have been focusing on the performance uner network capacity an latency constraints only. The capabilities of the propose PCsp algorithm in hanling banwith variations over ifferent route segments an meeting total operational bugets bring more avantages to the propose algorithm. These shall be verifie in Experiment II. In this experiment, we reuse the backbone an WAN network settings. As outline in Table III, a fixe number of 20 SGW, PGW, an MCC noes are attache to the network, each having a ranom operational cost between 0 an 2 with an average of 1 unit per time-slot. There are 1000 ranomly generate traffic flows. Each user request has 20 caniate resource noes on average, an must sequentially pass through one instance of SGW, MCC, an PGW before reaching the resource. The banwith may vary ranomly between 1/5 an 5 when passing through the MCC host. In Fig. 13, we compare the performance of various routing algorithms uner ifferent buget constraints. It can be seen that PCsp gains huge avantage over other alternatives when the buget is tight, because it gives preference to lessemaning traffic flows in this situation while all others are ignorant of the capacity an buget constraints. The avantage of buget awareness graually goes away as the buget increases an the performance becomes network-constraine. Another interesting phenomenon in this simulation is that the orer of performances here significantly iffers from our previous observations from Experiment I: Sp performs worse than PhSp, Ml performs worse than PhMl, ancsp performs worse than Ml. Such a contraiction seems surprising at first glance, but this is actually reasonable because all alternative algorithms other than PhCsp are just heuristics. Some heuristics may happen to perform better than others in certain situations, but the lack of simultaneous congestion an buget awareness makes them incapable of securing a guarantee performance improvement. Finally, the avantages of the propose PCsp algorithm over other alternatives are summarize in Table IV. VIII. CONCLUSION In this paper, we reveal the infrastructure inefficiencies an routing inflexibilities in current mobile networks, an propose an SDN-base architecture with clou computing to aress the potential bottlenecks an challenges of next generation mobile networking. In orer to enable policy-aware routing an provie QoS guarantee, we formulate a multi-commoity flow problem with sie constraints, an follow the primalual approach to obtain a log-competitive solution using a fast online approximation algorithm. Extensive simulations have been performe to investigate the impact of various parameters on the network performances. These results also show that our algorithm significantly outperforms prevalent heuristics ue to en-to-en optimization, congestion/buget awareness, an primal-ual approximation. REFERENCES [1] Network Architecture, ocument 3GPP TS Release 13, 3GPP, [2] V. Sekar et al., Design an implementation of a consoliate milebox architecture, in Proc. NSDI, 2012, pp [3] L. E. Li, Z. M. Mao, an J. Rexfor, Towar software-efine cellular networks, in Proc. EWSDN, Oct. 2012, pp [4] General Packet Raio Service (GPRS Enhancements for Evolve Universal Terrestrial Raio Access Network (E-UTRAN, ocument 3GPP TS Release 13, 3GPP, [5] J. Best. The Race to 5G: Insie the Fight for the Future of Mobile as we Know it. [Online]. Available: oes-the-worl-really-nee-5g, Dec

14 1444 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 3, JUNE 2017 [6] NGMN Alliance, 5G White Paper. [Online]. Available: ngmn.org/uploas/meia/ngmn_5g_white_paper_v1_0.pf, Feb [7] Open Networking Founation, Software-efine networking: The new norm for networks, ONF White Paper, Apr [8] X. Jin, L. E. Li, L. Vanbever, an J. Rexfor, SoftCell: Taking control of cellular core networks, Computer Science Department, Princeton Univ., Princeton, NJ, USA, Tech Rep. TR , [9] P. Demestichas et al., 5G on the horizon: Key challenges for the raioaccess network, IEEE Veh. Technol. Mag., vol. 8, no. 3, pp , Sep [10] A. Basta et al., Applying NFV an SDN to LTE mobile core gateways, the functions placement problem, in Proc. ACM Workshop All Things Cellular, 2014, pp [11] T. Taleb, Towar carrier clou: Potential, challenges, an solutions, IEEE Wireless Commun., vol. 21, no. 3, pp , Jun [12] B. G. Chun, S. Ihm, P. Maniatis, M. Naik, an A. Patti, Cloneclou: elastic execution between mobile evice an clou, in Proc. EuroSys, Apr. 2011, pp [13] M. Satyanarayanan, P. Bahl, R. Caceres, an N. Davies, The case for VM-base cloulets in mobile computing, IEEE Pervasive Comput., vol. 8, no. 4, pp , Oct./Dec [14] W. Zhang, Y. Wen, an D. O. Wu, Energy-efficient scheuling policy for collaborative execution in mobile clou computing, in Proc. IEEE INFOCOM, Apr. 2013, pp [15] Z. Cao, M. Koialam, an T. V. Lakshman, Traffic steering in software efine networks: Planning an online routing, in Proc. ACM SIGCOMM DCC, 2014, pp [16] A. Gember et al. (2013. Stratos: A network-aware orchestration layer for mileboxes in the clou. [Online]. Available: [17] J. Sherry et al., Making mileboxes someone else s problem: Network processing as a clou service, in Proc. ACM SIGCOMM, 2012, pp [18] A. Gember, R. Granl, A. Anan, T. Benson, an A. Akella, Stratos: Virtual mileboxes as first-class entities, Univ. Wisconsin-Maison, Maison, WI, USA, Tech. Rep. TR1771, [19] R. Soulé, S. Basu, R. Kleinberg, E. G. Sirer, an N. Foster, Managing the network with Merlin, in Proc. ACM HotNets, 2013, Art. no. 24. [20] L. E. Li et al., PACE: Policy-aware application clou embeing, in Proc. IEEE INFOCOM, Apr. 2013, pp [21] S. Jain et al., B4: Experience with a globally-eploye software efine WAN, in Proc. ACM SIGCOMM Comput. Commun. Rev., vol. 43, no. 4, pp. 3 14, [22] H. Wang, S. Chen, H. Xu, M. Ai, an Y. Shi, SoftNet: A software efine ecentralize mobile network architecture towar 5G, IEEE Netw., vol. 29, no. 2, pp , Mar./Apr [23] Policy an Charging Control Architecture, ocument 3GPP TS Release 13, 3GPP, [24] N. Garg an J. Konemann, Faster an simpler algorithms for multicommoity flow an other fractional packing problems, SIAM J. Comput., vol. 37, no. 2, pp , [25] N. Buchbiner an J. Naor, The esign of competitive online algorithms via a primal-ual approach, Foun. Trens Theoretical Comput. Sci., vol. 3, nos. 2 3, pp , [26] R. Hassin, Approximation schemes for the restricte shortest path problem, Math. Oper. Res., vol. 17, no. 1, pp , [27] M. Ziegelmann, Constraine shortest paths an relate problems, Ph.D. issertation, Int. Max Planck Res. School Comput. Sci., Saarlan Univ., Saarlan, Germany, Zizhong Cao (S 12 receive the B.S. egree in electrical engineering from Shanghai Jiao Tong University, China, in 2010, an the Ph.D. egree in electrical engineering from the Tanon School of Engineering, New York University, USA, in His research interests inclue network esign an optimization in switching, ata center, an mobile networks. He was a co-recipient of the Best Paper Awar for the IEEE INFOCOM 2014 an ACM SIGCOMM DCC Shivenra S. Panwar (S 82 M 85 SM 00 F 11 receive the B.Tech. egree in electrical engineering from the Inian Institute of Technology (IIT Kanpur, Kanpur, Inia, in 1981, an the M.S. an Ph.D. egrees in electrical an computer engineering from the University of Massachusetts, Amherst, MA, USA, in 1983 an 1986, respectively. He is currently a Professor an the Department Chair of Electrical an Computer Engineering with the Tanon School of Engineering, New York University, Brooklyn, NY, USA. He is also the Director of the New York State Center for Avance Technology in Telecommunications an a member of NYU WIRELESS. His research interests inclue the performance analysis an esign of networks. Current work inclues co-operative wireless networks, switch performance, an multimeia transport over networks. Dr. Panwar was a recipient of the IEEE Communication Society s Leonar G. Abraham Prize in the Fiel of Communication Systems for 2004 an the Best Paper Awar in Multimeia Communications He has serve as the Secretary of the Technical Affairs Council of the IEEE Communications Society. Murali Koialam (A 99 M 14 receive the Ph.D. egree from the Massachusetts Institute of Technology. He joine Bell Laboratories in 1992, where he is currently a Distinguishe Member of Technical Staff. He has authore over 80 referee conference an journal publications, an hols over 40 patents. His general research interests are in resource allocation in communication networks. He has worke on IP routing, switch scheuling an ata structures an algorithms for fast packet processing in wire an wireless networks. Dr. Koialam was an Associate Eitor of the IEEE/ACM TRANSACTIONS ON NETWORKING. He was a co-recipient of the 2008 IEEE Leonar Abraham Prize, the 2011 IEEE Bennett Prize, an the IEEE INFOCOM 2014 best paper awars. T. V. Lakshman (M 85 SM 98 F 05 receive the master s egree in physics from the Inian Institute of Science, Bangalore, Inia, in 1984, an the Ph.D. egree in computer science from the University of Marylan, College Park, in He is currently the Hea of the Networks Research Group at Nokia Bell Laboratories. His research contributions span a spectrum of networking topics, incluing switch architectures, network esign, TCP performance, traffic management, an software-efine networking. Dr. Lakshman is a Fellow of Bell Laboratories an ACM. He was a recipient of several IEEE an ACM awars, incluing the IEEE Leonar Abraham Prize, the IEEE Communication Society William R. Bennett Prize, the IEEE INFOCOM Achievement awar, the IEEE Fre W. Ellersick Prize Paper Awar, an the ACM SIGMETRICS Best Paper Awar. He also receive the 2010 Thomas Alva Eison Patent Awar from the R&D Council of New Jersey. He has been an Eitor of the IEEE/ACM TRANSACTIONS ON NETWORKING an the IEEE TRANSACTIONS ON MOBILE COMPUTING.