IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH

Transcription

1 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 7 A Game Theoretc Aroach for Dstrbuted Resource Allocaton and Orchestraton of Softwarzed Networks Salvatore D Oro, Student Member, IEEE, Laura Gallucco, Member, IEEE, Sergo Palazzo, Senor Member, IEEE, and Govann Schembra Abstract Softwarzaton of networks allows smlfyng deloyment, confguraton, and management of network functons. The drvng force toward ths evoluton s reresented by software defned networkng that allows more flexble and dynamc network resource allocaton and management. The effcent allocaton and orchestraton of network resources s of extreme mortance for ths softwarzaton rocess, and many centralzed solutons have been roosed. However, they are comlex and exhbt scalablty ssues. So, dstrbuted solutons are to be referred but, n order to be effectve, should quckly converge towards equlbrum solutons. In ths aer, we focus on makng dstrbuted resource allocaton and orchestraton a vable aroach, and rove convergence of the relevant mechansms. Secfcally, we exlot game theory to model nteractons between users requestng network functons and servers rovdng these functons. Accordngly, a two-stage Stackelberg game s resented, where servers act as leaders of the game and users as followers. Servers have conflctng nterests and try to maxmze ther utlty; users, on the other hand, use a relcator behavor and try to mtate other user s decsons to mrove ther beneft. The framework roves the exstence and unqueness of an equlbrum, and a learnng mechansm to converge to such equlbrum s roosed. Numercal results show the effectveness of the aroach. Index Terms Game theory, softwarzed networks, orchestraton, resource management. I. INTRODUCTION THE rolferaton of new servces and alcatons n the Internet wth dfferent requrements n terms of avalablty, servce qualty and reslence, s makng management of network nfrastructures a key challenge. In ths evolvng scenaro, Telco Oerators (TOs) show ncreasng nterest n softwarzng ther networks, so makng deloyment, confguraton, management and udatng of network functons faster and easer, and thus achevng numerous advantages n Manuscrt receved Arl 30, 06; revsed Setember 30, 06; acceted January 4, 07. Date of ublcaton February 0, 07; date of current verson Arl 6, 07. Ths work was suorted by the In-Network Programmablty for next-generaton ersonal cloud servce suort (INPUT) roject funded by the Euroean Commsson under the Horzon 00 Programme H00-ICT-04- under Grant The authors are wth the Dartmento d Ingegnera Elettrca, Elettronca ed Informatca, Unversty of Catana, 955 Catana, Italy (e-mal: salvatore.doro@dee.unct.t; laura.gallucco@dee.unct.t; sergo.alazzo@dee.unct.t; govann.schembra@dee.unct.t) Color versons of one or more of the fgures n ths aer are avalable onlne at htt://eeexlore.eee.org. Dgtal Object Identfer 0.09/JSAC terms of both Catal Exendture (CAPEX) and Oeratonal Exendture (OPEX). Two relevant key enablers of ths evoluton are Software- Defned Networkng (SDN) [] [3] and Network Functons Vrtualzaton (NFV) [4]. SDN allows a flexble management of the network resources thanks to ts ecularty of searatng the network control from the forwardng lane. NFV, on the other hand, brngs vrtualzaton concets from cloud comutng to the network n order to let software-based network functons, also called vrtualzed network functons (VNFs), run on commodty hardware nfrastructures. The ntroducton of the jont SDN/NFV aradgm s seen by Telco Oerators (TOs) as the way to move to more flexble networks where servces can be nstantly montored, controlled, blled, and managed on the fly, rather than requrng a set of comlex, manual changes [5]. However, as comared to urose-bult networkng hardware or mddle boxes devces, deterrents to ths aroach are the achevable erformance and the scalablty. Key elements for the desgn of these systems are resource allocaton, and network functon orchestraton. Although smlar desgn roblems have been studed n cloud comutng scenaros [6] [9], there are mortant dfferences stemmng from the fact that servers n data centers are connected to each other through hgh-caacty and hgh-seed networks, so makng the secfcs of the underlyng network less mortant. On the contrary, n network functon deloyment, network constrants (e.g. bandwdth and latency) are of crucal mortance. The choce of where runnng network functons has to be made by accountng not only for the ncreased load n the nodes hostng the functons, but also for the latency exerenced to reach these nodes, whch can be dfferent for each flow [0], []. The frst ste towards management and functon allocaton n an NFV scenaro was made n [], where the VNF-P algorthm was ntroduced to handle traffc load varatons by dynamcally nstantatng VNFs. In the same context [3] and [4] dscuss lacement olces for secfc VNFs. Instead, the work [5] consders a heterogeneous scenaro wth VNFs characterzed by dfferent scalablty, relablty, and avalablty requrements, and rooses an extenson of the Oenstack orchestrator to translate the ndvdual deloyment requrements nto a lacement of the VNFs n the cloud nfrastructure. Two works that are very close to ths aer IEEE. Personal use s ermtted, but reublcaton/redstrbuton requres IEEE ermsson. See htt:// for more nformaton.

2 7 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 Fg.. a) Case A: Centralzed market scenaro where all network functons are executed on a sngle server owned by the TO; b) Case B: Centralzed market scenaro where network functons are dstrbuted among several servers owned by the TO; c) Case C: Proosed dstrbuted market scenaro where network functons are executed on several thrd-artes servers. n some asects are [6] and [7]. In [6] VNF lacement s made n a way that mnmzes the overall network cost, exressed n terms of the dstance between users and the locatons where servces are rovded, and the cost of servce setu. Instead, the work n [7] consders the mult-commodty faclty locaton by consderng the exstence of more than one VNF nstance n the same network. The roblem of lacement of network functons mlemented as mddle boxes s consdered n [8], wth the target of otmzng network oeratonal costs and utlzaton, wthout volatng servce level agreements. Ths VNF Orchestraton Problem (VNF-OP) s addressed and formulated as an Integer Lnear Programmng (ILP) roblem that s then solved through heurstcs. Another asect that has to be consdered s system scalablty w.r.t. the number of functons and the customers. In fact, the comlex tasks of management, orchestraton and resource allocaton are n charge of only one entty, the Orchestrator, whch therefore requres sohstcated algorthms whch generally results n NP-hard roblems [6], [8], [9]. Ths makes the deloyment of the softwarzed aradgm unfeasble f TOs am at desgnng and managng ther networks whle otmzng costs and erformance. So centralzed solutons (see Case A n Fg. (a)) or vrtually centralzed solutons (see Case B n Fg. (b)) could result unfeasble, and dstrbuted aroaches to resource allocaton and orchestraton have to be ursued. Wth all ths n mnd, the reference scenaro addressed n ths aer addresses a sngle TO network doman where some customers, n the followng referred to as Servers, gve ther avalablty to suort the TO n rovdng network servces to the Users, not only sharng ther hardware facltes where runnng network functons, but also strongly allevatng management and orchestraton burden n the decson tasks of both lacng functons and assgnng them to each actve user flow. We consder that the Server role s layed by customers of the TO network and network functons are located n the CN remses (see Case C n Fg. (c)). So, both resource rovsonng and management are dstrbuted wthn the consdered TO network doman. Wth resect to ths, let us note that, at the best of our knowledge, there s no work so far where the man tasks erformed by the orchestrator, and thus related to management ssues, are dstrbuted among customers of the network,.e., the VNF Servers. A relevant ssue wth ths dstrbuted aroach s related to system convergence towards equlbrum. In fact, the ossblty to converge towards an equlbrum as well as the radness of ths convergence should be nvestgated to rove the feasblty of the aroach. Accordngly, game theory s the natural way to model and characterze the system. In artcular, a new market s assocated to the roosed system, where the man actors are: ) the Servers, that are the sellers of the network functons; ) the Users, that lay the role of buyers; 3) the TO, that coordnates the whole system. In such a context, Servers autonomously decde the rce and the bandwdth to be requested to the TO network n order to rovde the network servces. Users, on the other hand, accordng to the rce secfed by each Server, and the corresondng exected erformance n terms of both exerenced latency and rovded bandwdth, choose one Server for each VNF. In ths way the task of assocatng each flow to a Server s not decded by the Orchestrator, but n an autonomous and dstrbuted way, as a consequence of the nteracton between Users and Servers. To model and suort nteractons between Servers and Users, we exlot herarchcal and evolutonary game theoretc tools. Secfcally, snce Servers naturally act and make decsons by antcatng the Users, we defne a two-stage Stackelberg game where Servers act as the leaders of the game, and Users as the followers. Servers have conflctng nterests among themselves, as ther objectve s to ndvdually and selfshly maxmze a utlty functon. Also, as commonly assumed n mult-layer markets, Servers are exected not to cooerate wth each other, and do not exchange any nformaton wth other comettors. Therefore, ther nteractons are modeled by non-cooeratve game theory. Instead, Users are nfluenced by socal and mtaton behavor,.e., they observe other Users decsons and mtate those decsons f ths s exected to mrove ther beneft. Thus, ther nteractons are modeled by usng the relcator dynamcs from Evolutonary Game Theory (EGT) [0]. In more detal, we derve a closed-form soluton for the equlbrum condton of the relcator dynamcs whch s then used to solve the Stackelberg game. Accordngly, we show that the consdered game admts a Stackelberg Equlbra (SE), and we rove that the SE s unque. We also roose a renforcng learnng rocedure that rovably converges to the unque SE, and llustrate an algorthmc mlementaton. We show that the learnng rocedure can be mlemented n a rvacy-reservng and dstrbuted fashon. Fnally, we resent an extensve numercal result analyss to hghlght the deendency of the dynamc nteractons among layers on the man system arameters, and evaluate the roosed market model, n the vew of rovdng some nsghts n settng system arameters to maxmze revenues. A art of the numercal analyss s also amed to show that the roosed learnng rocedure s scalable w.r.t. the number of Servers, and quckly converges to the SE. At our best knowledge, ths s the frst work where a gametheoretc aroach s used n a softwarzed network scenaro to suort nteractons between Servers and Users. Moreover, exlotng the game-theoretc aroach allows us to demonstrate that adotng a dstrbuted framework, nstead of a tradtonal centralzed aroach, s benefcal for all the nvolved stakeholders. On the one hand, the smlfed

3 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 73 TABLE I NOTATION Fg.. Reference network scenaro. orchestraton and the ossblty to offload VNFs on thrdartes Servers s benefcal to the TO. Servers have economc beneft artcatng to the VNF Market as sellers. Fnally, Users can choose the Servers that most ft ther needs. The remanng of ths aer s organzed as follows. In Secton II, the consdered network scenaro s descrbed. In Secton III, the game-theoretc framework whch models the consdered resource allocaton and orchestraton roblem s roosed and studed. A numercal analyss of the roosed game-theoretc framework s resented n Secton IV. Fnally, n Secton V conclusons are drawn. II. SYSTEM MODEL The consdered reference scenaro s sketched n Fg.. It conssts of a network doman of a Telco Oerator (TO) that rovdes customers of ths doman wth network servces accordng to the NFV aradgm. The man roles n the system are layed by the Orchestrator, thevnf Servers, and the Users. Users are the customers that generate flows and request VNFs for each of ther flows. They are located n Customer Networks (CNs) where a Customer Premse Equment (CPE) devce allows them to be connected to an access node of the TO core network, n the fgure ndcated as Provder Edge (PE) node. VNF Servers are NFV-comlant nodes [], [] owned by network customers that have decded to run VNFs n order to serve the TO network doman they belong to, and obtan economc benefts. A VNF Server can be ether a stand-alone comuter, such as the one connected to the CN 6 n Fg., a set of servers organzed as a data center, whose resources are artally or totally dedcated to run VNFs, or an enhanced CPE (ecpe) node. As descrbed n [3], the latter s a CPE devce that s able to run VNFs n a vrtualzed envronment (e.g. the ecpe nodes connectng CN 5 and CN 7 to the TO network). Besdes the hardware facltes, VNF Servers need an amount of bandwdth that s rovded them by the TO network. A VNF Server can rovde more than one VNF and decde the sellng rces autonomously. In general, a VNF Server can also rovde, manage and sell an entre servce chan realzed by connectng local comonent VNFs. However, for the sake of smlcty and wthout loss n generalty, n the sequel we wll refer to VNF Servers as servers that rovde VNFs only. A very mortant role n the system s layed by the Orchestrator, whch s n charge of management and orchestraton of the whole system. It runs on a dedcated server and communcates wth all nodes through the TO network. The man tasks erformed by the Orchestrator are: Exosng a lst of VNFs that the TO wants to rovde to ts Users; Provdng the VNF Servers wth the VNF temlates, contanng the deloyment and oeratonal behavor requrements necessary to realze each VNF and manage ts lfecycle; Assgnng a slce of bandwdth to the VNF Servers accordng to ther bandwdth request; Provdng each User wth the current lst of VNF Servers that are runnng the requested VNFs, ncludng nformaton regardng the rce aled by each VNF Server and the relevant erformance arameters, n terms of exerenced latency and receved bandwdth. Allowng Users to choose a VNF Server for each requested VNF functon by settng the flow table of the SDN swtches n the TO network n such a way that User flows traverse the chosen VNF Servers. Polces for management and orchestraton of the resources are a key element of the system as they strongly nfluence ts erformance. The man erformance arameter that deends on the aled olces s the latency. In fact, lacng a VNF on a secfc node n the network determnes that all the flows usng t have to ass through that node; thus, f that node s very far from the sources of some flows, latency may result unaccetable for them. It s evdent that each VNF Server s characterzed by almost the same erformance latency arameter for all the Users that enter the network through the same PE node, or through dfferent PE nodes whch are close to each other n the core network,.e., are connected to each other by hgh-seed lnks of a few mles. Another mortant arameter s the bandwdth that each VNF Server rovdes to Users, whch deends on both the amount of The roblem of dstrbuted servce chan comoston s out of the scoe of ths aer and s addressed n [4].

4 74 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 7) The User u communcates to the Orchestrator the VNF Server chosen for f durng the revous ste. 8) The Vrtual Infrastructure Manager (VIM) block of the Orchestrator confgures the SDN swtches n the TO network n such a way that the secfc flow for whch the User u has chosen a gven Server, traverses ths Server. Fg. 3. Management rocess flow dagram. bandwdth the VNF Server requests to the TO and the number of User flows usng ts VNFs. In the followng, we use the term User Grou to ndcate the set of Users requestng the same VNF, whch are characterzed by the same latency from the VNF Servers rovdng that VNF, and exhbt the same requrements n terms of delay and bandwdth. A. Dstrbuted System Management In ths secton, we descrbe the management framework of the dstrbuted reference system. Let us defne F as the set of VNFs rovded by the TO. The man enttes nvolved n the management oeratons from ts onboardng to ts usage by User flows are sketched n Fg. 3. Accordngly, for each VNF f F, the relatve stes can be syntheszed as follows: ) The Network Manager (.e. a human oerator) creates a VNF temlate for f. All temlates are onboarded and stored on the Orchestrator, and lsted n a VNF Catalogue; ) Servers that are nterested n rovdng the VNF f, download the temlate from the Orchestrator, and create an nstance of f, assgnng to t a set of local resources. 3) Each VNF Server ndvdually decdes the amount of bandwdth requred to serve ts Users, artcatng to the game G (S) descrbed n Secton III-B; 4) Each VNF Server that has launched the VNF f, requests to the Orchestrator to be regstered on the VNF Server lst as rovdng the VNF f, also secfyng the decded rce to be aled to the Users and the requested bandwdth; 5) Each User that s nterested n the VNF f, n the followng ndcated as User u, contacts the Orchestrator to receve nformaton concernng the VNF market of f. More secfcally, the User receves the lst of all the VNF Servers that are runnng f and, for each of them, the bandwdth that ths VNF Server would assgn to u, the rce that t has decded to be aled to the Users of the same User Grou, and the IP address of the consdered VNF server, so that the User u can autonomously derve the latency from t. 6) Thanks to the nformaton receved n the revous tem, the User u chooses to whch VNF Server to connect to. Ths s done by artcatng to the game G (U) that wll be ntroduced n Secton III-A. B. The Market Model Let us now dscuss the market model that suorts the above management framework. Let F be the set of VNFs rovded by the network, and U the set of User Grous. For each VNF f F,let U a User Grou comosed by N Users that are nterested n f,ands be the set of VNF Servers that rovde t. Let d be the latency encountered by the flows of the Users belongng to to reach the VNF Server S. It s realstc to assume that User Grous act as dstnct tenants whch are allowed n rncle to connect to the same VNF Server, but, for securty reasons, cannot mx ther traffc wth those generated by other User Grous. Accordngly, requests from each User Grou have to be ndvdually accommodated. The enttes that wll artcate to ths market are the VNF Servers that have nstalled and run f, the Users that need f for some of ther flows, and the Orchestrator that has decded to rovde ts customers wth f and that ntend to gan some economc beneft from t. For each User flow traversng a gven VNF Server, ths Server has to allocate a gven amount of comutng and storage resources, and ths reresents a cost for the VNF Server. Consderng the generc VNF Server S, we wll refer to the ncremental cost ncurred by the VNF Server to guarantee the requred resources to a new flow requestng functon f as c. For examle, the cost c may be an energy cost as n [3], that deends on the rce aled by the energy rovder. Therefore, the cost for a VNF Server to manage all the User flows n, C (F ), s roortonal to the number of flows n,thats C (F ) = c n () Another cost for the VNF Servers s due to the bandwdth that they receve from the TO network accordng to the requests ssued to the Orchestrator. Let b be the bandwdth receved by the VNF Server to manage the User Grou, and (B) the bandwdth-unt rce aled by the TO network to the VNF Server. Note that the value of (B) does not deend on the User Grou. Accordngly, the cost of the overall bandwdth used by that VNF Server s: C (B) = (B) b () On the other hand, the revenue for the VNF Server assocated to the rovson of VNF f s roortonal to In Table I we rovde a lst of the symbols used throughout the aer wth ther meanng.

5 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 75 both the number n of Users that are usng t, and the rce ˆ (F ) aled by ths VNF Server. Now, f we assume that the VNF Servers have to ay a commsson (or fee) to the TO, reresented by the commsson arameter ψ [0, ], the actual revenue of the VNF Server related to the rovson of VNF f to the User Grou s R = (F ) n (3) where (F ) = ˆ (F ) ( ψ). The mechansm to decde the amount of bandwdth that each VNF Server requests to the TO network wll be dscussed n Secton III. It s amed at maxmzng a utlty functon defned as follows: [ ] U (S) (b ) = β R β C (F ) + C (B) (4) where b = (b, b,...,b M ) s the bandwdth vector that contans the bandwdth b requested to the TO network by each VNF Server, and β and β are arorate constants weghng the relatve relevance of revenues and costs. On the other hand, Users n User Grou, choose the VNF Server by also takng nto account the latency exerenced to reach t, d, and the current rce t s alyng to the VNF f. However, the hgher the number of User flows usng the same VNF Server, the lower the bandwdth allocated to each of them. Secfcally, the beneft functon of each user n s exected to be ncreasng n the amount of resource allocated to that user,.e., b /n ; and to be decreasng n both the rce ˆ (F ), and the latency d. Secfcally, and n lne wth standard economc assumtons [5], we assume that Users exerence dmnshng returns as the value of b /n ncreases. Such an assumton can be modeled through concave functons, e.g., logarthmc functons, and has been wdely used n economcs theory to reflect the concet of rsk-averson or satsfacton behavor of ratonal decsonmakers. Wth all ths n mnd, each User selects the VNF Server that maxmzes the followng utlty functon [5], [6]: U (U) (n ) = ln ( α b n ) α ˆ (F ) α 3 d (5) where n = (n, n,...,n M ) s the state vector that contans the number n of flows from the User Grou served by each VNF Server n S; α, α and α 3 are arorate constants that wegh the contrbutons to the utlty functon of the bandwdth receved, the rce aled by the VNF Server, and the latency encountered to reach that Server, resectvely. 3 In the followng of the aer, we wll refer to α, α, α 3, β and β as the weghng arameters. C. Telco Oerator revenues An mortant toc that deserves artcular attenton s related to the revenues generated by the TO. In centralzed markets such as n Cases A and B, dected n Fgs. (a) and (b), resectvely, the whole amount ad by 3 Note that the use of a logarthmc functon n (5) s justfed by the fact that such class of functons have been shown to be roortonally far. Users s receved by the TO tself. Therefore, the TO s able to monoolze revenues generated by the VNF rovsonng. Instead, by dstrbutng the VNF rovsonng rocess such as n Case C (see Fg. (c)), a art of the User ayments go to VNF Servers because they gve a commsson on the sale of VNFs, and also ay the TO to get the necessary amount of bandwdth to serve ts users. Accordngly, let U (TO) A, U (TO) B and U (TO) be the revenues n Cases A, B and C, resectvely. C In Case A, we have that all VNFs requred by all User Grous are executed on a sngle server. Therefore, the revenue of the TO can be exressed as follows: U (TO) A = ( ) N ˆ (F ) c (6) U where N s the number of Users n User Grou U, ˆ (F ) s the rce charged to users n User Grou to access functon f on the centralzed server, and c s the analogous of c for the centralzed server. In Case B, VNFs are all rovded by servers managed by the TO. In ths latter case, the revenue of the TO s: U (TO) B = U N ˆ (F ) S (P) ( ) n (OPT) c (7) where S (P) s the set of roretary servers and n (OPT) s the otmal number of users n the User Grou on the TOroretary server, forall U and S (P). Fnally, n Case C, VNF rovsonng s dstrbuted among dfferent thrd-arty VNF Servers, and the overall revenue of the TO s U (TO) C = U ( S C (B) + ψ S ˆ (F ) n The frst term n (8) deends on the bandwdth requested by the VNF Servers, whle the second term deends on the commssons ad by those VNF Servers to the TO. The analyss on the effcency of TO s revenues n all the above cases wll be nvestgated n Secton IV-E where we wll show that, n many cases, dstrbutng the VNF market s much more roftable than centralzng t. III. GAME MODEL In ths secton, we llustrate the roosed game-theoretc model of the nteractons between VNF Servers and Users n the dstrbuted management framework. Decsons taken by VNF Servers and Users deend on both ndvdualstc nterests, e.g., maxmze ther own utlty, and decsons taken by counterarts, e.g., oonents strateges. For examle, Users connect to one of the avalable VNF Servers deendng on the offered bandwdth and other relevant arameters such as roosed rce and exected communcaton delay. On the contrary, VNF Servers am to maxmze ther revenues and are not lkely to cooerate wth each other. Also, ther actons deend on the number of Users that are connected to them to use ther VNFs. In real scenaros, VNF Servers naturally act and make decsons by antcatng the Users. Accordngly, nteractons ) (8)

6 76 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 Fg. 4. Proosed game-theoretc framework for User Grou U. among VNF Servers and Users can be modeled as a twostage Stackelberg game where VNF Servers act as the leaders of the game and Users as the followers. In the addressed roblem we should also consder Users that relcate other Users decsons. Such relcatve behavor naturally arses n those scenaros where multle enttes make decsons by relcatng other Users behavor [7] [30]. among users of the same User Grou U where we exlot Evolutonary Game Theory (EGT) and relcator dynamcs to model the decson-makng rocess of Users. Then, n Secton III-B, we use non-cooeratve game theory to defne In Secton III-A, we frst defne a game G (U) whch models comettve nteractons among the VNF Servers to serve users n the User Grou. Fnally, n Secton III-C we roose a dstrbuted and rvacy reservng renforment learnng rocedure to comute the equlbrum of the game G (S) the game G (S). The llustrated games wll be layed each tme some condtons of the system change. More secfcally, a varaton of ether the latency of the VNF Servers from the Users of a User Grou, or the rce decded by one of the VNF Servers for a gven functon f, or the number of Users nterested n the functon f, determnes a varaton n the utlty functon of some User, and ths stmulates the Users to start layng the game to change the Server. The consequent dstrbuton varaton of the Users on the Servers, n ts turn, stmulates the Servers to lay the G (S) game n order to decde a modfcaton of the bandwdth to be requested to the TO network. The games are terated untl all the enttes n the system reach a new steady state. In Secton IV we wll numercally analyze ths transent erod, and show that t lasts some 0 teratons. The consdered game-theoretc model and ts herarchcal structure are shown n Fg. 4. G (U) A. Evolutonary Game G (U) Among Users Each User s ntrnscally selfsh as t makes decsons wth the am of maxmzng ts own utlty U (U), as defned n (5). However, the hgher the number n of Users n the User Grou connected to the -th VNF Server, the lower the utlty U (U) of that User. Therefore, the decson-makng rocess of each User s also nfluenced by decsons taken by the other Users belongng to the same User Grou. Also, f a User s aware that another User s achevng a better utlty, he can decde to mtate that User and mgrate to the same VNF Server to whch that User s connected [3]. In the rest of ths aer, we refer to ths henomenon as mtaton behavor. Imtaton behavor often arses when consderng nteractons among enttes that ratonally try to maxmze ther beneft by mtatng other enttes decsons that rovde better beneft. For examle, mtaton s at the bass of a varety of decson makng roblems n both wred and wreless networks [8] [30] that are often modeled by exlotng theoretcal tools from evolutonary game theory. In lne wth a vast body of lterature, we consder the well-known and wdely used relcator dynamcs [3] as the mtaton dynamcs whch descrbe the nteractons among Users. Accordngly, for each User Grou U, wedefne the evolutonary game G (U) as follows: Poulaton: t conssts of the set of the N Users n User Grou U. Strategy: t s defned as the choce of the VNF Server S to whom each User n the oulaton decdes to connect; the strategy set of each User s S. Utlty: the utlty, or beneft, acheved by each User connected to the VNF Server S s equal to U (U) as defned n (5). We can now defne the relcator equaton that descrbes how the number of Users n the oulaton that connect to avalable VNF Servers vares ṅ = n U (U) (n ) n j U (U) N j S j (n ) (9) where n n denotes the number of Users n the User Grou whch have chosen as a strategy to connect to the -th VNF Server. The frst term n the rght-hand sde of (9) reresents the utlty of a User that connects to the -th VNF Server, whle the second term reresents the average utlty of the oulaton whch deends on the current dstrbuton n of the oulaton. Therefore, the growth rate ṅ /n of the number of Users n the User Grou connected to the -th VNF Server s equal to the dfference between the beneft when choosng the strategy, and the average beneft of the whole oulaton. A general result from EGT shows that an equlbrum ont for the relcator dynamcs s a fxed ont of the relcator dynamcs such that all Users exerence the same beneft,.e., U (U) = U (U) j for all, j S. In Prooston, we wll show that the relcator equaton (9) for each User Grou admts a unque soluton for any bandwdth vector b. Furthermore, we characterze the equlbrum ont by dervng the resultng state vector n at the equlbrum. To ths urose, and for notaton uroses, let us defne an auxlary varable φ (), j as follows: ( ) [α ] φ (), j = e ˆ (F ) ˆ(F ) j +α 3(d d j) (0) From (0), t can be easly shown that the followng relatonshs hold for all, j, k S and U φ (), =, φ (), j = /φ () j,, and φ() k, j = φ(), j φ (),k () Prooston : For all U and any gven bandwdth vector b, the relcator equaton (9) admts a unque evolutonary equlbrum n. Also, the number of Users n n User

7 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 77 Grou connected to the generc VNF Server S at the equlbrum ont can be derved as follows: n = N j S b jφ () (), j where b b. Proof: The relcator equaton can be reduced to an equvalent system of ordnary dfferental equatons (ODEs). Thus, to show that the relcator dynamcs admts a unque equlbrum ont, t suffces to note that the rght-hand sde of the mean dynamc n (9) s contnuously dfferentable. Therefore, Lschtz contnuty, and thus unqueness of the equlbrum, follow [0]. Now, n order to determne the unque equlbrum, t s well known that t s reached when ṅ = 0. Such condton mles that U (U) = U (U) j for all, j S,.e., all Users receve the same beneft. Accordngly, we can buld a system of equatons wth N (N )/ equatons that can be solved by exlotng the relatonsh N = S n. Thus, after some easy analytcal dervatons, we obtan the result n (). For the sake of llustraton, n the followng we show how to derve () when N =. However, the more general case can be treated n a smlar way. From (5) and (0), and by mosng U (U) = U (U) we get ( ) b n ln = ln(φ (), b n ) (3) Recall that N = n + n. Thus, we get n = N b b + b φ (), b andn = N b b + b φ (), whch s a secfc case of (). An mortant result that stems from the unqueness of the equlbrum ont s that the relcator dynamcs converges towards a unque stable ont. Therefore, unqueness avods ossble oscllatons among two or more equlbrum onts. Also, snce the equlbrum s unque, comlex equlbrum selecton mechansms to determne the most effcent NE n the set of the feasble multle NEs are not requred. To estmate the relcator equaton n (9), each User n the User Grou has to evaluate ts utlty U (U) (n ) and get the average utlty N j S n ju (U) j (n ) of the grou. The utlty U (U) (n ) s defned n (5) and only requres the value of the rato b /n,.e., the amount of bandwdth that wll be assgned to the User. As already stated at ont 5) n Secton II.A, t s worth notng that the value of b /n s already avalable n the VNF Server Lst shown n Fg. 3. Instead, snce no User s aware of the strateges of the other Users n, the average utlty of Users n s broadcast by the Orchestrator to all Users. It s mortant to note that the above values do not carry any rvate nformaton about decsons taken by other Users. In fact, users do not know the number N of Users n the grou, therefore t s not ossble to derve any rvate nformaton from both b /n and N j S n ju (U) j (n ). B. Stackelberg Game G (S) Between VNF Servers and Users As already dscussed before, VNF Servers act as leaders of the game between VNF Servers and Users. Also, n Prooston we have derved the dstrbuton n of the oulaton U at the equlbrum of the relcator dynamcs. For the sake of notaton, let us frst defne the two followng auxlary varables = N ( β (F ) β c ) (4) and π = β (B) (5) Accordngly, we can ncororate (3), (), (4) and (5) n (4) to rewrte the utlty functon U (S) of the generc VNF Server S as follows: U (S) (b ) = b k S b kφ (),k π b (6) For each User Grou U, we defne the non-cooeratve game G (S) as follows: Player set: t conssts of the set S of VNF Servers. Strategy: t s defned as the amount of bandwdth b to be requested to the TO network to serve ts connected Users from User Grou. For each User Grou n U, we assume that such amount of bandwdth s bounded by B. Thus, the strategy set s B = S B,whereB =[0, B ] and dentfes the Cartesan roduct. 4 Utlty: the utlty of each VNF Server S s equal to U (S) as defned n (6). By calculatng the frst-order dervatve of (6), t can be easly shown that 0 leads to a non-ostve frst-order dervatve of the utlty functon U (S). In other words, the best strategy for the -th VNF Server s not to artcate n the game G (S) for User Grou,.e., b = 0. Therefore, those VNF Servers wth 0 ext the game and they can be removed from the layer set S. Accordngly, wthout loss of generalty, n our model we assume that the layer set S s comosed by only those VNF Servers such that > 0. and rovde useful results about ts equlbrum onts, referred to as SEs. Defnton : Let b B. The strategy rofle (b, n ) s a In the followng, we analyze the Stackelberg game G (S) SE for the game G (S) f for all b B and S, we have U (S) (b, n ) U (S) (b, n ) where n s defned as n (). Defnton : Let b = (b, b ),whereb s the bandwdth vector of all layers excet,.e., b = (b j ) j S, j = wth b j b. The strategy b = (b, b,...,b M ) s sad to be the Stackelberg strategy for the game G (S) f for all S we have that b = arg max b B U (S) (b, b, n ) 4 We do not consder the varable ˆ (F ) as a strategy for the VNF Server as we lmt our study to the case where, whle the bandwdth vares n tme, the rcng olcy remans constant durng each game executon.

8 78 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 Also, the value U (S) (b, n ) s denoted as the Stackelberg utlty of leader n game G (S). In Prooston, we rove that the game G (S) admts a unque SE. Prooston : The game G (S) admts a unque SE. Proof: The man stes of the roof are as follows. Frst, we rove the exstence of the equlbrum by exlotng concavty roertes of VNF Server utlty functons n (6). Then, we show that the Dagonal Strct Concavty (DSC) roerty holds. The DSC roerty mles that VNF Servers exerence dmnshng returns along any drecton,.e., along all b b. Fnally, we exlot results contaned n [33] and [34] to rove that a unque equlbrum exsts. Let the margnal utlty v (b ) of each layer S be (S) U defned as v (b ) = (b ) b. Therefore, from (6) t follows that the margnal utlty of the generc VNF Server s v (b ) = k S,k = b kφ (),k ( k S b kφ (),k ) π (7) where s defned n (4). To show that the DSC roerty holds, t must be shown that: ) U (S) (b ) s strctly concave n b ; ) U (S) (b ) s convex n b ; and ) the functon ρ(b, r ) defned as ρ(b, r ) = S r U (S) (b) (8) s concave n b for some r = ( r, r,...,r M ) such that r > 0forall S. From (6), t can be shown that roerty ) holds as U (S) (b ) s defned as the dfference between a strctly concave functon and a concave functon. To rove ), t suffces to note that the Hessan matrx of U (S) (b ) has all non-negatve egenvalues,.e., the Hessan matrx s ostve semdefnte. Let r = / for all S. Accordngly, (8) can be rewrtten as follows: ρ(b, r ) = b S b + j = b jφ () r π b, j S = b b + j = b jφ (), j + k = b k b k + j =k b jφ () k, j r π b (9) S From (), we have that ρ(b, r ) = b b + k = b kφ () + b k φ (),k,k k = b + j = b jφ (), j r π b = r π b (0) S S Observe that ρ(b, r ) s a concave functon n b as requred n ). Therefore, we have that DSC roerty holds and the general theory n [33] and [34] ensures the unqueness of the equlbrum. In (6), we use the equlbrum condton n (). Therefore, nteractons between Users (.e., the followers) and VNF Algorthm Exonental Renforcement Learnng (XL) Parameter: ste-sze sequence γ m (default: γ m = /m). Intalze: m 0; z 0 for all S. Reeat m m + ; for each VNF Server S do smultaneously requested bandwdth b B [ + ex( z ) ] ; measure margnal utlty v from (7); udate scores: z z + γ m v ; untl termnaton crteron s reached. Servers (.e., the leaders) modeled through the game G (S) roduce a unque SE (b, n ). However, recall that VNF Servers comete wth each other n the Stackelberg game. Accordngly, the strategy rofle b dscussed above also reresents a Nash Equlbrum (NE) [34] for the comettve game among VNF Servers. C. Renforcement Learnng Procedure for Game G (S) In Prooston, we have shown that the game G (S) admts a unque equlbrum. Unfortunately, we are not able to fnd a roer characterzaton of the equlbrum and rovde closed-form exressons. Thus, we need to rovde a robust mechansm to allow VNF Servers to ndvdually reach the equlbrum of the game. Accordngly, n the followng we roose an exonental renforcng learnng [35], [36] rocedure, whch rovably converges to the unque equlbrum of the game. For each VNF Server S that serves Users n the User Grou U, we defne the followng learnng rocedure { ( z (m + ) = z (m) + γ m v b (m) ) e b (m + ) = B z (m+) () +e z (m+) where m reresents the teraton ndex, b (m) s the bandwdth vector at teraton m, andγ m s the ste-sze of the learnng rocedure whose mortance wll be exlaned later. For each User Grou U, the algorthmc mlementaton of () s shownnalgorthm. In the followng Prooston 3, we show that the roosed exonental renforcng learnng rocedure converges to the equlbrum of the game. Prooston 3: Let γ m be the ste-sze of the learnng rocedure and m γ m < m γ m =+. For any feasble ntal condton n B and User Grou U, Algorthm always converges to the unque SE of G (S). The roof conssts n showng that ) the mean dynamc of (),.e., ts contnuous-tme verson, converges to the equlbrum of the game as tme goes to nfnty, and ) () s an asymtotc seudo-trajectory (APT) [37] for the contnuous-tme verson of (). For a detaled and rgorous roof, we refer the reader to V. From Prooston 3, we have that any varable ste-sze rule n the form γ m = /m β wth β (0.5, ] wll converge to the unque SE of the game G (S).

9 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 79 Fg. 5. Requested bandwdth and oulaton dstrbuton at the equlbrum as a functon of the rce (F ) charged by S (Sold lnes: d = 5 DUs and d = 40 DUs; Dashed lnes: d = d = 40 DUs). Let us note that, n order to comute b (m+) ( n (), each VNF Server S s requred to know v b (m) ) n (7), whch only deends on the term k S b k(m)φ (),k.toths urose, the Orchestrator has full access to the VNF Server arameters (e.g., d, (F ), etc...). Then, at each teraton and for each VNF Server S, the Orchestrator s able to comute the overall sum k S b k(m)φ (),k and send t to the corresondng -th VNF Server. Note that, by so dong, the -th VNF Server cannot extract any rvate nformaton on other VNF Servers from the sum k S b k(m)φ (),k. Thus, t means that the learnng rocedure () can be mlemented n a rvacy-reservng and dstrbuted fashon. IV. NUMERICAL ANALYSIS In ths secton, we resent a numercal analyss of the roosed dstrbuted orchestraton and resource allocaton scheme. In our smulatons, we assume a oulaton sze of N = 3000 Users and, unless otherwse stated, we consder the followng weghng arameters: α =, α = 0.05, α 3 = 0.035, β = andβ =. Fnally, and unless exlctly mentoned otherwse, we assume that the commsson arameter s ψ = 0, and the bandwdth-unt rce (B) s equal for all VNF Servers n S and s set to (B) = Prce Unts (PUs). For llustratve uroses, we rmarly focus on the two VNF Servers case (.e., M = ) as t allows us to hghlght the dynamcs of the nteractons among Users and VNF Servers together wth the mact of the varous system arameters on the outcome of the game G (S). Moreover, we also rovde extensve results also for the case M >, whch makes ossble to show the feasblty of the roosed learnng rocedure and to analyze the mact of latency when multle VNF Servers are wllng to rovde the consdered VNF. A. Imact of Prcng on the SE In ths secton, we relmnarly study the mact of the rcng aled by the VNF Servers. To ths urose, n Fg. 5 we show the outcome of the game as a functon of the rce (F ) charged by VNF Server S to ts Users, when the rce aled by VNF Server S s assumed constant and equal to (F ) = 60 PUs. Secfcally, we show the amount of bandwdth b that each VNF Server requests to the TO network and the number n of Users that connect to each VNF Server at the SE. Also, we consder two dfferent confguratons of the latences d exerenced by Users connected to the -th VNF Server. For the sake of generalty, we wll exress latency n terms of DUs. In more detal, sold lnes llustrate the outcome of the game when d = 5 DUs and d = 40 DUs, resectvely. Instead, dashed lnes refer to the case when d = d = 40 DUs. As exected, when (F ) s hgh, Users are lkely to connect to VNF Server S because t ales a lower rce (.e., (F ) = 60 PUs). Accordngly, Users get hgher ayoffs when they connect to VNF Server S ndeendently of the exerenced connecton latences d. On the contrary, when (F ) s low, n order to attract more Users, the strategy of VNF Server S conssts n requestng a hgh amount of bandwdth to the TO network. In ths way, as evdent from (5), the utlty of the Users ncreases as a consequence of the ncrease n the shared bandwdth. Such behavor holds for values of (F ) that are below a gven threshold, above whch requestng more bandwdth s no more the otmal choce. For values of (F ) hgher than ths threshold, 5 the otmal strategy of the VNF Servers conssts n reducng the amount of requested bandwdth. Such behavor s motvated by the fact that an ncrease n the requested bandwdth causes an ncrease n the costs, whch also leads to a reducton n the utltes acheved by the VNF Servers. Accordngly, when the cost to rovde more resources to the Users s hgher than the exected revenues, VNF Servers refer to reduce the amount of shared resources to reduce costs and kee hgh revenues. Fnally, t s worth notng that when d = 5 DUs and d = 40 DUs (sold lnes), both VNF Servers request to the TO network a lower amount of bandwdth than n the case when d = d = 40 DUs. Ths s due to the fact that, when latences are equal (or smlar), there s no monoolstc behavor and VNF Servers have to comete to attract more Users, whch results n hgher requested bandwdth. In Fg. 6 we show revenues, costs and utltes acheved by VNF Servers S and S at the SE as a functon of the rcng arameter (F ). More n detal, revenues and costs are defned as the frst and second terms n (4), resectvely. Instead, utltes are equal to U (S), and are defned as n (4). From (3), we have that revenues acheved by each VNF Server deend on R and are determned by the number n of Users that connect to that VNF Server as shown. Accordngly, Fg. 6 shows that revenues vary accordng to the dstrbuton of Users at the SE beng consdered n Fg. 5. On the contrary, from () and () we have that costs deend on both the number n of Users connected to VNF Server S and the requested bandwdth b at the SE. Therefore, as shown n Fg. 6 the resultng trend of exerenced costs s a combnaton of both n and b,whch are shown n Fg. 5. Note that, when the value of the rce arameter (F ) s hgh, the number of Users connected to S asymtotcally tends to N,.e., the whole Users oulaton s 5 In general, the threshold values are dfferent for the two VNF Servers.

10 730 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 Fg. 6. Revenues, costs and utltes of VNF Servers S and S as a functon of the rce (F ) charged by S (Sold lnes: d = 5 DUs and d = 40 DUs; Dashed lnes: d = d = 40 DUs). Fg. 8. Requested bandwdth of the two VNF Servers at the SE as a functon of the two weghts α and α 3 n (5). For examle, the requested bandwdth ncreases for low values of both α and α 3, and then decreases when ether α or α 3 decrease. Even though the behavor s smlar, the requested bandwdth consderably dffers for the two VNF Servers. In fact, Fg. 8 shows that the hghest value of b s , whle the maxmum value of b s 0 4. Fg. 7. Dstrbuton of Users at the SE as functon of the two weghts α and α 3 n (5). lkely nclned to connect to the VNF Server S whch rovdes better erformance for a lower rce (F ). Therefore, for hgh values of (F ), S requests a small amount of bandwdth such that ts revenues and costs asymtotcally tend to zero. B. Imact of the Weghng Parameters on the SE In ths secton, we estmate the mact of the weghts α and α 3 that aear n (5) on the outcome of the game G (S) when M =. To ths urose, n Fgs. 7 and 8 we show the dstrbuton of Users at the SE as a functon of α and α 3. In our smulaton we have assumed (F ) = 60 PUs, (F ) = 0 PUs, d = 5 DUs and d = 40 DUs. Fg. 7 llustrates the dstrbuton of Users at the equlbrum. When α s hgh and α 3 s low,.e., Users are much more concerned about the rce charged by VNF Servers than the exerenced latency, Users are much more attracted by the VNF Server S snce (F ) << (F ). On the contrary, when α 3 s hgh but α s low,.e., Users wegh the exerenced latency more than the cost to obtan a share of the resources, Users are more attracted by the VNF Server S. In Fg. 8, we show the bandwdth requested by both VNF Servers at the equlbrum as a functon of α and α 3. Fg. 8 shows that the behavor of both VNF Servers s smlar. C. Tme-Varyng and PE Postonng Analyss In ths secton, we dscuss the mact of the oulaton sze N, thecostc to rocess each flow, and the PE oston on the outcome of the game G (S). To ths urose, we smulated a scenaro where the number of Users requestng a gven VNF and the cost c vary n tme accordng to realstc nght/day usage atterns. More secfcally, let the number of Users N and the cost c vary n a 48-hours long temoral wndow as shownnfg.9(a). Fg. 9(b) llustrates both the strategy of each VNF Server,.e., the bandwdth requested to the TO network, and the dstrbuton of the oulaton at the equlbrum when (F ) (F ) = = 80 PUs, d = 5 DUs and d = 40 DUs. Observe that when the cost to rocess flows s low, VNF Servers can suort more User connectons. Accordngly, VNF Servers request more bandwdth to the network to attract a hgher number of Users. However, when d = 5 DUs and d = 40 DUs, even though the VNF Server S rovdes Users wth a hgher amount of bandwdth, t also has a hgh latency. Therefore, to reduce the exerenced latency, Users connect to the VNF Server S and thus n > n. Instead, when both VNF Servers have equal latency,.e., d = d = 40 DUs, Fg. 9(c) shows that the majorty of the oulaton chooses the VNF Server whch rovdes the hghest amount of bandwdth. To study the mact of the PE oston,.e., the User entrance onts to the network, w.r.t. the oston of VNF Servers on the outcome of the game G (S), we consder fve VNF Servers,.e., S ={S, S, S 3, S 4, S 5 }, and fve ossble ostons of the access PE, here denoted as PE k wth k =,,...,5. We assume ˆ (F ) = 60 PUs for all S. Each access PE oston corresonds to a dfferent latency confguraton. For examle, n Fg. 0(a) t s shown that VNF Servers S and S rovde low latences when Users

11 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 73 Fg. 9. a) Poulaton sze N and rce arameter c as a functon of tme; b) Requested bandwdth and dstrbuton of Users at the equlbrum as a functon of tme when d = 5 DUs and d = 40 DUs; c) Requested bandwdth and dstrbuton of Users at the equlbrum as a functon of tme when d = d = 40 DUs. Fg. 0. Latences (a), dstrbuton of Users (b), and requested bandwdth (c) at the SE for dfferent access PE ostons. access the network through rovder edges PE and PE,and hgh latences when Users access through PE 4 and PE 5.For VNF Servers S 4 and S 5, what haens s exactly the ooste, whereas S 3 rovdes low latences to Users ndeendently of the oston of the access PE. When Users access through PE and PE, Fg. 0(b) shows that the majorty of them decdes to connect to S and S. Thus, as shown n Fg. 0(c), to attract such an exectedly ncreasng number of Users, S and S request a hgh amount of bandwdth to the TO network. As exected, the contrary holds n the case Users access through PE 4 and PE 5. D. Convergence Analyss In ths secton, we nvestgate the convergence of the roosed learnng rocedure n (). Secfcally, we are nterested n analyzng the convergence seed of () and ts scalablty w.r.t. the number M of VNF Servers. Results shown n ths secton are averaged over 00 smulaton runs where we have assumed ˆ (F ) = 900 PUs for all S, whle the latency d and the cost c arameters have been randomly generated as llustrated below. Also, for llustratve uroses we frst focus on sngle User Grou case. Instead, n the sequel we wll also rovde results for the case of multle User Grous, whch makes ossble to show the adatablty to network confguraton changes of the roosed learnng rocedure. At each smulaton run, the latency arameters d are randomly generated from two Gaussan dstrbutons. Secfcally, the frst half of M/ VNF Servers are assocated wth a Gaussan dstrbuton wth mean values μ (d) = 300 and standard devatons equal to σ = /8μ (d). Instead, the second half of M/ VNF Servers are assocated to a Gaussan dstrbuton wth μ (d) = 450 and σ = 3/6μ (d).thecost arameters c are generated from a Gaussan dstrbuton wth mean value μ (c) = 850 and standard devaton σ = 30. To measure the convergence seed of the roosed learnng rocedure, at each teraton we consder the normalzed Eucldean dstance between the bandwdth vector b(m) comuted n () and the SE vector b as follows: d(b(m), b ) = ( b (m) b ) () B S In Fg. (a), we show how fast the roosed learnng rocedure converges to the unque SE of the game G (S) when M = 0, for dfferent ste-sze rules. Secfcally, we consder both varable ste-sze (.e., γ m = /m β ) wth β {0.5, }, and fxed ste-sze rules (.e., γ m {, 3}). It s shown that fxed ste-sze rules converge faster than varable ste-sze rules. In addton, the convergence seed s faster when hgh values of the fxed ste-sze are consdered,.e., γ m = 3. Recall that convergence of the learnng rocedure under varable ste-sze rules s ensured by Prooston 3. Unfortunately, the same s not true for fxed ste-sze rules, as n ths case convergence to the SE cannot be roven analytcally. It s worth notng that very large fxed ste-sze

12 73 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 Fg.. a) Dstance from the SE for dfferent ste-sze rules; b) Number of teratons needed to reach the SE for dfferent number M of VNF Servers and values of the ste-sze γ m ; c) Adatablty of the roosed learnng rocedure as a functon of dfferent ste-sze rules. are rone to generate oscllatons around the SE. 6 Therefore, to guarantee convergence to the SE whle achevng a fast convergence seed, a varable ste-sze γ m = /m β wth β = 0.5 should be consdered. Fnally, n Fg. (b), we show how many teratons the roosed learnng rocedure needs to reach the SE as a functon of the ste-sze γ m for dfferent values of the number M of VNF Servers when we consder a fxed ste-sze rule. More n detal, we let the learnng rocedure run untl the stong condton s reached,.e., d(b (m), b ) 0.0 for all S. As exected, an ncrease n the value of the ste-sze mroves the convergence seed of the learnng rocedure. Furthermore, n Fg. (b) we show the scalablty of the roosed learnng rocedure w.r.t. the number M of VNF Servers. It s mortant to note that an ncrease n the value of the ste-sze γ m allows to mrove the convergence seed of the learnng rocedure even when hgh number of VNF Servers are consdered, e.g., M = 50. Thus, by roerly ncreasng the value of the ste-sze, t s also ossble to mrove the scalablty of the learnng rocedure. Now, we nvestgate the adatablty of the roosed learnng rocedure when network arameters change over tme. Secfcally, we consder M = 0 VNF Servers whch serve fve dfferent User Grous each of whch s requestng a dfferent functon. Fg. (c) shows the er-user Grou average VNF Server utlty defned as U M S U U (S) at the SE and that acheved by usng the learnng rocedure for dfferent ste-sze rules as a functon of tme. For each User Grou U, at tme nstant t = 0, both the games G (U) and G (S) are layed and, as shown n Fg. (c), the SE s reached n few teratons. Then, at tme nstants t ={60, 80} we smulate a network confguraton change. More n detal, at t ={60, 80} we randomly generate a new latency arameter confguraton. Note that any change n the latency confguratons ushes users n each User Grou to re-dstrbute themselves accordng to the new network confguraton through the evolutonary game G (U). It follows that, to adat to new network and user dstrbuton condtons, and to evaluate the amount of 6 To avod oscllatons, f generated, aroaches smlar to Search-thenconverge (STC) [35] can be effectvely aled. bandwdth to be requested to the TO for each User Grou, VNF Servers need to re-execute the game G (S).Asshown n Fg. (c), at tme nstants t = {60, 80}, users wll rearrange themselves dfferently, thus causng a devaton from the revous SE. Accordngly, the learnng rocedure s reexecuted by VNF Servers and Fg. (c) shows that the system s able to quckly adat to system fluctuatons,.e., the roosed learnng rocedure s able to reach the SE n few teratons. As exected, the convergence rate s faster n the case of fxed ste-sze rules. Secfcally, the hgher the ste-sze, the faster the convergence seed and adatablty of the learnng rocedure. However, even though the consdered varable ste-sze rules show bad erformance n terms of number of teraton needed to reach the equlbrum, recall that they assures convergence. Instead, the same does not hold for fxed ste-sze rules whch are fast but ther convergence to the equlbrum cannot be analytcally roved. E. TO Revenue Effcency Analyss As ntroduced n Secton II, an mortant asect that deserves artcular attenton s related to the revenues generated by the TO. In the followng, to measure the effcency of the roosed dstrbuted mechansm w.r.t. the revenues of the TO we consder the effcency rato, defned as the rato between the revenues generated by the TO under our dstrbuted market model and that acheved under monoolstc and centralzed models,.e., Cases A and B n Fgs. (a), (b). Secfcally, the effcency rato n Case A and Case B s denoted as ξ A and ξ B, resectvely, and defned as follows: ξ A = U (TO) C, ξ B = U (TO) C U (TO) (3) B and U (TO) C are defned n (6), (7) U (TO) A where U (TO) A, U (TO) B and (8), resectvely. An effcency rato hgher than or equal to means that our roosed mechansm rovdes revenues to the TO whch are ether hgher or equal to that acheved n centralzed markets. In Case A, the oston of the unque centralzed server s of extreme mortance as t wll determne the latency that network users wll exerence when connectng to that

13 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 733 Fg.. Effcency rato ξ A as a functon of the commsson arameter ψ for dfferent values of the bandwdth-unt rce b. server and the cost arameter c. Therefore, to nvestgate the effcency of the roosed mechansms under dfferent network confguratons, we consder 50 ossble latency and cost confguratons. Secfcally, the VNF Server oston confguraton and the cost arameters used by each VNF Server, that s c, have been generated by usng the same Gaussan dstrbutons descrbed n Secton IV-D. In order the Case A to be comarable, also n ths case we consdered 50 smulaton runs, where the latences from the unque VNF Server and the costs, have been calculated as the average value of all the latences and the costs n the same run for the Case C. For llustratve uroses, n the followng we assume that the arameters (B), ˆ (F ) and ψ are fxed and equal for all VNF Servers S and we consder a sngle User Grou. We consder all the above ossble network confguratons and, for each network confguraton, we evaluate the effcency rato ξ A. Obtaned results are shown n Fg. where we show ξ A as a functon of the commsson arameter ψ for dfferent values of the bandwdth-unt rce (B). Fg. shows that, n most realzatons, dstrbutng VNF functons s more effcent than havng a sngle server that manages and controls the VNFs. Also, t s shown that only under few network confguratons the roosed mechansm s not effcent. Instead, n the majorty of the cases hgh effcency s obtaned, ξ A >, f comared to the centralzed market scheme n Case A. Furthermore, an nterestng result s related to the bandwdth-unt rce b. Secfcally, an ncrease n the value of b also ncreases the effcency of the roosed dstrbuted market model. Instead, n Fg. 3, we consder Case B, where VNFs are executed on M servers whch are owned by the TO, and we comare t wth our roosed mechansm where the same servers are, nstead, VNF Servers owned by customers. Accordngly, for each smulaton run we have consdered the same oston and rce confguratons of the relatve run of Case C. Fg. 3 shows the effcency rato ξ B as a functon of the commsson arameter ψ for dfferent values of the bandwdth-unt rce b. The roosed mechansm s not effcent for the TO for small values of the commsson arameter ψ. Secfcally, when the commssons on VNF rovsonng sent by VNF Servers s low,.e., U (TO) C, the roosed mechansm fals n mrovng the S C(B) Fg. 3. Effcency rato ξ B as a functon of the commsson arameter ψ for dfferent values of the bandwdth-unt rce b. revenues of the TO. Instead, t s worth notng that when small values of the commsson arameter are consdered, e.g., 4% 8% of the overall revenues generated by the sale of VNFs, our roosed soluton s more convenent for the TO than the centralzed ones. Ths s an mortant asect whch show that the roosed model for the dstrbuton of the VNF rovsonng s not only roftable for VNF Servers whch are now able to enter a new market, but t s also roftable for the TO. In fact, our dstrbuted market allows the TO to receve ayments from VNF Server w.r.t. both bandwdth requrements and commsson fees on ayments submtted by network users. Instead, Case B only allows ayments from network users. Accordngly, by allowng thrd-arty VNF Servers to access the VNF market, the TO can mrove ts revenues and also reduce the comutatonal cost whch s outsourced to those VNF Servers. Also, note that an ncrease n the value of the bandwdth-unt rce b mroves the effcency of the roosed mechansm aganst Case B. The numercal analyss of the roosed system shows that the three stakeholders can take advantage of the framework; ndeed the TO can smlfy the Orchestraton mechansm by usng the dstrbuted scheme and emloyng external servers for VNFs rovsonng. The VNF Servers on ther sde are sellers and can ncrease ther economc beneft. Fnally, Users can also ncrease ther beneft n terms of rce reducton and erformance mrovement. V. CONCLUSIONS In ths aer we have dscussed how game-theoretc tools can be effectvely used to address the roblem of dstrbuted management, resource allocaton and orchestraton of softwarzed networks. Secfcally, we exloted herarchcal game theory to desgn a dstrbuted SDN/NFV system where VNF Servers artcate n the VNF market as sellers of VNFs. The nteractons among VNF Servers and Users requestng VNFs have been modeled as a two-stage Stackelberg game where the former act as the leaders and the latter as the followers of the game. Unqueness of the SE has been roved, and a renforcng learnng rocedure whch rovably converges to the unque SE has been roosed. We accounted for mtatve and socal behavors of Users and we used the relcator

14 734 IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS, VOL. 35, NO. 3, MARCH 07 dynamcs equaton from evolutonary game theory to model ther nteractons. Furthermore, a closed-form equlbrum condton has been derved. Through the exlotaton of game theory, we have shown that a dstrbuted framework results benefcal for all the nvolved stakeholders. On the one hand, the smlfed orchestraton and the ossblty to offload VNFs on thrd-artes VNF Servers s benefcal to the TO. On the other hand, VNF Servers have economc beneft by artcatng to the VNF Market as sellers. Fnally, Users can choose the VNF Servers that most ft ther needs. The numercal analyss carred out has roved the feasblty and effcency of the roosed game-theoretc framework. In artcular, the results obtaned show that the framework s scalable and radly adats to network changes. APPENDIX PROOF OF PROPOSITION 3 Proof: For the sake of clarty, n the followng we wll omt the subscrt whch dentfes the User Grou U. The mean dynamcs of () s { ż = v (b) e b = B z (4) +e z At any gven tme t, letb(t) be a soluton for (4). In system theory, such soluton s often referred to as soluton orbt or trajectory of the system. In the followng, we show that ) b(t) converges to b as t +, and ) () s an asymtotc seudo-trajectory (APT) [37] for the mean dynamc (4), and converges to b f some mld condtons on the ste-sze are satsfed. From Prooston, we have that U (S) (b) s a strctly concave functon n b. Therefore, v (b)(b b ) < 0for all b [0, B ] by defnton. By exlotng ths latter result, t can be shown that the functon V (b) defned as V (b) = ( B b ) ( b B ln + b ) B b ln B b b B b (5) S s a strct Lyaunov functon for (4). In fact, we have that V = dv(b)/dt = S v (b)(b b )<0, V (b ) = 0and V (b) >0forallb = b. It can be shown that V (b) s radally unbounded,.e., V (b) when b. Therefore, the equlbrum ont b s also globally asymtotcally stable (GAS), whch mles that b(t) converges to b as t +. Now, we rove the second art of the rooston whch conssts n showng that also the dscrete-tme algorthm asymtotcally converges to the equlbrum. By decoulng (4), we get ḃ = db ( = b b ) v (b) (6) dt B The latter result wll be useful to show that the dscrete-tme algorthm tracks the contnuous-tme system u to a bounded error that asymtotcally tends to 0 as ncreases. A second-order Taylor exanson of () leads to ( b (m + ) = b (m)+γ m b(m) b (m) B ) v (b(m))+ μγ m (7) for some bounded μ. Note that μ s bounded because b v (b) s bounded by defnton. Intutvely, (7) s the dscrete verson of (6) u to a bounded error. Snce, by assumton, m γ m < m γ m =+, results n [37] show that b (m) s an APT for (4). It stll remans to rove( that b (m) b. By decoulng z and b, we obtan z = ln b B b ). By rewrtng V (b) n terms of z, we obtan V (z). By consderng a Taylor exanson of V (z), we obtan: V (z(m + )) = V (z(m)) + γ m + μ γ m S ( b (m) b ) v (b (m)) for some bounded μ > 0. Snce b s GAS, t follows that B s a basn of attracton for b. Therefore, there must exst a comact set L B contanng b,whereb s the strategy set of the game G (S).So, f we rove that there also exsts a large enough m such that b(m ) L, then, the roof s concluded. Assume ad absurdum that such m does not exsts. Recall that v (b)(b (m) b )<0 by defnton. Therefore, t must exst some β>0such that S v (b)(b (m) b ) β for a large enough m. It follows that whch yelds to V (z(m + )) V (z(m)) γ m β + μ γ m (8) V (z(m + )) V (z(0)) β γ m + μ γm (9) m m By assumton m γ m < m γ m =+. Thus, (9) leads to V (z(m +)), whch s a contradcton as V (z) s lower bounded by constructon. Therefore, [37] ensures that there must exst m such that b(m ) L and lm m + b(m) = b, whch concludes the roof. REFERENCES [] Software-Defned Networkng: The New Norm for Networks, Oen Networkng Foundaton, Palo Alto, CA, USA, Whte Paer, 0. [] M. Yu, L. Jose, and R. Mao, Software defned traffc measurement wth OenSketch, n Proc. NSDI, vol. 3. Ar. 03, [3] C. Monsanto, J. Rech, N. Foster, J. Rexford, and D. Walker, Comosng software defned networks, n Proc. NSDI, Ar. 03,. 3. [4] Network Functons Vrtualsaton-Introductory Whte Paer, accessed on Mar. 4, 07. [Onlne]. Avalable: htt://ortal.ets.org/nfv/nfv Whte Paer.df [5] S. Raynovch, Lfecycle servce orchestraton (LSO) market overvew, SDX Central Market Reort, Jan. 06. [6] F. Wuhb, R. Yanggratoke, and R. Stadler, Allocatng comute and network resources under management objectves n large-scale clouds, J. Netw. Syst. Manage., vol. 3, no.,. 36, Jan. 05. [7] M. Alcherry and T. V. Lakshman, Network aware resource allocaton n dstrbuted clouds, n Proc. IEEE INFOCOM, Mar. 0, [8] X. Meng, V. Paas, and L. Zhang, Imrovng the scalablty of data center networks wth traffc-aware vrtual machne lacement, n Proc. IEEE INFOCOM, Mar. 00,. 9. [9] J. W. Jang, T. Lan, S. Ha, M. Chen, and M. Chang, Jont VM lacement and routng for data center traffc engneerng, n Proc. IEEE INFOCOM, Mar. 0, [0] A. Basta, W. Kellerer, M. Hoffmann, H. J. Morer, and K. Hoffmann, Alyng NFV and SDN to LTE moble core gateways, the functons lacement roblem, n Proc. 4th Worksho All Thngs Cellular, Oer., Al. Challenges, Chcago, IL, USA, 04,

15 D ORO et al.: GAME THEORETIC APPROACH FOR DISTRIBUTED RESOURCE ALLOCATION AND ORCHESTRATION 735 [] A. Lombardo, A. Manzaln, V. Rccobene, and G. Schembra, An analytcal tool for erformance evaluaton of software defned networkng servces, n Proc. IEEE SDNMO, Krakow, Poland, May 04,. 7. [] H. Moens and F. de Turck, VNF-P: A model for effcent lacement of vrtualzed network functons, n Proc. CNSM, Nov. 04, [3] M. Bouet, J. Leguay, and V. Conan, Cost-based lacement of vrtualzed dee acket nsecton functons n SDN, n Proc. IEEE MILCOM, Nov. 03, [4] S. Gebert et al., osel, Demonstratng the otmal lacement of vrtualzed cellular network functons n case of large crowd events, n Proc. ACM SIGCOMM, 04, [5] M. Scholler, M. Stemerlng, A. Rke, and R. Bless, Reslent deloyment of vrtual network functons, n Proc. 5th Int. Congr. Ultra Modern Telecommun. Control Syst. Workshos (ICUMT), Se. 03, [6] R. Cohen, L. Lewn-Eytan, J. S. Naor, and D. Raz, Near otmal lacement of vrtual network functons, n Proc. IEEE INFOCOM, May 05, [7] R. Rav and A. Snha, Aroxmaton algorthms for multcommodty faclty locaton roblems, SIAM J. Dscrete Math., vol. 4, no., , 00. [8] M. F. Bar, S. R. Chowdhury, R. Ahmed, and R. Boutaba, On orchestratng vrtual network functons, n Proc. th IEEE Int. Conf. Netw. Servce Manage. (CNSM), Nov. 05, [9] A. Guta, M. F. Habb, P. Chowdhury, M. Tornatore, and B. Mukherjee, Jont vrtual network functon lacement and routng of traffc n oerator networks, Det. Comut. Sc., Unversty of Calforna Davs, One Shelds Avenue Davs, CA, USA, Tech. Re., 05. [0] W. H. Sandholm, Evolutonary game theory, n Encycloeda of Comlexty and Systems Scence. New York, NY, USA: Srnger, 009, [] ETSI NVF GS, Network Functons Vrtualzaton (NFV) Infrastructure Overvew, ETSI, Soha, France, Jan. 05. [] ETSI NFV GS, Network Functon Vrtualzaton (NFV) Management and Orchestraton, NFV-MAN 00 v0.8., ETSI, Soha, France, Nov. 04. [3] G. Farac and G. Schembra, An analytcal model to desgn and manage a green SDN/NFV CPE node, IEEE Trans. Netw. Servce Manage., vol., no. 3, , Se. 05. [4] S. D Oro, L. Gallucco, S. Palazzo, and G. Schembra, Exlotng congeston games to acheve dstrbuted servce channg n NFV networks, IEEE J. Sel. Areas Commun., vol. 35, no., , Feb. 07. [5] R. W. Shehard and F. Rolf, The Law of Dmnshng Returns, Producton Theory. Berln, Germany: Srnger, 974. [6] A. Mas-Colell et al., Mcroeconomc Theory, vol.. New York, NY, USA: Oxford Unv. Press, 995. [7] A. Bosch-Domènech and N. J. Vrend, Imtaton of successful behavour n cournot markets, Econ. J., vol. 3, no. 487, , 003. [8] G. Nan, Z. Mao, M. Yu, M. L, H. Wang, and Y. Zhang, Stackelberg game for bandwdth allocaton n cloud-based wreless lve-streamng socal networks, IEEE Syst. J., vol. 8, no., , Mar. 04. [9] J. Elas, F. Martgnon, A. Caone, and E. Altman, Non-cooeratve sectrum access n cogntve rado networks: A game theoretcal model, Comut. Netw., vol. 55, no. 7, , 0. [30] H. Tembne, E. Altman, R. El-Azouz, and Y. Hayel, Evolutonary games n wreless networks, IEEE Trans. Syst., Man, Part B, vol. 40, no. 3, , Jun. 00. [3] S. Gesendorf et al., The nfluence of nnovaton and mtaton on economc erformance, Econ. Issues, vol. 4, no.,. 65, 009. [3] P. D. Taylor and L. B. Jonker, Evolutonary stable strateges and game dynamcs, Math. Bosc., vol. 40, no., , 978. [33] J. C. Goodman, Note on exstence and unqueness of equlbrum onts for concave n-erson games, Econ., J. Econ. Soc., vol. 48, no.,. 5, Jan [34] J. B. Rosen, Exstence and unqueness of equlbrum onts for concave n-erson games, Econ., J. Econ. Soc., vol. 33, no. 3, , 965. [35] S. D Oro, P. Mertkooulos, A. L. Moustakas, and S. Palazzo, Interference-based rcng for oortunstc multcarrer cogntve rado systems, IEEE Trans. Wreless Commun., vol. 4, no., , Dec. 05. [36] P. Mertkooulos and A. L. Moustakas, The emergence of ratonal behavor n the resence of stochastc erturbatons, Ann. Al. Probab., vol. 0, no. 4, , 00. [37] M. Benaïm, Dynamcs of stochastc aroxmaton algorthms, n Semnare de Probabltes XXXIII. New York, NY, USA: Srnger, 999,. 68. Salvatore D Oro (S ) receved the B.S. degree n comuter engneerng and the M.S. degree n telecommuncatons engneerng, and the Ph.D. degree from the Unversty of Catana n 0, 0, and 05, resectvely. In 03 and 05, he was a Vstng Researcher wth Unversty Pars- Sud, Pars, France, and wth The Oho State Unversty, Columbus, OH, USA. He s currently a Post-Doctoral Research Fellow wth the Unversty of Catana. In 05, he organzed the Frst Worksho on COmettve and COoeratve Aroaches for 5G networks (COCOA), and served on the Techncal Program Commttee of the CoCoNet8 worksho at the IEEE ICC 06. In 03, he served on the Techncal Program Commttee of the 0th Euroean Wreless Conference (EW04). Laura Gallucco (M 0) receved the Laurea degree n electrcal engneerng and the Ph.D. degree n electrcal, comuter, and telecommuncatons engneerng from the Unversty of Catana, Italy, n 00 and 005, resectvely. In 005, she was Vstng Scholar wth the COMET Grou, Columba Unversty, New York, NY, USA. Snce 00, she has been wth the Italan Natonal Consortum of Telecommuncatons (CNIT) as a Research Fellow n the FIRB VICOM and NoE SATNEX rojects. Snce 00, she has been an Assstant Professor wth the Unversty of Catana. Her research nterests nclude unconventonal communcaton networks, software defned networks, and network erformance analyss. She serves on the Edtoral Board of the Elsever Ad Hoc Networks and the Wley Wreless Communcatons and Moble Comutng journals. Sergo Palazzo (M 9 SM 99) receved the degree n electrcal engneerng from the Unversty of Catana, Catana, Italy, n 977. In 994, he was wth the Internatonal Comuter Scence Insttute, Berkeley, as a Senor Vstor. In 003, he was wth the Unversty of Canterbury, Chrstchurch, New Zealand. Snce 987, he has been wth the Unversty of Catana, where he s currently a Professor of Telecommuncatons Networks. Hs current research nterests are n the modelng, otmzaton, and control of wreless networks, wth alcatons to cogntve and cooeratve networkng, SDN, and sensor networks. He was a recent of the Vstng Erskne Fellowsh from the Unversty of Canterbury. He has been servng on the Techncal Program Commttee of INFOCOM, the IEEE Conference on Comuter Communcatons, snce 99. He has been the General Char of some ACM conferences, ncludng MobHoc 006 and MobO 00, and currently s a member of the MobHoc Steerng Commttee. He has also been the TPC Co-Char of some other conferences, ncludng the IFIP Networkng 0, the IWCMC 03, and the Euroean Wreless 04. He also served on the Edtoral Board of several journals, ncludng the IEEE/ACM TRANSACTIONS ON NETWORKING, the IEEE TRANSACTIONS ON MOBILE COMPUTING,theIEEE Wreless Communcatons Magazne, the Comuter Networks, thead Hoc Networks, andthewreless Communcatons and Moble Comutng. Govann Schembra was wth the Telecommuncatons Research Grou, Cefrel, Mlan, Italy, from 99 to 99, where he was nvolved n traffc modelng and erformance evaluaton n broadband networks. He was nvolved n several natonal and EU rojects. In artcular, he worked for the Unversty of Catana n the Euroean roject DOLMEN (Servce Machne Develoment for an Oen Longterm Moble and Fxed Network Envronment), and has been actng as the WP Leader n the NoE Newcom. He s currently an Assocate Professor wth the Unversty of Catana. Hs research nterests manly concern wth SDNs, NFV, traffc modelng, cloud comutng and data center management, and moble cloud networks.