Load-Balanced Anycast Routng Chng-Yu Ln, Jung-Hua Lo, and Sy-Yen Kuo Department of Electrcal Engneerng atonal Tawan Unversty, Tape, Tawan sykuo@cc.ee.ntu.edu.tw Abstract For fault-tolerance and load-balance purposes, many modern Internet applcatons may requre that a group of replcated servers dspersed wdely over the world. The anycast as a new communcaton style defned n IPv6 provdes the capablty to route packets to the nearest server. Better qualty of servce (QoS) can be acheved by ths knd of computng paradgm. DS, Web Servce, and Dstrbuted Database System are three most well known examples. However, before anycastng can be realzed, more researches need to be done. The anycast routng scheme s one of the most mportant ssues. In ths paper, we propose a load-balanced anycast routng scheme based on the WRS (Weghted Random Selecton) method. We suggest that the server capablty should be propagated along wth other felds n the routng tables. An anycast routng algorthm should take nto account the network transmsson capablty as well as the server processng capablty for the selecton of a target server. Three weght determnaton strateges are gven. We also develop a smple algorthm to calculate the weghts of WRS to acheve optmzaton under both the heavy and the lght system traffc envronment. Our approach s locally optmzed to mnmze the average total delay and well balanced for the server load.. Introducton Many modern Internet applcatons may requre that a group of replcated servers dspersed n dfferent locatons. These servers have the same nformaton and provde the same servces. A request ssued from the clent can be sent to and satsfed by any one of them. Usually, the nearest one s the most desrable. Ths knd of systems are desgned for both fault-tolerance and loadbalance obectves. The network transmsson delay and the servce processng tme can be mnmzed to accomplsh a better servce qualty by carefully choosng the target server. Doman ame Servce (DS), Web Servce, and Dstrbuted Database System are three most well known examples. As compared wth uncast, broadcast, and multcast, network-layer anycast s a newer servce type defned n IPv6 to meet the user requrement. In IETF RFC 546 [], Partrdge, Mendez, and Mllken descrbe the anycast servce as below. A host transmts a datagram to an anycast address and the nternetwork s responsble for provdng best effort delvery of the datagram to at least one, and preferably only one, of the servers that accept datagrams for the anycast address. By movng the task of fndng an approprate server from clent software to network, anycast can greatly smplfy the effort of Internet applcatons. However, more researches stll need to be done before anycastng can be realzed. Furthermore, the routng scheme s one of the most mportant ssues. Recently research on anycastng ncludes many categores: anycast archtecture, anycast routng algorthm, server selecton polcy, and so on. The archtecture related topcs nclude network-layer anycast, applcaton-layer anycast [], anycast n wreless Ad Hoc networks, and actve anycast [3]. And the anycast routng algorthm related topcs usually combne wth the server selecton polcy together to acheve the obectves of loadbalance and Qualty of Servce (QoS) [4, 5]. A survey can be found n [6]. In [4], Xuan et al. proposed an anycast routng protocol that s composed of two sub-protocols: the routng table establshment sub-protocol and the packet forwardng sub-protocol. In the routng table establshment sub-protocol, they consdered four methods (Shortest-Shortest Path, Mnmum-Dstance, Sourcebased Tree, and Core-based Tree) to prevent the loop problem caused by multple paths among routers. In the packet forwardng sub-protocol, they take a Weghted Random Selecton (WRS) approach for mult-path selecton to balance the network traffc. The smulaton showed that the loop-preventon methods and the WRS approach have great mpact on the performance n terms of average end-to-end packet delay. They manly dealt wth the network congeston problem of anycast. They Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
also tred to dstrbute the network traffc as even as possble and hold the loop-free property smultaneously. In ther approach, routers select the target servers usng the dstance nformaton n ther routng tables. However, a router can do the same much better and only a slghtly extended effort s needed. Two man contrbutons of our paper are: () In our approach, we deal wth both the network capablty and the server capablty smultaneously. We suggest that the server capablty nformaton should be propagated along wth other nformaton contaned n routng tables. Ths makes routers have the potental capablty to provde better load-balance and Qualty of Servce. Three weght determnaton methods are also presented for WRS. () A smple weght calculaton algorthm s presented to acheve optmzaton under both lght traffc and heavy traffc envronments. Base on the queueng theory, our approach s locally optmal for load-balance and has the mnmum average total delay. The rest of ths paper s organzed as follows. In Secton, we brefly descrbe the basc knowledge of routng schemes. We also present the problem model descrpton and the theoretc dervatons n ths secton. Then we present our load-balanced anycast routng scheme n Secton 3. Some remarks are also dscussed here. Evaluatons and analyss are gven n secton 4. Secton 5 contans the conclusons.. Problem Model and Mathematcal Dervatons. Prelmnary The man functon of the network layer defned n OSI (Open Systems Interconnecton) Reference Model s routng packets from the source host to the destnaton host. The routng algorthm s the maor part of the network layer software responsble for determnng whch output lnk an ncomng packet should be transmtted on [7]. In 990, OSPF (Open Shortest Path Frst) protocol [8] proposed by Internet Engneerng Task Force (IETF) has become the standard of Internet routng protocol and supported by most router vendors. OSPF s an open standard, supports a varety of dstance metrcs (physcal dstance, delay, etc.), and s a dynamc algorthm. A routng scheme can be dvded nto two stages: () buld the routng table, () select the outgong nterface. In the frst stage, a lnk state advertsement sub-protocol s needed and the routng nformaton wll be propagated. The routng table ncludng the destnaton, cost, and next hop felds wll be establshed. In the second stage, when a packet s receved by a router, the router must apply ts routng algorthm to choose an outgong nterface and forward the packet. A good routng scheme should work both correctly and effcently. There are several strateges for the routng algorthm optmzaton. Mnmzng the mean packet delay and maxmzng the total network throughput are two good canddates. Mnmzng the mean packet delay can mprove the Qualty of Servce. Meanwhle, maxmzng the total network throughput can ncrease the network utlzaton. These two goals are not always attanable at the same tme. In realty, they are usually n conflct wth each other. For the tradtonal uncast routng scheme, the clent host tself determnes the destnaton address. The only task of the network s to route the packet to the assgned destnaton host as fast as possble. The optmal routng scheme s the one wth the mnmum transmsson delay. However, for an anycast routng scheme, the network may have more than one choce of the destnaton server for an anycast address. The server havng the mnmum transmsson tme but wth a heavy loadng may not be the best choce. On the other hand, the one havng the smallest servce processng tme but beng very far away may not be favorable ether. The user response tme ncludes the network transmsson tme and the server processng tme. From the Qualty of Servce vew, the fnal goal of an anycast routng scheme should mnmze the end-to-end user response tme. Both the network capablty and server capablty nformaton are benefcal for the routng scheme.. Problem Model A network consstng of a number of nodes and lnks s usually consdered as a connected graph G (V, E) where V s a set of vertces representng the hosts (and/or routers) and E s a set of edges representng the lnks. An ntermedate node s called router R whch s responsble for packet transmsson, and a boundary node s called host H. In our dscusson, a host can be a clent machne or a server machne. A clent machne s where the request packet s ssued, and a server machne processes the request and sends back the response packet. The sequence of routers through whch a packet s transmtted s called a path P. Each edge s assocated wth a numercal value called dstance d. The dstance s usually assgned wth the delay tme or the bandwdth of the lnk. Fgure shows the problem model. R s a router n the network. S to S are a group of servers wth a specfc anycast address and map to the entres of routng table on R. We assume that the packet arrval pattern s a Posson Process wth arrval rate λ. Usng the Weghted Random Selecton method, the ncomng packets are dstrbuted nto the outgong nterfaces I to I wth the destnaton Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
servers S to S accordng to the correspondng weghts W to W. Furthermore, each procedure that sends packets from R to S and processed at S can be modeled as a M/M/ queueng system [9, 0]. Posson Process arrval rate λ R λ W λ W λ W M / M / Fgure. Problem model.3 Mathematcal Dervatons We would lke to mnmze the average delay tme n the system as much as we can. From the queueng theory [9, 0], we have the average delay tme T of each M/M/ system as follows: T λ W, s the average servce rate. Therefore, we can model the optmzaton problem for anycast routng as below. The optmzaton problem can be represented as follows: Mnmze: W, () λw Subect to W, W 0,,,..., S S S. () To solve the above problem, the Lagrange multpler method [] can be appled. Eq. () and Eq. () can be smplfed as follows: Mnmze: L( W, W,..., W, a) W ( ) + a( W ) (3) λw By the Kuhn-Tucker condtons, the necessary condtons for a mnmum of Eq. (3) to exst are as follows: W a L( W, W,..., W, a) + a 0 (4) ( λw ) L( W, W,..., W, a) W 0 The soluton of W from Eq. (4) s / λ + / λ W,, /,...,. (5) Smlar dervatons can be found n the Appendx 3 of Xuan s work [4]. In secton 3.3, we wll propose a smple optmal algorthm to calculate W. 3. Load-Balanced Anycast Routng Scheme When an anycast packet arrves at a router, the router needs to select an outgong nterface accordng to the nformaton contaned n ts routng table. The suffcency of nput nformaton and the apposteness of algorthm determne the performance of the routng scheme. Based on WRS, we can mprove the routng scheme from three aspects. () We suggest that the server capablty should be propagated along wth other nformaton contaned n a routng table entry. By takng nto account the network lnk capablty and the server capablty smultaneously, the routng scheme get more nformaton to acheve better load-balance and Qualty of Servce. () We present three weght assgnment methods: the frst one takes account of network congeston only, the second one takes account of server load, and the thrd one takes account of both at the same tme. (3) We propose a smple algorthm to calculate the weghts of WRS. Ths algorthm holds the optmzaton property of WRS under both heavy loadng traffc and lght loadng traffc envronments. The frst aspect s appled to the routng table establshment stage, and the second and thrd aspects are appled to the outgong nterface selecton stage of the routng scheme. We explan our load-balanced anycast routng approach below. 3. Routng Table Establshment Stage The routng table establshment stage constructs the routng table to provde the needed nformaton n the routng scheme. An entry n the common routng table usually ncludes the destnaton address, dstance, and next-hop felds. The entres wth the addresses that match the destnaton feld n the packet are the canddates for the routng algorthm. The canddate entres for an anycast address can be multple n two ways: () a sngle anycast address wth multple target server addresses, () a target server wth multple routng paths. The dstance feld presents the nformaton of the transmsson tme needed from router to destnaton. The routng algorthm prefers to select the entry wth smallest dstance away from Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
canddates and forward the packet to the outgong nterface n the next-hop feld. Choosng the entry wth shortest path can mnmze the transmsson delay. However, t also tends to make the network congested and the server overloaded snce all the packets are sent to the same target server along the same routng path. Furthermore, the response tme s the summaton of transmsson delay and server processng tme. The mnmzaton of the response tme s a more approprate goal than the mnmzaton of the transmsson tme only. Therefore, to mnmze the average response tme as much as possble, we would lke to balance the network traffc and the server loadng by dstrbutng ncomng packets over more than one outgong lnk. To acheve that, the nformaton on the network lnk capablty and the server capablty are needed. So we suggest that the server capablty feld should be added and can be measured by the maxmum number of packets that can be processed per unt tme. Moreover the dstance feld can be used as a measurement of the network lnk capablty. We assume that a routng table or topology nformaton exchange protocol lke RIP or OSPF s ncluded n the routng scheme. Thus these extended nformaton can be propagated and updated. Some extensons may be needed for these table nformaton exchange protocols and are not ncluded n ths paper. Fgure s a sample network used for dscusson. H to H 4 are clent machnes that ssue anycast packets. R to R are routers. And S to S 4 are server machnes that have the same anycast address A and can deal wth the packets. Table s the modfed routng table at R. R 3 R5 R9 H S 6 5 8 R R3 6 R6 7 R7 8 0 0 R0 H S 3 R H3 S3 9 4 Fgure. Sample etwork. H4 S4 R4 7 R8 R Table. Routng Table at Router R. Destnaton Dstance ext Hop Server Capablty A, S 6 R5 6 A, S 6 R0 5 A, S3 6 R0 9 A, S4 35 R 8 3. Outgong Interface Selecton Stage Once the routng table has been bult on a router, t can be used to route packets. As mentoned above, the canddate entry could be multple for an anycast packet and we want packets be dstrbuted nto more than one outgong nterface. We ntroduce a routng algorthm called Weghted Random Selecton (WRS) method here. In WRS, every outgong nterface s assgned a weght and selected randomly. The probablty of an outgong nterface been selected s proportoned to ts correspondng weght. By carefully determnng the weghts, we can control the dstrbuton of packets among the outgong nterfaces. By ths way, the network traffc and the server loadng can be balanced. Eq. (5) n secton.3 gves us an optmal soluton for the determnaton of W. Assume that for an anycast address, there are canddate entres n the routng table of router R. Let us ndex these canddate entres by,,,. Then the values n the dstance felds of these entres can be denoted as D, D,, D. The values n the server capablty felds that are suggested to be added n secton 3. can be denoted as C, C,, C. The weghts of these entres are W, W,, W, respectvely. The weght assgnment method s the key step of the WRS algorthm. Usually we assume the packets arrve at the router R s a Posson process wth rate λ. Therefore, for each routng path from R to S modeled as a M/M/ system, the arrval rate λ λ W. There are several strateges to determne the servce rate. We descrbe three of them below. Method. D D s the needed transmsson tme from router R to server S. Method takes account of the network congeston problem only. The assgnment of W balances the traffc nto dfferent outgong lnks and locally optmzes the average transmsson delay. When the server loadng s lght, ths method works better. Method. C C s the number of packets that server S can serve per unt tme. Method consders the server overload Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
problem only. The assgnment of W balances the server load and locally optmzes the average server processng tme. When the network traffc s lght, ths method works more effcently. Method 3. D + C D + / C s the summaton of the network transmsson tme and the server processng tme and s called the total delay (or the response tme). Method 3 takes account of both the network congeston and the server overload problems. The assgnment of W balances the network traffc and the server load smultaneously and locally optmzes the average delay. Substtutng the servce rate nto Eq. (5), we get the weght assgnment equatons of the above three methods respectvely. We suggest that an anycast routng algorthm should adopt method 3 as ther routng strategy to acheve the best performance on QoS and server load-balance. The evaluatons and comparsons of these three methods are gven n secton 4. 3.3 Weght Calculaton Algorthm By applyng the Lagrange multpler method we can get soluton of the optmzaton problem. The weght W of the WRS method can be obtaned by substtutng related parameters nto Eq. (5). We need all W 0 snce the probablty of a packet transmtted nto nterface must be postve. However, ths requrement cannot be guaranteed by the Lagrange multpler method. Table gves an example to show ths problem. It occurs frequently under at a lght traffc (wth relatvely smaller arrval rate λ) envronment. To mnmze the average delay n Eq. () and stll satsfy the condton n Eq. (), the Lagrange multpler method forces the weght of the nterface wth longer delay to become negatve and obtans a bgger weght (maybe > ) on the nterface wth smaller delay. The smple algorthm we proposed below can prevent ths problem and acheve optmzaton whatever the arrval rate λ be. For general cases, ths algorthm ends n a couple of rounds. Table. Problem of Lagrange multpler method. Interface Servce Rate W (Eq. 5) W (Algorthm) 00-0.349 0.0000-0.0360 0.0000 3 0000.3509.0000 Arrval Rate λ 000 Algorthm: Step : Set r 0. Step : Calculate the followng equatons W r / λ + r / r Step 3: Rearrange the ndex such that W W... W 0 > W W / λ,,,..., r. k k+ k+ K W r Step 4: If k r then stop, else update r r + k and W, W, K, W 0. Step 5: Go to Step. k+ k+ r Moreover, the optmal soluton has the form: W r / λ + r W 0, otherwse. 3.4 Remarks / r / λ,,,..., r. We gve some remarks for our approach below.. Applcaton Layer vs. etwork Layer Approach Anycastng can be mplemented ether n the applcaton layer or the network layer. An applcaton layer approach determnes the destnaton address of the target server at the clent machne where the packet has been ssued. It provdes the potental for optmzng the end-to-end response tme. However, a large amount of extra works needs to be done by the applcaton tself. For example, the applcaton needs to mantan the nformaton on server load, network topology, transmsson tme, and so on. Several new protocols may be needed for the communcaton n the system. The network layer approach combnng wth the exstng IP protocols can sgnfcantly reduce the effort by the applcaton software. Due to the lack of global nformaton of the applcaton traffc, the router can only locally optmze the traffc drectly through t. However, from the theory of statstcs, a locally optmal soluton can provde a near optmal approach.. Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
. Dynamc vs. WRS Load-Balancng Approach A dynamc load-balancng approach montors the states of system and dynamcally dspatches tasks. It quckly reflects the change of system and can relatvely closes to the real optmzaton. However, t consumes more resources and usually s more complcated. The WRS approach s based on the probablty model and dspatches tasks accordng to the weghts. In our approach, the weght s pre-calculated accordng to the capablty of network lnk and server. In most cases, these two parameters do not change frequently. The nformaton update mechansm provded by the common routng table nformaton exchange protocol can be used as usual. 3. Determnng D and C A lot of works have been done on the measurement of the network lnk capablty and the server capablty. To explan the load-balanced anycast routng scheme, the recprocal of dstance feld n the modfed routng table (Table ) s selected to represent the network lnk capablty D. Ths makes the transferrng from the network lnk capablty to the transmsson delay become easy and possble. Meanwhle, the maxmum number of packets that can be processed per unt tme recorded n the server capablty feld s selected to represent the C. To apply ths load balanced anycast routng scheme, one may choose the most sutable measurement method. 4. Loop-Preventon Problem The canddate entres of a routng table for an anycast address may be multple for two reasons: () a sngle anycast address wth multple target server addresses, () a target server wth multple routng paths. That makes the routng path become multple. We need a mechansm to determne the order among routers. Four methods have been presented to do that, the Shortest-Shortest Path (SSP) method, the Mnmum Dstance (Mn-D) method, the Source- Based Tree (SBT) method, and the Core-Based Tree (CBT) method. All four methods can be combned wth our approach. We defne the canddate entres used n our approach are all the elgble ones that have been verfed by any one of the above four methods. 5. Flow Type Traffc Problem If a sequence of packets arrved on a router have been sent to the same server, these packets can be dentfed as a flow n IPv6. The leadng packet(s) s processed and routed by the routng algorthm normally. Once the flow s dentfed, the followng packet wth same flow label n ts IP header wll be routed accordng to the prevous result, and then transmtted to the same outgong nterface. Our approach s compatble wth ths mechansm and can be used n the routng process at the leadng packet(s) of a flow. 4. Evaluatons and Analyss Consderng the router R, we randomly generalze the entres n the routng table n our evaluaton model. The sze of an anycast group s assumed to be ten. Furthermore, the dstance and server capablty are both normalzed to be one. The average server load and the average delay of three weght determnaton methods are lsted n Table 3. And for comparson purpose, the fxed method wth r n [4] s recalculated and adusted to ft nto our evaluaton parameters. Method Method Method 3 T Average Delay D W ( T C ) + C W ( T + D ) D + C W T λ W Table 3. Evaluaton Parameters. Average Server Load λ W C λ W C λ W C Fgure 3, 4, and 5 depct the evaluaton results. Fgure 3 shows the relatonshp between the average server load and the arrval rate. Fgure 4 shows the tendency of the average delay versus the arrval rate. Fgure 5 s an enlarged verson at Y coordnate axs of Fgure 4 to show clearly the dfferences between varous methods. We pont out below some observatons from the fgures.. The assumptons and results of Method and fxed method n [4] are smlar. They both consder the network delay problem only and gnore the lmtatons of the server capablty. The only dfference between them s that Method uses the Algorthm to rearrange the weghts f ther values become to negatve. The curves of these two methods n three fgures are very close.. Snce both Method and fxed method n [4] do not take account of the server capablty problem, they may transmt too many packets to some servers and make these servers overloaded. In Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
Fgure 3, the average server loads of these two methods can exceed the normalzed value and those of other two methods won t. Method takes account of the server capablty only and Method 3 takes account both the server capablty and the network delay, so ther values of average server load are much smaller than others. 3. Fgure 4 shows the relatonshp between the average total delay and the arrval rate. The average total delay rapdly ncreases when the arrval rate becomes close to one. That means the total delay may become extremely large f the traffc s very heavy. Average Server Load.8.6.4 /D. C /(D+/C) 0.8 Fxed r 0.6 0.4 0. 0 0 0. 0.4 0.6 0.8 Arrval Rate Fgure 3. Average Server Load. The average server load can be taken as a measurement of the server load-balance characterstc. In the meanwhle, the average total delay can be taken as a measurement of the user Qualty of Servce. From the evaluaton results, we know the WRS-based routng algorthms for anycast are desrable and should consder both the network transmsson delay and the server processng delay. 5. Conclusons Average Delay 00 50 00 50 0 0 0. 0.4 0.6 0.8 Arrval Rate /D C /(D+/C) Fxed r To obtan a better performance on QoS and server load-balance, we mprove the routng algorthm from three aspects. We conclude that the server capablty should be propagated along wth other felds n the routng table. A strategy consderng both the network congeston and the server overload was also presented for the determnaton of the weghts for WRS method. Last but not least, we showed the problem of fndng the optmal weghts under lght traffc envronment and proposed a smple algorthm to solve the problem. Evaluatons and analyss were gven to show that our approach could acheve better performance on the average delay and the average server load. 6. References [] C. Partrdge, T. Mendez, and W. Mllken, Host Anycastng Servce, IETF RFC 546, ovember 993. [] E. Zegura, M. Ammar, Z. Fe, and S. Bhattacharee, Applcaton-Layer Anycastng: A Server Selecton Archtecture and Use n a Replcated Web Servce, ACM/IEEE Transactons on etworkng, 8(4): 455-466, 000. [3] H. Mura, and M. Yamamoto, Server Selecton Polcy n Actve Anycast, IEICE Transactons on Communcatons, Vol. E84-B. o. 0, October 00. Average Delay 50 40 30 0 0 0 Fgure 4. Average Delay. 0 0. 0.4 0.6 0.8 Arrval Rate /D C /(D+/C) Fxed r Fgure 5. Average Delay wth Enlargement. [4] D. Xuan, W. Ja, W. Zhao, and H. Zhu, A Routng Protocol for Anycast Messages, IEEE Transactons on Parallel and Dstrbuted Systems, Volume: Issue: 6, Jun 000. [5] W.T. Zaumen, S. Vutukury, and J.J. Garca-Luna- Aceve, Load-Balanced Anycast Routng n Computer etworks, Proceedngs Ffth IEEE Symposum on Computers and Communcatons (ISCC 000), July 000. [6] S. Yu, W. Zhou, and Y. Wu, Research on etwork Anycast, Proceedngs of the Ffth Internatonal Conference on Algorthms and Archtectures for Parallel Processng, 00. Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE
[7] Andrew S. Tanenbaum, Computer etworks, thrd edton, Prentce-Hall, 996. [8] J. Moy, OSPF Verson, IETF RFC 47, July 99. [9] Donald Gross, and Carl M. Harrs, Fundamentals of Queueng Theory, thrd edton, Wley-Interscence, 998. [0] Sheldon M. Ross, Introducton to Probablty Models, 7th edton, San Dego: Harcourt/Academc Press, 000. [] M. S. Bazaraa and C. M. Shetty, onlnear Programmng: Theory and Algorthm, John Wley & Sons, 993. Proceedngs of the Tenth Internatonal Conference on Parallel and Dstrbuted Systems (ICPADS 04) 5-9097/04 $ 0.00 IEEE