Journal of Chemical and Pharmaceutical Research, 2014, 6(10): Research Article. Study on the original page oriented load balancing strategy

Transcription

1 Avalable onlne Journal of hemcal and Pharmaceutcal Research, 2014, 6(10): Research Artcle IN : ODEN(UA) : JPR5 tudy on the orgnal page orented load balancng strategy Kunpeng Kang chool of omputer and Informaton Technology, hangqu Normal Unversty, hangqu, hna ABTRAT Load balance plays an mportant role to nformaton acquston system performance. Excellent load balancng strategy s a key for us to make full use of system memory and computng resources, to reduce the response tme of the dstrbuted operaton. Internal mechansm of orgnal page load balance s gven, based on analyss of the two recent commonly dynamc load balance methods. Fve orgnal page orented load balancng strateges are compared From the expermental and theoretcal perspectves On the premse of calculatng load ndex. Fnally, the concluson s drawn that date channel storage calculaton senstve partton s the most optmal load partton strategy. Keywords: Orgnal Page, Load Balance, Pollng, Load Balancng trategy INTRODUTION The load balancng n storage system of dstrbuted search engne. It can be dvded nto two aspects from the machne resources: hard dsk storage load balancng, ncludng the orgnal page, page content and ndex fles n dfferent nodes of the hard dsk storage. Load balance of computaton, whch refers to the load balancng of the calculaton among the storage nodes. Each node needs to carry out many computng tasks, ncludng the orgnal page wrtng dsk, the orgnal page content extracton, content page update, ndex update and ndex query etc. From the pont of vew of applcaton, the load balancng of these computng and storage resources may be dvded nto two stages. The frst stage s related wth the orgnal page, wth the crawler node wrtng the orgnal page drve on the system. For the storage t s the three data volume ncreasng and for the calculaton t s wrtng dsk ca by the orgnal page wrtng dsk, content extracton, ndex and content page update; the second stage s relevant to the ndex query, between storage nodes equlbrum ndex query load, to make the system have faster response tme of dstrbuted ndex query. The load balancng of the frst stage s crtcal for the equlbrum and full use of the storage capacty, as well as for the mprovement of the wrtng response tme of graspng subsystem. The load balancng of the second stage can optmze ndex query response tme. Ths paper manly studes the storage facng the orgnal page and computatonal load balancng. The second secton analyses the research status of the orgnal page load balance; the thrd secton puts forward the orgnal page load balance system; the fourth quarter puts forward fve strateges of the orgnal page load balancng; the ffth secton carry out the test and performance analyss for the load balancng strateges; the last secton s the summary. 2. Relevant Task The orgnal page related load dstrbuton occurs n dstrbuted acquston subsystem crawler node wrtng the page to the storage system. Before the crawler wrtes page to the storage node of the storage system, t needs to ask storage system for the nformaton of the storage nodes on the page. Load balancng strategy mplementaton begns at the moment when the management node wrtes the request to the crawler to dstrbute the storage nodes. The man task of the balance algorthm s to decde how to choose the next node, and then transmt the new servce request to 274

2 t. A good load balance algorthm s not omnpotent; generally speakng t s closely lnked wth applcaton scene. o we should consder the equalzaton algorthm comprehensvely accordng to the characterstcs of the process and make use dfferent algorthms and technques [1] [2] [3]. At present, there are two commonly dynamc load balance methods: 2.1 Pollng method In a task queue node, every member has the same status. Pollng method smply makes cycle selecton n turn n ths group. In load balancng envronment, the algorthm wll dstrbute new request next node n ths node queue, so contnuous, round and round. Each node s chosen n turn n equal status. Pollng method actvty s predctable, that s, each node s chance to be chosen s 1 / N (assumng there are N nodes). Pollng method s the most smple and most easy mplementaton approach. Ths method doesn't consder the machne somersm. 2.2 Load Index Method Load ndex algorthm calculates nodal Load Index LI (Load Index) based on node current Load condton. The Load Index consttutes the Load prorty queue of a storage node; t takes the team node from prorty queues and forwards servce request every tme when the servce request arrves. Load ndex s a dynamc estmate [4][5]. The dsadvantage of ths method s that the cost of the dynamc montorng of load ndex and the calculaton s too hgh. 3. The orgnal page load balance system The computaton load of ths part s the frst two stages of three processng phases of storage node, namely the orgnal page analyss storage and the content page processng. The content page processng needs to update the content page and ndex; storage load refers to the hard dsk storage of three data, of whch the storage of the orgnal page accounts for more than 80% of the storage; the weght of orgnal page load balance s maxmum. The frst target of the load balance s that the storage capacty of the machne can be fully and balanced, make full use of the system s hard dsk storage space; secondly, the computng power s equalzaton, avod calculaton overloadng of the node and make the whole storage system completes analyzng the receved orgnal page as soon as possble. Reply: Drectory crawler erver Request : news.sna.com.cn torage torage torage Informaton Form of torage Node Node IP Address alculatng occupancy torage occupancy LI Other Number rate rate nformaton Node % 50% 1.2 Node % 50% 1.15 Node % 50% 1.2 Node N % 40% 1.1 Fgure 1: The basc procedure of load balancng 275

3 Informaton Form of Fle drectory Date hannel torage Node News.sna.com.cn Node It.sohu.com.cn Node ports.qq.com Node News.sna.com.cn NodeN It.sohu.com.cn Node News.feng.com Node 1 Fgure 2: The logc drectory table Fgure 1 s the load balancng schematc dagram. Frst of all, grab subsystem s crawler nqury storage unt; then load balancng module chooses the storage node of whch the current load ndex beng the smallest as the storage destnaton of ths unt; At last, modfyng fgure 2 fles n a drectory nformaton sheet, recordng the unt storage nformaton for subsequent query, and returnng the unt storage node nformaton to the crawler. 4. The Orgnal Page Orented Load Balancng trategy The somersm of machne s not consdered n mple pollng method; on the other hand, even the somersm s consdered, there are dfferences n sze of the cells parttoned by task, n order to ensure the nstantanety of the nformaton collecton, we can't make the bg unt wrte request after the cache. There s great uncertanty as long as the storage unt s bg enough. The premse of load ndex method s to desgnate the dstrbuted system a load ndex whch can correctly reflect current load condton of the system. The defnton of the load ndex s crtcal. Lteratures[4] suggest usng resource utlzaton rather than resources queue length as the load ndex. Besdes, n the dstrbuted applcatons, processng memory, hard dsk, PU and I/O etc wll affect the overall speed. It s more practcal to defne the composte load ndex comprehensvely accordng to the nature of the task [6]. To balance the relevant storage and computatonal load of the orgnal page, we consdered the overall effect of the sze of the load cell, load balance method, load nformaton acquston and load balancng algorthm for storage system, and desgned a varety of load balance methods. In each method, the load ndex s defned (pollng method after consderng machne somersm transmttng nto load ndex method, just wthout dynamc montorng load nformaton). 4.1 Page Round Partton Page Round Partton P - RP (Page Round Partton) method regards the orgnal Page as the load partton unt, proportonally dstrbutes load accordng to the ntal storage capacty of each storage node. It does not take the balance of computng power nto consderaton. Its load ndex calculaton s shown as follows: pages LI = pages ( ) In the formulaton, t means at the moment t, the page numbers that the storage node has receved, refers to the hard dsk storage capacty of storage nodes. Ths method hypotheszes that the page sze s equal, due to the huge number of pages and pages beng small fles. The equalzaton of the number of page represents the equalzaton of the storage capacty, so t can balance capacty load very well. But ths strategy brngs much metadata traffc and yuan data storage to management node. Because the number of the page n the system memory s large, management node cannot stand these loads. 4.2 hannel Round Partton hannel round partton - RP (hannel Round Partton) method regards hannel as load partton unt, proportonally dstrbutes load accordng to ntal storage capacty of each storage node, and not take the balance of computng power nto consderaton. Its load ndex calculaton s as follows: channels LI = In the formulaton, channels means at the moment, the channel numbers that the storage node has receved, 276

4 refers to the hard dsk storage capacty of the storage nodes. Ths method hypotheszes the channel sze s equal, and ts load s uneven because the channel number s not large and the szes are dfferent; on the other hand, ts load dfference wll become worse wth the passage of tme because once a channel s assgned to a node, the data that the channel later grabs wll be stored n ths node, then the orgnal load dfference wll become greater wth tme goes on. But the related operatons of page content of ths method can only be carred out n the machne, makng the whole system smple n desgn. 4.3 hannel Date Round Partton hannel Date Round Partton D - RP (hannel Date Round Partton) methods regards the day hannel as load partton unt, proportonally dstrbutes load accordng to the ntal storage capacty of each storage node. It does not take the balance of computng power nto consderaton. Its load ndex calculaton s as follows: Datehannels LI = In the formulaton the Datehannels refers to the day channel number stored by the storage node at the refers to the hard dsk storage capacty of storage node. Ths method hypotheszes that the sze of moment oft, day channel s equal, though there s great dfference between the channel szes, day channel number s large, and at the same tme - RP load deteroraton does not exst. The same bg channel data wll be stored n dfferent nodes on dfferent dates, and the effect of load balance s good. But the same channel s dvded nto dfferent storage nodes. In the operaton of the content page t needs to transmt n dfferent nodes n order to have a judgment of new content. 4.4 hannel Date torage-enstve Partton hannel Date torage-enstve Partton D - P (hannel Date torage - enstve Partton) method regards hannel date as load partton unt, measurng the dsk occupaton of storage nodes real tme, dstrbutng load unt to those machnes of whch the dsk occupaton rate s low. Its load ndex calculaton s as follows: LI = In the formulaton, refers to the data quantty stored by the storage node at the moment t ; refers to the storage capacty of hard dsk of storage nodes. Ths method dstrbutes load accordng to the storage occupaton rate on the bass of D - RP, whch reflects the storage load condton of nodes. But n addton to the shortcomng D - RP, the management node needs to real-tme montorng dsk s occupaton quantty of the storage nodes, to ncrease the load balance mechansm expense. 4.5 hannel Date torage-ompute-enstve Partton hannel Date torage-ompute-enstve Partton D - P (hannel Date torage - ompute - enstve Partton) method regard hannel date as load partton unt, real-tme measurng the torage nodes dsk and computatonal load, calculatng the comprehensve load ndex, assgnng load unt to lower ndex. Its load ndex calculaton s as follows: LI = K ρ RT ( ) ρ t RT LI = u1ρ + u2ρ ρ < RT ( ) ( ) / 2 ρ t < RT ρ t RT LI = u1ρ others ρ = s the hard dsk occupancy rate of storage nodes at the moment of t ρ = ; refers to the calculaton occupancy rate of storage node at the moment t u ; 1 u 与 2 refers to storage and 277

5 calculatng adjustment factor Respectvely, equally large; RT 和 RT s storage and computng occupancy rate threshold Respectvely, beng less than and close to 1; K s a very large Number, on behalf of overload. When the node calculates or memory occupancy rate s more than threshold, the load ndex ndcates overload; In the premse of not more than the threshold value, f the node calculaton load up to more than half calculaton threshold value, t wll add the calculaton load to load ndex, otherwse, t only takes capacty load as load ndex. Ths method adds calculaton load regulaton factor based on D - P method, gves full consderaton to the nfluence on the system load of storage and calculaton. The shortcomng s load nformaton acquston quantty ncreases. 4.6 dynamc load nformaton acquston The premse to calculate load ndex s the load nformaton acquston, fast and smple load nformaton acquston mechansm ensures the tmely and effectve load ndex. The frst three knds of methods do not need to acquston the dynamc load nformaton, because management node has dstrbuton unt of each node [7]. The later two strateges need to obtan dynamc load nformaton: storage load and computatonal load. torage load s hard dsk occupancy rate; each node obtans the dsk occupaton through the system call, and then takes the nformaton real-tme feedback to management node. Because the content page analyss s the most onerous lnk of storage node calculaton. The computaton load takes the orgnal page cache data quantty to be analyzed of the storage nodes as the measurement, and takes the buffer occupancy rate as calculatng load to sent to management node at true tme. 5. Experment nce the Page Round Partton method s nfeasble n actual deployment, we do not make further expermental analyss. Ths paper makes further expermental measurement wth the later four methods about ther equlbrum effect. Two experments are carred out, and each experment takes sx days. In each experment, the storage node s 8, storage node operatng system s Red Hat Enterprse Lnux A release 4. The former one somorphsm storage nodes, the machne s processng speed s the same between hard dsk and PU; the later one s heterogeneous storage nodes, the machne s processng speed s dfferent between hard dsk and PU. No matter whch group of machne, the network card s MB Enterc card,the sze of cache area of content analyss s 1 G. At the end of the frst day of each experment, the load s recorded, and at the end of the sxth day, the load s recorded agan. oeffcent of Varaton s statstcal mathematcal concepts, ts value s the mean square error dvded by average of the statstc data, representng the dstrbuton of the data. If the coeffcent of varaton s smaller, the dfference between data s smaller, and the dstrbuton s more average; on the contrary, f the dfference s bgger, the dstrbuton s uneven. In ths paper, the load balance s measured, calculaton or storage load of each machne s quantzed, and then coeffcent of varaton of these values s calculated. By usng load coeffcent of varaton we ndcate storage system load balance. The load coeffcent of varaton s the smaller, the load s well-dstrbuted. tatstcs are made wth three knds of load oeffcent of Varaton. torage load oeffcent of Varaton ( - V): oeffcent of Varaton of hard dsk space occupancy of all storage nodes, ths value represents storage load dstrbuton. omputaton load oeffcent of Varaton ( - V): oeffcent of Varaton of content analyss buffer occupancy of all storage nodes, ths value represents computaton load dstrbuton. Total load oeffcent of Varaton (T - V): Average of - V and - V values, ndcatng total load dstrbuton of system. Ths paper made statstcs on the dstrbuton of date channel partton unt. Fgure 3 s date channel data dstrbuton, comparson s made between ts dstrbuton and random data dstrbuton. In random data, medan and mean s nearly equal; coeffcent of varaton of all values s 0.5, well-dstrbuted. n the actual date channel data, small channels are many, the greater the channel, the sparser the dstrbuton; the coeffcent of varaton of all the values s large, up to 2.9; The medan value s much smaller than the mean, ndcatng the date tends to the small. The dfference between maxmum and mnmum of the actual date channel data s great; the bggest reach 2G, and the mnmum s 100 k. 278

6 Fgure 3: The data dstrbuton of the unt of channel date Fgure 4 and fgure 5 are the result of the frst group of experments. Wth - RP method, storage load s deteroraton, the effect of long-term operaton s the worst; wth D - RP method, although the load coeffcent s larger n the frst day, t declned n sx day, whch s the adjustment by the Date hannel Partton ; - RP and D - RP are two methods based on the storage round, - V s much greater compared wth non-round method; D - P method and D - P method are the most load balanced, they are almost the same. D - P method s a lttle better n the calculaton of loadng. Wth D - P method, the man reason for calculatng the coeffcent of varaton s the somorphsm of machnes, the result of storage balance s that the calculaton s nearly equal for each machne (calculaton s wrtten by the orgnal page drve). Fgure 4: One day of somorphsm stores Fgure 5: x days of somorphsm stores Fgure 6 and fgure 7 are the results of the second set of experment. ompared wth the frst set of experments, the - V of ths group of data of the former three methods s partcularly bg, because wth the former three methods t only carres out storage load balancng, wthout consderng computaton load, to make the machne calculaton overload, whch wll as a result ncrease the crawler wrtng response tme. D - P method stll has the least load coeffcent of varaton, although ts V s bgger than D - P, ts - V s much smaller, and n D - P method, - V and - V both are about 0.3. The system can accept ths uneven load. hown by the comparson of two groups of the experments, D - P method has the most equlbrum computng and storage load dstrbuton n both somorphsm and heterogeneous fleet. D - P method takes the second place, and n the heterogeneous network t could form serous uneven dstrbuton of calculaton. Round method performs the worst. 279

7 Fgure 6: One day of heterogeneous stores Fgure 7: x days of heterogeneous stores ONLUION Load balance plays a key role to nformaton acquston system performance. only load balance s well done, can we make full use of system memory and computng resources, to reduce the response tme of the dstrbuted operaton. Ths paper deals wth balance processng to storage and calculaton related load brought by the orgnal page wrte system, and puts forward a varety of load balancng strateges, and further measures the load balance effect of each scheme through experments. Fnally, we fnd that date channel storage calculaton senstve partton s the most optmal load partton strategy. Acknowledgements Ths paper s supported by cence and Technology Department of Henan Provnce (No ). REFERENE [1] Pol R; Kennedy J; Blackwell T. warm Intellgence, 2007, 1(1), [2] He Jalang; Ouyang Dantong; Zhu X; J Jnchao, AI, 2012, 4(7), [3] Banks A; Vncent J; Anyakoha. Natural omputng, 2007, 45(6), [4] ZHANG Deyu; LI hunyan.jdta, 2012,6(23), [5] Jpng L; houyn Lu; hxun Wu. IJAT, 2012, 4(12), [6] Janguo Wang; Zhje Zhang; Wenxng Zhang. JIT, 2013, 8(8), [7] Zhaohu Jang; Jng Zhang; hunsheng Wang. IJAT, 2013, 5(8),