An Empirical Study of MPI over PC Clusters

Transcription

1 An mpirical Sudy of MPI over PC Clusers Fazal Noor*, Majed Alhaisoni*, Anonio Lioa+ *Compuer Science and Sofware ngineering Deparmen Universiy of Hail, Saudi Arabia +Deparmen of lecrical ngineering and Deparmen of Mahemaics and Compuer Science Technische Universiei indhoven The Neherlands Absrac Message Passing Inerface (MPI) is an imporan mechanism in P2P. Herein we assess how differen ypes of MPI collecive communicaion funcions perform on a Gigabi herne Homogeneous Beowulf PC cluser. In his way we provide an insigh on he facors ha affec P2P applicaions over an enerprise conex such as he emerging Cloud-based services. By conras o he lieraure, which includes mosly heoreical sudies, we carry ou an empirical sudy. We show ha he bahaviour of scaer and gaher are mos unpredicable in comparison wih oher collecive funcions. Keywords - MPI benchmark, Homogeneous PC Cluser, herne, Collecive Communicaions, Laency, Bandwidh, Allo-all, Gaher, Scaer, Broadcas. I. INTRODUCTION In recen years Peer-o-Peer (P2P) neworks have become an acive area of research [1]-[4]. Tradiional neworks use he clien/server paradigm where dedicaed servers offer cliens services. P2P neworks are characerized by all peers having he capabiliy of boh being a clien and a server. P2P neworks can suppor many applicaions such as sharing of resources, e.g. communicaion services, file sharing, query search, disribued compuing, ec. In P2P opology, MPI (Message Passing Inerface) is considered as a common communicaion proocol for various P2P sysems. Therefore, i is considered as a good mechanism wih is goals are o have high performance, scalabiliy, and porabiliy. Having low delay wih reasonable hroughpu is imporan for compuing clusers, due o a lack of shared memory implies large amouns of nework daa ransfer. However, porabliliy is very imporan for MPI. The scalabiliy of MPI is mainly due o MPI being he real sandard in disribued compuing. In his paper, a simulaion of P2P is done using MPI collecive communicaions rouines on a PC cluser o measure and evaluae he performance of P2P sysem. A homogeneous PC cluser is defined as one having idenical hardware (including nework hardware such as swiches) and operaing sysem on all he machines in he nework. I is considered heerogeneous if PC hardware and/or sofware is differen from each oher in a cluser. One of he reasons o sudy homogeneous PC cluser is o gain insigh ino he behaviour of collecive communicaion models used as models usually are made under he assumpion of homogeneiy. The objecive of his paper is wofold, firs o sudy how a variey of MPI communicaion models perform over PC clusers. Second, we define and measure in pracical seings he execuion ime of herne neworks. We pinpoin he overheads and how hese affec link efficiency. Among all echnologies, including Infiniband [5], Quadrics [6], and Myrine [7], we have decided o focus on herne which is readily available for experimenaion. The paper is organized as follows, in Secion II some relaed work is presened. In Secion III experimenal mehods are presened and in Secion IV he resuls are presened. Secion V conains discussion of he resuls presened. Finally in Secion VI conclusion and fuure work is presened. II. RLATD WORK There has been a lo of work on MPI communicaions performance of PC cluser. Mos of work is performed on a heerogeneous Beowulf PC clusers. In our work we focus on sudying he performance of MPI collecive communicaions on homogeneous Beowulf PC cluser consising of 20 idenical machines. In [9] he auhors have used MPIBench a sofware for benchmarking he performance of MPI funcions using a highly accurae, globally synchronized clock. In [10] he auhors have developed a MagPIe library which opimizes MPI s collecive communicaion and have used a LogP model for shor messages and LogGP model for long messages (Table 1). In [11] a nice comparison is made among he common 65

2 parallel communicaon models appearing in he vas lieraure, such as Hockney, LogP, LogGP, and PlogP. All hese models appearing in he lieraure make an assumpion of a homogeneous environmen and are lised in Table 1 [11]. Table 1: Common Communicaion Models Model Time Parameers Hockney Log P Log GP P Log P T = l + m/b T= o + ( k/w -1)*max (g,o) + L + o T= o + (k-1)g + L + o Same as Log P bu wih each parameer being dependen on message size. l : is laency of nework b: is bandwidh o: overhead ime o ransmi or receive g: gap min ime inerval beween message p: number of processors l: upper bound on laency k: number of byes in a message w: size of he nework package in byes G: gap per bye for long messages Same as Log P bu dependan on message lengh m L: end o end laency However analyical models can no ruly replace acual performance measuremens. In our work we presen empirical resuls of he MPI collecive communicaion rouines. Such funcions provide insigh o communicaing on boh wireless and wired Peer-o-Peer sysems. III. XPRIMNTAL MTHODS A. Performance Measuremen The facors which affec performance are many and may be lised as: 1. Hardware relaed: CPU clock speed and number of CPUs on moherboard, memory, and nework adapers. 2. Nework relaed: Type of hardware, cable, faseherne, swiches, and rouers. Proocols used TCP/IP, UDP/IP and ohers. 3. Sofware relaed: Operaing sysem ype, user buffering, kernel buffering of daa, ypes of MPI rouines used for example collecive communicaions. MPI eager and rendezvous proocols, efficiency of algorihms, and polling or inerrups. From he above we can define execuion ime as a funcion of boh hardware and sofware. xecuion Time will be a funcion of opology, number of nodes, message size, swich, rouer, nework adaper, ype of algorihm, link ype, overhead compuer and operaing sysems, physical medium, MPI relaed, TCP/IP relaed, and herne relaed. This is indeed a very complex funcion in which laencies of boh hardware and sofware have o be considered. Hard disk, RAM, and cache access imes wih nework inerface card, PCI, and PCI xpress ransporaion imes need o be considered in calculaion of execuion ime. Also he laencies of nework devices should be considered such as he swiches which are in he 2 o 20 µsec range [8]. In rouers he processing delay due o sofware processes would be considerably higher. DSL or Cable inerne connecions have less han 100 milliseconds (ms) delays bu less han 25 ms are desired. Saellie Inerne connecions have an average of 500 ms or higher laency. The peak heoreical bandwidh of a nework connecion is fixed by he echnology used bu he acual hroughpu (bandwidh) varies wih ime and is affeced by high laencies. xcessive laencies on he nework causes bolenecks ha hinder flow of daa herefore decreasing effecive bandwidh. The oal ime of sending a message from one peer o anoher peer compuer can be represened in erms of execuion imes and communicaion ime as, Time = + +, (1) execa comm where exec ime can be defined as, execb exec = K C / f. (2) where K is insrucions per program, C is clock cycles per insrucion and f is CPU frequency. The ime involves he message s journey from he ransmiing compuer s memory, user space o kernel space o he nework inerface, hrough he physical medium, o he swich, and hen o he receiver compuer s nework inerface, and up o he applicaion. In Peer-o-Peer applicaions collecive operaions are rampan. Broadcas, scaer, and gaher rouines are common and heir communicaion ime depends on he size of each message, number of messages, inerconnecion srucure, and nework conenion. The communicaion ime can be wrien as comm = s + m + c. (3) In he above equaion s is he message laency assumed consan and includes he overhead ime a he source and desinaion; is he ransmission ime compued as 1/B where B is link bandwidh given in Q bis/sec; and m represens message o send. The conenion ime c is burs dependen and usually removed for simpliciy. The ime complexiy is O(m) for m daa iems. Usually one sends messages from one compuer o muliple desinaions. The 1-o-N fan-ou broadcas is when he same message is sen o N desinaions sequenially, hen he communicaion ime is 66

3 comm = N( s + m ), (4) and for he scaer and gaher communicaion models is, comm = N( s + m / n). (5) In scaer a unique message is sen from source o every oher desinaion and in gaher a unique message is received from every oher nodes. The ime complexiy is O (Nm) for one source connecing o N desinaions. In (5), n is he oal number of nodes and N = n-1. For a ree ype srucure he ime complexiy of 1-o-N fan-ou will depend on number of nodes a each level and he number of levels. The disadvanage of a binary ree implemenaion is if one node fails hen all he nodes below i will no receive he message. For an herne LAN nework, communicaion ime can be defined as: = + m (6) comm m where all erms depend on message size and message ime a Media Access Conrol (MAC); m is dependan on message size and would be same for a homogeneous nework and would be some facor muliplied by message size, α msize ; m represens he number of bis in an herne packe; q is he queuing delay; p is he propagaion delay defined as d/c, where d is he lengh of he link and c is speed of ligh in medium having a value 8 less han 3x10 m/sec (e.g. Copper wire.77c); is he herne inerframe gap which is 12 byes (96 bis). The raffic inensiy can be defined as T q ( m p) p i / B = (7) where p is he average packe arrival rae. Therefore T i approaching close o 0 will indicae small delay; Ti approaching o 1 is an indicaion of large delay. The communicaion overhead can affec ransmission efficiency. The ransmied herne packe has a payload which is TCP/IP encapsulaed in addiion o he applicaion header. TCP header consiss of 20 byes and he IP header consiss of a leas 20 byes. The ransmied herne frame has a preamble of 8 byes and an inerframe gap of 12 byes. The number of herne packes per second on he link will be, hernepackespersecond = B / [ 8 *(FrameSize + Inerframe gap 12 byes + Preamble 8 byes ) ] (8) The herne proocol efficiency is defined as, fficiency = Payload size/ Frame size (9) and he hroughpu is Throughpu = fficiency x B (10) Therefore for every herne packe on he link, a 96 bi inerframe gap and 64 bis of preamble would be overhead. If he link has he capaciy of 1 Gbps hen for a minimum herne frame size of 64 byes ransmied, he link will consis of 7.62 x 10^8 bis/sec due o herne frame and an overhead of 2.38 x 10^8 bis/sec due o inerframe gap and preamble combined. For a Gigabi herne he minimum frame size would be 512 byes when operaing in halfduplex mode. The herne proocol efficiency is low for small packes (e.g % for 64 byes) and high (e.g % for 1518 byes) for large packes and hence he hroughpu is low for small packes and high for large packes. Laency can have derimenal affecs lasing few seconds or can be persisen depending on source of delays. Boh bandwidh and laency are wo main eniies o measure nework performance. Since sofware relaed laencies are hard o measure and define, one resors o empirical mehods as done in he nex secion. B. Beowulf PC cluser Specificaions Our esbed consiss of a PC cluser including 20 Lenovo machines wih he following specificaions: Inel Core 2 Duo CPU, GHz, 1.00 GB of RAM. Nework Card: Broadcom Nelink, Gigabi herne, Driver dae 8/28/2006 version The PC are conneced o a Gigabi D-Link herne swich. ach machine has RedHa nerprise AS Linux operaing sysem insalled, and use LAM 7.0.6/MPI 2. C. MPI Benchmarks The Message Passing Inerface (MPI) is a sandard inerface ha is broadly used wih disribued compuing applicaions [12]-[15]. The following MPI-based benchmarks are used o es he communicaion performance of he nodes: a) All-o-all: every node sends a message o every oher node. b) Broadcas: one node sends one message o every oher node. c) Gaher: all nodes send a differen message o a single node. d) Scaer: a single node sends a differen message o every oher node. e) Poin-o-Poin: a single message is sen/received beween 2 specific nodes. The implemenaion deails of he above collecive communicaions are usually unknown o he programmer. 67

4 The MP1-1 sandard specified he blocking collecive communicaions only while he MPI-2 sandard defines non-blocking rouines which perform beer wih some applicaions. MPI relaed sofware performance will depend on he ype of message passing proocols, eager (asynchronous communicaion) or rendezvous (synchronous), ype of message buffering (user and sysem), sender-receiver synchronizaion (e.g. polling or inerrup), and efficiency of he algorihms used o implemen he collecive communicaion rouines. There are many benchmarking sofware available on he inerne such as MPIBench or mppes. However, mos benchmarks available for collecive communicaion basically use he following procedure o measure he execuion ime. 1. All processes arrive a Barrier 2. Sar ime 3. MPI_collecive_fn 4. All processes arrive a Barrier 5. nd ime 6. PTime = nd_ ime Sar_ime where MPI_collecive_fn is one of he MPI funcions all-oall, gaher, scaer, broadcas, and Poin-o-Poin. D. Analysis of xecuion Time (PTime) In he benchmark procedure above, PTime consiss of ime o execue he MPI_collecive_fn funcion and wice he ime of Barrier. Firs, he ime spen a he ransmiing compuer would involve sending daa by he kernel of sysem o he nework inerface card (NIC) and he ime NIC akes o pack byes in an herne frame o send he frame on he physical wire. Depending on ype of NIC archiecure, a ypical herne NIC would have specificaions as given in Table 2. Table 2. Gigabi herne Nework Inerface Specificaions. Speed Inerface Daa Widh Clock Time for 1 bye ransfer 1 Gbps GMII 8 bis 125 MHz 8 ns 10 Gbps XGMII 32 bis MHz.8 ns To ransmi an herne frame he ime o ransmi from Medium Access Conrol (MAC) o Physical (PHY) for a 1 Gbps link would be T = 8 ns * herne_frame_size (10) One hing abou Gigabi herne is is clock rae a 125 MHz bu more daa is ransmied per ime. The ransfer rae is higher since 125 MHz x 2 bis per signal (i.e. per wire pair in Ca 5 cable) x 4 signals per ime = 1 Gbps. Therefore, on he moherboard if PCI xpress is available hen wih a maximum ransfer rae of up o 250 MB/s hen full speed of Gigabi herne is achievable. Nex, he swich receives he herne frame, and processes he frame wih a ypical delay of 2 o 20 µsecs [8]. I is hen sen ou o he desinaion compuer where again he NIC on receiving compuer processes i (PHY aking anywhere from 200 o 300 nanoseconds depending on echnology) and sends i o he MAC layer, hen ono he TCP/IP layers up o he applicaion. The process of ransmiance, recepion, and acknowledgemen is repeaed according o TCP/IP, herne framing, and depending upon he applicaion s insrucions, i.e. MPI collecive_fn funcion and ime of Barrier. The Barrier is used o explicily conrol he flow of execuion. There are a leas 3 ypes of Barrier implemenaions [9], namely, a) Couner implemenaion (linear barrier) b) Tree implemenaion c) Buerfly barrier The ime complexiy of barrier wih couner implemenaion is O (n). For boh he ree and he buerfly implemenaions he ime complexiy is O (log n), where n is he number of nodes. From he above, PTime depends on how MPI collecive_fn and Barrier are implemened in LAM 7.0.6/MPI 2, as summarized in Table 3 (for large number of nodes). Table 3. Time Complexiy Linear Model PTime Barrier Implemenaion a) Couner b) Tree and c) Buerfly MPI_alloall O(nNm) + O(n) O(nNm) + O(log n) MPI_Bcas O(Nm) + O(n) O(Nm) + O(log n) MPI_Gaher O(Nm) + O(n) O(Nm) + O(log n) MPI_Scaer O(Nm) + O(n) O(Nm) + O(log n) IV. RSULTS The MPI benchmarks are run on a PC cluser by firs fixing he number of nodes in a communicaion group o 2 and varying he size of messages from x Kbyes o y Mbyes n ( 2 where n = 0,1,2,3,4,). Then he number of nodes in communicaion group is ieraively incremened up o 20 nodes. The figures show Minimum Round Trip (MRT) ime measured in granulariy of µsecs for messages of sizes ranging from 256 KB o 2 MB. In Figure 1 all-o-all minimum round rip ime is ploed versus number of nodes in a PC cluser. Fig. 1 show MRT is almos consan (wih lile variaion) for a communicaion group consising of anywhere from 2 o 9 nodes in comparison wih 10 o 20 nodes which shows MRT linearly increasing. In Figure 2, broadcas MRT ime is ploed versus number of nodes. From he figure we observe MRT increases wih increase in he number of nodes again wih a seeper slope for large message sizes wihin each group. Noe, from he figure i shows somewha a sep wise incremen in MRT values. 68

5 In Figures 3 and 4, gaher and scaer MRT ime is ploed versus number of nodes, respecively. Boh gaher and scaer have a similar shape for MRT. The shape is more pronounced for large message sizes wihin each group. For gaher and scaer communicaions i is seen from he figures MRT for small number of nodes e.g. 2, 3 and for large number of nodes, e.g akes on values which are much larger han he MRT of he number of nodes in beween. MRT akes on a minimum for 8, 9 nodes and slighly higher for 10 nodes. I is also ineresing o noe ha skewed shape flaens ou as he message size is reduced. The figures of scaer and gaher show unexpeced behavior in MRT for nodes up o 9. The reason for his is presened under he discussion secion. In Figure 5, Poin-o-Poin benchmark is ploed wih logarihmic scale for MRT and messages sizes ranging from 4 byes o 8 Mbyes. Comparing all-o-all communicaion wih he ohers i is seen all-o-all has he highes MRT as expeced. Figure 3. Gaher benchmark showing MRT for message sizes of 256KB, 512KB, 1MB, 2MB and for differen size of PC cluser. Figure 4. Scaer benchmark showing MRT for message sizes of 256KB, 512KB, 1MB, 2MB and for differen size of PC cluser Figure 1. All-o-all benchmark showing MRT for message sizes of 256K B, 512KB, 1MB, 2MBand for differen size of PC cluser. Figure 2. Broadcas benchmark showing MRT for message sizes of 256KB, 512KB, 1MB and for differen size of PC cluser. Figure 5. Poin-o poin benchmark showing MRT log scale versus message sizes of 4 byes o 8 Mbyes. V. DISCUSSION In his secion we discuss he resuls of all-o-all, broadcas, gaher, scaer, and Poin-o-Poin funcions used in he benchmarks. Firs noe, he message size being ransmied is fixed in he case of broadcas and Poin-o- Poin. In he second case of all-o-all, gaher, and scaer, he message size is divided equally wihin he communicaion group and depends on he number of paricipaing nodes. Le m denoe he message size and n denoe he number of nodes, s i = m / n i for i = 2,3,..., 20 (11) where s i is he acual message size being ransmied or received by each communicaing node. Therefore, s 1 > s2 > s3 L > s20 since n i < n i+ 1. As he number of nodes are increased he message size being sen or received goes down. From (5) one knows ha laency depends on message size (and of course is a funcion of ime): as he message size s i+ 1 is less han s i herefore he per message laency of s i is larger han s i+ 1. However his is no always he case as seen in he scaer and gaher rouines. When a hos applicaion ransmis o is desinaion, non-blocking sends are posed by MPI, 69

6 reducing laency for cerain nodes (e.g. 9 in he scaer figure) and he TCP proocol is used for reliabiliy. I mus wai for a period of ime o receive an acknowledgmen. If he reply does no arrive wihin he expeced period he daa is reransmied. On herne LAN he wai ime is no more han a few µsecs. Thus, he overhead ime has a major affec on laency. A number of facors ogeher are involved in having a major affec on MRT as shown in he scaer and gaher plos. Firs, varying he size of he message has affec. Second, he implemenaion of LAM-MPI rouines are no known o he programmer, which model is being used wheher linear or ree ype. Third, which proocol MPI is using eiher eager or rendezvous proocols. The LAM-MPI consrucs he message and sends i hrough he nework o he desinaion compuer which mus accep and ac upon he message conens. LAM-MPI uses eiher he eager proocol or he rendezvous proocol depending on message size. Wih he eager proocol, as soon as a message is posed boh he envelope and daa are sen o he desinaion. If on he desinaion he receive operaion is no posed hen buffering has o be done; his buffering involves an addiional daa duplicaion. In he rendezvous proocol, when a message is posed he envelope is sen o he desinaion and buffered. As soon as a receive is posed he desinaion sends an acknowledgemen o he sender which hen only will be he daa send by he sender In his case buffering of he daa is avoided and used for large messages. Fourh, TCP/IP proocol is being used by LAM-MPI proocols on op of TCP/IP proocols which cause higher laencies in communicaing a message from sender o desinaion. The acual applicaion byes packed in an herne frame are much less due o MPI applicaion, TCP/IP, and herne headers. In some cases, as depiced in he figures of scaer and gaher, when he message size decreases among he communicaion nodes i can happen ha MRT decrease down o a cerain value and, as he message size is furher decreased beyond he minimum MRT poin, he MRT will sar o increase again. The explanaion of such a behavior is due o he facors menioned above (i.e. he implemenaion of he scaer and gaher algorihms in MPI, eager and rendezvous proocol swiching depending on message size, plus TCP overhead and he increase of overhead raio as he herne frame size decreases). VI. CONCLUSION AND FUTUR WORK In his paper we have sudied he performance of MPI collecive communicaions rouines on a gigabi herne LAN. The experimens were performed o represen a Peero-Peer scenario in which one has differen sizes of nodes in a group and variaions in message size. All he benchmark rouines presened closely represen ypical communicaions of a Peer-o-Peer sysem i.e. broadcasing, gahering, scaering, all-o-all, and poin-o-poin. The resuls show in he case of gaher and scaer ha as he message size is decreased among he increasing nodes here is no se predicable paern MRT akes. We have seen ha many facors affec he performance of collecive communicaions in a wireline (herne) environmen. Our nex arge is o exend our sudy o he area of Peer-o-Peer over wireless neworks. ACKNOWLDGMNT The auhors would like o hank CSS Research Cener for carrying ou experimens on he PC cluser. The auhors would also hank he anonymous reviewers for heir valuable and insighful commens. RFRNCS [1] S. Lin, A. Pan, R. Guo, and Z. Zhang., Simulaing Large- Scale Peer-TO-Peer Sysems wih WiDS Toolki, Whie Paper, Microsof, Jan [2] M. Li, W. Lee, and A. Sivasubramaniam, fficien Peer-o- Peer Informaion Sharing over Mobile Ad Hoc Neworks, In MobA, [3] X. M. Huang, C.Y. Chang, and M.S. Chen, PeerCluser: A Cluser Based Peer-o-Peer Syem, I Transacions on Parallel and Disribued Sysems, vol. 17, No. 10, Oc. 2006, pp [4] B. Parviz and K. Miremadi, Building a Peer o Peer Message Passing nvironmen by Uilizing Reflecion in.nt., In Proceedings of PDPTA'2006. pp.1096~1102. [5] InfiniBand Trade Organizaion., hp:// [6] hp://en.wikipedia.org/wiki/quadrics, Sep. 3, [7] Myricom Inc., hp:// Sep 3, [8] CISCO Inc., hp:// Sep 3, [9] F. A. Vaughan, D. A. Grove, and P. D. Coddingon, Communiaion Issues for Two Cluser Compuers, ACSC 03 Proceedings of he 26 h Ausrailasian compuer science conference, vol 16, [10] T. Kielmann and H.. Bal, Fas Measurmen of LogP Parameers for Message Passing Plaforms, 4h Workshop on Runime Sysems for Parallel Programming (RTSPP), pp , held in conjuncion wih IPDPS 2000, Cancun, Mexico, May 1-5, Lecure Noes in Compuer Science, Vol [11] J. P. Grbovic, e al, Performance Analysis of MPI Collecive Operaions, Journal Cluser Compuing, Vol 10, Issue 2, June 2007, pp [12] R. Riesen, Communicaon Paerns, Parallel and Disribued Processing Symposium, April [13] A. Leko, e al, Pracical xperiences wih Modern Parallel Performance Analysis Tools : An valuaion, Parallel and Disribued Processing, IPDPS 2008 I Symposium April 2008, Miami, Fl, pp [14] B. Wilkinson and M. Allen, Parallel Programming Techniques and Applicaions Using Neworked Worksaions and Parallel Compuers, Second diion, Pearson Prenice Hall, [15] F. Noor and S. Misbahuddin, Using MPI on PC Cluser o Compue igenvalues of Hermiian Toepliz Marices, Lecure Noes in Compuer Science, 2010, vol 6081, pp