Pin-down Cache: A Virtual Memory Management Technique for Zero-copy Communication

Transcription

1 Pin-down Cche: A Virtul Memory Mngement Technique for Zero-copy Communiction Hiroshi Tezuk, Frncis 0 Cnoll, Atsushi Hori, nd Yutk Ishikw Rel World Computing Prtnership { tezuk, ocrroll, hori, Abstrct The overhed of copying dt through the centrl processor by messge pssing protocol limits dt trnsfer bndwidth. If the network intelfce directly trnsfers the user s memory to the network by issuing DMA, such dt copies my be eliminted. Since the DMA fcility ccesses the physicl memory ddress spce, user virtul memory must be pinned down to physicl memory loction before the messge is sent or received. If ech messge trnsfer involves pin-down nd relese kernel primitives, messge trnsfer bndwidth will decrese since those primitives re quite expensive. We propose zero copy messge trnsfer with pin-down cche technique which reuses the pinneddown re to decrese the number of clls to pin-down nd relese primitives. The proposed fcility hs been implemented in the PM low-level communiction librry on our RWC PC Cluster II, consisting of 64 Pentium Pro 200 MHz CPUs connected by Myricom Myrinet network, nd running NetBSD. The PM chieves MByteshec for 100 % pin-down cche hit rtio nd 78.7MByteskec for ll pindown cche miss. The MPI librry hs been implemented on top of PM. According to the NAS Prllel benchmrks result, n ppliction is still better performnce in cse tht cche miss rtio is very high. 1. Introduction We hve been designing nd developing high performnce network driver clled PM on Myricom Myrinet network which hs dedicted processor consisting of DMA engine tht cn trnsfer dt between the host memory nd network. In order to eliminte the dt copy between user nd kernel memory spces nd to eliminte issuing operting system primitives, user memory mpped communiction fcility hs been relized[9]. In user memory mpped communiction, the communiction buffer re, hndled by the netw wre, is mpped to the user virtul ddress spce etwork hrdwre is controlled by the user progrm of by the operting system. PM chieves not only high performnce communiction but lso enbles multiuser e shring mnner using gng-scheduling[9,4, re is reserved s specil virtul ddress e physicl re is never pged out, so tht the ne wre cn trigger DMA. This re is clled t llow the user progrm to us user requests cn n progrm will benefit from high performnce uniction if the prog progrm usully does not ccess the buffer directly. For exmple, high-level communiction librry such s MPI my be implemented in wy such tht memory in the user progrm must be copied to the PM Though the PM 1.O low-level communiction li dt copy, the high-level communiction lib Our experiments show tht the MPI dt trn is only 42 MBytes/sec while the PM dt trnsfer bndwidth Myrinet network, nd r incorported with the pinned-down re is designed in order to void memory copies between the user specified memory re nd communiction e. The API provides the pin-down nd relese for the ppliction specified memory re nd zer sge trnsfer opertions. Though the user is requ e pin-down nd relese opertions, pinned-do controlled by the system. A pinned-down eed until the totl re pinned-down exceeds the some mximum size. If the totl size of pinned-down res exceeds the mximum size, some pinned-down res re freed in the LRU fshion. Thus we cll it the pin-down cche. The proposed mechnisms hve /98 $ IEEE 308

2 been implemented in PM 1.2. PM 1.2 is relized by userlevel librry, kernel driver, nd communiction hndler sitting on the Myrinet interfce. No operting system kernel modifictions hve been done to implement the system except fixing some bugs. The orgniztion of this pper is s follows: In the following section, issues of low-level communiction functions will be presented ccording to our experiments in PM 1.0. A zero copy messge trnsfer protocol with the pin-down cche technique is proposed in sections 3 nd 4. A new version of PM, PM 1.2, bsed on the technique is described in section 5. PM 1.2 is evluted in section 6. Section 7 describes relted work nd finlly the pper is concluded in section 8. U 2. Bckground Receiver - Figure 1. PM Dt Trnsfer Flow U Recent network hrdwre such s Myrinet hs dedicted processor with DMA engine. The hrdwre enbles the implementtion of low-level communiction fcility which then implements higher level communiction fcilities such s MPI. In this section, the requirements of low-level communiction fcility re discussed ccording to experiments on PM 1.O on the Myrinet network. PM consists of user-level librry, device driver, nd Myrinet communiction hndler which exists on the Myrinet network crd. The Myrinet communiction hndler controls the Myrinet network hrdwre nd relizes network protocol. PM coopertes with n operting system to support low ltency nd high bndwidth in multi-user environment[9]. Higher-level communiction lyers including MPI[8] nd MPC++ MTTL (Multiple Threds Templte Librry)[6] hve been implemented on top of PM. To chieve high performnce communiction, user process directly ccesses the Myrinet hrdwre nd polls dt without system cll or interrupt. The dt flow between nodes in PM is depicted in Figure 1 nd described s follows: 1) An ppliction progrm stores dt into the messge buffer on the min memory t the sender node. Since the virtul ddress of the messge buffer is mpped to the ppliction progrm, the dt is copied by the processor without issuing kernel primitive. 2) The messge buffer is trnsferred to the Myrinet SRAM re by the Myrinet DMA fcility. 3) The messge is sent to the receiver node by the Myrinet network. 4) The receivedmessge in the Myrinet SRAM re is trnsfered to the messge buffer on the min memory by the Myrnet DMA fcility t the receiver node. 5) Dt in the messge buffer is copied to the ppliction specified re. Since the DMA engine ccesses physicl memory ddress, the messge buffer re s physicl memory must never be pged out. An operting system gurntees such n re in the form of pinned-down re. PM 1.O provides buffer re s the pinned-down re so tht the user progrm does not mke new pinned-down re for communiction buffer. The reson for not llowing the users to pin n re to physicl memory loction is tht the mlicious user s pin-down requests my exhust physicl memory. Providing the messge buffer re hs the following dvntges: i) retrnsmission of messge is esily implemented t the congestion control level, nd ii) synchronous communiction is implemented. The synchronous communiction cpbility t low-level communiction librry is for to implementing higher level communiction librries. For exmple, in n MPI implementtion, synchronous communiction is used not only for the MPI synchronous messge pssing clls but lso for pssing control messges to implement other MPI fetures. The PM 1.0 bndwidth on our RWC PC Cluster II, consisting of 64 Pentium Pro 200 MHz processors which hve 512KB internl cche with the Myrinet LANi 4.1 PCI bord, running NetBSD, is shown in Figure 2. The PM 1.O bndwidth for n 8 KBytes messge is 119 MBytes/sec excluding dt copy to the ppliction specified re while only 50 MBytedsec bndwidth is chieved including dt copy to the ppliction specified re. This evlution shows tht for short messges, the overhed of copying dt to the messge buffer is not so crucil. But in for bulk dt trnsfer, the copy overhed is dominnt. The overhed of CPU memory copy is further investigted in Tble 1. In this evlution, i) the bcopy C librry function is used, ii) dt lrger thn the externl cche is trnsfered to eliminte cche effect, nd iii) severl trils re done by sliding the copy destintion ddress to exmine 309

3 Receiver c, fi (d m 25 0 Figure 3. Zero Copy Messge Trnsfer Dt size (bytes) Figure 2. Dt Trnsfer bndwidth in PM 1.O the cche lignment effect. The result shows tht memory copy bndwidth is hevily mchine dependent. If n Ultr Sprc 170 MHz is the trget mchine, the memory copy overhed is not crucil for bndwidth. But copying dt yields communiction ltency. Thus, copy elimintion t the low-level communiction level is needed to chieve high performnce communiction for ny mchine. n synchronous communiction with zero copy messge trnsfer mechnism is required t the lowlevel communiction level. However, since synchronous communiction requires buffer re, n extr copy between buffer re nd user specified re cnnot be voided. Thus, the new PM version, PM 1.2, supports both synchronous communiction nd zero copy messge trnsfer primitives. The zero copy messge trnsfer primitive is bsed on remote memory write. A higher-level communiction librry my combine these primitives to implement the librry fetures. Tble 1. Dt Copy Bndwidth (MByteds) SS20/71 (75 MHz) Alph PC (433 MHz) Pentium (166MHz) Pentium Pro (200MHz) 3. Zero Copy Messge Trnsfer As shown in Figure 3, zero copy messge trnsfer is relized by the Myrinet DMA cpbility. The ccesses only virtul ddresses while the DM ccesses physicl memory ddresses. To re issued messge trnsfer, both sender nd receiver physicl memory ddresses must be known nd must be pinned to physicl memory. Two designs re considered: 1. Messge Pssing Rendezvous When n ppliction progrm is messge receive primitive, the messge nd stores dt to the destintio ified by the messge receive primitive. plementtion, the sender does not need to know the remote memory ddress where the dt is stored. 2. Remote Memory Write The sender sends messge to the specific destintion ddress on the receiver so tht the re the messge nd stores dt to the desti We choose the ltter design for the following reson: Messge pssing rendezvous is only effective if it is gurnteed tht both the receiver nd the sender re i chronous messge communiction phse. This beh not cceptble for low-level communiction primi cuse we ssume tht synchronous nd synchronous sge pssing primitives my be simu eously issued by high-level communiction librry in ulti-threded environment. In remote memory write, the receiver s receive ddress is lwys explicitly defined by the sender. Messge retrnsmission nd flow control due to the lck of receiver s buffer is voided nd thus messge trnsfer overhed is reduced. On the other hnd, the sender node must hve informtion ed-down re before sending bsed on remote memory wr performed s follows: 1) A memory re, where messge is to be received, is pinned down nd mpping informtion for the re is sent to the sender by the receiver. 3 10

4 2) The sender receives the mpping informtion. 3) The sender s memory re is pinned down. The communiction hndler mintins pge tble entry to trnslte virtul ddress to its physicl ddress. 4) To pss the messge, the sender psses the informtion of the sender s dt ddress nd receiver s mpping informtion to the Myrinet communiction hndler. 5) The Myrinet communiction hndler sends messge, whose heder hs the receiver s mpping informtion, by triggering the DMA function t the sender. 6) When the receiver receives the messge, the Myrinet communiction hndler stores dt ccording to the mpping informtion. 7) The pinned-down re for receiving messge is relesed t the receiver. 8) The pinned-down re for sending messge is relesed t the sender. In the protocol bove, pin-down procedures nd negotitions between sender nd receiver in steps l), 2), 3), 7), nd 8) re not needed if the sme memory res lwys prticipte in messge trnsfers. It seems tht procedures l), 2), nd 3) re only need to be done during progrm initiliztion. However, we do not employ it becuse i) one ppliction occupying huge physicl memory is not cceptble becuse we ssume tht multiple users pplictions re running in time shring fshion on our cluster environment, nd ii) the Myrinet communiction hndler would hve to keep huge tble for the ddress mp in order to llow DMA. If pin-down nd relese kernel primitives were issued for ech dt trnsfer, higher bndwidth would not be chieved becuse the cost of pin-down kernel primitive is reltively high. To overcome the problem, the pin-down cche technique is proposed in the next section. 4. Pin-Down Cche The overhed of the pin-down opertion cn be reduced if n ppliction progrm repetedly trnsfers dt from the sme memory re without relesing the pinned-down re. When request to relese pinned-down memory re is issued, the ctul relese is postponed until totl physicl memory reserved for pinned-down res is exceeded. If the relese primitive for pinned-down re hs been issued but in fct the re remins pinned down, subsequent request to pin down the sme re cn be nswered immeditely, voiding the overhed of pin-down kernel primitive cll. We cll this technique Pin-Down Cche. Pin-down cche works s follows: User Address Spce lock list pge tble free list Offset of P e Tble 1\1 L h Virtul Address Length Offset of Pge Tble \ Kernel Address Spce L Pinned-down Pge Tble Figure 4. Dt Structure At pin-down request, 1. Check whether or not the requested pge hs been pinned down. 2. If the pge hs been pinned down, the pin-down informtion is updted nd the cll returns. 3. If the pge hs not been pinned down, () Check whether or not the totl pinned-down re exceeds the mximum size llowed to the user process, nd check whether or not the pge tble kept by the communiction hndler is full. If either condition is true, some pinned-down re(s) re relesed bsed on some replcement lgorithm such s LRU. (b) The requested re is pinned down. 0 At relese request, 1. Actul pin-down opertion is postponed. The entry of the pin-down tble for the requested pge re is mrked for relese. In this wy the pin-down cche postpones the ctul pindown relese opertion. If the mximum pinned-down re vilble is lrger thn the ggregte size of the ppliction s dt trnsfer res, the pin-down cche lwys hits. Except for the overhed of checking the cche, this becomes exctly s though dt trnsfer re(s) were pinned down stticlly. 311

5 5. Implementtion In PM, the zero copy communiction using pin-down cche is relized by user-level librry, device driver of the operting system kernel, nd Myrinet communiction hndler. Figure 4 shows the dt structures used in the implementtion. The physicl ddress informtion of the pinned-down re is kept by the pinned-down pge tble in the Myrinet interfce memory so tht the user cnnot directly modify the tble. The pinned-down pge tble is ccessed by the Myrinet network hndler. In the user memory spce, there re i) the pinned-down re list, ech entry of which contins the virtul ddress, length nd n offset into the pinned-down pge tble for the pinned-down re, nd ii) the free list of the pinned-down pge tble. In the current implementtion, there re 1,024 pge tble entries for the Myrinet communiction hndler nd the pge size is 4 KBytes. Thus, the user my pin t most 4 MBytes re down t time. Device Driver The device driver provides the bsic mechnism of pin-downhelese opertions for communiction. The mechnism is relized by the Unix U0 control fcility. The VM primitives of NetBSD kernel re used to mnge the pge tble used by the communiction hndler nd pin-downhelese pges. User-Level Librry The user-level librry relizes the pindown cche strtegy nd remote memory write opertions. Opertions -pmmlock nd pmmunlock support pin-downhelese primitives with the pin-down cche strtegy by using the pinned-down re list nd the pin-down pge tble free list. The pin-down cche replcement lgorithm is bsed on LRU. Three remote memory write opertions, pmwr i te, -pmwri tedone, nd -pmsenddone, re provided. Opertion -pmwrite only enqueues the DMA request to the Myrinet interfce. Tht is, when it returns it does not gurntee tht the dt trnsfer is finished. Opertion -pmwr i tedone gurntees tht previously written user dt hs been trnsferred to the memory re in the network interfce. The opertion does not gurntee tht dt hs been trnsfered to the remote mchine. Opertion -pmsenddone gurntees tht the receiver hs received the messge nd stored the dt in the destintion. Myrinet Communiction Hndler The LANi processor on the Myrinet interfce performs the DMA trnsfer between host memory nd Myrinet SRAM re ccording to the pge tble informtion. It lso mnges sendinglseceiving messges tolfrom other nodes. 6. Evlution Pin-down overhe zero-copy messge trnsfer bndwidth, nd MPI th on top of PM re shown. Moreover, results of NAS prllel benchmrk suites using MPI re shown. The evlution ws done us WCPC Cluster II consisting of 64 Pentium Pro 200 cessors running NetBSDli Bsic Performnce Tble 2 shows the overhed of the pin-down cche opertions. In this evlution, pin-downhelese opertions, i.e., -pmmlock nd -pmmunlock, re issued 100,000 times for the sme memory re nd the result time is divided by 100,000. In the evlution of pin-down cche misses, the progrm is modified so tht the pin-down cche lwys misses. significnt penlty Tble 2. Pin-down Overhed 47 p seconds. Of course, the pin-down cost depend processors nd operting in Figure 5. In this n, trnsmitted dt size vries from 4 bytes to 1 M e bndwidth for ech dt size is clculted s the result of 100,000 times trnsfers. In evlution, the receiver pins the receiving memory re o nd sends the informtion to the sender. The pinned-down re is reused during the evlution. The sender clls the pin-downhelese opertions ech time. By modifying the source progrm, the cche hit rtio vries from 0% to 100%. The mximum b.8 MBytes/sec for 1 MBytes for 100 % pin-down cche hit rtio. When pindown cche is lwys mis, the mximum bndwidth is still 78.7 MBytesIsec. CO red to old implementtions, when messge is lrger thn 8 KBytes, the bndwidth of zero copy messge trnsfer is thn the bndwidth of dt trnsfer with dt ing the PM messge buffer. But it is lwys bndwidth of messge trnsfer without dt copy. This penlty is due to the overhed of virtul to physicl ddress trnsltion nd the 3 12

6 pin-down cche opertion, even when the pin-down cche hits 100%. Note tht the one-wy messge cost for 4 bytes messge is 7.5 p seconds'. This performnce for such short messge is relized by the PM synchronous communiction fcility, i.e., dt copy to messge buffer E-- ox m 100 \ 50% r3 v o- 100% * msg+cory I I i 0 le+001e+01 le+o2le+031e+04 1E+051E+06 Dt size (bytes) Figure 5. Bndwidth of zero copy memory trnsfer in PM 1.2 I I i i l i 100 m \ -: 75.E; 50.A 3 E i 25 p1 0 --D- 100% % --o--. msg+co I (MO$) I (Mop/s) 1 up I Dt size (bytes) Figure 6. New MPI Bndwidth Tble 3. NAS Prllel Benchmrk (Clss A) Result 64 PES prog. I Messge I VWrite 1 speed I hit I miss I rtio I size I % I (KB) FT( I I I 847 I 16.0 I MPI Performnce MPI hs been implemented using the MPICH implementtion on top of PM[7, 81. We implemented two versions of MPI: i) MPI implemented using the PM 1.2 synchronous communiction primitives nd ii) MPI implemented using the PM 1.2 zero copy messge trnsfer primitive. Those implementtions re clled old MPI nd new MPI in this pper. According to the MPICH implementtion, if the dt trnsfer size is less thn or equl to 8 KBytes, synchronous communiction primitives re used in both two versions. Figure 6 shows the bndwidth of synchronous messge sendreceive of MPI. In this evlution, the pin-down cche hit rtio vries between 0 % nd 100 %. For comprison, old MPI dt trnsfer bndwidth, msg+copy in the figure, is lso shown in this figure. The synchronous messge trnsfer is better performnce thn the zero copy messge trnsfer for smller thn 8 KBytes messge size in cse of 100 % cche hit. In cse of 0 % cche hit, the zero copy messge trnsfer gives better performnce thn synchronous messge trnsfer for messge sizes lrger thn 20 KBytes. 'This one-wy ltency is clculted s hlf of the round-trip time.. The mximum bndwidth of MPI using the old PM is bout 42.8 MBytes/sec for 128 KBytes while the new MPI chieves 71.6 MBytes/sec(pin-down cche 0% hit) to 99.1 MBytes/sec( 100%) for the sme dt size. Further, the peek bndwidth of new MPI is MB/s for 1 MBytes. The MPI bndwidth using zero copy messge trnsfer is lower thn the PM zero copy messge trnsfer shown in Figure 5. The reson is tht pin-down opertion nd senderheceiver negotition re performed in MPI dt trnsfer. The IS nd FT benchmrks from the NAS prllel benchmrk[ 13 NPB2.3 is used to evlute MPI on PM 1.2. In this evlution, benchmrk progrms written in Fortrn re utomticlly trnslted to C codes. The C codes were compiled by GNU gcc version with -04 option. The 64 processors' result is shown in Tble 3. Column Messge is the totl performnce result (in Mop/s) of the old MPI while column Write is the time for the new MPI. The speed up column shows how much the new MPI is fster thn the old MPI. The pin-down cche sttistics re shown in columns hit, miss, rtio. The men messge size is shown in columns size. Although the cche miss rtio is very high in FT, the performnce is better thn the old MPI implementtion. The reson is tht the trnsfer bndwidth for more thn 16 KBytes messges is better thn the old one s shown in 313

7 Figure 6, nd the men messge length on FT is 32 KBytes on 64 processors. The IS nd FT benchmrk clss A progrms under the new MPI run from 1.18 to 1.20 times fster since the communiction cost is reltively high in those benchmrks on the Pentium Pro 200 MHz. 7. Relted Works According to our design clssifiction of zero copy messge trnsfer techniques described in section 3, VMMC-2[3] nd BIP(Bsic Interfce for Prllelism)[2] of the LHPC re implemeneted bsed on rendezvous. The VMMC-2 introduces the redirection technique if the receiver hs not issued the receive primitive when messge rrives. To pin-down the user virtul ddress spce, the VMMC-2 is lso the sme technique in terms of the pge tble mngement by kernel primitives nd user librry. As long s we understnd, the pin-down cche technique is not employed in the VMMC-2 but some pge tble cche technique for the Myrinet SRAM nd host memory is employed in the VMMC-2. The VMMC-2 implementtion is bsed on the sme hrdwre of our PC cluster. The mximum bndwidth is 84 Mbytesls nd one-wy ltency is 20 p seconds in VMMC- 2 while the PM chieves MBytesIsec for 1 MBytes in cse of 100 % pin-down cche hit rtio nd 78.7 MBytesIsec in cse of ll pin-down cche miss. The 7.5 p seconds one-wy ltency is chived in the PM. The BIP(Bsic Interfce for Prllelism) is implemented on Pentium Pro (200MHz) with Myrinet nd Linux. They chieved 4.3 p seconds ltency (dt size is zero) nd 126 MByteshec bndwidth for 8 Mbytes dt size. As long s we know, the detiled design hs not been published. 8. Conclusion We hve proposed the technique of zero copy messge trnsfer with pin-down cche to eliminte dt copy between user specific dt re nd messge buffer. The pin-down cche is technique to reuse the pinned-down re to decrese the number of pin-down nd relese opertions since the pin-downirelese kernel primitives re very high cost. The proposed technique hs been implemented in PM 1.2. In PM 1.2, the mximum bndwidth is 109 MBytesIsec for 1 MBytes in cse of 100 % pin-down cche hit rtio, In cse tht pin-down cche is lwys miss, the mximum bndwidth is still 78.7 MBytesisec which is higher thn the bndwidth of dt trnsfer with dt copy, i.e., using the PM messge buffer. PM lso chieves low ltency, 7.5 p seconds one-wy ltency. The results of the IS nd FT benchmrk from the NAS prllel benchmrk 2.3 hve been shown to demonstrte the effectiveness of pin-down cche. Although the cche miss rtio is very high in FT, the performnce is better thn the old MPI implementtion. A progrms under the new fster since the communi those benchmrks on the Pentium Pro 200 MHz. We re currently extending the RWC PC Cluster 11 to 128 Pentium Pro 200 MHz processors nd softwre DSM to investigte the impct of the zero copy messge trnsfer with the pin-down cche technique. Reders interested in our clusters my visit the following URL: We re currently distributing PM 1.2 nd MPC++ MTTL on PM. The other softwres, MPI on PM nd Score-D[S], will soon be distributed. References [ 11 [2] [3] C. Dubnicki, A. Bils, Y. Chen, S. Dminkis, nd K. Li. VMMC-2: Efficient Support for Relible, Connection- Oriented Communiction. In HOT Interconnects V, pges 3746,1997. [4] A. Hori, H. Tezuk, nd Y. Ishikw. Globl Stte Detection using Network Preemption. In IPPS 97 Workshop on Job Scheduling Strtegies for Prllel Processing, April [SI A. Hori, H. Tezuk, Y. Ishikw, N. Sod, H. Konk, nd M. Med. Implementtion of Gng-Scheduling on Worksttion Cluster. In D. G. Feitelson nd L. Rudolph, editors, IPPS 96 Workshop on Job Scheduling Strtegies for Prllel Processing, volume 1162 of Lecture Notes in Computer Science, pges [6] Y. Ishikw. Multi T sion 2.0 Level 0 Document -. Technicl Report TR-96012, RWC, September This technil report is obtined vi rwcp. oejp/lb/mpsl b/mpc+ +/mpc+ i. html. [7] E B. O Crroll, A. Hori, Y. Ishikw, nd S. Mtsuok. Implementing MPI in High-Performnce, Multithreded Lnguge MPC++. In SWoPP 96, pges , suok. Performnce Myrinet. In Cluster ting Conference 19 [9] H. Tezuk, A. Hori, Y. Ishikw, nd M. Sto. 97,