Lecture 4. Programming with Message Passing: Applications and Performance

Transcription

1 Lecure 4 Programming wih Message Passing: Applicaions and Performance

2 Announcemens Quiz #1 in secion on 10/13 Miderm: evening of 10/30, 7:00 o 8:20 PM Wednesday s office hours sar a 1.30 (res of he quarer) 10/3/06 Sco B. Baden / CSE 160 / Fall

3 Today s lecure Inerconnec Our firs MPI applicaions Numerical quadraure Measuring communicaion performance N-body simulaion Reporing Performance Under he hood of MPI 10/3/06 Sco B. Baden / CSE 160 / Fall

4 10/3/06 Sco B. Baden / CSE 160 / Fall Inerconnec Generic inerconnec hides he opology Ring Toroidal mesh e x e x e x e x e x e x e x e x

5 Performance characerisics Diameer: maximum disance beween any 2 poins in he nework Bisecion bandwidh: collecive bandwidh beween wo halves of he nework; spli he graph ino wo equal pars and measure he capaciy of he cu edges 10/3/06 Sco B. Baden / CSE 160 / Fall

6 10/3/06 Sco B. Baden / CSE 160 / Fall Ring P elemen array wih end-around connecion Diameer is cu in half by he end around connecion Bisecion bandwidh? Broadcas running ime? e x e x e x e x e x e x e x e x

7 Mesh P P array if square Diameer? Bisecion bandwidh? Broadcas algorihm? 10/3/06 Sco B. Baden / CSE 160 / Fall

8 Toroidal mesh End around connecions on rows and columns Diameer? Bisecion bandwidh? Broadcas running ime? 10/3/06 Sco B. Baden / CSE 160 / Fall

9 Crossbar Expensive all poins conneced Diameer? Bisecion bandwidh? Broadcas algorihm? 10/3/06 Sco B. Baden / CSE 160 / Fall

10 Muli-sage neworks Swiching is performed in sages Someimes he sages are inducively consruced Ofen redundan pahs 10/3/06 Sco B. Baden / CSE 160 / Fall

11 An Omega Nework The nework is consruced from swich modules A module can swap he inpus, or pass hem hrough unchanged 10/3/06 Sco B. Baden / CSE 160 / Fall

12 Swich conenion If wo messages require he same modules a he same ime, hey conend for ha module Performance penaly, since access is serialized 10/3/06 Sco B. Baden / CSE 160 / Fall

13 Today s lecure Inerconnec Our firs MPI applicaions Numerical quadraure Measuring communicaion performance Under he hood of MPI 10/3/06 Sco B. Baden / CSE 160 / Fall

14 Numerical Quadraure Compue a numerical approximaion o he definie inegral b a f(x) dx using he rapezoidal rule 10/3/06 Sco B. Baden / CSE 160 / Fall

15 How he rapezoidal rule works Divide he inerval [a,b] ino n segmens of size h=1/n Approximae he area under an inerval using a rapezoid Area under he i h rapezoid (f(a+i h)+f(a+(i+1) h)) h Area under he enire curve sum of all he rapezoids h a+i*h a+(i+1)*h 10/3/06 Sco B. Baden / CSE 160 / Fall

16 Reference maerial For a discussion of he rapezoidal rule hp://meric.ma.ic.ac.uk/inegraion/echniques/definie/numericalmehods/rapezoidal-rule A apple o carry ou inegraion hp:// Code (from Pacheco hard copy ex) PUB = /expor/home/cs160x-public Serial Code PUB/Pacheco/ppmpi_c/chap04/serial.c Parallel Code PUB/Pacheco/ppmpi_c/chap04/rap.c 10/3/06 Sco B. Baden / CSE 160 / Fall

17 Serial code (Following Pacheco) main() { floa f(floa x) { reurn x*x; } // Funcion we're inegraing floa h = (b-a)/n; floa inegral = (f(a) + f(b))/2.0; // h= rapezoid base widh // a and b: endpoins // n = # of rapezoids floa x; in i; for (i = 1, x=a; i <= n-1; i++) { x += h; inegral = inegral + f(x); } inegral = inegral*h; } 10/3/06 Sco B. Baden / CSE 160 / Fall

18 The parallel algorihm Decompose he inegraion inerval ino subinervals, one per processor Each processor compues he inegral on is local subdomain Processors combine heir local inegrals ino a global one 10/3/06 Sco B. Baden / CSE 160 / Fall

19 Firs version of he parallel code local_n = n/p; // Number of rapezoids; assume p divides n evenly floa local_a = a + my_rank*local_n*h, local_b = local_a + local_n*h, inegral = Trap(local_a, local_b, local_n, h); if (my_rank == 0) { // Sum he inegrals calculaed by all he processes oal = inegral; for (source = 1; source < p; source++) { MPI_Recv(&inegral, 1, MPI_FLOAT, source, ag, WORLD, &saus); oal += inegral; } } else MPI_Send(&inegral, 1, MPI_FLOAT, des, ag, WORLD); 10/3/06 Sco B. Baden / CSE 160 / Fall

20 Can we improve he running ime? The resul does no depend on he order in which he sums are aken We use a linear ime algorihm o accumulae conribuions, bu here are oher orderings } for (source = 1; source < p; source++) { MPI_Recv(&inegral, 1, MPI_FLOAT, MPI_ANY_SOURCE, ag, WORLD, &saus); oal += inegral; 10/3/06 Sco B. Baden / CSE 160 / Fall

21 Collecive communicaion We can ofen improve performance by aking advanage of global knowledge abou communicaion Insead of using poin o poin communicaion operaions o accumulae he sum, use collecive communicaion 10/3/06 Sco B. Baden / CSE 160 / Fall

22 Collecive communicaion in MPI Collecive operaions are called by all processes in a communicaor Broadcas: disribue daa from a designaed roo process o all ohers in a communicaor MPI_Bcas(in, coun, ype, roo, comm) Reduce: combine daa from all processes in communicaor and reurns i o one process MPI_Reduce(in, ou, coun, ype, op, roo, comm) 10/3/06 Sco B. Baden / CSE 160 / Fall

23 Broadcas One process ransmis of m pieces of daa o all he (p-1) ohers Linear algorihm performs p-1 sends of lengh m Cos is (p-1)( + m) Anoher approach is o use he hypercube algorihm, which has a logarihmic running ime 10/3/06 Sco B. Baden / CSE 160 / Fall

24 Wha is a hypercube? A hypercube is a d-dimensional graph wih 2 d nodes A 0-cube is a single node, 1-cube is a line connecing wo poins, 2-cube is a square, ec Each node has d neighbors 10/3/06 Sco B. Baden / CSE 160 / Fall

25 Properies of hypercubes A hypercube wih p nodes has lg(p) dimensions Inducive consrucion: we may consruc a d-cube from wo (d-1) dimensional cubes Diameer: Wha is he maximum disance beween any 2 nodes? Bisecion bandwidh: How many cu edges (mincu) 10/3/06 Sco B. Baden / CSE 160 / Fall

26 Bookkeeping Label nodes wih a binary refleced grey code hp:// Neighboring labels differ in exacly one bi posiion 001 = 101 e 2, e 2 = 100 e 2 =100 10/3/06 Sco B. Baden / CSE 160 / Fall

27 Hypercube broadcas algorihm wih p=4 Processor 0 is he roo, sends is daa o is hypercube buddy on processor 2 (10) Proc 0 & 2 send daa o respecive buddies /3/06 Sco B. Baden / CSE 160 / Fall

28 Reducion We may use he hypercube algorihm o perform reducions as well as broadcass Use varian of reducion Allreduce( ) Everyone obains a copy of he reduced resul This is equivalen o a Reduce( ) + Bcas( ) A clever algorihm performs an Allreduce in one phase raher han having perform separae reduce and broadcas phases 10/3/06 Sco B. Baden / CSE 160 / Fall

29 in local_n = n/p; Improved parallel code floa local_a = a + my_rank*local_n*h, local_b = local_a + local_n*h, inegral = Trap(local_a, local_b, local_n, h); MPI_Allreduce( &inegral, &oal, coun=1, MPI_FLOAT, MPI_SUM, WORLD) 10/3/06 Sco B. Baden / CSE 160 / Fall

30 How do we model communicaion performance? The simples communicaion cos model is Alpha-Bea Model: ransfer ime = + n = message sarup ime = 1/ peak bandwidh (byes per second) n = message lengh send Laency recv Sender Nework inerfaces Recvr 10/3/06 Sco B. Baden / CSE 160 / Fall

31 Wha are sarup and bandwidh? The sarup erm dominaes when he message is sufficienly shor > n n < / The bandwidh erm dominaes when he message is sufficienly long n > / We refer o his message hreshold as he half power poin n 1/2 10/3/06 Sco B. Baden / CSE 160 / Fall

32 Half power poin Gives a relaionship beween sarup and bandwidh n 1/2 = message size required o achieve peak bandwidh (1/ ) In heory, his occurs when = n 1/2 Bu look closely! For NPACI Blue Horizon, he acual value of n 1/2 100 KB 10/3/06 Sco B. Baden / CSE 160 / Fall

33 Evaluaing communicaion performance wih he Ring microbenchmark Trea he p processors as if conneced in a logical ring and pass messages around Neighbors of processor k are ( k + 1 ) mod p ( k + p -1 ) mod p p0 P 0 imes he cos of sending a message around he ring p3 p1 Wha are we measuring? Wha are we ignoring? p2 10/3/06 Sco B. Baden / CSE 160 / Fall

34 Communicaion Bandwidh on Blue Horizon 390 MB/sec N = 4MB N 1/2 100KB 10/3/06 Sco B. Baden / CSE 160 / Fall

35 Performance measuremen Use MPI_Wime( ) o measure wall clock ime Ignore ransien behavior Measure sufficienly long periods of represenaive seady sae behavior Warm up he program by running i firs wihou collecing iming informaion Repea he measuremens several imes, and repor he shores imes Noe any ouliers 10/3/06 Sco B. Baden / CSE 160 / Fall

36 Measuremen echnique wih Ring for (in len = 1, l=0; len <= maxsize; len *= 2, l++) if (myid == 0) { // (WARM UP CODE) cons double sar = MPI_Wime( ); for (in i = 0; i < rips; i++) { PROCESSOR 0 CODE } cons double dela = MPI_Wime( ) - sar; Bandwidh = (long)((rips*len*nodes)/ dela /1000.0); } else { // myid!= 0 // (WARM UP CODE) for (in i = 0; i < rips; i++) { ALL OTHER PROCESSORS } } 10/3/06 Sco B. Baden / CSE 160 / Fall

37 The Ring program Processor 0: MPI_Reques req; MPI_Irecv(buffer, len, MPI_CHAR, (rank + p - 1)%p, ag, MPI_COMM_WORLD, &req); MPI_Send(buffer, len, MPI_CHAR, (rank + 1) % p, ag, MPI_COMM_WORLD); MPI_Saus saus; MPI_Wai(&req,&saus); All ohers: MPI_Saus saus1; MPI_Recv(buffer, len, MPI_CHAR, (rank + p - 1)%p, ag, MPI_COMM_WORLD, &saus1); MPI_Send(buffer, len, MPI_CHAR, (rank+1)%p, ag, MPI_COMM_WORLD); 10/3/06 Sco B. Baden / CSE 160 / Fall

38 Reporing and Displaying Performance Give he viewer sufficien informaion o Draw heir own conclusions Reproduce your resuls Tabulae and display he resuls fairly Avoid misleading echniques See he Bailey paper for examples of how no o display and repor performance daa hp://crd.lbl.gov/~dhbailey/dhbpapers/mislead.pdf 10/3/06 Sco B. Baden / CSE 160 / Fall

39 Challenges o measuring performance Reproducibiliy Transien sysem operaing condiions Differing sysems or program configuraion Measuremens are imprecise Heisenberg uncerainy principle: measuremen echnique may affec performance Variaions in performance are ineviable; OK if we can explain and olerae hem Overheads and inaccuracy Explain anomalous behavior, bu ignore anomalies ha are no significan 10/3/06 Sco B. Baden / CSE 160 / Fall

40 Complicaions Cos of measuring a full run is prohibiive Ignore sarup code if you plan o run for a much longer ime in producion Transien behavior Repea your measuremens Warm up he coded before collecing measuremens Ignore ouliers unless heir behavior is imporan o you Average ime, maximum ime, minimum ime? 10/3/06 Sco B. Baden / CSE 160 / Fall

41 Measuremen collecion Repor he bes imings Repea resuls 3 o 5 imes unil a leas 2 measures agree o wihin 5%, 10% Repor he minimum ime Also repor ouliers A scaer plo or error bar can be useful Compue Communicae /3/06 Sco B. Baden / CSE 160 / Fall

42 Measures of ime Timing collecion Elapsed, or wall clock ime CPU ime = sysem + user ime Overhead, resoluion, and quanizaion effecs Measuremen ools Unix ime command does a reasonable job for long-running programs Hardware performance moniors Sysem clocks Ofen plaform dependen, especially library rouines MPI provides MPI_Wime( ), elapsed or wall clock ime 10/3/06 Sco B. Baden / CSE 160 / Fall

43 Qualifying measuremens Specify version and opions Compiler Operaing sysem Numerical libraries Esablish appropriae operaing condiions Program inpus Sysem environmen variables Dedicaed sysem access uname -a Linux valkyrie.ucsd.edu ELsmp #1 SMP Wed Apr 20 00:16:40 BST 2005 i686 i686 i386 GNU/Linux g++ -v Reading specs from /usr/lib/gcc/i386-redhalinux/3.4.5/specs Configured wih:. hos=i386-redha-linux Thread model: posix gcc version (Red Ha ) 10/3/06 Sco B. Baden / CSE 160 / Fall

44 Under he hood of MPI If here is no a pending receive, hen an incoming message is placed in an anonymous sysem buffer When he receive ges posed, he message is moved ino he user specified buffer Ring uses non-blocking communicaion o avoid exra copy For more informaion see Buffering and Safey in MPI: The Complee Reference, by Marc Snir e al. Recv( ) 10/3/06 Sco B. Baden / CSE 160 / Fall

45 Rendezvous proocol When a long message is o be sen, MPI firs checks if he recipien has sufficien sorage o receive he message If so, hen i sends he message? 10/3/06 Sco B. Baden / CSE 160 / Fall

46 Eager limis In an eager implemenaion, we jus send he message In pracice, MPI implemenaions swich beween he wo modes The eager limi is he longes message ha can be sen in eager mode 10/3/06 Sco B. Baden / CSE 160 / Fall