COSC 4397 Parallel Computation. Performance Modeling. Edgar Gabriel. Spring Motivation

Size: px

Start display at page:

Download "COSC 4397 Parallel Computation. Performance Modeling. Edgar Gabriel. Spring Motivation"

Anna Wood
5 years ago
Views:

1 COSC 4397 Perfrmance Mdeling Spring 2010 Mtivatin Can we estimate the csts fr a parallel cde in rder t Evaluate quantitative and qualitative differences between different implementatin alternatives Understand the parameters effecting the perfrmance f the applicatin Understanding relevant hardware characteristics Restrictins: Any analytical mdel can nt replace real measurements since parallel systems are t cmplex and unpredictable 1

2 Hw t mdel cllective peratins? Eg MPI_Bcast: strngly depending n the algrithm used t implement the peratin One prcess (rt prcess) distributes the same data items t all members within a prcess grup (cmmunicatr) inear Algrithm: the rt prcess sends ne message t each prcess in the cmmunicatr if (rank == rt ) { fr (i=0; i<size; i++ ) if ( i!= rt ) MPI_Send (buf, cnt, dat, i, TAG, cmm); else MPI_Recv (buf, cnt, dat, rt, TAG, cmm, &stat); inear Algrithm (I) 1 Hckney s Mdel: t( s) = l+ s b s: message size l: latency b: bandwidth Estimate f the executin time accrding t Hckney s mdel fr p prcesses: 1 ( s, p) = ( l+ s)*( p 1) b t (4:1) 2

3 inear Algrithm (II) Using nn-blcking peratins: if (rank == rt ) { fr (i=0; i<size; i++ ) MPI_Isend (buf, cnt, dat, i, TAG, cmm, &req[i]); MPI_Recv (buf, cnt, dat, rt, TAG, cmm, &stat); if (rank == rt ) { MPI_Waitall ( size, req, statuses); Frmula (4:1) is nw arbitrarily wrng Several cmmunicatins simultaneusly nging Maximum (ptimal) number f messages depending n message size and netwrk parameters Hw des cmmunicatin really wrk (I) Tw prtcls usually used internally: Eager prtcl: message is sent immediately t the receiver, withut waiting fr the accrding receive t be psted Usually used fr shrt messages (eg 1 KB in Open MPI) Rendezvus prtcl: Send a header t receiver Wait fr an acknwledgment receive has started Send message data Avids having t buffer large messages n the receiver prcess ( unexpected messages) 3

4 Hw cmmunicatin really wrks (II) Three levels f buffering Applicatin level (eg MPI_Bsend) MPI library level unexpected message queues System buffering System buffering wrks similarly t file systems eg fr sckets: data is cpied int scket buffer befre sending MPI_Send returns as sn as data is in the scket buffer! N way t alternate this data anymre, s it is safe t return cntrl t the applicatin Hw cmmunicatin really wrks (III) Fr a shrt message ( < scket buffer size (=sbsize) ) Data cpied int scket buffer write peratin n the accrding scket called MPI_Send returns cntrl t the applicatin in a time which is shrter than the netwrk latency! Fr a lng message arge message is split int chunks f size sbsize A chunk f the data is cpied int scket buffer and sent As sn as the receiving prcess acknwledges the receipt f the data chunk, the next chunk is cpied int scket buffer etc 4

5 Hw cmmunicatin really wrks (IV) S transfer f a large message lks like Sending a small chunk Wait Sending a small chunk Wait This behavir is nt mdeled by Hckney, but eg by the ggp mdel Based n ggp, ne shuld split a large message int smaller chunks and send them simultaneusly fr a bcast peratin Hide the gap by using a different channel Multi-segmented linear algrithm nmgs = cnt/scnt; if (rank == rt ) { fr (j=0; j<nmsgs; j++ ) { tbuf = buf + (j*scnt ); fr (i=0; i<size; i++ ) MPI_Isend (tbuf, scnt, dat, i, TAG, cmm,&req[2*j+i]); fr (j=0; j<nmsgs; j++ ) { tbuf = buf + (j*scnt); MPI_Irecv (tbuf, scnt, dat, rt, TAG, cmm, &rreq[j]); if (rank == rt ) { MPI_Waitall ( size*nmsgs, req, statuses); MPI_Waitall ( nmsgs, rreq, rstatuses ); 5

6 Binary and Binmial Trees Number f messages increase with every iteratin netwrk saturated starting frm a certain number f messages message segmenting can imprve the perfrmance as well Chain Algrithms Segment a message and pass them frm ne prcess t anther Perfrms very well fr very large messages n n n n n 6

7 eg k=5 k-chain Algrithm Hckney s Mdel t(s) = l + s/b (14:1) l: latency f the netwrk b: bandwidth f the netwrk Hw can we determine the latency and the bandwidth? Ping-png benchmark: prcess A sends a message t prcess B, prcess B sends message back Advantage: des nt require synchrnized clcks between A and B Disadvantage: assumes symmetric cmmunicatin perfrmance ( csts (A->B) == csts (B->A) T determine latency: execute ping-png benchmark fr cnt=0 7

8 15 Ping png benchmark cmm = MPI_COMM_WORD; fr (i=1; i< MAX_MSG_EN; i*=2 ) { t1 = MPI_Wtime(); fr ( j=0; j<max_measurements; j++ ) { if ( rank == 0 ) { MPI_Send (buf, i, MPI_INT, 1, 1, cmm); MPI_Recv (buf, i, MPI_INT, 1, 1, cmm, &status); else if ( rank == 1 ) { MPI_Recv (buf, i, MPI_INT, 0, 1, cmm, &status); MPI_Send (buf, i, MPI_INT, 0, 1, cmm ); t2 = MPI_Wtime(); if ( rank == 0 ) { printf( Msg len: %d avg exec%lf bandw %d \n, i, (t2-t1)/(2*max_measurements), i*sizef(int)/((t2-t1)/(2*max_measurements)); Ping-png benchmark (II) 8

9 Ping-png benchmark (II) T determine bandwidth: have t determine the saturatin pint Required message length des depend n the netwrk bandwidth 17 gp Mdel published by Culler et all Parameters: : upper bund n the latency : verhead, defined as the length f the time that a prcess is engaged in the transmissin r receptin f a message During this time, the prcess can nt perfrm ther peratins g: gap, defined as the minimum time interval between cnsecutive message transmissins r receptins The reciprcal time f g crrespnds t the per-prcess cmmunicatin bandwidth P: number f prcessrs 9

10 Sender Start sending gp (II) Message enters netwrk Receiver Message leaves netwrk End receiving Csts fr sending a messages: t + 2 = (19:1) Start sending g Sender g gp (II) Receiver End receiving Csts fr sending tw messages: t= + g+ 2 (20:1) 10

11 gp(iii) Please nte: atency in the gp mdel is different than the latency in the Hckney mdel atency f Hckney includes the verhead In the frmula (20:1), we assumed that < g which is typically crrect The frmulas shuld hwever be instead t +max( g, ) + 2 = (21:1) gp(iii) gp assumes, that any large message can be decmpsed t a series f shrt messages eg sending a message f k bytes takes k/ w t = + 1)*max( g, ) + + ( (22:1) with w being the size f the netwrk package in bytes fr which gp still hlds gp assumes, that the verhead is equal fr the sender and the receiver side Mre fine grained appraches use different values, eg s and r 11

12 ggp Extensin f gp taking int accunt, that large message can ften be transferred mre efficiently than what gp predicts, due t special hardware supprt Additinal parameter: G: Gap per bytes fr lng messages Sending a k byte message with ggp: cycles until the first byte enters the netwrk G cycles fr each subsequent byte cycles n the receiver side t = + k 1) G+ + ( (23:1) ggp Sender GGG g GGG Receiver End receiving Csts fr sending tw k-byte messages: t = + ( k 1) G+ g+ ( k 1) G+ + = + 2( k 1) G+ g+ 2 (24:1) 12

13 PgP Extensin f the PgP mdel making the parameters g, s and r dependent n the message length m g(m), s (m) and r (m) atency is cnsidered t be an end-t-end latency gp/ggp PgP +g(1)- s (1)- r (1) ( s (1)+ r (1))/2 g G P g(1) g(m)/m, fr sufficiently large m P PgP(II) Sender g(m) s (m) ggp Receiver PgP r (m) g(m) Receiver spends +g(m) cycles in a recv peratin 13

14 PgP(III) Hw can we determine the parameters f gp, ggp and PgP Since we can determine the parameters f gp/ggp using the PgP mdel, we will nly fcus n PgP Idea: execute a series f measurements, whse perfrmance yu can mdel using PgP, and which lead t a set f linearly independent equatins Determine the parameters frm the equatins PgP(IV) Test 1: Send n very small messages (m=0) and wait fr a single acknwledgement Measure the Time t send n messages f length 0: n*g(0) (7) (8) RundTripTime (RTT n ) = 2(+g(0)) Test 2: Send a message f length m and wait fr an ack f length 0 Measure the Time t send a message f length m : s (m) (9) RTT(m) = +g(m)++g(0) (10) Test 3: send a message f length 0, wait fr >RTT(m) and receive than a message f length m Since >RTT(m) we knw that the message is available, and thus we really measure r (m) (11) 14

15 PgP(V) Test 1 n*g(0) 2(+g(0)) Sender g(0) Receiver g(0) Please nte, that since g is a netwrk parameter (nt sftware) n has t be sufficiently large t saturate the netwrk PgP(VI) Test 2 +g(m)++g(0) Sender s (m) g(0) Receiver g(m) 15

16 PgP(VII) Test 3 Sender s (0) Δ>RTT(m) r (m) Receiver g(0) Example: linear bradcast g 0 GGG GGG

17 Example: linear bradcast Executin time accrding t gp: First message takes cycles t push int the netwrk All subsequent messages take g cycles The last message takes + cycles t be received t(p) = +(P-2)g++ Executin time accrding t ggp: First message takes +(k-1)g cycles Subsequent messages take g+(k-1)g cycles ast message takes + cycles t be received t(k,p) = +(P-2)g+(P-1)(k-1)G++ Example: nn-segmented chain bradcast 0 GGG 1 GGG 2 17

18 Example: nn-segmented chain bradcast Executin time accrding t gp: Rt prcess takes cycles t push the message int the netwrk A prcess takes + cycles t receive the message and cycles t push the message int the netwrk ast prcess takes + cycles t receive the message t(p)=+(p-2)(+2)++ = (P-1)( + 2) Similarly fr ggp: t(k,p)=+(k-1)g+(p-2)(+2+(k-1)g)++ = (P-1)(+2+(k-1)G) 18

COSC 6374 Parallel Computation. Performance Modeling (II) Edgar Gabriel. Spring Hockney s Model. t(s) = l + s/b

COSC 6374 Parallel Computation. Performance Modeling (II) Edgar Gabriel. Spring Hockney s Model. t(s) = l + s/b COSC 6374 Parallel Cmputatin Perfrmance Mdeling (II) Spring 2007 Hckney s Mdel t(s) = l + s/b (1) l: latency f the netwrk b: bandwidth f the netwrk Hw can we determine the latency and the bandwidth? Ping-png