Cluster Computing Spring 2004 Paul A. Farrell

Transcription

1 Cluster Computig Sprig 004 3/18/004 Parallel Programmig Overview Task Parallelism OS support for task parallelism Parameter Studies Domai Decompositio Sequece Matchig Work Assigmet Static schedulig Divide work ito pieces which will take equal time where is umber of workers Dyamic schedulig If tasks are of widely differet sizes (times) there is a load balacig problem Assig subset of sub-tasks to slaves Whe slave fiished assig aother sub-task Observatios: Still load balacig problem at ed Miimize by makig sub-tasks small If sub-tasks too small commuicatio overhead will impact performace adversely Cluster Computig 1 Cluster Computig 3 Master/Slave paradigm Divide task ito early parallel sub-tasks Start the master Start the slaves Master commuicates sub-task specificatios to slaves Slaves perform sub-tasks Slaves commuicate results to master Master esures that all results have bee collected Shut dow slaves Shut dow master Uix OS Cocepts for Parallel Programmig Uix Process (task) Executable code Istructio poiter Stack Logical registers Heap Private address space Task forkig to create depedet processes thousads of clock cycles Thread lightweight process Logical registers Stack Shared address space Hudreds of clock cycles to create/destroy/sychroize threads Cluster Computig Cluster Computig 4 Ket State Uiversity 1

2 Cluster Computig Sprig 004 3/18/004 Local Process executio All processes childre of iit Processes spawed usig fork-exec combiatio Fork creates a copy of the process Differs from paret oly i retured value from fork 0 i child, pidof child i paret Exec substitute aother program executable for the curret program image If ot familiar with this read 7.. i book for details Iterprocess Commuicatio with Sockets See sectio 7..5 i book Also Cluster Computig 5 Cluster Computig 7 Remote Process Executio/File Access Rsh Ssh Note that sshslower due to ecriptio Ssh ca do X forwardig usually sytax to tur this off (-x) Ca be problems with NFS mouted file system due to all odes tryig to write.xauthorityfile NFS Rcp Scp ftp/sftp Rdist maitai idetical copies of files across hosts Rsyc detect differeces betwee files o differet hosts ad oly trasfer diffs Parameter Studies Ru same sequetial program multiple times with differet iput data (parameters) Trivially parallel Commo where oe wats to see which set of parameters give the best approximatio to kow (experimetal or theoretical results) Also where oe wats to documet the effects of parameters o results See example i book (sectio 7.3) o testig compiler optimizatio flags Cluster Computig 6 Cluster Computig 8 Ket State Uiversity

3 Cluster Computig Sprig 004 3/18/004 Parallel Search Sequece matchig i Computatioal Biology databases of ucleotide (RNA or DNA) or amio acid sequeces Ecoded as strigs of characters Iformatio derived by matchig give strig exactly or approximately agaist large database Example program for matchig: BLAST Uses data i FASTAformat A program formatdb will build a idexed database from them Oe ca the use BLAST to search for similarities Get list of matches ad similarities Geeric Parallel Programmig Models Sigle Program Multiple Data Stream (SPMD) Each CPU accesses same object code Same applicatio ru o differet data Data exchage may be hadled explicitly/implicitly Natural model for SIMD machies Most commoly used geeric parallel programmig model Message-passig Shared-memory Usually uses process/task ID to differetiate Focus of remaider of this sectio Multiple Program Multiple Data Stream (MPMD) Each CPU accesses differet object code Each CPU has oly data/istructios eeded Natural model for MIMD machies Cluster Computig 9 Cluster Computig 11 BLAST i Parallel Database ca be large 1.4 millio sequeces Use parallelism to compare all agaist all Use master/slave paradigm Distribute etire db to each slave Slaves ru BLAST with iput file of subset of db chuck Chuk set to slave over socket Slaves are persistet Output of slaves copies to a output directory usig scp Master listes o a socket for added slaves so they ca come ad go If slaves die they ca be replaced with miimal impact Master keeps track of chuk status ad checkpoits so restart is possible. Parallel Architectures Mappig Hardware Models to Programmig Models Cotrol Mechaism Memory Model Programmig Model SIMD shared -memory SPMD MIMD Hybrid (SMP cluster) distributed-memory MPMD Cluster Computig 10 Cluster Computig 1 Ket State Uiversity 3

4 Cluster Computig Sprig 004 3/18/004 Methods of Problem Decompositio for Parallel Programmig Wat to map ( Problem + Algorithms + Data) to Architecture Coceptualize mappig via e.g., pseudocode Realize mappig via programmig laguage Data Decompositio - data parallel program Each processor performs the same task o differet data Example - grid problems Task (Fuctioal ) Decompositio - task parallel program Each processor performs a differet task Example - sigal processig addig/subtractig frequecies from spectrum Other Decompositio methods Desigig ad Buildig Parallel Applicatios Attributes of Parallel Algorithms Cocurrecy - May actios performed simultaeously Modularity - Decompositio of complex etities ito simpler compoets Locality - Wat high ratio of of local memory access to remote memory access Usually wat to miimize commuicatio/computatio ratio Performace Measures of algorithmic efficiecy Executio time Complexity usually ~ Executio Time Scalability Cluster Computig 13 Cluster Computig 15 Categories of Parallel Problems Geeric Parallel Problem Architectures ( after G Fox) Ideally Parallel (Embarrassigly Parallel, Job-Level Parallel ) Same applicatio ru o differet data Could be ru o separate machies Example: Parameter Studies Almost Ideally Parallel Similar to Ideal case, but with miimum coordiatio required Example: Liear Mote Carlo calculatios, itegrals Pipelie Parallelism Problem divided ito tasks that have to be completed sequetially Ca be trasformed ito partially sequetial tasks Example: DSP filterig Sychroous Parallelism Each operatio performed o all/most of data Operatios deped o results of prior operatios All processes must be sychroized at regular poits Example: Modelig Atmospheric Dyamics Loosely Sychroous Parallelism similar to Sychroous case, but with miimum itermittet data sharig Example: Modelig Diffusio of cotamiats through groudwater Desigig ad Buildig Parallel Applicatios Partitioig - Break dow mai task ito smaller oes either idetical or disjoit. Commuicatio phase - Determie commuicatio patters for task coordiatio, commuicatio algorithms. Agglomeratio - Evaluate task ad/or commuicatio structures wrt performace ad implemetatio costs. Tasks may be combied to improve performace or reduce commuicatio costs. Mappig - Tasks assiged to processors; maximize processor utilizatio, miimize commuicatio costs. Mappig may be either static or dyamic. May have to iterate whole process util satisfied with expected performace Cosider writig applicatio i parallel, usig either SPMD message-passig or shared-memory Implemetatio (software & hardware) may require revisit, additioal refiemet or re-desig Cluster Computig 14 Cluster Computig 16 Ket State Uiversity 4

5 Cluster Computig Sprig 004 3/18/004 Desigig ad Buildig Parallel Applicatios Partitioig Geometric or Physical decompositio (Domai Decompositio) - partitio data associated with problem Fuctioal (task) decompositio partitio ito disjoit tasks associated with problem Divide ad Coquer partitio problem ito two simpler problems of approximately equivalet size iterate to produce set of idivisible subproblems Programmig Methodologies - Practical Aspects Bulk of parallel programs writte i Fortra, C, or C++ Geerally, best compiler, tool support for parallel program developmet Bulk of parallel programs use Message-Passig with Performace, portability, mature compilers, libraries for parallel program developmet Data ad/or tasks are split up oto differet processors by: Distributig the data/tasks oto differet CPUs, each with localmemory (MPPs,) Distribute work of each loop to differet CPUs (SMPs,OpeMP, Pthreads) Hybrid - distribute data oto SMPs ad the withi each SMP distribute work of each loop (or task) to differet CPUs withi the box (SMP-Cluster, &OpeMP, LAM) Cluster Computig 17 Cluster Computig 19 Geeric Parallel Programmig Software Systems Message-Passig Local tasks, each ecapsulatig local data Explicit data exchage Supports both SPMD ad MPMD Supports both task ad data decompositio Most commoly used Process -based, but for performace, processes should be ruig o separate CPUs Example API:, PVM Message-Passig libraries MP systems, i particular,, will be focus of remaider of workshop Data Parallel Usually SPMD Supports data decompositio Data mappig to cpus may be either implicit/explicit Example: HPF compiler Shared-Memory Tasks share commo address space No explicit trasfer of data - supports both task ad data decompositio Ca be SPMD, MPMD Thread-based, but for performace, threads should be ruig o separate CPUs Example API : OpeMP, Pthreads Hybrid - Combiatio of Message-Passig ad Shared -Memory - supports both task ad data decompositio Example: OpeMP + Typical Data Decompositio for Parallelism Example: Solve -D Wave Equatio: Origial partial Ψ Ψ Ψ differetial equatio: = D + B t x y Fiite Differece Approximatio: y + 1 f fi 1, j fi, j fi 1, j fi, j 1 i i + + f t = D x PE #0 PE #1 PE # PE #4 PE #5 PE #6 + B PE #3 PE #7 f f i, j y i, j 1 Cluster Computig 18 x Cluster Computig 0 Ket State Uiversity 5

6 Cluster Computig Sprig 004 3/18/004 Fiite Differece Approximatio: Sedig Data Betwee CPUs + 1 f fi 1, j fi, j fi 1, j f i, j 1 i i + + = D + B f t Sample Pseudo Code if (taskid =0) the li = 1 ui = 5 lj = 1 uj = 5 sed(1:5)=f(5,1:5) elseif (taskid=1)the... elseif (taskid=) the... elseif(taskid=3) the... ed if do j = lj,uj do i = li,ui work o f(i,j) ed do ed do y x PE #0 PE #1 i=1,5 i=1,5 j=1,5 j=6,50 i=1-5, j=6 i=6-50,j=5 PE #3 i=6,j=1-5 i=6,j=6-50 PE #4 i=6,50 i=6,50 j=1,5 j=6,50 i=6-50,j=5 x f i, j y f i=1-5, j=6 i, j 1 i=5,j=1-5 i=5,j=6-50 Basics of Task Parallel Decompositio - SPMD Same program will ru o differet CPUs Task decompositio aalysis has defied tasks (a ad b) to be doe by CPUs program.f: iitialize... if TaskID=A the do task a elseiftaskid=b the do task b ed if. ed program Task A Executio Stream program.f: Iitialize do task a ed program Task B Executio Stream program.f: Iitialize do task b ed program Cluster Computig 1 Cluster Computig 3 Typical Task Parallel Decompositio Process 0 Process 1 Process Multi-Level Task Parallelism Proc set #1 Proc set # Proc set #3 Proc set #4 SPECTRUM IN Subtract Frequecy f1 Subtract Frequecy f Subtract Frequecy f3 SPECTRUM OUT Sigal processig Use oe processor for each idepedet task Ca use more processors if oe is overloaded v Program tskpar Implicit oe (declaratios) Do loop #1 Ed task #1 Do loop # Ed task # threads Program tskpar Implicit oe (declaratios) Do loop #1 Ed task #1 Do loop # Ed task # Program tskpar Implicit oe (declaratios) Do loop #1 Ed task #1 Do loop # Ed task # Program tskpar Implicit oe (declaratios) Do loop #1 Ed task #1 Do loop # Ed task # Cluster Computig etwork Implemetatio: ad OpeMP Cluster Computig 4 Ket State Uiversity 6

7 Cluster Computig Sprig 004 3/18/004 Parallel Applicatio Performace Cocepts Parallel Speedup Parallel Efficiecy Parallel Overhead Limits o Parallel Performace Limits of Parallel Computig Theoretical upper limits Amdahl s Law Gustafso s Law Practical limits Commuicatio overhead Sychroizatio overhead Extra operatios ecessary for parallel versio Other Cosideratios Time used to re-write (existig) code Cluster Computig 5 Cluster Computig 7 Parallel Applicatio Performace Cocepts Parallel Speedup - ratio of best sequetial time to parallel executio time S() = ts/tp Parallel Efficiecy - fractio of time processors i use E() = ts/(tp*) = S()/ Parallel Overhead Commuicatio time (Message -Passig) Process creatio/sychroizatio (MP) Extra code to support parallelism, such as Load Balacig Thread creatio/coordiatio time (SMP) Limits o Parallel Performace Parallel Computig - Theoretical Performace Upper Limits All parallel programs cotai: Parallel sectios Serial sectios Serial sectios limit the parallel performace Amdahl s Law provides a theoretical upper limit o parallel performace for size-costat problems Cluster Computig 6 Cluster Computig 8 Ket State Uiversity 7

8 Cluster Computig Sprig 004 3/18/004 Amdahl s Law Amdahl s Law places a strict limit o the speedup that ca be realized by usig multiple processors Effect of multiple processors o ru time for size -costat problems Effect of multiple processors o parallel speedup, S: Where fs = serial fractio of code fp = parallel fractio of code N = umber of processors t1 = sequetial executio time t = ( f p / N + f s )t 1 S = 1 fs + fp / N Speedup Amdahl s Law (Cotiued) Amdahl's Law (Actual) Number CPUs f=0.0 f=0.01 Actual Cluster Computig 9 Cluster Computig 31 Amdahl s Law Gustafso s Law Speedup Amdahl's Law (Ideal Case) Number CPUs f=0.0 f=0.01 f=0.05 f=0.1 Cosider scalig problem size as processor cout icreased Ts = serial executio time Tp(N,W ) = parallel executio time for same problem, size W, o N CPUs S(N,W) = Speedup o problem size W, N CPUs S(N,W) = (Ts + Tp(1,W) )/( Ts + Tp(N,W ) ) Cosider case where Tp(N,W ) ~ W*W/N S(N,W) -> (N*Ts + N*W*W)/(N*Ts + W*W) -> N Gustafso s Law provides some hope for parallel applicatios to deliver o the promise. Cluster Computig 30 Cluster Computig 3 Ket State Uiversity 8

9 Cluster Computig Sprig 004 3/18/004 Parallel Programmig Aalysis - Example Cosider solvig -D Poisso s equatio by iterative method o a regular grid with M poits Cluster Computig 33 Ket State Uiversity 9