Outline: Embarrassingly Parallel Problems. Example#1: Computation of the Mandelbrot Set. Embarrassingly Parallel Problems. The Mandelbrot Set

Transcription

1 Outline: Embarrassingly Parallel Problems Example#1: Computation of the Mandelbrot Set what they are Mandelbrot Set computation cost considerations static parallelization dynamic parallelizations and its analysis Monte Carlo Methods parallel random number generation Ref: Lin and Snyder Ch 5, Wilkinson and Allen Ch 3 Admin: reminder - pracs this week, get your NCI accounts!; parallel programming poll News: Improving Energy Efficiency and Exploiting Parallelism with Processing in Memory and Near-Data Processing COMP4300/8300 L8: Embarrassingly Parallel Problems COMP4300/8300 L8: Embarrassingly Parallel Problems Embarrassingly Parallel Problems computation can be divided into completely independent parts for execution by separate processors (correspond to totally disconnected computational graphs) infrastructure: Blocks of Independent Computations (BOINC) project SETI@home and Folding@Home are projects solving very large such problems part of an application may be embarrassingly parallel distribution and collection of data are the key issues (can be non-trivial and/or costly) frequently uses the master/slave approach (p 1 speedup) The Mandelbrot Set set of points in complex plane that are quasi-stable computed by iterating the function z k+1 = z 2 k + c z and c are complex numbers (z = a + bi) z initially zero c gives the position of the point in the complex plane iterations continue until z >2 or some arbitrary iteration limit is reached z = a 2 + b 2 enclosed by a circle centered at (0,0) of radius 2 COMP4300/8300 L8: Embarrassingly Parallel Problems COMP4300/8300 L8: Embarrassingly Parallel Problems

2 Evaluating 1 Point Cost Considerations on NCI s Raijin t2 str t 1{ t r 1 st t 1 t r t 1 1 { t t 1 3 { t t t sq { t 3 r 3 r 3 3 r 3 3 r 3 3 r t t sq 3 r 3 r 3 3 t sq < t < 1 t r r t r t 10 flops per iteration maximum 256 iterations per point approximate time on one Raijin core: /( ) 0.12usec between two nodes the time to communicate single point to slave and receive result 2 2usec (latency limited) conclusion: cannot parallelize over individual points also must allow time for master to send to all slaves before it can return to any given process COMP4300/8300 L8: Embarrassingly Parallel Problems COMP4300/8300 L8: Embarrassingly Parallel Problems Building the Full Image Parallelisation: Static Define: P min. and max. values for (usually -2 to 2) number of horizontal and vertical pixels r 1 1 < t 1 r 2 2 < t 2 { r r t 1 1 r r t t 2 1 t r 1 s r P Summary: t t totally independent tasks each task can be of different length COMP4300/8300 L8: Embarrassingly Parallel Problems COMP4300/8300 L8: Embarrassingly Parallel Problems

3 Master: Static Implementation r s r s < r s { s r s r r t r r 1 1 < t t 1 { r 1 2 r 2 r ss r s r Slave: st t st r r r rstr st r str rstr t r t r 1 1 < t 1 r 2 rstr 2 < str 2 { r r t 1 1 r r t t 2 1 t r 1 s 1 2 r st r COMP4300/8300 L8: Embarrassingly Parallel Problems t Processor Farm: Master r r s s < r { s r s t t r { r s r r 2 r r s t t t r < t { s r s t t r s s r s t r t r t s 2 t r r r t > COMP4300/8300 L8: Embarrassingly Parallel Problems Dynamic Task Assignment discussion point: why would we expect static assignment to be sub-optimal for the Mandelbrot set calculation? Would any regular static decomposition be significantly better (or worse)? pool of over-decomposed) tasks that are dynamically assigned to next requesting process Processor Farm: Slave st t st r r r 2 st r 2 t s r t s r t t t { t 2 1 t r 1 1 < t 1 { r r t 1 1 r r t r 1 1 s 2 2 r st r r s t t r 2 st r s r t COMP4300/8300 L8: Embarrassingly Parallel Problems COMP4300/8300 L8: Embarrassingly Parallel Problems

4 Analysis Let p, m, n, I denote r t t 1 t r: sequential time: (t f denotes floating point operation time) t seq I mn t f = O(mn) parallel communication 1: (neglect t h term, assume message length of 1 word) t com1 = 2(p 1)(t s +t w ) parallel computation: t comp I mn p 1 t f parallel communication 2: t com2 = overall: m p 1 (t s +t w ) t par I mn p 1 t f + (p 1+ m p 1 )(t s +t w ) Discussion point: What assumptions have we been making here? Are there any situations where we might still have poor performance, and how could we mitigate this? COMP4300/8300 L8: Embarrassingly Parallel Problems evaluation of integral (x 1 x i x 2 ) example r < { 1r r 1 1 s 1r 1r 1r r s 1 1 Monte Carlo Integration x2 1 N area = f (x)dx = lim x 1 N N f (x i )(x 2 x 1 ) i=1 x2 I = (x 2 3x)dx x 1 where r 1 1 computes a pseudo-random number between 1 and 1 COMP4300/8300 L8: Embarrassingly Parallel Problems Example#2: Monte Carlo Methods use random numbers to solve numerical/physical problems evaluation of π by determining if random points in the unit square fall inside a circle area of circle area of square = π(1)2 2 2 = π 4 Parallelization only problem is ensuring each process uses a different random number and that there is no correlation one solution is to have a unique process (maybe the master) issuing random numbers to the slaves P COMP4300/8300 L8: Embarrassingly Parallel Problems COMP4300/8300 L8: Embarrassingly Parallel Problems

5 Parallel Code: Integration Master (process 0): r < { r < 1r r 1 1 r 2 r r q t s r s 1r sr t r < r { r r q t s st t r s r Slave: st t st r r s st r r q t r 1r st r t t t { r < s 1r 1r 1r s r q t r 1r st r t r s P r Question: performance problems with this code? COMP4300/8300 L8: Embarrassingly Parallel Problems Parallel Random Numbers linear congruential generators x i+1 = (ax i + c) mod m (a, c, and m are constants) using property x i+p = (A(a, p,m)x i +C(c,a, p,m)) mod m, we can generate the first p random numbers sequentially to repeatedly calculate the next p numbers in parallel Summary: embarrassingly parallel problems defining characteristic: tasks do not need to communicate non-trivial however: providing input data to tasks, assembling results, load balancing, scheduling, heterogeneous compute resources, costing static task assignment (lower communication costs) vs. dynamic task assignment + overdecomposition (better load balance) Monte Carlo or ensemble simulations are a big use of computational power! the field of grid computing arose to solve this issue COMP4300/8300 L8: Embarrassingly Parallel Problems