Parallel Numerical Algorithms

Transcription

1 Parallel Numerical Algorithms [ 4 ] Scheduling Theory Parallel Numerical Algorithms / IST / UTokyo 1

2 PNA16 Lecture Plan General Topics 1. Architecture and Performance 2. Dependency 3. Locality 4. Scheduling MIMD / Distributed Memory 5. MPI: Message Passing Interface 6. Collective Communication 7. Distributed Data Structure MIMD / Shared Memory 8. OpenMP 9. Cache Performance Special Lectures 5/30 How to use FX10 (Prof. Ohshima) 6/6 Dynamic Parallelism (Prof. Peri) SIMD / Shared Memory 10. GPU and CUDA 11. SIMD Performance Parallel Numerical Algorithms / IST / UTokyo 2

3 Scheduling Window Curtain Design Base Build Door Lock Complete Furniture Carpet Parallel Numerical Algorithms / IST / UTokyo 3

4 In Parallel Computing Problem Analyze Locality Choose Algorithm Define Subtasks Analyze Parallelism Scheduling Algorithm Assign tasks to processors Order tasks in computations Parallel Numerical Algorithms / IST / UTokyo 4

5 Definitions Set of Tasks or Jobs TT 1, TT 2,, TT nn nn represents number of tasks ii and jj are used for task indices Set of Processors or Machines PP 1, PP 2,, PP mm mm represents the number of processors pp and qq are used for processor indices pp iiii : Processing Time of task ii on processor pp Parallel Numerical Algorithms / IST / UTokyo 5

6 Processor Types Identical Processors (P) pp iiii = pp ii Processing time is same for all processors Uniform Processors (Q) pp iiii = ww ii /ss pp ww ii : Work of task ii ss pp : Speed of processor pp Unrelated Processors (R) No restrictions on pp iiii pp iiii = means, processor pp cannot compute task ii Homogeneous Heterogeneous Parallel Numerical Algorithms / IST / UTokyo 6

7 Processor Types Examples Q (uniform) P (identical) TT 11 TT 22 TT 33 PP PP PP TT 11 TT 22 TT 33 PP PP PP R (unrelated) TT 11 TT 22 TT 33 PP PP PP Parallel Numerical Algorithms / IST / UTokyo 7

8 Precedence Precedence TT ii TT jj Task jj cannot start before task ii finishes TT ii TT jj time Independent Tasks There is no precedence constraints among the tasks Parallel Numerical Algorithms / IST / UTokyo 8

9 Schedule A schedule consists of: Task Assignment Which processor computes each task Task Ordering In which order each processor computes assigned tasks Must meet all constraints Completion Time CC ii Time when the task ii finishes Makespan or Schedule Length CC mmmmmm CC mmmmmm = max{cc ii } Parallel Numerical Algorithms / IST / UTokyo 9

10 1 Gantt Chart pp CC 1 CC 4 CC 6 CC 8 0 CC 7 Parallel Numerical Algorithms / IST / UTokyo 10

11 Graham s Notation αα ββ γγ αα : processor type P for identical parallel processors Q for uniform parallel processors R for unrelated parallel processors Number of processors may be specified ββ : constraints Empty for independent tasks prec for existence of precedence constraints Other constraints can be inserted here γγ : criterion CC mmmmmm for (minimizing) makespan Parallel Numerical Algorithms / IST / UTokyo 11

12 PP CC mmmmmm Independent tasks on identical parallel processors No need of ordering: only assignment Theorem: PP2 CC mmmmmm is NP-hard There are 2 nn possible assignments Approximate algorithms! Parallel Numerical Algorithms / IST / UTokyo 12

13 List Scheduling PP CC mmmmmm Put tasks into a list Assign task to processor, one by one Assign the task to the processor that can start it at earliest time Parallel Numerical Algorithms / IST / UTokyo 13

14 List Scheduling PP CC mmmmmm What happens if the last task is long? Parallel Numerical Algorithms / IST / UTokyo 14

15 LPT PP CC mmmmmm Longest Processing Time first Sort tasks in decreasing order of processing time Parallel Numerical Algorithms / IST / UTokyo 15

16 How good? PP CC mmmmmm Let makespan by LPT be CC LLLLLL Let the optimal makespan be CC oooooo (unknown) Theorem (Graham): LPT for PP CC mmmmmm gives CC LLLLLL mm CC oooooo Only 33% worse than the optimal Coefficient is called Worse-Case Performance Ratio There are algorithms of better performance ratios Parallel Numerical Algorithms / IST / UTokyo 16

17 PP pppppppp CC mmmmmm List scheduling Now we cannot sort the tasks by their length List must consistent with precedence relation Parallel Numerical Algorithms / IST / UTokyo 17

18 How good? PP pppppppp CC mmmmmm Theorem (Graham): Any list scheduling for PP pppppppp CC mmmmmm gives CC llllllll 2 1 mm CC oooooo Not worse than twice of the optimal Parallel Numerical Algorithms / IST / UTokyo 18

19 PP iiiiiiiiiiii, pp jj = 1 CC mmmmmm The processing time of all the tasks are equal (UET : Unit Execution Time) Precedence graph is a tree (in-tree or out-tree) Parallel Numerical Algorithms / IST / UTokyo 19

20 Level and Critical Path Length of a path = sum of processing times Level of a task = length of the path from the task to the end task Critical Path = longest path Level 5 Level 4 Level 3 Level 2 Level 1 Parallel Numerical Algorithms / IST / UTokyo 20

21 Level Scheduling repeat All predecessors have been finished Make a list of Ready Tasks sorted by levels Assign first mm tasks to the processors Remove assigned tasks until all tasks have been scheduled Parallel Numerical Algorithms / IST / UTokyo 21

22 Level Scheduling repeat All predecessors have been finished Make a list of Ready Tasks sorted by levels Assign first mm tasks to the processors Remove assigned tasks until all tasks have been scheduled Parallel Numerical Algorithms / IST / UTokyo 22

23 Level Scheduling repeat All predecessors have been finished Make a list of Ready Tasks sorted by levels Assign first mm tasks to the processors Remove assigned tasks until all tasks have been scheduled Parallel Numerical Algorithms / IST / UTokyo 23

24 Level Scheduling repeat All predecessors have been finished Make a list of Ready Tasks sorted by levels Assign first mm tasks to the processors Remove assigned tasks until all tasks have been scheduled Parallel Numerical Algorithms / IST / UTokyo 24

25 Level Scheduling repeat All predecessors have been finished Make a list of Ready Tasks sorted by levels Assign first mm tasks to the processors Remove assigned tasks until all tasks have been scheduled Parallel Numerical Algorithms / IST / UTokyo 25

26 UET results Level Scheduling (Hu s Algorithm) is optimal for PP iiiiiiiiiiii, pp jj = 1 CC mmmmmm and PP oooooooooooooo, pp jj = 1 CC mmmmmm PPP pppppppp, pp jj = 1 CC mmmmmm can be solved in OO nn 2 PP pppppppp, pp jj = 1 CC mmmmmm is NP-hard PP pp jj = 1 CC mmmmmm is trivial Parallel Numerical Algorithms / IST / UTokyo 26

27 5 minutes break Parallel Numerical Algorithms / IST / UTokyo 27

28 Clustering approach Second major approach (next to list scheduling) Theorem: PPP pppppppp CC mmmmmm is solvable in polynomial time Infinite processors: use as many processors as we need Makespan is determined by the critical path Assign a critical to a processor, etc. Parallel Numerical Algorithms / IST / UTokyo 28

29 Some Techniques Task insertion In list scheduling If there is an idle time slot for a task, then insert the task (so the order of tasks is different from the list) Task duplication Communication time can be reduced Without duplication With duplication Parallel Numerical Algorithms / IST / UTokyo 29

30 Heterogeneous Processors PPPP, QQQQ or RRRR mm is arbitrary but fixed (treated as a constant) Q: Uniform processors QQQQ CC mmmmmm : FPTAS exists QQ CC mmmmmm : LPT is approx.; PTAS exists QQ pppppppp CC mmmmmm : an OO log mm -approx. algorithm (F)PTAS: (Fully) Polynomial Time Approximation Scheme R: Unrelated processors RRRR CC mmmmmm : FPTAS exists RR CC mmmmmm : a 2-approx. algorithm exists RR pppppppp CC mmmmmm : almost no theoretical result, many heuristics such as MH, GDL, BIL, HEFT and LDBS Parallel Numerical Algorithms / IST / UTokyo 30

31 Online Scheduling All of the tasks are not always known a priori Dynamic scheduling (in HPC terminology) Several versions of problem definitions Competitive Ratio ρρ Worst case ratio of the resulting makespan to optimal (omniscient) makespan PP pppppppp, oooooooooooo CC mmmmmm : list scheduling ρρ = 2 1 mm QQ oooooooooooo CC mmmmmm : an algorithm ρρ = 5.8 RR oooooooooooo CC mmmmmm : an algorithm ρρ = OO log mm Parallel Numerical Algorithms / IST / UTokyo 31

32 Simpler models as task scheduling is so hard Parallel Numerical Algorithms / IST / UTokyo 32

33 Divisible Load Theory Assume: tasks are identical and really many Approximate it as if a continuous material You can divide the task into arbitrary size Communication time is proportional to task size Usually Manager-Worker Model as follows Parallel Numerical Algorithms / IST / UTokyo 33

34 Single-round scheme Manager send send send Worker1 recv Worker2 recv Worker3 recv Optimal solution: all processors completes at the same time Closed formula can be derived Parallel Numerical Algorithms / IST / UTokyo 34

35 Multi-round scheme Manager Worker1 Worker2 Worker3 Worker can start earlier, better than single round Hard to optimize: number of rounds, task sizes Parallel Numerical Algorithms / IST / UTokyo 35

36 Repeat identical rounds Manager Prologue Steady-state rounds Epilogue kk = 4 Worker1 Worker2 Worker3 Schedule steady-state rounds so that it takes time CC oooooo /kk + αα (kk: number of rounds, αα: constant) Total makespan CC mmmmmm 1 + 1/kk CC oooooo + kkkk By letting kk = CC oooooo αα, we have CC mmmmmm CC oooooo = 1 + OO 1 CC oooooo Parallel Numerical Algorithms / IST / UTokyo 36

37 Loop Scheduling Almost identical tasks, but time varies a little 1. Equal sized subtasks time variation 2. Divide tasks into many chunks, and use dynamic assignment subtask overheads Parallel Numerical Algorithms / IST / UTokyo 37

38 Loop Scheduling 3. Divide task into decreasing size of subtasks Guided Self Scheduling Chunk size = remaining size / num. processors Factoring Chunk size = remaining size / (αα num. processors) αα: constant 2 Parallel Numerical Algorithms / IST / UTokyo 38

39 Summary Task scheduling Combinatorial problem, NP-hard in many cases Good approximate algorithms with guaranteed performance exist There is much room to improve Simpler models Divisible Load Theory, and Loop scheduling Rough approximation, but practically useful Parallel Numerical Algorithms / IST / UTokyo 39

40 Proof of Graham s Theorem Th: Let CC max and CC oooooo be makespans of list scheduling and minimum for a PP CC max problem. Then CC mmmmmm 2 1 mm CC oooooo Proof: Let pp tttttt = ii pp ii be the sum of execution times. As it is processed by mm processors, it takes at least pp tttttt /mm, which implies pp tttttt /mm CC oooooo Let TT ll be the task that completes last: CC ll = CC mmmmmm. Starting time of TT ll is SS ll = CC mmmmmm pp ll. Before SS ll, there were no idle processor, otherwise TT ll could start earlier. So all processors were busy till SS ll to process tasks other than TT ll. So we have mmss ll pp tttttt pp ll. Of course we have CC mmmmmm pp ll. Combining these inequalities we have the result we want. Parallel Numerical Algorithms / IST / UTokyo 40

41 Optimality of Hu s algorithm Hu s algorithm gives optimal schedule for PP iiii tttttttt, pp jj = 1 CC mmmmmm. Proof: Note that the number of ready tasks monotonically decreases. At some time, the number of ready tasks becomes less than mm. After that, it continues to be less than mm, and the highest level decreases by one for each step. Before that, all processors are busy, so there is no way to reduce the time. (incomplete) Parallel Numerical Algorithms / IST / UTokyo 41

42 PNA16 Lecture Plan General Topics 1. Architecture and Performance 2. Dependency 3. Locality 4. Scheduling MIMD / Distributed Memory 5. MPI: Message Passing Interface 6. Collective Communication 7. Distributed Data Structure MIMD / Shared Memory 8. OpenMP 9. Cache Performance Special Lectures 5/30 How to use FX10 (Prof. Ohshima) 6/6 Dynamic Parallelism (Prof. Peri) SIMD / Shared Memory 10. GPU and CUDA 11. SIMD Performance Parallel Numerical Algorithms / IST / UTokyo 42