Thinking parallel. Decomposition. Thinking parallel. COMP528 Ways of exploiting parallelism, or thinking parallel

Transcription

1 COMP528 Ways of exploiting parallelism, or thinking parallel Alexei Lisitsa Dept of computer science University of Liverpool Thinking parallel Think about inherent parallelism in the job/problem at hand first That will drive The choice of the algorithm; The choice of the implementation language The implementation itself... Thinking parallel Different aspects: Decomposition: how to decompose your problem into concurrent tasks; Scaling: how to ensure that there are enough concurrent tasks to keep all the processor cores busy; Correctness: how to ensure that the result is free from errors; Abstractions and Patterns: how to choose and utilize algorithms... Decomposition How do you find the parallelism? Data parallelism Task parallelism Pipelines Mixed solutions 1

2 Data parallelism Given: a data set and an operation that can be applied element by element; What to do: apply the same task concurrently to each element Task parallelism Several different independent tasks to be applied to the same data a b c d e f g h Average Minimum Maximum Geom- mean A B C D E F G H Pipelines Special kind of task/data parallelism: many independent tasks to be applied to a stream of data; Each data item is processed by stages as they passed through; Different items can pass through different stages and be processed at the same time; Pipelines data1 Data2 Data3 task 1 Task 2 2

3 Simple case study: Mixed solutions Task of folding, stuffing, sealing, addressing, stamping and mailing letters There are 6 people to perform the task Several natural solutions and their combinations are possible Mixed solutions (cont.) Pipeline: each person doing exactly one task (folding, stuffing, etc) Fine-grained parallelism: small individual tasks and frequent interactions Data parallelism: each person takes a portion of the envelops and performs all six tasks for this portion Coarse-grained parallelism: large individual tasks and infrequent interactions Mixed solutions (cont.) Mixed solution: Person 1 is doing folding and stuffing Person 2 is doing sealing Persons 3,4,5 are doing addressing Person 6 is doing stamping and mailing Neither data parallelism, nor task (pipeline) parallelism, but rather a combination of both Scalability and Speedup Speedup is a ratio of the time it takes to run a program without parallelism vs the time it runs in parallel; Scalability is a measure of how much speedup the program gets as one adds more processor cores; The program does not scale beyond certain point when adding more processors does not result in additional speedup. 3

4 How much parallelism is there? Amdahl s law (1967) The computer program will never go faster than the sum of the parts that do not run in parallel (the serial portions), no matter how many processors we have How much parallelism is there? J. Gustafson s observations (around Amdahl s law): For many problems as the problem size grows the work required for the parallel part grows faster then the work required for the serial part The serial fraction decreases, so by Amdahl s law scalability improves Amdahl vs Gustafson Both are right, but Amdahl s point is pessimistic: for a fixed problem/program there is limit of its acceleration by the parallelization; Gustafson s point is optimistic: for growing problems there is chance to keep up using parallelization Correctness Correctness of sequential programs is not a trivial issue, even more so is correctness of parallel programs If a program is executed in parallel, i.e using multiple threads of control, the precise order of operations potentially be different That may lead to.. 4

5 That may lead to Correctness (cont.) Non-deterministic behaviour Different results from run to run, not all of them may be correct; Round-off errors Failures in coordination of tasks Deadlocks Race conditions Concurrent access and Locks Locking is a mechanism for concurrent access control Acquiring a lock gives a tread an exclusive access to a data structure (e.g. variable) Locks can be used to ensure correct (required) behaviour of the multiple-threads programs Example of using locks Example of using locks (cont.) Thread A Thread B Value of X LOCK(X) (waiting) 44 Read X(44) (waiting) 44 add 10 (waiting) 44 Write X(54) (waiting) 54 UNLOCK(X) (waiting) 54 LOCK(X) 54 Read X (54) 54 subtract Write X(42) 42 UNLOCK(X) 42 Several variants ( = 2) of execution possible here, but The final value of X = 42 is guaranteed here 5

6 Race conditions Race conditions occur when multiple tasks read from and write to the same memory without proper synchronization; Depending on application can lead to incorrect results Example of race conditions Tread A Thread B Value of X Read X(44) 44 Add 10 Read X(44) 44 Write X(54) subtract Write X(32) 32 Tread A Tread B Value of X Read X(44) 44 Read X (44) subtract add 10 Write X(32) 32 Write X(54) 54 Deadlocks Deadlock occurs when at least two tasks wait for each other and each will not resume until the other task proceeds may occur when code requires the acquisition of multiple locks; Dining Philosophers example; Conclusion All discussed aspects of Thinking Parallel approach are non-trivial We need tools and instruments which support Thinking Parallel at appropriate level of abstraction 6