Taxonomy of Parallel Computers, Models for Parallel Computers. Levels of Parallelism

Transcription

1 Taxonomy of Parallel Computers, Models for Parallel Computers Reference : C. Xavier and S. S. Iyengar, Introduction to Parallel Algorithms 1 Levels of Parallelism Parallelism can be achieved at different levels Job level parallelism ( Program level parallelism) e.g. 10 jobs given, run them in 10 machines Subtask level parallelism ( Subprogram level parallelism) e.g. Each job(task) can be divided into smaller subtasks(subprograms), they may be executed in parallel. Statement level parallelism e.g. Each program(or subprogram) there are several statements. These statement may be done in parallel. 2 1

2 Levels of Parallelism Parallelism can be achieved at different levels Operational level parallelism e.g. In a statement, several operations (add, subtract, store) are carried out, and they can be parallelized. Micro-Operational level parallelism e.g. Adding the content of 2 variables A and B and storing the result of C. These consist of the following micro-operations. 1. Load the accumulator with the content of A 2. Add the content of B with the content of the accumulator 3. Store the content of the accumulator in the variable C 3 Levels of Parallelism Job (Program) level parallelism Subtask (Subprogram) level parallelism Statement level parallelism Operational level parallelism Micro-Operational level parallelism 4 2

3 Statement Level Parallelism Consider the following program O(n) time sequentially For i=1 to n do x i x i + 1 This can be parallelized. O(1) time using n processors For i=1 to n do in parallel x i x i + 1 End Parallel Using n processing elements, p 1, p 1,, p n, they work simultaneously and it is completed in O(1) time. 5 Based on Flynn s Taxonomy Multiple/Single Instruction (MI/SI) Multiple/Single Data (MD/SD) 4 main architectural classes Single Instruction Single Data (SISD) Single Instruction Multiple Data (SIMD) Multiple Instruction Single Data (MISD) Multiple Instruction Multiple Data (MIMD) 6 3

4 Models of Parallel Computation Any parallel algorithm is designed with an assumption of an architecture of a parallel computer. A number of very different models of machines have been assumed for designing parallel algorithms in the literature. Binary tree Network Hypercubes 7 Binary tree Models Processing of the whole problem is represented in the form of a binary tree, in which non-leave nodes have two children. Non-leave node: operation Example of finding the sum of 8 numbers

5 Binary tree Models The task of adding 8 numbers carried out by 4 processors in 3 units of time. Complete binary tree with n nodes is of height log 2 n, so the work of adding n numbers could be done by n/2 processors in log 2 n time units. These processors could now be scheduled for the operations represented by the internal nodes. SHC assigns for each internal node an ordered pair (p,t) where p represents the processor number and t represents the time at which this operation takes place. SCHEDULE SCH(1) = (1,1) SCH(2) = (2,1) SCH(3) = (3,1) SCH(4) = (4,1) SCH(5) = (1,2) SCH(6) = (2,2) SCH(7) = (1,3) MEANING Processor 1 does at time 1 Processor 2 does at time 1 Processor 3 does at time 1 Processor 4 does at time 1 Processor 1 does at time 2 Processor 2 does at time 2 Processor 1 does at time 3 9 Network Models Several processors are interconnected using physical links and the following assumptions are made: Each has its own local memory No common memory that can be accessed by all processors Connected directly with physical links and the interconnection topology is called the network topology If two processors are adjacent, data can be moved form one processor to the other directly. In one clock cycle, a unit operation takes place in any processor A processor can communicate a data to any of its adjacent processor in a unit time. 10 5

6 Network Topology Unfortunately, there is no universal topology that is ideal for all the problems. So we must investigate the problem and propose the interconnection topology. Examples are: Chain Ring Mesh Torus Tree Star Cube Hypercube 11 Language Structure for Parallel Algorithms Structure 1 For variable i =1 to n do in parallel S i End Parallel The instructions S 1, S 2,, are all carried out by each of the n processors simultaneously. The running variable i in the For statement denotes the processor index. 12 6

7 Language Structure for Parallel Algorithms Structure 2 For x S do in parallel End Parallel Statements x S are all carried out by each of the processors simultaneously. Here the number of processors working simultaneously is the number of elements in S. 13 7