Parallel Computing Ideas

Transcription

1 Parallel Computing Ideas K. 1 1 Department of Mathematics 2018

2 Why When to go for speed Historically: Production code Code takes a long time to run Code runs many times Code is not end in itself 2010: If it is worth writing compiled code, it is worth doing in parallel... All CPUs now have multiple processors All applications will one day take advantage

3 Why Uses of Parallelism 1980s - Fluid Mechanics Single solution, large matrices, gather at every step 1990s - Large nonlinear problems Large matrices, repeated operations, gather at every step 2000s - Larger nonlinear problems and Big Data Multiple solutions, more independent processes, gathered results 2010s - Simulations and Data Analytics Multiple solutions, independent processes, gathered results Move from solving one large problem to many time-consuming problems.

4 Why Nomenclature CPU - Central Processing Unit Register - place in CPU where operands reside during use Cache - memory that resides on the CPU IF - Instruction Fetch ID - Instruction Decode EX - Execute WB - Write Back

5 Why Nomenclature Core A single processor on one of the CPUs of a node Processor Usually means a core, or hyperthread core Process A program that runs on a processor. Possibly many processes per processor.

6 Why Central Processing Unit From hardwaresecrets.com/ how-a-cpu-works/4/ accessed March 2018

7 Why How to speed? Avoid cache misses Try to access contiguous memory Do all computations on a block before moving on Example...

8 Why No Pipeline... All pipeline figures from

9 Pipelines Vector Pipelines Simple form of parallel processing Single CPU can perform two different operations at once Single instruction, sequential data

10 Pipelines Instruction Pipeline... Fetch, Decode, Execution, and so on, can all work at same time.

11 SIMD Single Instruction, Multiple Data Single CPU with many Arithmetic units ID step fills registers for each ALU EX step does computation simultaneously on all ALUs

12 SIMD SIMD Pipeline... After instructions are decoded, the same operations can be executed on a vector array of numbers.

13 SIMD Disadvantages Slower to fill ALU registers Many ALUs idle during EX

14 SIMD Single Instruction, Multiple Thread Many CPUs Main program spawns many threads for one instruction GPU computing

15 MIMD Multiple Instruction, Multiple Data Many CPUs Asynchronous Redundant work Much more versatile than most SIMD

16 MIMD Shared Memory E.g. Quad Core CPU Bus-based Limited bandwidth Scales poorly Switch-based Expensive Still does not scale well

17 SPMD SPMD Single Program, Multiple Data You write one (1) program......that program runs in every process Processes perform tasks based on conditions and messages Processes have different inputs, outputs

18 SPMD Python If we really want speed, should compile... For simulations and Monte Carlo, maybe... Forking processes is expensive match it to number of processors.

19 When CPUs were expensive: Pipelines As chips became denser SIMD As CPUs become commodities: MIMD As GPUs could be dense: GPU As processors automated everything: Scripts

20 Goal Hope to show that we can modify programs easily to take advantage of modern processors Getting speedups is more problematic

21 Resources Solitary - Two cpus, four cores each, 8GB RAM runs prime1 on 6 cores in.038 seconds on OpenMPI Cluster - Five nodes, one cpu per node, six cores per cpu, 8GB RAM per node runs prime1 on 6 cores in.041 seconds on MPICH2 Labs - 20 to 32 nodes, two to eight cores per node