Parallel Numerics, WT 2013/ Introduction

Transcription

1 Parallel Numerics, WT 2013/ Introduction page 1 of 122

2 Scope Revise standard numerical methods considering parallel computations! Required knowledge Numerics Parallel Programming Graphs Literature Dongarra, Duff, Sorensen, van der Vorst: Numerical Linear Algebra for High-Performance Computers Pacheco: A User s Guide to MPI (web) Parallel Programming with MPI Schüle: Paralleles Rechnen Source for some images: comp/ page 2 of 122

3 Computer Science Aspects of Parallel Numerics Numerical Problems Data Dependency Graphs Loop Manipulations Why parallel computing? SETI, weather prediction, quantum simulation TOP500 SuperMUC page 3 of 122

4 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations 2 Elementary Linear Algebra Problems 2.1 BLAS 2.2 Matrix-Vector Operations 2.3 Matrix-Matrix-Product 3 Linear Systems of Equations with Dense Matrices 3.1 Gaussian Elimination 3.2 Parallelization 3.3 QR-Decomposition with Householder matrices 4 Sparse Matrices 4.1 General Properties, Storage 4.2 Sparse Matrices and Graphs 4.3 Reordering 4.4 Gaussian Elimination for Sparse Matrices 5 Iterative Methods for Sparse Matrices 5.1 Stationary Methods 5.2 Nonstationary Methods 5.3 Preconditioning 6 Domain Decomposition 6.1 Overlapping Domain Decomposition 6.2 Non-overlapping Domain Decomposition 6.3 Schur Complements page 4 of 122

5 1.1. Computer Science Aspects of Parallel Numerics Pipelining in the CPU Elementary operations are carried out in pipelines: Divide task into smaller subtasks Each subtask is executed on piece of hardware that operates concurrently with other stages of the pipeline Addition Pipeline: page 5 of 122

6 Visualization Pipelining page 6 of 122

7 Visualization Pipelining page 7 of 122

8 Visualization Pipelining page 8 of 122

9 Visualization Pipelining page 9 of 122

10 Visualization Pipelining page 10 of 122

11 Visualization Pipelining page 11 of 122

12 Visualization Pipelining Startup time = k(= 4) clock units Lateron: per clock unit one result Total time: k u + n u page 12 of 122

13 Advantages of Pipelines Pipeline filled: per clock unit one result is achieved. All additions should be organized such that pipeline is always filled! Pipeline nearly empty: e.g., in the beginning of the computations, it is not efficient! Major task for CPU: Organize all operations such that operands are just in time at the right position to fill the pipeline and keep it full. page 13 of 122

14 Computer Science Aspects of Parallel Numerics Numerical Problems Data Dependency Graphs Loop Manipulations CPU Pipelining page 14 of 122

15 Computer Science Aspects of Parallel Numerics Numerical Problems Data Dependency Graphs Loop Manipulations CPU page 15 of 122

16 Pipelining: General Steps Instruction Fetch: Get the next command Decoding: Analyse instruction, compute addresses of operands Operand Fetch: Get the values of the next operands Execution step: Carry out command on operands Result Write: Write result in memory Pipelining of these steps, and also inside each step. page 16 of 122

17 Special case: Vector scaling For set of data the same operation has to be executed on all components. y 1 x 1 y 2. = α x 2. y n x n for j = 1, 2,..., n : y j = αx j ; Total costs: Startup time + vector length clock time (pipeline length + vector length ) T page 17 of 122

18 Example of a modern CPU architecture Source: A. Heinecke and C. Trinitis, Cache-oblivious Matrix Algorithms in the Age of Multi- and Many-Cores, Concunrrency and Computation: Practice and Experience, pp. 1 20, 2012, DOI: /cpe page 18 of 122

19 Problem: Data Dependency Example: Fibonacci numbers x 0 = 0, x 1 = 1, x 2 = x 0 + x 1,..., x i = x i 2 + x i 1 After filling in x 0 and x 1 the next pair needs x 2 which is known only after the first computation is finished! Pipeline contains always only one pair is nearly empty all the time! Similar Problem for recursive subroutine calls. page 19 of 122

20 Memory Organization page 20 of 122

21 Cache Idea Cache as memory buffer between large, slow and small, fast memory. Considering data flow (last used data), try to predict which data will be requested in next step: keep last used data in fast cache because it is likely that the same data will be used again keep the neighborhood of last used data in fast cache. Memory is organized in chunks. Together with current data, put surroundings into cache (cache line). page 21 of 122

22 Cache Idea Cache hit: The data requested from the higher level is found in the cache: use the cached data and transfer it to the higher level. Done. Cache miss: The data requested from the higher level is not found in the cache: Look for data in the lower (slower) memory. Copy the related data to the cache (removing the oldest cache entry) and transfer it to the higher level. Temporal locality: Data referenced at some point in time will again be referenced in near future. If data needs to be reused, do it as soon as possible! Spatial locality: Data close to recently referenced data is probable to be referenced soon. Work blockwisely to ensure neighbouring! page 22 of 122

23 Mapping between Memory Direct mapping: Adress in cache, modulo Adress Adress Disadvantage: Immediately replacing of data in cache Associative mapping: Partition cache in blocks. Write data to direct mapped address in one of the blocks. Replace oldest data in block. page 23 of 122

24 Cyclic data distribution Main memory is often organized in banks connected by bus: Per cycle n operands can be fetched out of the n banks. Storing vectors! x 1 in bank1 x 2 in bank2,.... Allows one step access to x. page 24 of 122

25 Parallel Processors Classical von Neumann model: Code and data in memory! Control unit fetches instructions and data from memory and sequentially coordinates the operations. page 25 of 122

26 Flynn Taxonomie Classification of computer systems by number of instruction streams and data streams: SISD: Single instruction stream, single data stream. Conventional serial computers SIMD: Single instruction stream can handle multiple data streams. All PEs synchronously executing same instruction stream on different sets of data. E.g., SSE (Streaming SIMD Extensions) MISD: Machine is capable to process single data stream using multiple instruction streams. Uncommon architecture. Heterogenous systems operate on same data stream and must agree on result e.g., fault tolerance. MIMD: Multiple instruction streams on multiple data streams. General purpose parallel computers. page 26 of 122

27 Parallel Computation MIMD architecture: multiple instructions multiple data (compare to SISD = single instructions single data, etc.) page 27 of 122

28 Shared Memory Parallelization (SMP) page 28 of 122

29 Cache Coherence parallel for(i=proc_number; i<n; i=i+2) x[i]=2.; 2 threads, proc_number 0 and 1: Thread 1 changes x(0,2,4,...) and thread2 changes x(1,3,5,...). Each changing step of thread 1 also changes data that is contained in the cache of thread 2 (and vice versa). Otherwise data in the two caches is not consistent anymore! To retain the right values in both caches after each changing step, also the value in the other cache has to be renewed! Leads to dramatic increase of computational time, ev. slower than sequential computation! page 29 of 122

30 Distributed Memory Parallelization (DMP) Virtual shared memory: Distributed data but organized as shared memory. page 30 of 122

31 Nonuniform Memory Access (NUMA) Cluster of multiple CPU processors Different types of communication shared memory and distributed memory! page 31 of 122

32 Topology of the processor/memory interconnection Mesh (Array, Grid): p processors, longest path 2( p 1) page 32 of 122

33 Topology of the processor/memory interconnection Time for sending data from one processor to another depends on the connection network topology: Topology worst case time Mesh 2( p 1) Vector p 1 Ring/Torus p 2 Tree 2(ld(p) 1) Hypercube ld(p) Definition: diameter = largest distance page 33 of 122

34 Communication Topology diameter = 2( p 1) Manhattan distance page 34 of 122

35 Hypercube Diameter page 35 of 122

36 Hypercube Diameter page 35 of 122

37 Different Topologies Topology G p diameter(g) degree(g) edges(g) G 1 (n), (vector) n n 1 2 n 1 T 1 (n), (ring) n n 2 2 n G 2 (n, n), (mesh) n 2 2n 2 4 2n 2 2n T 2 (n, n) n 2 2 n 2 4 2n2 BT (h) 2 h+1 1 2h 3 2 h+1 2 HC(k) 2 k k k 2 k 1 k G: Grid, T : Torus, BT : binary tree, HC: hypercube page 36 of 122

38 Network based on Switches - 3-level Omega network Blocking network. Simultaneous connection P0 P6 and P1 P7 is not possible! Turn-over of switches necessary! page 37 of 122

39 Network based on Switches - Crossbar Direct, independent connection between all processors. Nonblocking! page 38 of 122

40 Performance Analysis Definitions: f : parallel fraction of program 1 f : sequential fraction of program t p : wall clock time to execute the job on p parallel processors Using p processors in parallel we can achieve speedup. Speedup S p := t 1 tp is the ratio of execution time with 1 vs. p PEs In the ideal case we observe t 1 = p t p. Efficiency using p processors: E p = S p p, 0 E p 1 E p 1: very good parallelizable, because then S p p or t 1 p t p. (problem scales) E p 0: bad, because E p = Sp p = t 1 p t p and t 1 p t p. page 39 of 122

41 Amdahl s Law Assumption: for fixed problem size program can be separated into parallel fraction f and sequential fraction 1 f. t p = f t 1 f + (1 f )p + (1 f )t 1 = t 1 (1 f )t 1 p }{{} p }{{} strongly sequential ideally parallel Amdahl s law: S p = t 1 t p = 1 f +(1 f )p p = p f + (1 f )p 1 1 f E p = S p p = 1 f + (1 f )p 1 p E p = 0 (1 f )p We always have a small portion of our algorithm that is not parallelizable the efficiency will always be zero in the limit! page 40 of 122

42 Amdahl s Law Speedup depending on p. Saturation 1 1 f. page 41 of 122

43 Gustafson s Law Assume that problem can be solved in 1 unit of time on parallel machine with p processors. Fraction f is good parallelizable, 1 f not Compared with this parallel implementation an uniprocessor would perform (1 f ) + fp for the same job. Speedup: Efficiency: S pf = t 1 = 1 f + fp = p + (1 p)(1 f ) t p 1 E pf = S pf p = 1 f p + f p f page 42 of 122

44 Example: f = 0.99, p = 100 or p = 1000 law p speedup efficiency Amdahl S = p f +(1 f )p E = 1 f +(1 f )p 100 S 100 = E 100 = S 1000 = E 1000 = 0.1 Gustafson S = 1 f + fp E = 1 f p + f 100 S 100 = E 100 = S 1000 = E 1000 = page 43 of 122