Parallel Computing Architectures

Size: px

Start display at page:

Download "Parallel Computing Architectures"

Ashlee Rich
5 years ago
Views:

1 Parallel Computing Architectures Moreno Marzolla Dip. di Informatica Scienza e Ingegneria (DISI) Università di Bologna

Copyright 2013 2018 Moreno Marzolla, Università di Bologna, Italy http://www.moreno.marzolla.name/teaching/hpc/ This work is licensed under the Creative Commons Attribution-ShareAlike 4.

2 Copyright Moreno Marzolla, Università di Bologna, Italy This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). To view a copy of this license, visit or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. 2

3 An Abstract Parallel Architecture Processor Processor Processor Processor Interconnection Network Memory Memory Memory How is parallelism handled? What is the exact physical location of the memories? What is the topology of the interconnect network? 3

4 Why are parallel architectures important? There is no "typical" parallel computer: different vendors use different architectures There is currently no universal programming paradigm that fits all architectures Parallel programs must be tailored to the underlying parallel architecture The architecture of a parallel computer limits the choice of the programming paradigm that can be used 4

5 Von Neumann architecture and its extensions 5

6 Von Neumann architecture Data Address Control Bus R0 R1 ALU Memory Memory Rn-1 PC IR PSW Control 6

7 The Fetch-Decode-Execute cycle The CPU performs an infinite loop Fetch Decode Fetch the opcode of the next instruction from the memory address stored in the PC register, and put the opcode in the IR register The content of the IR register is analyzed to identify the instruction to execute Execute The control unit activates the appropriate functional units of the CPU to perform the actions required by the instruction (e.g., read values from memory, execute arithmetic computations, and so on) 7

8 How to limit the bottlenecks of the Von Neumann architecture Reduce the memory access latency Hide the memory access latency Rely on CPU registers whenever possible Use caches Multithreading and context-switch during memory accesses Execute multiple instructions at the same time Pipelining Multiple issue Branch prediction Speculative execution SIMD extensions 8

9 CPU times compared to the real world 1 CPU cycle Level 1 cache access Main memory access Solid-state disk I/O Rotational disk I/O Internet: SF to NYC Internet: SF to UK Internet: SF to Australia Physical system reboot 0.3 ns 0.9 ns 120 ns μs 1-10 ms 40 ms 81 ms 183 ms 1m 1s 3s 6 min 2-6 days 1-12 months 4 years 8 years 19 years 6 millennia Source:

10 Caching 10

11 Cache hierarchy Large memories are slow; fast memories are small CPU L1 Cache L2 Cache L3 Cache (possible) interconnect bus DRAM 11

12 Cache hierarchy of the AMD Bulldozer architecture Source: 12

13 CUDA memory hierarchy Block Block Shared Memory Registers Thread Local Memory Registers Thread Local Memory Shared Memory Registers Thread Local Memory Registers Thread Local Memory Global Memory Constant Memory Texture Memory 13

14 How the cache works Cache memory is very fast Often located inside the processor chip Expensive, very small compared to system memory The cache contains a copy of the content of some recently accessed (main) memory locations If the same memory location is accessed again, and its content is in cache, then the cache is accessed instead of the system RAM 14

15 How the cache works If the CPU accesses a memory location whose content is not already in cache the content of that memory location and "some" adjacent locations are copied in cache in doing so it might be necessary to purge some old data from the cache to make room to the new data The smallest unit of data that can be transferred to/from the cache is the cache line On Intel processors, usually 1 cache line = 64B 15

16 Example CPU Cache RAM a b c d e f g h i j k l Cache line 16

17 Example CPU Cache RAM a b c d e f g h i j k l Cache line 17

18 Example CPU a b c d e f g h i j k l Cache RAM a b c d e f g h i j k l 18

19 Example CPU a b c d e f g h i j k l Cache RAM a b c d e f g h i j k l 19

20 Example CPU a b c d e f g h i j k l Cache RAM a b c d e f g h i j k l 20

21 Example CPU a b c d e f g h i j k l Cache RAM a b c d e f g h i j k l 21

22 Spatial and temporal locality Cache memory works well when applications exhibit spatial and/or temporal locality Spatial locality Accessing adjacent memory locations is OK Temporal locality Repeatedly accessing the same memory location(s) is OK 22

23 Example: matrix-matrix product Given two square matrices p, q, compute r = p q j j i i p q r void matmul( double *p, double* q, double *r, int n) { int i, j, k; for (i=0; i<n; i++) { for (j=0; j<n; j++) { r[i*n + j] = 0.0; for (k=0; k<n; k++) { r[i*n + j] += p[i*n + k] * q[k*n + j]; } } } } 23

24 Matrix representation Matrices in C are stored in row-major order Elements of each row are contiguous in memory Adjacent rows are contiguous in memory p[0][0] p[1][0] p[4][4] p[2][0] p[3][0] p[0][0] p[1][0] p[2][0] p[3][0] 24

25 Row-wise vs column-wise access Row-wise access is OK: the data is contiguous in memory, so the cache helps (spatial locality) Column-wise access is NOT OK: the accessed elements are not contiguous in memory (strided access) so the cache does NOT help 25

26 Matrix-matrix product Given two squared matrices p, q, compute r = p q j j i i p OK; rowwise access q r NOT ok; columnwise access 26

27 Optimizing the memory access pattern p00 p01 p02 p03 p10 p11 p12 p13 p20 p21 p22 p23 p30 p31 p32 p33 p q00 q10 q02 q03 q10 q11 q12 q13 q20 q21 q22 q23 q30 q31 q32 q33 q r00 r01 r02 r03 r10 r11 r12 r13 r20 r21 r22 r23 r30 r31 r32 r33 r 27

28 Optimizing the memory access pattern p00 p01 p02 p03 p10 p11 p12 p13 p20 p21 p22 p23 q00 q10 q02 q03 p30 p31 p32 p33 q10 q11 q12 q13 q20 q21 q22 q23 r00 r01 r02 r03 q30 q31 q32 q33 p r10 r11 r12 r13 r20 r21 r22 r23 r30 r31 r32 r33 q r Transpose q p00 p01 p02 p03 p10 p11 p12 p13 p20 p21 p22 p23 p30 p31 p32 p33 p q00 q10 q20 q30 q01 q11 q21 q31 q02 q12 q22 q32 q03 q13 q23 q33 qt r00 r01 r02 r03 r10 r11 r12 r13 r20 r21 r22 r23 r30 r31 r32 r33 r 28

29 But... Transposing q requires time. Is it worth it? See matmul.c./matmul-plain vs./matmul-transpose To measure the cache performance you can use perf On Debian/Ubuntu: sudo apt-get install linux-tools-common linux-tools-generic Which performance counters can be measured by perf is system-dependent The meaning of the performance counters is systemdependent 29

30 Measuring cache stats perf stat -B -e \ task-clock,cycles,cache-references,\ cache-misses,l1-dcache-loads,l1-dcache-load-misses,\ L1-dcache-stores,L1-dcache-store-misses \./exec_name $./cache-stats.sh./matmul-plain Starting plain matrix-matrix multiply (n=1000)... done Low level cache references and misses r[0][0] = elapsed time = s Performance counter stats for './matmul-plain': 3909, task-clock (msec) cycles cache-references cache-misses L1-dcache-loads L1-dcache-load-misses L1-dcache-stores L1-dcache-store-misses # 0,997 CPUs utilized # 3,465 GHz # 32,235 M/sec # 26,401 % of all cache refs # 3597,174 M/sec # 8,03% of all L1-dcache hits # 0,000 K/sec # 0,455 M/sec 3, seconds time elapsed L1 data cache references and misses 30

31 Measuring cache stats perf stat -B -e \ task-clock,cycles,cache-references,\ cache-misses,l1-dcache-loads,l1-dcache-load-misses,\ L1-dcache-stores,L1-dcache-store-misses \./exec_name $./cache-stats.sh./matmul-transpose Starting transpose matrix-matrix multiply (n=1000)... done Low level cache references and misses r[0][0] = elapsed time = s Performance counter stats for './matmul-transpose': 2678, task-clock (msec) cycles cache-references cache-misses L1-dcache-loads L1-dcache-load-misses L1-dcache-stores L1-dcache-store-misses # 0,999 CPUs utilized # 3,461 GHz # 1,234 M/sec # 39,092 % of all cache refs # 5253,280 M/sec # 0,91% of all L1-dcache hits # 0,000 K/sec # 0,732 M/sec 2, seconds time elapsed L1 data cache references and misses 31

32 Instruction-Level Parallelism (ILP) 32

33 Instruction-Level Parallelism Uses multiple functional units to increase the performance of a processor Pipelining: the functional units are organized like an assembly line, and can be used strictly in that order Multiple issue: the functional units can be used whenever required Static multiple issue: the order in which functional units are activated is decided at compile time (example: Intel IA64) Dynamic multiple issue (superscalar): the order in which functional units are activated is decided at run time 33

34 Instruction-Level Parallelism IF ID IF ID EX MEM WB EX Integer WB Pipelining Instruction Fetch Instruction Decode Execute Memory Access Write Back Instruction Fetch and Decode Unit Integer MEM Floating Point Commit Unit Multiple Issue In-order issue Floating Point In-order commit Load Store Out of order execute 34

35 Pipelining Instr1 IF ID EX MEM WB Instr2 Instr1 IF ID EX MEM WB Instr3 Instr2 Instr1 IF ID EX MEM WB Instr4 Instr3 Instr2 Instr1 IF ID EX MEM WB Instr5 Instr4 Instr3 Instr2 Instr1 IF ID EX MEM WB 35 Flusso di istruzioni

36 Control Dependency z = x + y; if ( z w } else w } > 0 ) { = x ; { = y ; The instructions w = x and w = y have a control dependency on z > 0 Control dependencies can limit the performance of pipelined architectures z = x + y; c = z > 0; w = x*c + y*(1-c); Don't do this by hand; ideally, compilers should take care of these optimizations 36

37 In the real world... From GCC documentation Built-in Function: long builtin_expect(long exp, long c) You may use builtin_expect to provide the compiler with branch prediction information. In general, you should prefer to use actual profile feedback for this (-fprofile-arcs), as programmers are notoriously bad at predicting how their programs actually perform. However, there are applications in which this data is hard to collect. The return value is the value of exp, which should be an integral expression. The semantics of the built-in are that it is expected that exp == c. For example: if ( builtin_expect (x, 0)) foo (); 37

$h> int main( void ) { int A[1000000]; size_t i; const size_t n = sizeof(a) / sizeof(a[0]); for ( i=0; builtin_expect( i<n, 1 ); i++ ) { A[i]$

38 Branch Hint: Example #include <stdlib.h> int main( void ) { int A[ ]; size_t i; const size_t n = sizeof(a) / sizeof(a[0]); for ( i=0; builtin_expect( i<n, 1 ); i++ ) { A[i] = i; } return 0; } Don't do this by hand; ideally, compilers should take care of these optimizations 38

39 Hardware multithreading Allows the CPU to switch to another task when the current task is stalled Fine-grained multithreading A context switch has essentially zero cost The CPU can switch to another task even on stalls of short durations (e.g., waiting for memory operations) Requires CPU with specific support, e.g., Cray XMT Coarse-grained multithreading A context switch has non-negligible cost, and is appropriate only for threads blocked on I/O operations or similar The CPU is less efficient in the presence of stalls of short duration 39

40 Source: 40

41 Hardware multithreading Simultaneous multithreading (SMT) is an implementation of fine-grained multithreading where different threads can use multiple functional units at the same time HyperThreading is Intel's implementation of SMT Each physical processor core is seen by the Operating System as two "logical" processors Each logical processor maintains a complete set of the architecture state: general-purpose registers control registers advanced programmable interrupt controller (APIC) registers some machine state registers Intel claims that HT provides a 15 30% speedup with respect to a similar, non-ht processor 41

42 HyperThreading Queue Queue MEM WB Queue EX Queue ID Queue IF Queue Queue The pipeline stages are separated by two buffers (one for each executing thread) If one thread is stalled, the other one might go ahead and fill the pipeline slot, resulting in higher processor utilization Queue Hyper-Threading Technology Architecture and Microarchitecture 42

43 HyperThreading See the output of lscpu ("Thread(s) per core") or lstopo (hwloc Ubuntu/Debian package) Processor without HT Processor with HT 43

44 44

45 Von Neumann architecture Flynn's Taxonomy Single Multiple Instruction Streams Data Streams Single Multiple SISD SIMD Single Instruction Stream Single Data Stream Single Instruction Stream Multiple Data Streams MISD MIMD Multiple Instruction Streams Single Data Stream Multiple Instruction Streams Multiple Data Streams 45

46 SIMD SIMD instructions apply the same operation (e.g., sum, product ) to multiple elements (typically 4 or 8, depending on the width of SIMD registers and the data types of operands) Time This means that there must be 4/8/... independent ALUs LOAD A[0] LOAD A[1] LOAD A[2] LOAD A[3] LOAD B[0] LOAD B[1] LOAD B[2] LOAD B[3] C[0] = A[0] + B[0] C[1] = A[1] + B[1] C[2] = A[2] + B[2] C[3] = A[3] + B[3] STORE C[0] STORE C[1] STORE C[2] STORE C[3] 46

47 SSE Streaming SIMD Extensions Extension to the x86 instruction set Provide new SIMD instructions operating on small arrays of integer or floating-point numbers Introduces 8 new 128-bit registers (XMM0 XMM7) SSE2 instructions can handle 2 64-bit doubles, or 2 64-bit integers, or 4 32-bit integers, or 8 16-bit integers, or 16 8-bit chars bit XMM0 XMM1 XMM7 47

48 SSE (Streaming SIMD Extensions) 4 lanes 32 bit 32 bit 32 bit 32 bit X3 X2 X1 X0 Y3 Y2 Y1 Y0 Op Op Op Op X3 op Y3 X2 op Y2 X1 op Y1 X0 op Y0 48

49 Example m128 a = _mm_set_ps( 1.0, 2.0, 3.0, 4.0 ); m128 b = _mm_set_ps( 2.0, 4.0, 6.0, 8.0 ); m128 ab = _mm_mul_ps(a, b); 32 bit 32 bit 32 bit 32 bit a b _mm_mul_ps(a,b) ab

50 GPU Modern GPUs (Graphics Processing Units) have a large number of cores, and can be regarded as a form of SIMD architecture Fermi GPU die (source: 50

51 CPU vs GPU The difference between CPUs and GPUs can be appreciated by looking at how the chip surface is used ALU ALU ALU ALU Control Cache DRAM controller DRAM controller CPU GPU 51

52 GPU core A single CPU core contains a fetch/decode unit shared among multiple ALUs ALU ALU ALU ALU ALU ALU ALU ALU Ctx Ctx Ctx Ctx Ctx Ctx Ctx Ctx Ctx Ctx Ctx Ctx If there are 8 ALU, each instruction can operate on 8 values simultaneously Each GPU core maintains multiple execution contexts, and can switch between them at virtually zero cost Fetch / Decode Fine-grained parallelism 52

53 GPU Example: 12 instruction streams 8 ALU = 96 operations in parallel 53

54 MIMD In MIMD systems there are multiple execution units that can execute multiple sequences of instructions Each execution unit generally operates on its own input data Time Multiple Instruction Streams Multiple Data Streams LOAD A[0] CALL F() a = 18 w=7 LOAD B[0] z=8 b=9 t = 13 C[0] = A[0] + B[0] y = 1.7 if ( a>b ) c = 7 k = G(w,t) STORE C[0] z=x+y a=a-1 k=k+1 54

55 MIMD architectures Shared Memory CPU A set of processors sharing a common memory space Each processor can access any memory location CPU A set of compute nodes connected through an interconnection network The most simple example: cluster of PCs connected via Ethernet Nodes can share data through explicit communications CPU Interconnect Memory Distributed Memory CPU CPU CPU CPU CPU Mem Mem Mem Mem Interconnect 55

56 AMD Ryzen Intel core i7 56

57 Hybrid architectures Many HPC systems are based on hybrid architectures Each compute node is a shared-memory multiprocessor A large number of compute nodes is connected through an interconnect network GPU GPU GPU GPU GPU GPU GPU GPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU Mem Mem Mem Mem Interconnect 57

Distributed memory system IBM BlueGene / Q @ CINECA Architecture 10 BGQ Frame Model IBM-BG/Q Processor Type IBM PowerA2, 1.

58 Distributed memory system IBM BlueGene / CINECA Architecture 10 BGQ Frame Model IBM-BG/Q Processor Type IBM PowerA2, 1.6 GHz Computing Cores Computing Nodes RAM 1GByte / core Internal Network 5D Torus Disk Space 2PByte scratch space Peak Performance 2PFlop/s 58

59 59

60 60

61 (June 2018) 61

62 (June 2018) 62

63 SANDIA ASCI RED Date: 1996 Peak performance: 1.8Teraflops Floor space: 150m2 Power consumption: Watt 63

000 Watt Sony PLAYSTATION 3 Date: 2006 Peak

64 SANDIA ASCI RED Date: 1996 Peak performance: 1.8Teraflops Floor space: 150m2 Power consumption: Watt Sony PLAYSTATION 3 Date: 2006 Peak performance: >1.8Teraflops Floor space: 0.08m2 Power consumption: <200 Watt 64

65 Inside SONY's PS3 Cell Broadband Engine 65

66 Empirical rules When writing parallel applications (especially on distributed-memory architectures) keep in mind that: Computation is fast Communication is slow Input/output is incredibly slow 66

67 Recap Shared memory Advantages: Easier to program Useful for applications with irregular data access patterns (e.g., graph algorithms) Distributed memory Advantages: Disadvantages: The programmer must take care of race conditions Limited memory bandwidth Highly scalable, provide very high computational power by adding more nodes Useful for applications with strong locality of reference, with high computation / communication ratio Disadvantages: Latency of interconnect network Difficult to program 67

Parallel Computing Architectures

Parallel Computing Architectures Moreno Marzolla Dip. di Informatica Scienza e Ingegneria (DISI) Università di Bologna http://www.moreno.marzolla.name/ 2 An Abstract Parallel Architecture Processor Processor