CAMA: Modern processors. Memory hierarchy: Caches. Gerhard Wellein, Department for Computer Science and Erlangen Regional Computing Center

Transcription

1 AMA: Modern processors Memory hierarchy: aches Gerhard Wellein, Department for omputer Science and Erlangen Regional omputing enter Johannes Hofmann/Dietmar Fey, Department for omputer Science University Erlangen-Nürnberg, Sommersemester 2015

2 Schematic view of modern memory hierarchy & cache logic U/Arithmetic unit issues a LOAD request to transfer a data item to a register ache logic automatically checks all cache levels if data item is already in cache GB/s If data item is in cache ( cache hit ) it is loaded to register GB/s If data item is in no cache level ( cache miss ) data item is loaded from main memory and a copy is held in cache. AMA D. Fey and G. Wellein 2

3 Memory hierarchies: Latency problem & ache line Two quantities characterize the quality of each memory hierarchy: Latency (T lat ): Time to set up the memory transfer from source (main memory or caches) to destination (registers). Bandwidth (BW): Maximum amount of data which can be transferred per second between source (main memory or caches) and destination (registers). Transfer time: T = T lat + (amount of data) / BW Effective bandwidth: BW eff = (amount of data) / T Low amount of data: Large amount of data: BW eff << BW BW eff ~ BW AMA D. Fey and G. Wellein 3

4 Memory hierarchies: Latency problem & ache line ~ Typical values for modern microprocessors: = T lat =100 ns - BW=4 GB/s amount of data=8 Byte (double) T=102 ns (100 ns from latency!) Data transfer rate: 8 B / 102 ns = B/ns = GB/s Data access is organized in cache lines (L) that are transferred as a whole e.g. amount of data = 128 Byte (16 doubles) T=132 ns (100 ns from latency) Data transfer rate: 128 B / 132 ns = 0.97 B/ns = 0.97 GB/s Data transfers between memory and cache as well as between caches always happens at the L granularity! Still not sufficient to hide most of the memory latency Multiple non-blocking cache line transfers are supported Automatic hardware prefetcher AMA D. Fey and G. Wellein 4

5 Memory hierarchies: ache lines If one item is loaded from main memory ( cache miss ), whole cache line it belongs to is loaded to the caches ache lines are contiguous in main memory, i.e. neighboring items can then be used from cache Iteration LD ache miss : Latency Use data ache line size: 4 words LD LD Use data LD Use data LD Use data t ache miss : Latency do i=1,n s = s + a(i)*a(i) enddo LD Use data LD Use data Use AMA D. Fey and G. Wellein 5

6 Memory Hierarchies: ache line Spatial locality ache line addresses latency problem not bandwidth bottleneck ache line use is optimal for contiguous access ( stride 1 ) STREAMING Non-consecutive reduces performance; Access with wrong stride (e.g. with cache line size) can lead to disastrous performance breakdown Typical L sizes: 64 Byte or 128 Byte alculations get cache bandwidth inside the cache line, but main memory bandwidth still limits the speed of the cache line transfer Spatial locality : Ensure accesses to neighboring data items GOOD ( Streaming ) do i=1,n s = s + a(i)*a(i) enddo BAD ( Strided ) do i=1,n,2 s = s + a(i)*a(i) enddo If a(1:n) is loaded from main memory: same runtime! erformance of strided loop is half of the continuous one AMA D. Fey and G. Wellein 6

7 Memory Hierarchies: ache size Temporal locality If cache is full an old data items need to be removed if new data items come in ache lines wear out Age of cache line Last access time (Remember cache replacement strategies: LRU, ) Efficient use of caches requires some locality of reference, i.e. a data item loaded to cache needs to be reused several times soon ( Temporal locality ) before it gets old. Assume large N A(1:N)=B(1:N)+Z(1:N) (1:N)=(1:N)*Z(1:N) E(1:N)=Z(1:N)+A(1:N)*(1:N) DO I = 1,N A(I) = B(I)+Z(I) (I) = (I)*Z(I) E(I) = Z(I)+A(I)*(I) ENDDO AMA D. Fey and G. Wellein 7

8 Memory Hierarchies: Data Layout & contiguous access How to traverse multidimensional arrays?! Example: Initialize matrix A with A(i,j) = i*j What is the storage order of multidimensional-data structure? It depends, e.g. 2-dimensional 3x3 array A of doubles FORTRAN: column by column ( column major order ) 0 B Memory layout 71 B A(1,1) A(2,1) A(3,1) A(1,2) A(2,2) A(3,2) A(1,3) A(2,3) A(3,3) /++: row by row ( row major order ) 0 B Memory layout 71 B A[0][0] A[0][1] A[0][2] A[1][0] A[1][1] A[1][2] A[2][0] A[2][1] A[2][2] AMA D. Fey and G. Wellein 8

9 Memory Hierarchies: Data Layout & contiguous access Default layout for FORTRAN: column by column (column major order) do i=1,n do j=1,n a(j,i)=i*j enddo enddo ontinuous access! do j=1,n do i=1,n a(j,i)=i*j enddo enddo Stride n access! FORTRAN: Inner loop must access innermost/left array index Data arrangement is transpose of the usual matrix layout AMA D. Fey and G. Wellein 9

10 Memory Hierarchies: Data Layout & contiguous access Default layout for /++: row by row (row major order) for(i=0; i<n; ++i) { for(j=0; j<n; ++j) { a[i][j] = i*j; } } ontinuous access! for(j=0; j<n; ++j) { for(i=0; i<n; ++i) { a[i][j] = i*j; } } Stride N access! In : Inner loop must access outermost/rightmost array index AMA D. Fey and G. Wellein 10

11 Memory Hierarchies: Data Layout & contiguous access 3-dimensional arrays in /++ for(i=0; i<n; ++i) { for(j=0; j<n; ++j) { for(k=0; k<n; ++k) { a[i][j][k] = i*j*k; } } } ontinuous access! for(k=0; k<n; ++k){ for(j=0; j<n; ++j) { for(i=0; i<n; ++i) { a[i][j][k] = i*j*k; } } } Stride N*N access! /++: Always start with rightmost index as inner loop index if possible! Sometimes there are problems. (spatial blocking may improve the situation here cf. later) for(i=0; i<n; ++i) { for(j=0; j<n; ++j) { a[i][j] = b[j][i]; } } AMA D. Fey and G. Wellein 11

12 Memory Hierarchies: ache Mapping (see D. Fey) ache Mapping airing of memory locations with cache locations e.g. mapping 1 GB of main memory to 1 MB of cache Memory (10 9 Byte) ache line x ache line y ache (10 6 Byte) Static Mapping Directly Mapped caches vs. m-way set associative caches Mapping substantially impacts the flexibility of replacement strategies Reduces the potential set of evict/replace locations May incur additional data transfer ( cache thrashing)! AMA D. Fey and G. Wellein 12

13 Memory Hierarchies: ache Mapping Directly mapped (see D. Fey) Directly mapped cache: Every memory location can only be mapped to exactly one cache location If cache size=n, i-th memory location can be stored at cache location mod(i,n) Easy to implementation & fast lookup, e.g. Mapping of 1 MB to 1 KB Memory Address (20 Bit) ache Address (10 Bit) No penalty for stride-one access Memory access with stride=cache size will not allow caching of more than one line of data, i.e. effective cache size is one line! AMA D. Fey and G. Wellein 13

14 Memory Hierarchies: ache Mapping Directly Mapped (see D. Fey) N-1 N N+1 N N-1 Memory... ache Example: Directly mapped cache. Each memory location can be mapped to one cache location only. E.g. Size of main memory= 1 GByte; ache Size= 256 KB 4096 memory locations are mapped to the same cache location AMA D. Fey and G. Wellein 14

15 Memory Hierarchies: ache Mapping Associative aches Set-associative cache: m-way associative cache of size m x n: each memory location i can be mapped to the m cache locations j*n+mod(i,n), j=0..m-1 E.g.: 2-way set associative cache of size 256 KBytes: Way KB 128KB+1 acheline 256 KB Set Address within Number of sets: 256 KB/ 64 Byte /2 = 2048 cache line Memory address (32 Bit): Ideal world: Fully associative cache where every memory location is mapped to any cache line Thrashing nearly impossible The higher the associativity, the larger the overhead, e.g. latencies increase AMA D. Fey and G. Wellein 15

16 Memory hierarchies: ache Mapping Associative aches (see D. Fey) N-1 N N+1 N N-1 Memory... ache Example: 2-way associative cache. Each memory location can be mapped to two cache locations ( ways ) within the same set: E.g. Size of main memory= 1 GByte; ache Size= 256 KB 8192 memory locations are mapped to two cache locations AMA D. Fey and G. Wellein 16

17 Memory hierarchies: itfalls & roblems If many memory locations are used that are mapped to the same set, cache reuse can be very limited even with m-way associative caches Warning: Using powers of 2 in the leading array dimensions of multi-dimensional arrays should be avoided! (ache Thrashing) double precision A(16384,16384) A(1,1) A(1,2) A(1,3) If cache / m-ways are full and new data comes in from main memory, data in cache (full cache line) must be invalidated or written back to main memory Ensure spatial and temporal data locality for data access! AMA D. Fey and G. Wellein 17

18 Memory hierarchies: ache thrashing - Example Example: 2D square lattice At each lattice point the 4 velocities for each of the 4 directions are stored N=16 real*8 vel(1:n, 1:N, 4) s=0.d0 do j=1,n do i=1,n s=s+vel(i,j,1)-vel(i,j,2)+vel(i,j,3)-vel(i,j,4) enddo enddo AMA D. Fey and G. Wellein 18

19 Memory hierarchies: ache thrashing - Example Memory to cache mapping for vel(1:16, 1:16, 4) ache: 256 byte (=32 double) / 2-way associative / ache line size=32 byte 1,1,1 2,1,1 3,1,1 4,1,1. 1,1,2 2,1,2 3,1,2 4,1,2. 1,1,3 2,1,3 3,1,3 4,1,3. 1,1,4 2,1,4 3,1,4 4,1,4 i=1, j=1 1,1,1 2,1,1 3,1,1 4,1,1. 1,1,2 2,1,2 3,1,2 4,1,2. 1,1,3 2,1,3 3,1,3 4,1,3. 1,1,4 2,1,4 3,1,4 4,1,4. Vel(1:16,1:16,1) Vel(1:16,1:16,2) Vel(1:16,1:16,3) Vel(1:16,1:16,4) ache: 2 ways 1,1,1 1,1,3 2,1,1 2,1,3 3,1,1 3,1,3 4,1,1 4,1,3 1,1,2 1,1,4 2,1,2 2,1,4 3,1,2 3,1,4 4,1,2 4,1,4 with 16 double each Each cache line must be loaded 4 times from main memory to cache! AMA D. Fey and G. Wellein 19

20 Memory hierarchies: ache thrashing - Example Memory to cache mapping for vel(1:16+2, 1:16+2, 4) ache: 256 byte (=32 doubles) / 2-way associative / ache line size=32 byte 1,1,1 2,1,1 3,1,1 4,1,1. 1,1,2 2,1,2 3,1,2 4,1,2. 1,1,3 2,1,3 3,1,3 4,1,3. 1,1,4 2,1,4 3,1,4 4,1,4 i=1, j=1 ache: 2 way 1,1,1 2,1,1 3,1,1 4,1,1 15,18,1 16,18,1 17,18,1 18,18,1 1,1,1 2,1,1 3,1,1 4,1,1 1,1,2 2,1,2 3,1,2 4,1,2. 1,1,3 2,1,3. 3,1,3 4,1,3.. 1,1,4 2,1,4. 3,1,4 4,1,4 1,1,2 2,1,2 3,1,2 4,1,2 1,1,3 2,1,3 3,1,3 4,1,3 1,1,4 2,1,4 3,1,4 4,1,4.. with 16 doubles each Each cache line needs only be loaded once from memory to cache! AMA D. Fey and G. Wellein 20

21 Memory hierarchies: ache management details ache misses: LOAD misses: If data item (e.g. a[2]) to be loaded to a register is not available in cache, the full cache line (e.g. a[0:7]) holding the data item is loaded from main memory to cache. STORE miss: Data item to be modified (e.g. a[2]=0.0) is not in cache? One cache line is the minimum data transfer unit between main memory and cache (e.g. a[0:7]). Load cache line from main memory to cache ( WRITE ALLOATE ) Modify data item in cache Later evict/write back full cache line to main memory ( store to main memory) Overall data transfer volume increases up to 2x! (NT stores: no increase) do i=1,n do j=1,n a(j,i)= 0.0 enddo enddo n 2 words are loaded from main memory to cache (WRITE ALLOATE) and n 2 words are evicted/written back to main memory! AMA D. Fey and G. Wellein 21

22 Memory hierarchies: ache management details How does data travel from memory to the U and back? Example: Array copy A(:)=(:) Special store instruction to avoid WA! LD (1) MISS U registers ST A(1) MISS LD (2..N cl ) ST A(2..N cl ) HIT LD (1) MISS U registers NTST A(1) LD (2..N cl ) NTST A(2..N cl ) HIT L ache L L ache write allocate evict (delayed) L L L L 3 L (:) A(:) (:) A(:) Memory transfers Standard stores (WRITE ALLOATE) Memory Nontemporal (NT) stores 2 L transfers 50% performanc e boost for OY AMA D. Fey and G. Wellein 22

23 Memory management: aches management details (see D. Fey) Inclusive: ache line copy in all levels Reduced effective size in outer cache levels heap eviction for unmodified cache lines Higher latency: cache lines have to load through hierarchy All Intel processors Exclusive: Only one cacheline copy in cache hierarchy Full aggregate effective cache size Eviction is expensive (copy back) Lower latency: Data can be directly loaded in L1 cache All AMD processors Write back : A modified cache line is evicted to the next (lower) cache/memory level before it is overwritten by new data Write through : When a cache line is updated then the cache line copy in the next (lower) cache/memory level is updated as well AMA D. Fey and G. Wellein 23

24 Memory Hierarchies: Intel vs. AMD (current generations) Single core specs eak perf. lock freq. Intel Xeon Sandy Bridge 21.6 GFlop/s 2.7 GHz AMD Opteron Interlagos 22.4 GFlop/s 2.8 GHz Q1/2012: Intel Sandy Bridge: 8 cores AMD Interlagos: 16 cores # F Registers 16/32 16/32 Size 32 KB 16 KB L1 D BW ~130 GB/s ~90 GB/s Latency 4 cycles 4 cycles Size 256 KB 2 MB (2 cores) L2 BW ~90 GB/s ~90 GB/s L3 Mem. Latency 12 cycles >20 cycles Size 20 MB (shared) 8 MB (8 cores) BW ~300 GB/s ~40 GB/s Latency ~30 cycles 48 (?) cyles Socket BW ~36 GB/s (measured) ~32 GB/s (measured) Latency ~150 cycles ~150 cycles ache Associativity Intel AMD L1 8 4 L L AMA D. Fey and G. Wellein 24

25 haracterization of Memory Hierarchies Determine performance levels with low level benchmark: Vector Triad DOUBLE REISION, dimension(size):: A,B,,D DOUBLE REISION :: S,E,MFLOS! Input N.le. SIZE DO i=1,n A(i) = 0.d0; B(i)=1.d0; (i)=2.d0; D(i)=3.d0! initialize ENDDO call get_walltime(s) DO ITER=1, NITER DO i=1, N A(i) = B(i) + (i) * D(i)! 3 loads + 1 store; 2 FLO ENDDO IF(A(2).lt.0) call dummy(a,b,,d)! revent loop interchange ENDDO call get_walltime(e) MFLOS = NITER * N * 2.d0 /( (E-S) * 10 6 ) AMA D. Fey and G. Wellein 25

26 Memory Hierarchies: Measure performance levels Vector Triad single core performance: A[1:N]=B[1:N]+[1:N]*D[1:N] an we explain performance based on hardware features? L1 cache L2 cache L3 cache AMA D. Fey and G. Wellein 26

27 AMA: Multicore processors There is no way back Modern multi-/manycore chips Basic ompute architecture Gerhard Wellein, Department for omputer Science and Erlangen Regional omputing enter Dietmar Fey, Department for omputer Science University Erlangen-Nürnberg, Sommersemester 2013

28 Moore s law continues NVIDIA Fermi: ~3.0 billion Intel SNB E: ~2.2 billion Intel orp Electronics Magazine, April 1965: The complexity for minimum component costs has increased at a rate of roughly a factor of two per year ertainly over the short term this rate can be expected to continue, if not to increase. AMA D. Fey and G. Wellein 28

29 but the free lunch is over Moore s law run smaller transistors faster Faster clock speed Higher Throughput (Instructions/s) for free Frequency [MHz] Intel x86 processor clock speed Single core: Instruction level parallelism: Superscalarity Single Instruction Multiple Data (SIMD) SSE / AVX 10 Investing the transistor budget: 1 Multi-ore/Threading 0, Year omplex on chip caches New on-chip functionalities (GU, Ie, ) AMA D. Fey and G. Wellein 29

30 ower consumption the root of all evil By courtesy of D. Vrsalovic, Intel N transistors Dual-ore 1.73x erformance ower 1.13x 1.00x 2N transistors 1.73x 1.02x ower envelope: Max W ower consumption: = f * (V core ) 2 V core ~ V Over-clocked (+20%) Max Frequency Dual-core (-20%) Same process technology: ~ f 3 AMA D. Fey and G. Wellein 30

31 Modern multi- and manycore chips Intel Sandy Bridge AMD Interlagos/Bulldozer NVIDIA GK110 / K20 Intel Xeon hi Be prepared for more cores with less complexity and slower clock!

32 The x86 multicore evolution so far Intel Single-Dual-/Quad-/Hexa-/Octo-ores (single socket view) hipset Memory 2005: Fake dual-core hipset Memory Woodcrest ore2 Duo 65nm 2006: True dual-core hipset Memory Other socket hipset Memory Other socket Harpertown ore2 Quad 45nm 2008: Simultaneous MultiThreading (SMT) T0 T1 T0 T1 T0 T1 T0 T1 T0 T1 2010: 6-core chip T0 T1 T0 T1 T0 T1 T0 T1 T0 T1 2012: Wider SIMD units AVX: 256 Bit T0 T1 T0 T1 T0 T1 T0 T1 T0 T1 T0 T1 T0 T1 T0 T1 MI Other socket MI Other socket MI Memory Memory Memory Nehalem E ore i7 45nm Westmere E ore i7 32nm Sandy Bridge E ore i7 32nm AMA D. Fey and G. Wellein 32

33 There is no longer a single driving force for chip performance! Floating oint (F) erformance: = n core * F * S * ν n core number of cores: 8 F F instructions per cycle: 2 (1 MULT and 1 ADD) Intel Xeon E ( Sandy Bridge ) (4,6 core variants also available) S F ops / instruction: 4 (dp) / 8 (sp) (256 Bit SIMD registers AVX ) ν lock speed : 2.5 GHz TO = 160 GF/s (dp) / 320 GF/s (sp) But: =5.0 GF/s (dp) for serial, non-vectorized code AMA D. Fey and G. Wellein 33

34 omplex socket topologies: AMD Interlagos / Bulldozer Up to 16 cores (8 modules 2.6 GHz Each module: 2 lightweight cores FU: 4 MULT & 4 ADD /cycle (dp) 16 kb dedicated L1D cache 2 MB shared L2 cache 6 MB shared L3 cache = GF/s = GF/s (dp) 2 2 DDR channels: GB/s ccnuma: 2 NUMA domains per socket AMA D. Fey and G. Wellein 34

35 NVIDIA Kepler GK110 Block Diagram (GGU) Architecture 7.1B Transistors 15 big cores 15 SMX units with 192 (sp) units each 192 single precision ops/ instruction block > 1 TFLO D peak 1.5 MB L2 ache 3:1 S:D performance NVIDIA orp. Used with permission. AMA D. Fey and G. Wellein

36 Intel Xeon hi block diagram Architecture 3B Transistors 16 single precision ops/instruction 60+ cores with 512 bit SIMD unit each 1 TFLO D peak 0.5 MB L2/core GDDR5 2:1 S:D performance 64 byte/cy AMA D. Fey and G. Wellein

37 omparing accelerators Intel Xeon hi 60+ IA32 cores each with 512 Bit SIMD FMA unit 480/960 SIMD D/S tracks lock Speed: ~1000 MHz Transistor count: ~3 B (22nm) ower consumption: ~250 W eak erformance (D): ~ 1 TF/s Memory BW: ~250 GB/s (GDDR5) Threads to execute: rogramming: Fortran//++ +OpenM + SIMD NVIDIA Kepler K20 15 SMX units each with 192 cores 960/2880 D/S cores lock Speed: ~700 MHz Transistor count: 7.1 B (28nm) ower consumption: ~250 W eak erformance (D): ~ 1.3 TF/s Memory BW: ~ 250 GB/s (GDDR5) Threads to execute: 10,000+ rogramming: UDA, OpenL, (OpenA) AMA D. Fey and G. Wellein

38 Trading single thread performance for parallelism: GGUs vs. Us GU vs. U light speed estimate: 1. ompute bound: 2-10x 2. Memory Bandwidth: 1-5x Intel ore i ( Sandy Bridge ) Intel Xeon E D node ( Sandy Bridge ) NVIDIA K20x ( Kepler ) ores@lock 3.3 GHz 2 x 2.7 GHz 0.7 GHz erformance + /core 52.8 GFlop/s 43.2 GFlop/s 1.4 GFlop/s Threads@STREAM <4 <16 >8000 Total performance GFlop/s 691 GFlop/s 4,000 GFlop/s Stream BW 18 GB/s 2 x 40 GB/s 168 GB/s (E=1) Transistors / TD 1 Billion* / 95 W 2 x (2.27 Billion/130W) 7.1 Billion/250W + Single recision * Includes on-chip GU and I-Express omplete compute device AMA D. Fey and G. Wellein

39 Basic compute node architecture From UMA to ccnuma

40 Single hip is not enough! Basic architecture of shared memory compute nodes Hardware/software layers (HT/QI): Shared address space and ensure data coherency A(1: ) Separate memory controllers scalable performance Single shared address space ease of use ache-coherent Non-Uniform Memory Architecture (ccnuma) HT / QI :scalable bandwidth at the price of ccnuma: Where does my data finally end up? AMA D. Fey and G. Wellein

41 There is no longer a single flat memory: From UMA to ccnuma 2-way nodes Yesterday: Dual-socket Intel ore2 node: Uniform Memory Architecture (UMA): Flat memory ; symmetric Ms But: system anisotropy Shared Address Space within the node! Today: Dual-socket Intel (Westmere) node: ache-coherent Non-Uniform Memory Architecture (ccnuma) HT / QI provide scalable bandwidth at the expense of ccnuma architectures: Where does my data finally end up? On AMD it is even more complicated ccnuma within a chip! AMA D. Fey and G. Wellein 41

42 arallel computers Shared-Memory Architectures Basic lassification Shared memory computers provide a single shared address space (memory) for all processors All processors share the same view of the address space! U U Shared Memory Two basic categories of shared memory systems U U Uniform Memory Access (UMA): Memory is equally accessible to all processors with the same performance (Bandwidth & Latency) cache-coherent Non Uniform Memory Access (ccnuma): Memory is physically distributed but appears as a single address space: erformance (Bandwidth & Latency) is different for local and remote memory access opies of the same cache line may reside in different caches ache coherence protocols guarantees consistency all time (for UMA & ccnuma) ache coherence protocols do not alleviate parallel programming for shared-memory architectures! AMA D. Fey and G. Wellein 42

43 arallel computers: Shared-memory: UMA UMA Architecture: switch/bus arbitrates memory access Special protocol ensures cross-u cache data consistency Flat memory also known as Symmetric Multi-rocessor (SM) U 1 U 2 U 3 U 4 ache ache ache ache... Switch/Bus... Memory AMA D. Fey and G. Wellein 43

44 arallel shared memory computers: ccnuma/node Layout ccnuma: Single address space although physically distributed memory through proprietary hardware concepts (e.g. NUMALink in SGI systems; QI for Intel; HT for AMD) Advantages: Aggregate memory bandwidth is scalable Systems with more 1024 cores are available (SGI) Disadvantages: ache oherence hard to implement / expensive erformance depends on access to local or remote memory Examples: All modern multi-socket compute nodes SGI Altix/UV Memory Memory Memory Memory AMA D. Fey and G. Wellein 44

45 Basic challenges for shared memory architectures ache coherence for UMA & ccnuma! Multiple copies of the same cache line in multiple caches how to keep them coherent? ccnuma: Data Locality M M M M "Golden Rule" of ccnuma: A memory page gets mapped into the local memory of the processor that first touches it! AMA D. Fey and G. Wellein 45