CS 240 Stage 3 Abstractions for Practical Systems

Transcription

1 CS 240 Stage 3 Abstractions for Practical Systems Caching and the memory hierarchy Operating systems and the process model Virtual memory Dynamic memory allocation Victory lap

2 Memory Hierarchy: Cache Memory hierarchy Cache basics Locality Cache organization Cache-aware programming

3 How does execution time grow with SIZE? int array[size]; fillarrayrandomly(array); int s = 0; for (int i = 0; i < ; i++) { for (int j = 0; j < SIZE; j++) { s += array[j]; } } TIME SIZE 4

4 reality Time SIZE 5

5 Processor-Memory Bottleneck Processor performance doubled about every 18 months CPU Reg Cache Bus bandwidth evolved much slower Main Memory Bandwidth: 256 bytes/cycle Latency: 1-few cycles Example Bandwidth: 2 Bytes/cycle Latency: 100 cycles Solution: caches 6

6 Cache English: n. a hidden storage space for provisions, weapons, or treasures v. to store away in hiding for future use Computer Science: n. a computer memory with short access time used to store frequently or recently used instructions or data v. to store [data/instructions] temporarily for later quick retrieval Also used more broadly in CS: software caches, file caches, etc. 7

7 General Cache Mechanics CPU Block: unit of data in cache and memory. (a.k.a. line) Cache Data is moved in block units Smaller, faster, more expensive. Stores subset of memory blocks. (lines) Memory Larger, slower, cheaper. Partitioned into blocks (lines). 8

8 Cache Hit CPU Cache Request: Request data in block b. 2. Cache hit: Block b is in cache. Memory

9 Cache Miss CPU Cache Request: Request data in block b. 2. Cache miss: block is not in cache 12 9 Request: Cache eviction: Evict a block to make room, maybe store to memory. Memory Cache fill: Fetch block from memory, store in cache. Placement Policy: where to put block in cache Replacement Policy: which block to evict 10

10 Locality: why caches work Programs tend to use data and instructions at addresses near or equal to those they have used recently. Temporal locality: Recently referenced items are likely to be referenced again in the near future. Spatial locality: Items with nearby addresses are likely to be referenced close together in time. block block How do caches exploit temporal and spatial locality? 11

11 Locality #1 sum = 0; for (i = 0; i < n; i++) { sum += a[i]; } return sum; What is stored in memory? Data: Temporal: sum referenced in each iteration Spatial: array a[] accessed in stride-1 pattern Instructions: Temporal: execute loop repeatedly Spatial: execute instructions in sequence Assessing locality in code is an important programming skill. 12

12 Locality #2 row-major M x N 2D array in C int sum_array_rows(int a[m][n]) { int sum = 0; } for (int i = 0; i < M; i++) { for (int j = 0; j < N; j++) { sum += a[i][j]; } } return sum; a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] a[2][0] a[2][1] a[2][2] a[2][3] 1: a[0][0] 2: a[0][1] 3: a[0][2] 4: a[0][3] 5: a[1][0] 6: a[1][1] 7: a[1][2] 8: a[1][3] 9: a[2][0] 10: a[2][1] 11: a[2][2] 12: a[2][3] stride 1 13

13 Locality #3 row-major M x N 2D array in C int sum_array_cols(int a[m][n]) { int sum = 0; } for (int j = 0; j < N; j++) { for (int i = 0; i < M; i++) { sum += a[i][j]; } } return sum; a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] a[2][0] a[2][1] a[2][2] a[2][3] 1: a[0][0] 2: a[1][0] 3: a[2][0] 4: a[0][1] 5: a[1][1] 6: a[2][1] 7: a[0][2] 8: a[1][2] 9: a[2][2] 10: a[0][3] 11: a[1][3] 12: a[2][3] stride N 14

14 Locality #4 int sum_array_3d(int a[m][n][n]) { int sum = 0; } for (int i = 0; i < N; i++) { for (int j = 0; j < N; j++) { for (int k = 0; k < M; k++) { sum += a[k][i][j]; } } } return sum; What is "wrong" with this code? How can it be fixed? 15

15 Cost of Cache Misses Miss cost could be 100 hit cost. 99% hits could be twice as good as 97%. How? Assume cache hit time of 1 cycle, miss penalty of 100 cycles Mean access time: 97% hits: 1 cycle * 100 cycles = 4 cycles 99% hits: 1 cycle * 100 cycles = 2 cycles hit/miss rates 16

16 Cache Performance Metrics Miss Rate Fraction of memory accesses to data not in cache (misses / accesses) Typically: 3% - 10% for L1; maybe < 1% for L2, depending on size, etc. Hit Time Time to find and deliver a block in the cache to the processor. Typically: 1-2 clock cycles for L1; 5-20 clock cycles for L2 Miss Penalty Additional time required on cache miss = main memory access time Typically cycles for L2 (trend: increasing!) 17

17 memory hierarchy why does it work? small, fast, power-hungry, expensive registers explicitly programcontrolled L1 cache (SRAM, on-chip) L2 cache (SRAM, on-chip) Memory L3 cache (SRAM, off-chip) large, slow, power-efficient, cheap main memory (DRAM) persistent storage (hard disk, flash, over network, cloud, etc.)

18 Cache Organization: Key Points Block Fixed-size unit of data in memory/cache Placement Policy Where in the cache should a given block be stored? direct-mapped, set associative Replacement Policy What if there is no room in the cache for requested data? least recently used, most recently used Write Policy When should writes update lower levels of memory hierarchy? write back, write through, write allocate, no write allocate

19 (byte) Blocks address Memory Divide address space into fixed-size aligned blocks. power of 2 Example: block size = 8 full byte address Block ID address bits - offset bits offset within block log 2 (block size) block 0 block 1 block 2 block 3 Note: drawing address order differently from here on! remember withinsameblock? (Pointers Lab)...

20 Placement Policy Block ID Memory Large, fixed number of block slots. Mapping: index(block ID) =??? Index Cache S = # slots = 4 Small, fixed number of block slots.

21 Placement: Direct-Mapped Block ID Memory Mapping: index(block ID) = Block ID mod S (easy for power-of-2 block sizes...) Index Cache S = # slots = 4 22

22 Placement: mapping ambiguity Block ID Memory Mapping: index(block ID) = Block ID mod S Index Cache Which block is in slot 2? S = # slots = 4 23

23 Placement: Tags resolve ambiguity Block ID Memory Mapping: index(block ID) = Block ID mod S Index Cache Tag Data Block ID bits not used for index. S 24

24 Address = Tag, Index, Offset Disambiguates slot contents. What slot in the cache? Where within a block? a-bit Address Tag Index Offset (a-s-b) bits s bits b bits Block ID bits - Index bits Tag log 2 (# cache slots) Index full byte address Block ID Address bits - Offset bits Offset within block log 2 (block size) = b # address bits

25 Placement: Direct-Mapped Block ID Memory Why not this mapping? index(block ID) = Block ID / S (still easy for power-of-2 block sizes...) Index Cache 26

26 A puzzle. Cache starts empty. Access (address, hit/miss) stream: (10, miss), (11, hit), (12, miss) block size >= 2 bytes block size < 8 bytes What could the block size be? 27

27 Placement: direct mapping conflicts Block ID Index What happens when accessing in repeated pattern: 0010, 0110, 0010, 0110, ? cache conflict Every access suffers a miss, evicts cache line needed by next access. 28

28 Placement: Set Associative One index per set of block slots. Store block in any slot within set. sets S = # slots in cache Mapping: index(block ID) = Block ID mod S Set way 8 sets, 1 block each Set way 4 sets, 2 blocks each Set way 2 sets, 4 blocks each Set 0 8-way 1 set, 8 blocks direct mapped fully associative Replacement policy: if set is full, what block should be replaced? Common: least recently used (LRU) but hardware usually implements not most recently used 29

29 Example: Tag, Index, Offset? 4-bit Address Tag Index Offset Direct-mapped 4 slots 2-byte blocks tag bits set index bits block offset bits index(1101) =

30 Example: Tag, Index, Offset? E-way set-associative S slots 16-byte blocks 16-bit Address Tag Index Offset Set E = 1-way S = 8 sets Set E = 2-way S = 4 sets Set 0 1 E = 4-way S = 2 sets tag bits set index bits block offset bits index(0x1833) tag bits set index bits block offset bits index(0x1833) tag bits set index bits block offset bits index(0x1833)

31 Replacement Policy If set is full, what block should be replaced? Common: least recently used (LRU) (but hardware usually implements not most recently used Another puzzle: Cache starts empty, uses LRU. Access (address, hit/miss) stream (10, miss); (12, miss); (10, miss) 12 is not in the same block as s block replaced 10 s block associativity direct-mapped of cache cache? 32

32 General Cache Organization (S, E, B) Powers of 2 E lines per set ( E-way ) set block/line S sets v tag B-1 cache capacity: S x E x B data bytes address size: t + s + b address bits valid bit B = 2 b bytes of data per cache line (the data block) 33

33 Cache Read E = 2 e lines per set Locate set by index Hit if any block in set: is valid; and has matching tag Get data at offset in block Address of byte in memory: t bits s bits b bits S = 2 s sets tag set index block offset data begins at this offset 1 tag B-1 valid bit B = 2 b bytes of data per cache line (the data block) 34

34 Cache Read: Direct-Mapped (E = 1) This cache: Block size: 8 bytes Associativity: 1 block per set (direct mapped) v tag Address of int: t bits S = 2 s sets v tag v tag find set v tag

35 Cache Read: Direct-Mapped (E = 1) This cache: Block size: 8 bytes Associativity: 1 block per set (direct mapped) valid? + match?: yes = hit Address of int: t bits v tag block offset int (4 Bytes) is here If no match: old line is evicted and replaced 36

36 Direct-Mapped Cache Practice 12-bit address 0x lines, 4-byte block size Direct mapped 0xA20 Offset bits? Index bits? Tag bits? Index Tag Valid B0 B1 B2 B3 Index Tag Valid B0 B1 B2 B A D 0 2 1B A 2D DA 3B B 0B D 8F 09 C D F0 1D D E B D C2 DF 03 F

37 Example (E = 1) int sum_array_rows(double a[16][16]){ double sum = 0; } for (int r = 0; r < 16; r++){ for (int c = 0; c < 16; c++){ sum += a[r][c]; } } return sum; int sum_array_cols(double a[16][16]){ double sum = 0; } Locals in registers. Assume a is aligned such that &a[r][c] is aa...a rrrr cccc 000 for (int c = 0; c < 16; c++){ for (int r = 0; r < 16; r++){ sum += a[r][c]; } } return sum; Assume: cold (empty) cache 3-bit set index, 5-bit offset aa...arrr rcc cc000 0,4: 2,0: 1,0: 0,0: aa...a001 aa...a ,0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 0,a 0,b 0,c 0,d 0,e 0,f 1,0 1,1 1,2 1,3 1,4 1,5 1,6 1,7 1,8 1,9 1,a 1,b 1,c 1,d 1,e 1,f 32 bytes = 4 doubles 4 misses per row of array 4*16 = 64 misses 32 bytes = 4 doubles every access a miss 16*16 = 256 misses 0,0 4,0 2,0 0,1 4,1 2,1 0,2 4,2 2,2 0,3 4,3 2,3 3,0 1,0 3,1 1,1 3,2 1,2 3,3 1,3 38

38 Example (E = 1) int dotprod(int x[8], int y[8]) { int sum = 0; } for (int i = 0; i < 8; i++) { sum += x[i]*y[i]; } return sum; 16 bytes = 4 ints block = 16 bytes; 8 sets in cache How many block offset bits? How many set index bits? Address bits: ttt...t sss bbbb B = 16 = 2 b : b=4 offset bits S = 8 = 2 s : s=3 index bits Addresses as bits 0x : x : x000000A0: if x and y are mutually aligned, e.g., 0x00, 0x80 x[0] y[0] x[1] y[1] x[2] y[2] x[3] y[3] if x and y are mutually unaligned, e.g., 0x00, 0xA0 x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7] y[0] y[1] y[2] y[3] y[4] y[5] y[6] y[7] 39

39 Cache Read: Set-Associative (Example: E = 2) This cache: Block size: 8 bytes Associativity: 2 blocks per set Address of int: t bits v tag v tag v tag v tag find set v tag v tag v tag v tag

40 Cache Read: Set-Associative (Example: E = 2) This cache: Block size: 8 bytes Associativity: 2 blocks per set compare both Address of int: t bits valid? + match: yes = hit v tag v tag block offset int (4 Bytes) is here If no match: Evict and replace one line in set. 41

41 Example (E = 2) float dotprod(float x[8], float y[8]) { float sum = 0; } for (int i = 0; i < 8; i++) { sum += x[i]*y[i]; } return sum; If x and y aligned, e.g. &x[0] = 0, &y[0] = 128, can still fit both because each set has space for two blocks/lines 2 blocks/lines per set x[0] x[1] x[2] x[3] y[0] y[1] y[2] y[3] x[4] x[5] x[6] x[7] y[4] y[5] y[6] y[7] 4 sets 43

42 Types of Cache Misses Cold (compulsory) miss Conflict miss Capacity miss Which ones can we mitigate/eliminate? How? 44

43 Writing to cache Multiple copies of data exist, must be kept in sync. Write-hit policy Write-through: Write-back: needs a dirty bit Write-miss policy Write-allocate: No-write-allocate: Typical caches: Write-back + Write-allocate, usually Write-through + No-write-allocate, occasionally 45

44 Write-back, write-allocate example eax = 0xCAFE ecx = T edx = U 1. mov $T, %ecx 2. mov $U, %edx 3. mov $0xFEED, (%ecx) a. Miss on T. Cache/memory not involved Cache U 0xCAFE 0 tag dirty bit Memory T U 0xFACE 0xCAFE 46

45 Write-back, write-allocate example Cache eax = 0xCAFE ecx = T edx = U tag T 0xFEED 0xFACE 01 dirty bit 1. mov $T, %ecx 2. mov $U, %edx 3. mov $0xFEED, (%ecx) a. Miss on T. b. Evict U (clean: discard). c. Fill T (write-allocate). d. Write T in cache (dirty). 4. mov (%edx), %eax a. Miss on U. Memory T U 0xFACE 0xCAFE 47

46 Write-back, write-allocate example Cache Memory eax = 0xCAFE ecx = T edx = U tag U T 0xCAFE 0xFACE 0xFEED 0 dirty bit 1. mov $T, %ecx 2. mov $U, %edx 3. mov $0xFEED, (%ecx) a. Miss on T. b. Evict U (clean: discard). c. Fill T (write-allocate). d. Write T in cache (dirty). 4. mov (%edx), %eax a. Miss on U. b. Evict T (dirty: write back). c. Fill U. d. Set %eax. 5. DONE. U 0xCAFE 48

47 Example Memory Hierarchy Processor package Core 0 Regs Core 3 Regs Typical laptop/desktop processor (c.a. 201_) L1 i-cache and d-cache: 32 KB, 8-way, Access: 4 cycles L1 d-cache L1 i-cache L1 d-cache L1 i-cache L2 unified cache: 256 KB, 8-way, Access: 11 cycles L2 unified cache L3 unified cache (shared by all cores) L2 unified cache L3 unified cache: 8 MB, 16-way, Access: cycles Block size: 64 bytes for all caches. Main memory slower, but more likely to hit 49

48 Aside: software caches Examples File system buffer caches, web browser caches, database caches, network CDN caches, etc. Some design differences Almost always fully-associative Often use complex replacement policies Not necessarily constrained to single block transfers 50

49 Cache-Friendly Code Locality, locality, locality. Programmer can optimize for cache performance Data structure layout Data access patterns Nested loops Blocking (see CSAPP 6.5) All systems favor cache-friendly code Performance is hardware-specific Generic rules capture most advantages Keep working set small (temporal locality) Use small strides (spatial locality) Focus on inner loop code 51