Cache Memories. Andrew Case. Slides adapted from Jinyang Li, Randy Bryant and Dave O Hallaron

Transcription

1 Cache Memories Andrew Case Slides adapted from Jinyang Li, Randy Bryant and Dave O Hallaron 1

2 Topics Cache memory organiza3on and opera3on Performance impact of caches 2

3 Cache Memories Cache memories are small, fast SRAM- based memories Hold frequently accessed blocks of main memory Managed automaecally in hardware CPU looks first for data in caches (e.g., L1, L2, and L3), then in main memory. Typical system structure: CPU chip Register file Cache memories ALU System bus Memory bus Bus interface I/O bridge Main memory 3

4 General Cache Organiza3on (S, E, B) E = lines per set (2 e ) S = num sets (2 s ) line set valid bit v tag B- 1 Cache size: C = S x E x B data bytes most significant memory bits B = bytes per cache block (the data) (2 b ) 4

5 Cache Read E = 2 e lines per set Locate set Check if any line in set has matching tag If Match + valid: hit Locate data starang at offset Address of word: t bits s bits b bits S = 2 s sets tag set index block offset MSB memory address (address_size s - b) data begins at this offset v tag B- 1 valid bit B = 2 b bytes per cache block (the data) 5

6 Example: Direct Mapped Cache (E = 1) Direct mapped: One line per set Assume: cache block size 8 bytes S = 2 s sets v tag v tag v tag Address of int: t bits find set v tag

7 Example: Direct Mapped Cache (E = 1) Direct mapped: One line per set Assume: cache block size 8 bytes valid? + match = hit Address of int: t bits v tag block offset 7

8 Example: Direct Mapped Cache (E = 1) Direct mapped: One line per set Assume: cache block size 8 bytes valid? + match = hit Address of int: t bits v tag block offset int (4 Bytes) is here On cache miss: old line is evicted and replaced 8

9 Direct- Mapped (E=1) Cache Simula3on t=1 s=2 b=1 x xx x M=16 byte addresses B=2 bytes/block S=4 sets E=1 Blocks/set Address trace (reads, one byte per read): 0 [ ], miss 1 [ ], hit 7 [ ], miss 8 [ ], miss 0 [ ] miss Set 0 Set 1 Set 2 Set 3 v Tag Block ? M[8-9] M[0-1]? 1 0 M[6-7] 9

10 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 10

11 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 11

12 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Miss ra3o =? 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 12

13 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Miss ra3o = B = 4 doubles (line size) AdministraAve flag and tag bits not shown 13

14 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] a[0][4] a[0][5] a[0][6] a[0][7] Miss ra3o =? 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 14

15 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; a[0][0] Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][1] a[0][2] a[0][3] a[0][4] a[0][5] a[0][6] a[0][7] a[0][8] a[0][9] a[0][10] a[0][11] a[0][12] a[0][13] a[0][14] a[0][15] a[1][0] a[1][1] a[1][2] a[1][3] a[1][4] a[1][5] a[1][6] a[1][7] Miss ra3o =? a[1][8] a[1][9] a[1][10] a[1][11] a[1][12] a[1][13] a[1][14] a[1][15] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 15

16 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] a[0][4] a[0][5] a[0][6] a[0][7] a[0][8] a[0][9] a[0][10] a[0][11] a[0][12] a[0][13] a[0][14] a[0][15] a[1][0] a[1][1] a[1][2] a[1][3] a[1][4] a[1][5] a[1][6] a[1][7] Miss ra3o = 0.25 a[1][8] a[1][9] a[1][10] a[1][11] a[1][12] a[1][13] a[1][14] a[1][15] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 16

17 A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Ignore the variables sum, i, j (stored in registers) a[0][0] swapped for a[2][0] a[2][0] a[2][1] a[2][2] a[2][3] a[0][4] a[0][5] a[0][6] a[0][7] a[0][8] a[0][9] a[0][10] a[0][11] a[0][12] a[0][13] a[0][14] a[0][15] a[1][0] a[1][1] a[1][2] a[1][3] a[1][4] a[1][5] a[1][6] a[1][7] Miss ra3o = 0.25 a[1][8] a[1][9] a[1][10] a[1][11] a[1][12] a[1][13] a[1][14] a[1][15] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 17

18 A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o =? a[0][0] a[0][1] a[0][2] a[0][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 18

19 A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o = 1.0 a[0][0] a[0][1] a[0][2] a[0][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 19

20 A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o =? a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 20

21 A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o = 1.0 a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 21

22 A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o =? a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; a[2][0] a[2][1] a[2][2] a[2][3] a[3][0] a[3][1] a[3][2] a[3][3] a[4][0] a[4][1] a[4][2] a[4][3] a[5][0] a[5][1] a[5][2] a[5][3] a[6][0] a[6][1] a[6][2] a[6][3] a[7][0] a[7][1] a[7][2] a[7][3] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 22

23 A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o = 1.0 a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; a[2][0] a[2][1] a[2][2] a[2][3] a[3][0] a[3][1] a[3][2] a[3][3] a[4][0] a[4][1] a[4][2] a[4][3] a[5][0] a[5][1] a[5][2] a[5][3] a[6][0] a[6][1] a[6][2] a[6][3] a[7][0] a[7][1] a[7][2] a[7][3] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 23

24 A Higher Level Example Ignore the variables sum, i, j (stored in registers) a[0][0] is swapped with a[8][0] Miss ra3o = 1.0 a[8][0] a[8][1] a[8][2] a[8][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; a[2][0] a[2][1] a[2][2] a[2][3] a[3][0] a[3][1] a[3][2] a[3][3] a[4][0] a[4][1] a[4][2] a[4][3] a[5][0] a[5][1] a[5][2] a[5][3] a[6][0] a[6][1] a[6][2] a[6][3] a[7][0] a[7][1] a[7][2] a[7][3] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 24

25 E- way Set Associa3ve Cache (Here: E = 2) E = 2: Two lines per set Assume: cache block size 8 bytes Address of short int: t bits v tag v tag v tag v tag find set v tag v tag v tag v tag

26 E- way Set Associa3ve Cache (Here: E = 2) E = 2: Two lines per set Assume: cache block size 8 bytes compare both Address of short int: t bits valid? + match: yes = hit v tag v tag block offset 26

27 E- way Set Associa3ve Cache (Here: E = 2) E = 2: Two lines per set Assume: cache block size 8 bytes compare both Address of short int: t bits valid? + match: yes = hit v tag v tag short int (2 Bytes) is here block offset On cache miss: One line in set is selected for evic3on and replacement Replacement policies: random, least recently used (LRU), 27

28 2- Way Set Associa3ve Cache Simula3on t=2 s=1 b=1 xx x x M=16 byte addresses B=2 bytes/block, S=2 sets E=2 blocks/set Address trace (reads, one byte per read): 0 [ ], miss 1 [ ], hit 7 [ ], miss 8 [ ], miss 0 [ ] hit Set 0 v Tag Block 01? 00? M[0-1] M[8-9] Set M[6-7] 0 28

29 E- Way Set Associa3ve Cache Simula3on Increased associa3vity decreases miss rate But with diminishing returns Simula3on of a system with 64KB D- cache, 16- word blocks, SPEC way: 10.3% 2- way: 8.6% 4- way: 8.3% 8- way: 8.1% 29

30 What about writes? Mul3ple copies of data exist: L1, L2, Main Memory, Disk What to do on a write- hit? Write- through (write immediately to memory) Write- back (write to memory on eviceon) Dirty bit (keeps track if line is different from memory) What to do on a write- miss? Write- allocate (load into cache, update line in cache) Good if more writes to the locaeon follow No- write- allocate (writes immediately to memory) Typical Write- through + No- write- allocate Write- back + Write- allocate 30

31 Intel Core i7 Cache Hierarchy Processor package Core 0 Regs Core 3 Regs L1 i- cache and d- cache: 32 KB, 8- way, Access: 4 CPU cycles L1 d-cache L1 i-cache L1 d-cache L1 i-cache L2 unified cache: 256 KB, 8- way, Access: 11 cycles L2 unified cache L2 unified cache L3 unified cache: 8 MB, 16- way, Access: cycles L3 unified cache (shared by all cores) Block size: 64 bytes for all caches. Main memory i instruceon d data 31

32 Topics Cache organiza3on and opera3on Performance impact of caches The memory mountain Rearranging loops to improve spaeal locality Using blocking to improve temporal locality 32

33 Cache Performance Miss Rate FracEon of memory references not found in cache Typical numbers (in percentages): 3-10% miss rate for L1 can be quite small (e.g., < 1%) for L2, depending on L2 size, etc. Hit Time Time to deliver a line in the cache to the processor includes Eme to determine whether the line is in the cache Typical numbers: 1-2 clock cycle for L clock cycles for L2 (~10x L1 hit Eme) Miss Penalty AddiEonal Eme required because of a miss typically cycles for main memory (Trend: increasing!) 33

34 Think about those numbers Huge difference between a hit and a miss Could be 100x, if just L1 and main memory Would you believe 99% hits is twice as good as 97%? Consider: cache hit Eme of 1 cycle miss penalty of 100 cycles Average access Eme: 97% hits: 1 cycle * 100 cycles = 4 cycles 99% hits: 1 cycle * 100 cycles = 2 cycles 34

35 Wri3ng Cache Friendly Code Make the common case go fast Focus on the inner loops of the core funceons Minimize the misses in the inner loops Repeated references to variables are good (temporal locality) Stride- 1 reference paperns are good (spaeal locality) Key idea: Locality is quan3fied through understanding of cache memories. 35

36 The Memory Mountain Based on Intel Core i7 Read throughput (MB/s) s1 s3 s5 s7 s9 Element size (x8 byte strides) s11 s13 s15 s32 64M 8M 1M 128K 16K 2K Data size (bytes) 36

37 The Memory Mountain Based on Intel Core i7 Read throughput (MB/s) Slopes of spaaal locality s1 s3 s5 s7 s9 Element size (x8 byte strides) s11 s13 s15 s32 64M 8M 1M 128K 16K 2K Data size (bytes) 37

38 The Memory Mountain Based on Intel Core i7 Read throughput (MB/s) Slopes of spaaal locality s1 s3 s5 s7 s9 Element size (x8 byte strides) s11 s13 Mem s15 L3 s32 64M L1 L2 8M 1M 128K 16K 2K Data size (bytes) Ridges of Temporal locality 38

39 Topics Cache organiza3on and opera3on Performance impact of caches The memory mountain Rearranging loops to improve spaeal locality Using blocking to improve temporal locality 39

40 Miss Rate Analysis for Matrix Mul3ply Assume: Line size = 32B (big enough for four 64- bit words) Matrix dimension (N) is very large Cache size > 1 row Cache size < muleple rows Analysis Method: Look at access papern of inner loop k j j i * k = i A B C 40

41 Matrix Mul3plica3on Example Variables i, j, sum held in registers Descrip3on: MulEply N x N matrices O(N 3 ) total operaeons N reads per source element /* ijk */ for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; 41

42 Layout of C Arrays in Memory (review) C arrays allocated in row- major order each row in coneguous memory locaeons Stepping through columns in one row: accesses successive elements exploit spaeal locality! Stepping through rows in one column: accesses distant elements no spaeal locality! 42

43 Matrix Mul3plica3on (ijk) /* ijk */ for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; Misses per inner loop iteraeon: A B C Inner loop: (i,*) (*,j) (i,j) A B C Row- wise Column- wise Fixed 43

44 Matrix Mul3plica3on (kij) /* kij */ for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; Inner loop: (i,k) (k,*) (i,*) A B C Fixed Row- wise Row- wise Misses per inner loop iteraeon: A B C

45 Matrix Mul3plica3on (jki) /* jki */ for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; Inner loop: (*,k) (k,j) (*,j) A B C Column- wise Fixed Column- wise Misses per inner loop iteraeon: A B C

46 Summary of Matrix Mul3plica3on for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; ijk (& jik): 2 loads, 0 stores misses/iter = 1.25 kij (& ikj): 2 loads, 1 store misses/iter = 0.5 jki (& kji): 2 loads, 1 store misses/iter =

47 Core i7 Matrix Mul3ply Performance 60 jki / kji Cycles per inner loop iteration ijk / jik kij / ikj jki kji ijk jik kij ikj Array size (n) 47

48 Topics Cache organiza3on and opera3on Performance impact of caches The memory mountain Rearranging loops to improve spaeal locality Using blocking to improve temporal locality 48

49 Using blocking to improve temporal locality c = (double *) calloc(sizeof(double), n*n); /* Example: Multiply n x n matrices a and b */ void mmm(double *a, double *b, double *c, int n) { int i, j, k; for (i = 0; i < n; i++) for (j = 0; j < n; j++) for (k = 0; k < n; k++) c[i*n+j] += a[i*n + k]*b[k*n + j]; c = i a * b j 49

50 Cache Miss Analysis Assume: Matrix elements are doubles Cache block = 8 doubles Cache size C << n (much smaller than n) First itera3on: n/8 + n = 9n/8 misses = * n Aterwards in cache: = * 8 wide C A B 50

51 Cache Miss Analysis Assume: Matrix elements are doubles Cache block = 8 doubles Cache size C << n (much smaller than n) Second itera3on: Again in cache: n/8 + n = 9n/8 misses = * n Total misses: 9n/8 * n 2 = (9/8) * n 3 8 wide C A B 51

52 Blocked Matrix Mul3plica3on c = (double *) calloc(sizeof(double), n*n); /* Multiply n x n matrices a and b */ void mmm(double *a, double *b, double *c, int n) { int i, j, k; for (i = 0; i < n; i+=b) for (j = 0; j < n; j+=b) for (k = 0; k < n; k+=b) /* B x B mini matrix multiplications */ for (i1 = i; i1 < i+b; i++) for (j1 = j; j1 < j+b; j++) for (k1 = k; k1 < k+b; k++) c[i1*n+j1] += a[i1*n + k1]*b[k1*n + j1]; c = i1 a * b j1 + c Block size B x B 52

53 Cache Miss Analysis Assume: Cache block = 8 doubles Cache size C << n (much smaller than n) Three blocks fit into cache: 3B 2 < C First (block) itera3on: B 2 /8 misses for each block 2n/B * B 2 /8 = nb/4 (omivng matrix c) = * n/b blocks Aterwards in cache = Block size B x B * 53

54 Cache Miss Analysis Assume: Cache block = 8 doubles Cache size C << n (much smaller than n) Three blocks fit into cache: 3B 2 < C Second (block) itera3on: Same as first iteraeon 2n/B * B 2 /8 = nb/4 = * n/b blocks Total misses: nb/4 * (n/b) 2 = n 3 /(4B) Block size B x B 54

55 Summary No blocking: (9/8) * n 3 Blocking: 1/(4B) * n 3 Suggest largest possible block size B, but limit 3B 2 < C! Reason for drama3c difference: Matrix muleplicaeon has inherent temporal locality: Input data: 3n 2, computaeon 2n 3 Every array elements used O(n) Emes! But program has to be wripen properly 55

56 Concluding Observa3ons Programmer can op3mize for cache performance How data structures are organized How data are accessed Nested loop structure Blocking is a general technique All systems favor cache friendly code Gevng absolute opemum performance is very plaworm specific Cache sizes, line sizes, associaeviees, etc. Can get most of the advantage with generic code Keep working set reasonably small (temporal locality) Use small strides (spaeal locality) 56