Cache Memories. Andrew Case. Slides adapted from Jinyang Li, Randy Bryant and Dave O Hallaron

Cache Memories Andrew Case Slides adapted from Jinyang Li, Randy Bryant and Dave O Hallaron 1

Topics Cache memory organiza3on and opera3on Performance impact of caches 2

Cache Memories Cache memories are small, fast SRAM- based memories Hold frequently accessed blocks of main memory Managed automaecally in hardware CPU looks first for data in caches (e.g., L1, L2, and L3), then in main memory. Typical system structure: CPU chip Register file Cache memories ALU System bus Memory bus Bus interface I/O bridge Main memory 3

General Cache Organiza3on (S, E, B) E = lines per set (2 e ) S = num sets (2 s ) line set valid bit v tag 0 1 2 B- 1 Cache size: C = S x E x B data bytes most significant memory bits B = bytes per cache block (the data) (2 b ) 4

Cache Read E = 2 e lines per set Locate set Check if any line in set has matching tag If Match + valid: hit Locate data starang at offset Address of word: t bits s bits b bits S = 2 s sets tag set index block offset MSB memory address (address_size s - b) data begins at this offset v tag 0 1 2 B- 1 valid bit B = 2 b bytes per cache block (the data) 5

Example: Direct Mapped Cache (E = 1) Direct mapped: One line per set Assume: cache block size 8 bytes S = 2 s sets v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 Address of int: t bits 01 100 find set v tag 0 1 2 3 4 5 6 7 6

Example: Direct Mapped Cache (E = 1) Direct mapped: One line per set Assume: cache block size 8 bytes valid? + match = hit Address of int: t bits 01 100 v tag 0 1 2 3 4 5 6 7 block offset 7

Example: Direct Mapped Cache (E = 1) Direct mapped: One line per set Assume: cache block size 8 bytes valid? + match = hit Address of int: t bits 01 100 v tag 0 1 2 3 4 5 6 7 block offset int (4 Bytes) is here On cache miss: old line is evicted and replaced 8

Direct- Mapped (E=1) Cache Simula3on t=1 s=2 b=1 x xx x M=16 byte addresses B=2 bytes/block S=4 sets E=1 Blocks/set Address trace (reads, one byte per read): 0 [0000 2 ], miss 1 [0001 2 ], hit 7 [0111 2 ], miss 8 [1000 2 ], miss 0 [0000 2 ] miss Set 0 Set 1 Set 2 Set 3 v Tag Block 0 1 10? M[8-9] M[0-1]? 1 0 M[6-7] 9

A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 10

A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Miss ra3o =? 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 12

A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Miss ra3o =.25 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 13

A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; a[0][0] Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][1] a[0][2] a[0][3] a[0][4] a[0][5] a[0][6] a[0][7] a[0][8] a[0][9] a[0][10] a[0][11] a[0][12] a[0][13] a[0][14] a[0][15] a[1][0] a[1][1] a[1][2] a[1][3] a[1][4] a[1][5] a[1][6] a[1][7] Miss ra3o =? a[1][8] a[1][9] a[1][10] a[1][11] a[1][12] a[1][13] a[1][14] a[1][15] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 15

A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here a[0][0] a[0][1] a[0][2] a[0][3] a[0][4] a[0][5] a[0][6] a[0][7] a[0][8] a[0][9] a[0][10] a[0][11] a[0][12] a[0][13] a[0][14] a[0][15] a[1][0] a[1][1] a[1][2] a[1][3] a[1][4] a[1][5] a[1][6] a[1][7] Miss ra3o = 0.25 a[1][8] a[1][9] a[1][10] a[1][11] a[1][12] a[1][13] a[1][14] a[1][15] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 16

A Higher Level Example int sum_array_rows(double a[16][16]) { int i, j; double sum = 0; for (i = 0; i < 16; i++) for (j = 0; j < 16; j++) sum += a[i][j]; return sum; Ignore the variables sum, i, j (stored in registers) a[0][0] swapped for a[2][0] a[2][0] a[2][1] a[2][2] a[2][3] a[0][4] a[0][5] a[0][6] a[0][7] a[0][8] a[0][9] a[0][10] a[0][11] a[0][12] a[0][13] a[0][14] a[0][15] a[1][0] a[1][1] a[1][2] a[1][3] a[1][4] a[1][5] a[1][6] a[1][7] Miss ra3o = 0.25 a[1][8] a[1][9] a[1][10] a[1][11] a[1][12] a[1][13] a[1][14] a[1][15] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 17

A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o =? a[0][0] a[0][1] a[0][2] a[0][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 18

A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o = 1.0 a[0][0] a[0][1] a[0][2] a[0][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 19

A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o =? a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; a[2][0] a[2][1] a[2][2] a[2][3] a[3][0] a[3][1] a[3][2] a[3][3] a[4][0] a[4][1] a[4][2] a[4][3] a[5][0] a[5][1] a[5][2] a[5][3] a[6][0] a[6][1] a[6][2] a[6][3] a[7][0] a[7][1] a[7][2] a[7][3] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 22

A Higher Level Example Ignore the variables sum, i, j (stored in registers) assume: cold (empty) cache, a[0][0] goes here Miss ra3o = 1.0 a[0][0] a[0][1] a[0][2] a[0][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; a[2][0] a[2][1] a[2][2] a[2][3] a[3][0] a[3][1] a[3][2] a[3][3] a[4][0] a[4][1] a[4][2] a[4][3] a[5][0] a[5][1] a[5][2] a[5][3] a[6][0] a[6][1] a[6][2] a[6][3] a[7][0] a[7][1] a[7][2] a[7][3] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 23

A Higher Level Example Ignore the variables sum, i, j (stored in registers) a[0][0] is swapped with a[8][0] Miss ra3o = 1.0 a[8][0] a[8][1] a[8][2] a[8][3] a[1][0] a[1][1] a[1][2] a[1][3] int sum_array_cols(double a[16][16]) { int i, j; double sum = 0; for (j = 0; i < 16; i++) for (i = 0; j < 16; j++) sum += a[i][j]; return sum; a[2][0] a[2][1] a[2][2] a[2][3] a[3][0] a[3][1] a[3][2] a[3][3] a[4][0] a[4][1] a[4][2] a[4][3] a[5][0] a[5][1] a[5][2] a[5][3] a[6][0] a[6][1] a[6][2] a[6][3] a[7][0] a[7][1] a[7][2] a[7][3] 32 B = 4 doubles (line size) AdministraAve flag and tag bits not shown 24

E- way Set Associa3ve Cache (Here: E = 2) E = 2: Two lines per set Assume: cache block size 8 bytes Address of short int: t bits 01 100 v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 find set v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 25

E- way Set Associa3ve Cache (Here: E = 2) E = 2: Two lines per set Assume: cache block size 8 bytes compare both Address of short int: t bits 01 100 valid? + match: yes = hit v tag 0 1 2 3 4 5 6 7 v tag 0 1 2 3 4 5 6 7 short int (2 Bytes) is here block offset On cache miss: One line in set is selected for evic3on and replacement Replacement policies: random, least recently used (LRU), 27

2- Way Set Associa3ve Cache Simula3on t=2 s=1 b=1 xx x x M=16 byte addresses B=2 bytes/block, S=2 sets E=2 blocks/set Address trace (reads, one byte per read): 0 [0000 2 ], miss 1 [0001 2 ], hit 7 [0111 2 ], miss 8 [1000 2 ], miss 0 [0000 2 ] hit Set 0 v Tag Block 01? 00? M[0-1] 10 10 M[8-9] Set 1 10 01 M[6-7] 0 28

E- Way Set Associa3ve Cache Simula3on Increased associa3vity decreases miss rate But with diminishing returns Simula3on of a system with 64KB D- cache, 16- word blocks, SPEC2000 1- way: 10.3% 2- way: 8.6% 4- way: 8.3% 8- way: 8.1% 29

What about writes? Mul3ple copies of data exist: L1, L2, Main Memory, Disk What to do on a write- hit? Write- through (write immediately to memory) Write- back (write to memory on eviceon) Dirty bit (keeps track if line is different from memory) What to do on a write- miss? Write- allocate (load into cache, update line in cache) Good if more writes to the locaeon follow No- write- allocate (writes immediately to memory) Typical Write- through + No- write- allocate Write- back + Write- allocate 30

Intel Core i7 Cache Hierarchy Processor package Core 0 Regs Core 3 Regs L1 i- cache and d- cache: 32 KB, 8- way, Access: 4 CPU cycles L1 d-cache L1 i-cache L1 d-cache L1 i-cache L2 unified cache: 256 KB, 8- way, Access: 11 cycles L2 unified cache L2 unified cache L3 unified cache: 8 MB, 16- way, Access: 30-40 cycles L3 unified cache (shared by all cores) Block size: 64 bytes for all caches. Main memory i instruceon d data 31

Topics Cache organiza3on and opera3on Performance impact of caches The memory mountain Rearranging loops to improve spaeal locality Using blocking to improve temporal locality 32

Cache Performance Miss Rate FracEon of memory references not found in cache Typical numbers (in percentages): 3-10% miss rate for L1 can be quite small (e.g., < 1%) for L2, depending on L2 size, etc. Hit Time Time to deliver a line in the cache to the processor includes Eme to determine whether the line is in the cache Typical numbers: 1-2 clock cycle for L1 5-20 clock cycles for L2 (~10x L1 hit Eme) Miss Penalty AddiEonal Eme required because of a miss typically 50-200 cycles for main memory (Trend: increasing!) 33

Think about those numbers Huge difference between a hit and a miss Could be 100x, if just L1 and main memory Would you believe 99% hits is twice as good as 97%? Consider: cache hit Eme of 1 cycle miss penalty of 100 cycles Average access Eme: 97% hits: 1 cycle + 0.03 * 100 cycles = 4 cycles 99% hits: 1 cycle + 0.01 * 100 cycles = 2 cycles 34

Wri3ng Cache Friendly Code Make the common case go fast Focus on the inner loops of the core funceons Minimize the misses in the inner loops Repeated references to variables are good (temporal locality) Stride- 1 reference paperns are good (spaeal locality) Key idea: Locality is quan3fied through understanding of cache memories. 35

The Memory Mountain Based on Intel Core i7 Read throughput (MB/s) 7000 6000 5000 4000 3000 2000 1000 0 s1 s3 s5 s7 s9 Element size (x8 byte strides) s11 s13 s15 s32 64M 8M 1M 128K 16K 2K Data size (bytes) 36

The Memory Mountain Based on Intel Core i7 Read throughput (MB/s) 7000 6000 5000 4000 3000 2000 Slopes of spaaal locality 1000 0 s1 s3 s5 s7 s9 Element size (x8 byte strides) s11 s13 s15 s32 64M 8M 1M 128K 16K 2K Data size (bytes) 37

The Memory Mountain Based on Intel Core i7 Read throughput (MB/s) Slopes of spaaal locality 7000 6000 5000 4000 3000 2000 1000 0 s1 s3 s5 s7 s9 Element size (x8 byte strides) s11 s13 Mem s15 L3 s32 64M L1 L2 8M 1M 128K 16K 2K Data size (bytes) Ridges of Temporal locality 38

Topics Cache organiza3on and opera3on Performance impact of caches The memory mountain Rearranging loops to improve spaeal locality Using blocking to improve temporal locality 39

Miss Rate Analysis for Matrix Mul3ply Assume: Line size = 32B (big enough for four 64- bit words) Matrix dimension (N) is very large Cache size > 1 row Cache size < muleple rows Analysis Method: Look at access papern of inner loop k j j i * k = i A B C 40

Matrix Mul3plica3on Example Variables i, j, sum held in registers Descrip3on: MulEply N x N matrices O(N 3 ) total operaeons N reads per source element /* ijk */ for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; 41

Layout of C Arrays in Memory (review) C arrays allocated in row- major order each row in coneguous memory locaeons Stepping through columns in one row: accesses successive elements exploit spaeal locality! Stepping through rows in one column: accesses distant elements no spaeal locality! 42

Matrix Mul3plica3on (ijk) /* ijk */ for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; Misses per inner loop iteraeon: A B C 0.25 1.0 0.0 Inner loop: (i,*) (*,j) (i,j) A B C Row- wise Column- wise Fixed 43

Matrix Mul3plica3on (kij) /* kij */ for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; Inner loop: (i,k) (k,*) (i,*) A B C Fixed Row- wise Row- wise Misses per inner loop iteraeon: A B C 0.0 0.25 0.25 44

Matrix Mul3plica3on (jki) /* jki */ for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; Inner loop: (*,k) (k,j) (*,j) A B C Column- wise Fixed Column- wise Misses per inner loop iteraeon: A B C 1.0 0.0 1.0 45

Summary of Matrix Mul3plica3on for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; ijk (& jik): 2 loads, 0 stores misses/iter = 1.25 kij (& ikj): 2 loads, 1 store misses/iter = 0.5 jki (& kji): 2 loads, 1 store misses/iter = 2.0 46

Core i7 Matrix Mul3ply Performance 60 jki / kji Cycles per inner loop iteration 50 40 30 20 10 ijk / jik kij / ikj jki kji ijk jik kij ikj 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 Array size (n) 47

Topics Cache organiza3on and opera3on Performance impact of caches The memory mountain Rearranging loops to improve spaeal locality Using blocking to improve temporal locality 48

Using blocking to improve temporal locality c = (double *) calloc(sizeof(double), n*n); /* Example: Multiply n x n matrices a and b */ void mmm(double *a, double *b, double *c, int n) { int i, j, k; for (i = 0; i < n; i++) for (j = 0; j < n; j++) for (k = 0; k < n; k++) c[i*n+j] += a[i*n + k]*b[k*n + j]; c = i a * b j 49

Cache Miss Analysis Assume: Matrix elements are doubles Cache block = 8 doubles Cache size C << n (much smaller than n) First itera3on: n/8 + n = 9n/8 misses = * n Aterwards in cache: = * 8 wide C A B 50

Cache Miss Analysis Assume: Matrix elements are doubles Cache block = 8 doubles Cache size C << n (much smaller than n) Second itera3on: Again in cache: n/8 + n = 9n/8 misses = * n Total misses: 9n/8 * n 2 = (9/8) * n 3 8 wide C A B 51

Blocked Matrix Mul3plica3on c = (double *) calloc(sizeof(double), n*n); /* Multiply n x n matrices a and b */ void mmm(double *a, double *b, double *c, int n) { int i, j, k; for (i = 0; i < n; i+=b) for (j = 0; j < n; j+=b) for (k = 0; k < n; k+=b) /* B x B mini matrix multiplications */ for (i1 = i; i1 < i+b; i++) for (j1 = j; j1 < j+b; j++) for (k1 = k; k1 < k+b; k++) c[i1*n+j1] += a[i1*n + k1]*b[k1*n + j1]; c = i1 a * b j1 + c Block size B x B 52

Cache Miss Analysis Assume: Cache block = 8 doubles Cache size C << n (much smaller than n) Three blocks fit into cache: 3B 2 < C First (block) itera3on: B 2 /8 misses for each block 2n/B * B 2 /8 = nb/4 (omivng matrix c) = * n/b blocks Aterwards in cache = Block size B x B * 53

Cache Miss Analysis Assume: Cache block = 8 doubles Cache size C << n (much smaller than n) Three blocks fit into cache: 3B 2 < C Second (block) itera3on: Same as first iteraeon 2n/B * B 2 /8 = nb/4 = * n/b blocks Total misses: nb/4 * (n/b) 2 = n 3 /(4B) Block size B x B 54

Summary No blocking: (9/8) * n 3 Blocking: 1/(4B) * n 3 Suggest largest possible block size B, but limit 3B 2 < C! Reason for drama3c difference: Matrix muleplicaeon has inherent temporal locality: Input data: 3n 2, computaeon 2n 3 Every array elements used O(n) Emes! But program has to be wripen properly 55

Concluding Observa3ons Programmer can op3mize for cache performance How data structures are organized How data are accessed Nested loop structure Blocking is a general technique All systems favor cache friendly code Gevng absolute opemum performance is very plaworm specific Cache sizes, line sizes, associaeviees, etc. Can get most of the advantage with generic code Keep working set reasonably small (temporal locality) Use small strides (spaeal locality) 56