Improving Cache Performance. Dr. Yitzhak Birk Electrical Engineering Department, Technion

Transcription

1 Improving Cache Performance Dr. Yitzhak Birk Electrical Engineering Department, Technion 1

2 Cache Performance CPU time = (CPU execution clock cycles + Memory stall clock cycles) x clock cycle time Memory stall clock cycles = (Reads x Read miss rate x Read miss penalty + Writes x Write miss rate x Write miss penalty) Simplification (combining reads and writes): Memory stall clock cycles = Memory accesses x Miss rate x Miss penalty 2

3 Cache Performance CPUtime = IC x (CPI execution + Mem accesses per instruction x Miss rate x Miss penalty) x Clock cycle time Misses per instruction = Memory accesses per instruction x Miss rate CPUtime = IC x (CPI execution + Misses per instruction x Miss penalty) x Clock cycle time Convention: include cache hit time as part of CPI execution (because it always takes place) 3

4 Improving Cache Performance Average memory-access time = Hit time + Miss rate x Miss penalty (ns or clocks) Improve performance by: 1. Reducing the miss rate 2. Reducing the miss penalty 3. Reducing the hit time in the cache Remark: need to be careful with computation of savings whenever latency can be masked (processor can continue work on other things) 4

5 Memory Hierarchy Misses, 3 Cs and 8 Ways to reduce Miss Rate 1. Larger block size 2. Higher associativity 3. Victim cache 4. Way prediction 5. Trace Caches 6. Hardware prefetching 7. Software prefetching 8. Other compiler techniques 5

6 Classifying Misses: 3 Cs Compulsory The first access to a block is not in the cache, so the block must be brought into the cache. These are also called cold start misses or first reference misses. (Misses in Infinite Cache) Capacity If the cache cannot contain all the blocks needed during execution of a program, capacity misses will occur due to blocks being discarded and later retrieved. (Misses in optimally-organized Finite Cache) Conflict If the block-placement strategy is set associative or direct mapped, conflict misses (in addition to compulsory and capacity misses) will occur because a block can be discarded and later retrieved if too many blocks map to its set. These are also called collision misses or interference misses. (Misses in N-way Associative, Finite Cache) 6

7 3Cs: Absolute Miss Rate way 2-way 4-way 8-way Capacity Cache Size (KB) Compulsory 7

8 3Cs: Relative Miss Rate 100% 80% 60% 1-way 2-way 4-way 8-way Conflict 40% 20% Capacity 0% Cache Size (KB) Compulsory 8

9 Reducing Miss Rate? Change Block Size? Which of 3Cs affected? Change Associativity? Which of 3Cs affected? Change Cache size? Which of 3Cs affected? Change Compiler? Which of 3Cs affected? 9

10 1. Reduce miss rate via Larger Block Size 25% Miss Rate 20% 15% 10% 5% 0% 1K 4K 16K 64K 256K Block Size (bytes) 10

11 2. Reduce Miss rate via Higher Associativity 2:1 Cache Rule: Miss Rate with a Direct Mapped cache of size N = Miss Rate with 2-way Associative cache of size N/2 Beware: Execution time is the only final measure! Will Clock Cycle time increase? Hill [1988] suggested hit time external cache +10%, internal + 2% for 2-way vs. 1-way 11

12 Example: Avg. Memory Access Time vs. Miss Rate Example: assume CCT = 1.10 for 2-way, 1.12 for 4-way, 1.14 for 8-way vs. CCT direct mapped Cache Size Associativity (KB) 1-way 2-way 4-way 8-way (Red: A.M.A.T. not improved by higher associativity) 12

13 3. Reducing Miss Rate via Victim Cache How to combine fast hit time of Direct Mapped yet still avoid conflict misses? Add buffer to place data discarded from cache Jouppi [1990]: 4-entry victim cache removed 20% to 95% of conflicts for a 4 KB direct mapped data cache Tag =? Data Victim cache FIGURE 5.15 Placement of victim cache in the memory hierarchy. =? CPU address Data Data in out Write buffer Lower level memory 13

14 4. Reducing Miss Rate via Way Prediction General idea: combine fast hit time of Direct Mapped and have the lower conflict misses of n-way SA The set is guessed based on prior accesses Multiplexer is set to read a block from the likely option (according to the guess) If the tag matches, we effectively have the performance of direct mapped. If not, there is a penalty. Benefit: Hit Time Pseudo Hit Time Can be faster Less energy consumption Time Miss Penalty Example: Alpha instruction cache; MIPS R

15 5. Hit time Reduction via Trace Caches Cache a sequence of recently accessed instructions, including chosen branch targets Cache a sequence of recently accessed data, including non-contiguous addresses More efficient use of cache space More complicated to manage Example: Intel NetBurst architecture (used in Pentium 4) 15

16 6. Reducing Miss Rate by HW Prefetching of Instruction & Data E.g., Instruction Prefetching Alpha fetches 2 blocks on a miss Extra block placed in stream buffer On miss, check stream buffer Works with data blocks too: Jouppi [1990] 1 data stream buffer got 25% misses from 4KB cache; 4 streams got 43% Palacharla & Kessler [1994] for scientific programs for 8 streams got 50% to 70% of misses from two 64KB, 4-way set associative caches Prefetching relies on extra memory bandwidth that can be used without penalty 16

17 7. Reducing Miss Rate by SW Prefetching of Data Data Prefetch Load data into register (HP PA-RISC loads) Cache Prefetch: load into cache (MIPS IV, PowerPC, SPARC v. 9) Special prefetching instructions cannot cause faults; a form of speculative execution Issuing Prefetch Instructions takes time Is cost of prefetch issues < savings in reduced misses? 17

18 8. Reducing Miss Rate by Compiler Optimizations Instructions Data Reorder procedures so as to reduce misses Profiling to look at conflicts McFarling [1989] reduced caches misses by 75% on 8KB direct mapped cache with 4 byte blocks Merging Arrays: improve spatial locality by single array of compound elements instead of two arrays Loop Interchange: change nesting of loops to access data in order stored in memory Loop Fusion: Combine two independent loops that have same looping and some variables overlap Blocking: Improve temporal locality by accessing blocks of data repeatedly vs. going down whole columns or rows 18

19 Merging Arrays: Example /* Before */ int val[size]; int key[size]; /* After */ struct merge { int val; int key; }; struct merge merged_array[size]; Reducing conflicts between val & key 19

20 Loop Interchange: Example /* Before */ for (k = 0; k < 100; k = k+1) for (j = 0; j < 100; j = j+1) for (i = 0; i < 5000; i = i+1) x[i][j] = 2 * x[i][j]; /* After */ for (k = 0; k < 100; k = k+1) for (i = 0; i < 5000; i = i+1) for (j = 0; j < 100; j = j+1) x[i][j] = 2 * x[i][j]; Sequential accesses Instead of striding through memory every 100 words 20

21 Loop Fusion Example (H&Pe2: p. 407) /* Before */ for (i = 0; i < N; i = i+1) for (j = 0; j < N; j = j+1) a[i][j] = 1/b[i][j] * c[i][j]; for (i = 0; i < N; i = i+1) for (j = 0; j < N; j = j+1) d[i][j] = a[i][j] + c[i][j]; /* After */ for (i = 0; i < N; i = i+1) for (j = 0; j < N; j = j+1) { a[i][j] = 1/b[i][j] * c[i][j]; d[i][j] = a[i][j] + c[i][j];} 2 misses per access to a & c vs. one miss per access 21

22 Blocking Example (H&P2e: p. 408) /* Before */ for (i = 0; i < N; i = i+1) for (j = 0; j < N; j = j+1) {r = 0; for (k = 0; k < N; k = k+1){ r = r + y[i][k]*z[k][j];}; x[i][j] = r; }; Two Inner Loops: Read all NxN elements of z[] Read N elements of 1 row of y[] repeatedly Write N elements of 1 row of x[] Capacity Misses a function of N & Cache Size: 3 NxN => no capacity misses; otherwise... Idea: compute on BxB submatrix that fits 22

23 Blocking Example /* After */ for (jj = 0; jj < N; jj = jj+b) for (kk = 0; kk < N; kk = kk+b) for (i = 0; i < N; i = i+1) for (j = jj; j < min(jj+b-1,n); j = j+1) {r = 0; for (k = kk; k < min(kk+b-1,n); k = k+1) { r = r + y[i][k]*z[k][j];}; x[i][j] = x[i][j] + r; }; Capacity Misses from 2N 3 + N 2 to 2N 3 /B +N 2 B called Blocking Factor Conflict Misses Too? 23

24 Reducing Conflict Misses by Blocking Direct Mapped Cache Fully Associative Cache Blocking Factor Conflict misses in caches not FA vs. Blocking size Lam et al [1991] a blocking factor of 24 had a fifth the misses vs. 48 despite both fitting in cache 24

25 Summary of Compiler Optimizations to Reduce Cache Misses vpenta (nasa7) gmty (nasa7) tomcatv btrix (nasa7) mxm (nasa7) spice cholesky (nasa7) compress Performance Improvement merged arrays loop interchange loop fusion blocking 25

26 Summary CPUtime = IC CPI Execution + Memory accesses Instruction Miss rate Miss penalty 3 Cs: Compulsory, Capacity, Conflict Clock cycle time 26

27 Memory Hierarchy Five Ways to Reduce Miss Penalty (Second Level Cache) 1. Reducing the miss rate 2. Reducing the miss penalty 3. Reducing the hit time in the cache 27

28 Review: Summary CPUtime = IC x (CPIexecution + Mem Access per instruction x Miss Rate x Miss penalty) x clock cycle time 3 Cs: Compulsory, Capacity, Conflict Misses Reducing Miss Rate 1. Reduce Misses via Larger Block Size 2. Reduce Conflict Misses via Higher Associativity 3. Reducing Conflict Misses via Victim Cache 4. Reducing Conflict Misses via Pseudo-Associativity 5. Reducing Misses by HW Prefetching Instr, Data 6. Reducing Misses by SW Prefetching Data 7. Reducing Capacity/Conf. Misses by Compiler Optimizations Remember danger of concentrating on just one parameter when evaluating performance Next: reducing Miss penalty & Hit time 28

29 1. Reducing Miss Penalty: Read Priority over Write on Miss Write back with write buffers offer RAW conflicts with main memory reads on cache misses Waiting for write buffer to empty might increase read miss penalty by 50% (old MIPS 1000) Check write buffer contents before read; if no conflicts, let the memory access continue Write Back? Read miss replacing dirty block Normal: Write dirty block to memory, and then do the read Instead, copy the dirty block to a write buffer, then do the read, and then do the write CPU stall less since it restarts as soon as do read 29

30 2. Subblock Placement to Reduce Miss Penalty Don t have to load full block on a miss Have bits per subblock to indicate valid (Originally invented to reduce tag storage) 30

31 3. Early Restart and Critical Word First Don t wait for full block to be loaded before restarting CPU Early restart As soon as the requested word of the block arrives, send it to the CPU and let the CPU continue execution Critical Word First Request the missed word first from memory and send it to the CPU as soon as it arrives; let the CPU continue execution while filling the rest of the words in the block. Also called wrapped fetch and requested word first Generally useful only in large blocks, Spatial locality a problem; tend to want next sequential word, so not clear if benefit by early restart 31

32 4. Non-blocking Caches to reduce stalls on misses Non-blocking cache or lockup-free cache allowing the data cache to continue to supply cache hits during a miss Hit under miss reduces the effective miss penalty by being helpful during a miss instead of ignoring the requests of the CPU Hit under multiple miss or miss under miss may further lower the effective miss penalty by overlapping multiple misses Significantly increases the complexity of the cache controller as there can be multiple outstanding memory accesses. 32

33 Value of Hit Under Miss for SPEC Hit Under i Misses >1 1->2 2->64 Base Hit under n Misses 0 eqntott espresso xlisp Integer compress mdljsp2 ear fpppp tomcatv swm256 doduc su2cor wave5 mdljdp2 Floating Point hydro2d alvinn nasa7 spice2g6 ora FP programs on average: AMAT= > > > 0.26 Int programs on average: AMAT= > > > KB Data Cache, Direct Mapped, 32B block, 16 cycle miss 33

34 Value of Hit Under Miss for SPEC 100% 90% 80% 70% 60% Performance 50% 40% 30% 20% 10% 0% swm256 tomcatv fpppp su2cor hydro2d mdljsp2 nasa7 doduc wave5 ear mdljdp2 alvinn spice2g6 ora xlisp espresso compress eqntott Benchmarks Hit under 1 miss Hit under 2 misses Hit under 64 misses FIGURE 5.22 Ratio of the average memory stall time for a blocking cache to hitunder-miss schemes as the number of outstanding misses is varied for 18 SPEC92 programs. 34

35 5th Miss Penalty Reduction: Second Level Cache L2 Equations AMAT = Hit Time L1 + Miss Rate L1 x Miss Penalty L1 Miss Penalty L1 = Hit Time L2 + Miss Rate L2 x Miss Penalty L2 AMAT = Hit Time L1 + Miss Rate L1 x (Hit Time L2 + Miss Rate L2 x Miss Penalty L2 ) Definitions: Local miss rate misses in this cache divided by the total number of memory accesses to this cache (Miss rate L2 ) Global miss rate misses in this cache divided by the total number of memory accesses generated by the CPU (Miss Rate L1 x Miss Rate L2 ) 35

36 Multilevel Cache Design Considerations L1 cache: Tied directly to CPU clock cycle: must be small and fast L2 cache: Latency is less critical (sees only L1 misses) Can be optimized differently:» Larger» Higher degree of associativity» Main purpose: prevent need to go to main memory Joint: Get speed from small, direct-mapped L1 cache Get low probability of having to go to main memory from large, hit-rate optimized L2 cache (e.g., higher associativity) Inclusive vs. Exclusive: exclusive iff L2 is not much larger than L1 If inclusive: use L2 tag checking mechanism, which is underutilized, for cache coherence protocols in shared memory multi-processor machines. 36

37 Comparing Local and Global Miss Rates 32 KByte 1st level cache; Increasing 2nd level cache Global miss rate close to single level cache rate provided L2 >> L1 Don t use local miss rate Cost & A.M.A.T. Generally Fast Hit Times and fewer misses Since hits are few, target miss reduction Miss rate Miss rate 80.0% 72% 72% 71% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 53% 38% 10.0% 8% 6% 4% 3% 2% 1% 3% 3% 3% 2% % 10.0% 1.0% Cache size (KB) 0.1% Cache size (KB) FIGURE 5.23 Miss rates versus cache size. 28% 22% 18% 16% 15% 15% Cache Size Linear 1% 1% 1% 1% 1% Single cache miss rate Global miss rate Local miss rate Local miss rate Log Single cache miss rate Global miss rate Cache Size 37

38 Reducing Misses: Which apply to L2 Cache? Reducing Miss Rate 1. Reduce Misses via Larger Block Size 2. Reduce Conflict Misses via Higher Associativity 3. Reducing Conflict Misses via Victim Cache 4. Reducing Conflict Misses via Pseudo-Associativity 5. Reducing Misses by HW Prefetching Instr, Data 6. Reducing Misses by SW Prefetching Data 7. Reducing Capacity/Conf. Misses by Compiler Optimizations 38

39 L2 cache block size & A.M.A.T. Relative CPU Time Block Size 32KB L1, 8 byte path to memory 39

40 Reducing Miss Penalty Summary Five techniques Read priority over write on miss Subblock placement Early Restart and Critical Word First on miss Non-blocking Caches (Hit Under Miss) Second Level Cache Can be applied recursively to Multilevel Caches Danger is that time to DRAM will grow with multiple levels in between 40

41 Improving Cache Performance 1. Reduce the miss rate 2. Reduce the miss penalty 3. Reduce the hit time in the cache 41

42 1. Fast Hit times via Small and Simple Caches Alpha has 8KB Instruction and 8KB data cache + 96KB second level cache Direct Mapped, on chip 42

43 2. Fast hits by Avoiding Address Translation Cache memories can be either physically addressed or virtually addressed. If the cache is physically addressed then the address provided by the CPU needs to undergo a process of translation (can be very long). The concept of virtual memory will be introduced in the next chapter where we will extensively discuss this problem. 43

44 3. Fast Hit Times Via Pipelined Writes Pipeline Tag Check and Update Cache as separate stages; current write tag check & previous write cache update Only Write in the pipeline; empty during a miss =? CPU address Data Data in out Tag Delayed write buffer =? M u x Data Write buffer Lower level memory FIGURE 5.28 The hardware organization of pipelined writes. In color is Delayed Write Buffer; must be checked on reads; either complete write or read from buffer 44

45 4. Fast Writes on Misses Via Small Subblocks If most writes are 1 word, subblock size is 1 word, & write through then always write subblock & tag immediately Tag match and valid bit already set: Writing the block was proper, & nothing lost by setting valid bit on again. Tag match and valid bit not set: The tag match means that this is the proper block; writing the data into the subblock makes it appropriate to turn the valid bit on. Tag mismatch: This is a miss and will modify the data portion of the block. As this is a write-through cache, however, no harm was done; memory still has an up-to-date copy of the old value. Only the tag to the address of the write and the valid bits of the other subblock need be changed because the valid bit for this subblock has already been set Doesn t work with write back due to last case 45

46 Cache Optimization Summary Technique MR MP HT Complexity Larger Block Size + 0 Higher Associativity + 1 Victim Caches + 2 Way prediction and pseudoassoc. + 2 Trace caches + 2 HW Prefetching of Instr/Data + 2 Compiler Controlled Prefetching + 3 Other Compiler techniques + 0 Priority to Read Misses + 1 Subblock Placement Early Restart & Critical Word 1st + 2 Non-Blocking Caches + 3 Second Level Caches + 2 Small & Simple Caches + 0 Avoiding Address Translation + 2 Pipelining Writes