Why Care About Memory Hierarchy?

Transcription

1 Performance EEC 8 Computer Architecture Memory Hierarchy Design (I) Department of Electrical Engineering and Computer Science Cleveland State University Why Care About Memory Hierarchy? Processor-DRAM Performance Gap grows % / year Processor %/year Moore s Law (X/. years) CPU DRAM DRAM 9%/year (X/ years) Time

2 An Unbalanced System Source: Bob Colwell keynote ISCA 9 Memory Issues Latency Time to move through the longest circuit path (from the start of request to the response) Bandwidth Number of bits transported at one time Capacity Size of memory Energy Cost of accessing memory (to read and write)

3 Model of Memory Hierarchy Reg File SRAM L Data cache L Inst cache L Cache DRAM Main Memory DISK Levels of the Memory Hierarchy Capacity Access Time Cost CPU Registers s Bytes < ns Cache K Bytes - ns -. cents/bit Main Memory M Bytes ns- ns $.-. cents /bit Disk G Bytes, ms (,, ns) cents/bit Tape infinite sec-min -8 Registers Instr. Operands Cache Cache Lines Memory Pages Disk Files Tape Staging Transfer Unit Compiler -8 bytes Cache controller 8-8 bytes Operating system -K bytes User Mbytes Upper Level faster This Lecture Larger Lower Level

4 Topics covered Why do caches work Principle of program locality Cache hierarchy Average memory access time (AMAT) Types of caches Direct mapped Set-associative Fully associative Cache policies Write back vs. write through Write allocate vs. No write allocate Principle of Locality Programs access a relatively small portion of address space at any instant of time. Two Types of Locality: Temporal Locality (Locality in Time): If an address is referenced, it tends to be referenced again e.g., loops, reuse Spatial Locality (Locality in Space): If an address is referenced, neighboring addresses tend to be referenced e.g., straightline code, array access Traditionally, HW has relied on locality for speed Locality is a program property that is exploited in machine design. 8

5 Example of Locality int A[], B[], C[], D; for (i=; i<; i++) { C[i] = A[i] * B[i] + D; } C[] C[] C[] C[] C[] C[] B[] B[] B[9] B[8] B[] B[] B[] B[] B[] B[] A[99] A[98] D C[99] C[98] A[] A[] A[] A[] A[] A[] A Cache Line (One fetch) C[9] C[] B[] A[9] A[] C[9] C[] B[] A[9] A[] 9 Modern Memory Hierarchy By taking advantage of the principle of locality: Present the user with as much memory as is available in the cheapest technology. Provide access at the speed offered by the fastest technology. Processor Control Datapath Registers L I L D Cache Cache Second Level Cache (SRAM) Third Level Cache (SRAM) Main Memory (DRAM) Secondary Storage (Disk) Tertiary Storage (Disk/Tape)

6 Example: Intel Core Duo L KB, 8-Way, Byte/Line, LRU, WB Cycle Latency IL Core DL DL Core IL L. MB, -Way, Byte/Line, LRU, WB Cycle Latency L Cache Source: Example : Intel Itanium MB Version 8nm mm MB Version nm mm

7 Intel Nehalem MB MB Core MB MB Core MB MB MB MB MB L Example : STI Cell Processor Local Storage SPE = M transistors (M array; M logic)

8 Cell Synergistic Processing Element Each SPE contains 8 x8 bit registers, KB, -port, ECC-protected local SRAM (Not cache) Cache Terminology Hit: data appears in some block Hit Rate: the fraction of memory accesses found in the level Hit Time: Time to access the level (consists of RAM access time + Time to determine hit) Miss: data needs to be retrieved from a block in the lower level (e.g., Block Y) Miss Rate = - (Hit Rate) Miss Penalty: Time to replace a block in the upper level + Time to deliver the block to the processor Hit Time << Miss Penalty From Processor Upper Level Memory Blk X Lower Level Memory Blk Y To Processor 8

9 Average Memory Access Time Average memory-access time = Hit time + Miss rate x Miss penalty Miss penalty: time to fetch a block from lower memory level access time: function of latency transfer time: function of bandwidth b/w levels Transfer one cache line/block at a time Transfer at the size of the memory-bus width Memory Hierarchy Performance clk clks Hit Time First-level Cache Miss % * Miss penalty Average Memory Access Time (AMAT) = Hit Time + Miss rate * Miss Penalty = T hit (L) + Miss%(L) * T(memory) Example: Cache Hit = cycle Miss rate = % =. Miss penalty = cycles AMAT = +. * = cycles Can we improve it? Main Memory (DRAM) 8 9

10 Reducing Penalty: Multi-Level Cache clk clks Second clks clks Level Cache First-level Cache L L Third Level Cache Main Memory (DRAM) On-die L Average Memory Access Time (AMAT) = T hit (L) + Miss%(L)* (T hit (L) + Miss%(L)* (T hit (L) + Miss%(L)*T(memory) ) ) 9 AMAT of multi-level memory = T hit (L) + Miss%(L)* T miss (L) = T hit (L) + Miss%(L)* { T hit (L) + Miss%(L)* (T miss (L) } = T hit (L) + Miss%(L)* { T hit (L) + Miss%(L)* (T miss (L) } = T hit (L) + Miss%(L)* { T hit (L) + Miss%(L) * [ T hit (L) + Miss%(L) * T(memory) ] }

11 AMAT Example = T hit (L) + Miss%(L)* (T hit (L) + Miss%(L)* (T hit (L) + Miss%(L)*T(memory) ) ) Example: Miss rate L=%, T hit (L) = cycle Miss rate L=%, T hit (L) = cycles Miss rate L=%, T hit (L) = cycles T(memory) = cycles AMAT =?. (compare to with no multi-levels).x speed-up! Assume that main memory accesses take ns and that memory accesses are % of all instructions. The system memory uses an L data cache with miss rate.%. CPU s clock rate is.ghz Assuming a base CPI of.. Calculate the total CPI? If we add an L cache with local miss rate 98%, hit time.ns, what is the total CPI in this case?

12 .GHz ~.ns Memory access cycles = /. = CPI = +.*. * CPI = +.*(.*./.+.*.98*) Types of Caches Type of cache Direct mapped (DM) Setassociative (SA) Fullyassociative (FA) Mapping of data from memory to cache A memory value can be placed at a single corresponding location in the cache DM and FA can be thought as special cases of SA DM -way SA FA All-way SA A memory value can be placed in any of a set of locations in the cache A memory value can be placed in any location in the cache Complexity of searching the cache Fast indexing mechanism Slightly more involved search mechanism Extensive hardware resources required to search (CAM)

13 Direct Mapping Tag Index Data x xf xaa xf Direct mapping: A memory value can only be placed at a single corresponding location in the cache Set Associative Mapping (-Way) Tag Index Data Way Way x xf xaa xf Set-associative mapping: A memory value can be placed in any location of a set in the cache

14 Fully Associative Mapping Tag Data x xf xaa xf Fully-associative mapping: A memory value can be placed anywhere in the cache Cache Two issues: How do we know if a data item is in the cache? If it is, how do we find it? Our first example: block size is one word of data "direct mapped" For each item of data at the lower level, there is exactly one location in the cache where it might be. e.g., lots of items at the lower level share locations in the upper level 8

15 Direct Mapped Cache Address 8 9 A B C D E F Memory Cache Index DM Cache A Cache Line (or Block) Cache location is occupied by data from: Memory locations,, 8, and C Which one should we place in the cache? How can we tell which one is in the cache? 9 Three (or Four) Cs (Cache Miss Terms) Compulsory Misses: cold start misses (Caches do not have valid data at the start of the program) Capacity Misses: Increase cache size Conflict Misses: Increase cache size and/or associativity. Associative caches reduce conflict misses Coherence Misses: In multiprocessor systems (later lectures ) Processor Processor Processor x x x Cache x8 x9b x Cache x8 x9b x Cache

16 : # of set : : Example: KB DM Cache, -byte Lines The lowest M bits are the Offset (Line Size = M ) Index = log (# of sets) Tag Address 9 Index Ex: x Offset Ex: x Valid Bit : Cache Tag : Cache Data Byte Byte Byte Byte : Byte Byte Byte Byte 99 Direct Mapped Cache Mapping: address is modulo the number of blocks in the cache Cache Memory

17 Direct Mapped Cache For MIPS: Address (showing bit positions) Byte offset H it Tag D ata Index V alid T ag D ata Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88

18 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 8

19 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 -byte block, drop low bits for byte offset! Only matters for byteaddressable systems # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is Next log (8) bits mod 8 = Index # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 8 9

20 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is Next log (8) bits mod 8 = Index # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 9 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88

21 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy

22 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy

23 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #8 is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy

24 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #8 is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [b] copy Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [b] copy 8

25 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [b] copy 9 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [b] copy It s valid! How to tell it s the wrong address?

26 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [b] copy The tags don t match! It s not what we want to access! Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy

27 Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [c] copy Example: DM$, 8-Entry, B lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # #8 [a] [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [c] copy

28 Example: DM$, 8-Entry, B Q: What if the machine is only word-addressable? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [c] copy Example: DM$, 8-Entry, B Q: What if the machine is only word-addressable? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [c] copy 8

29 Example: DM$, 8-Entry, B Q: What if the machine is only word-addressable? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # is # [a] #8 [b] # [c] V alid T ag D ata ( bytes) #88 [a] copy [c] copy Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] Write Through Vs. Write Back # [c] V alid T ag D ata ( bytes) Write Through = write to both memory and cache Write Back = write to cache block only for the time being then write to memory when block is replaced.. [a] copy #88 [c] copy 8 9

30 Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] # [c] OLD V alid T ag D ata ( bytes) #88 [a] copy [c] NEW Do we update memory now? Or later? 9 Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] # [c] OLD V alid T ag D ata ( bytes) #88 [a] copy [c] NEW Assume later...

31 Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] # [c] OLD V alid T ag D ata ( bytes) #88 [a] copy [c] NEW Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #88 # [a] #8 [b] # [c] OLD V alid T ag D ata ( bytes) #88 [a] copy [c] NEW Now What? How do we know to write back?

32 Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #88 # [a] #8 [b] # [c] OLD V alid T ag D ata ( bytes) [a] copy [c] NEW Dirty Need extra state! The dirty bit! #88 Dirty Bit is used to track whether data from cache was written into memory Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #88 # [a] #8 [b] # [c] NEW V alid T ag D ata ( bytes) Dirty #88 [a] copy [c] NEW

33 Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #88 # [a] #8 [b] # [c] NEW V alid T ag D ata ( bytes) Dirty #88 [a] copy Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 #88 # [a] #8 [b] # [c] NEW V alid T ag D ata ( bytes) Dirty #88 [a] copy copy

34 Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] # [c] NEW V alid T ag D ata ( bytes) Dirty #88 [a] copy copy Example: DM$, 8-Entry, B Q: What about writing back to memory? lw $, # [a] lw $, #8 [b] sw $, # [c] sw $, #88 # [a] #8 [b] Write Back more complex.. Must have a miss and dirty =, in order to maintain data, cannot instantly write over cache like a write through V alid T ag D ata ( bytes) Dirty # #88 [c] NEW OLD [a] copy NEW Dirty Bit indicates value hasn t been written to memory yet (write-back method) 8

35 DM$: Thoughts Trade-Offs: Write-back or Write-Through? Write-Alloc or No-Write-Alloc? How does Tag change with # of Entries? How does minimum machine word size impact tag? H it Address (showing bit positions) Byte offset Tag Index V alid T ag D ata D ata What kind of locality are we taking advantage of? No write alloc = only processor reads cached 9 Direct Mapped Cache Taking advantage of spatial locality: Address (showing bit positions) Hit Tag Byte offset Index Block offset Data bits 8 bits V Tag Data K entries Mux

36 Hits vs. Misses Read hits this is what we want! Read misses stall the CPU, fetch block from memory, deliver to cache, restart Write hits: can replace data in cache and memory (write-through) write the data only into the cache (write-back the cache later) Write misses: read the entire block into the cache, then write the word? Writes are hard read: concurrently check tag and read data write is destructive, so it must be slower Write strategies when to update memory? - on every write (write-through) - when a modified block is replaced (write-back) what to do on a write miss? - fetch the block to cache (write allocate), used with write-back - do not fetch (write no-allocate), used with write-through Trade-offs - Write-back uses bus bandwidth more efficiently saves power since it uses lower level memory less often attractive to both multiprocessors and embedded applications. Write-through easier to implement keeps MM consistent with CM, also good for multiprocessors since data is coherent in the memory hierarchy.

37 Performance Increasing the block size tends to decrease miss rate: % % M iss ra te % % % % KB 8 KB KB KB KB % % % Block size (bytes) Program Block size in words Instruction miss rate Data miss rate Effective combined miss rate gcc.%.%.%.%.%.9% spice.%.%.%.%.%.% Use split caches because there is more spatial locality in code: Performance Simplified model: execution time = (execution cycles + stall cycles) * cycle time stall cycles = # of instructions * miss ratio * miss penalty Two ways of improving performance: decreasing the miss ratio decreasing the miss penalty What happens if we increase block size?

38 Decreasing miss ratio with associativity One-way set associative (direct mapped) Block Tag Data Set Two-way set associative Tag Data Tag Data Set Four-way set associative Tag Data Tag Data Tag Data Tag Data Eight-way set associative (fully associative) Tag Data Tag Data Tag Data Tag Data Tag Data Tag Data Tag Data Tag Data Compared to direct mapped, give a series of references that: results in a lower miss ratio using a -way set associative cache results in a higher miss ratio using a -way set associative cache (assuming least recently used replacement strategy) An implementation Address Index V Tag Data V Tag Data V Tag Data V Tag Data -to- multiplexor Hit Data 8

39 Set-Associative Cache Multiple cache blocks (lines) can be allocated into the same set When full, needs to evict some block out of the cache Need to consider the locality Replacement policy Last-In First-Out (LIFO), like a stack Random First-In First-Out (FIFO) Least Recently Used (LRU) Example of Caches Given a MB, direct-mapped physical caches, line size=byte Support up to -bit physical address Tag size? Now change it to -way, Tag size? How about if it s -bit physical address? 8 9

40 Show cache addressing for a byte-addressable memory with -bit addresses. Cache line W = B. Cache size L = 9 lines ( KB). Tag bits? 9 Show cache addressing for a byte-addressable memory with -bit addresses. Cache line W = B. Cache size L = 9 lines ( KB). Tag bits? Solution Byte offset in line is log = b. Cache line index is log 9 = b. This leaves = b for the tag. -bit line index in cache -bit line tag -bit byte offset in line -bit address Byte address in cache 8

41 Show cache addressing scheme for a byteaddressable memory with -bit addresses. Cache line width W = B. Set size S = lines. Cache size L = 9 lines ( KB). 8 Show cache addressing scheme for a byteaddressable memory with -bit addresses. Cache line width W = B. Set size S = lines. Cache size L = 9 lines ( KB). Solution Byte offset in line is log = b. Cache set index is (log 9/) = b. This leaves = b for -bit set index in cache the tag. -bit address -bit line tag Address in cache used to read out two candidate items and their control info -bit byte offset in line 8

42 A KB four-way set-associative cache is byte-addressable and contains B lines. Memory addresses are b wide. How wide are the tags in this cache? Solution Address ( B) = b byte offset + 9 b set index + 8 b tag. -bit address Tag width = 9 = 8 Tag Set index Offset 8 bits 9 bits bits Set size = B = 8 B Number of sets = / = 9 Line width = B = B 8 Example: KB DM Cache, -byte Lines lw from xffc8 Tag Index Offset FFC8 = Tag array Data array DM Cache 8

43 Index DM Cache Speed Advantage Tag and data access happen in parallel Faster cache access! Tag array Tag Index Offset Data array 8 Associative Caches Reduce Conflict Misses Set associative (SA) cache multiple possible locations in a set Fully associative (FA) cache any location in the cache Hardware and speed overhead Comparators Multiplexors Data selection only after Hit/Miss determination (i.e., after tag comparison) 8

44 Valid Set Associative Cache (-way) Cache index selects a set from the cache The two tags in the set are compared in parallel Data is selected based on the tag result Additional circuitry as compared to DM caches Makes SA caches slower to access than DM of comparable size Cache Index Cache Tag Cache Data Cache Data Cache Line : : : Cache Line : Cache Tag Valid : : Adr Tag Compare Sel Mux Sel Compare Hit OR Cache Line 8 Set-Associative Cache (-way) bit address lw from xffc8 Tag Index offset Tag array Data aray Data array Tag array 88

45 Fully Associative Cache tag = = = Associative Search offset Tag Data = Multiplexor Rotate and Mask 89 Address Fully Associative Cache Write Data Tag offset Tag Data Tag Data Tag Data Tag Data compare compare compare compare Additional circuitry as compared to DM caches More extensive than SA caches Read Data Makes FA caches slower to access than either DM or SA of comparable size 9

46 Cache Write Policy Write through -The value is written to both the cache line and to the lower-level memory. Write back - The value is written only to the cache line. The modified cache line is written to main memory only when it has to be replaced. Is the cache line clean (holds the same value as memory) or dirty (holds a different value than memory)? 9 Write-through Policy x8 x x Processor Cache Memory 9

47 Write Buffer Processor Cache DRAM Write Buffer Processor: writes data into the cache and the write buffer Memory controller: writes contents of the buffer to memory Write buffer is a FIFO structure: Typically to 8 entries Desirable: Occurrence of Writes << DRAM write cycles Memory system designer s nightmare: Write buffer saturation (i.e., Writes DRAM write cycles) 9 Writeback Policy x9abc x8 x8 x x????? Processor Write miss Cache Memory 9

48 On Write Miss Write allocate The line is allocated on a write miss, followed by the write hit actions above. Write misses first act like read misses No write allocate Write misses do not interfere cache Line is only modified in the lower level memory Mostly use with write-through cache 9 Quick recap Processor-memory performance gap Memory hierarchy exploits program locality to reduce AMAT Types of Caches Direct mapped Set associative Fully associative Cache policies Write through vs. Write back Write allocate vs. No write allocate 9 8

49 Cache Replacement Policy Random Replace a randomly chosen line FIFO Replace the oldest line LRU (Least Recently Used) Replace the least recently used line NRU (Not Recently Used) Replace one of the lines that is not recently used In Itanium L Dcache, L and L caches 9 LRU Policy Access C Access D Access E Access C Access G MRU MRU- LRU+ LRU A B C D C A B D D C A B E D C A C E D A G C E D MISS, replacement needed MISS, replacement needed 98 9

50 LRU From Hardware Perspective LRU Way Way Way Way A B C D State machine Access update Access D LRU policy increases cache access times Additional hardware bits needed for LRU state machine 99 LRU Algorithms True LRU Expensive in terms of speed and hardware Need to remember the order in which all N lines were last accessed N! scenarios O(log N!) O(N log N) LRU bits -ways AB BA = =! -ways ABC ACB BAC BCA CAB CBA = =! Pseudo LRU: O(N) Approximates LRU policy with a binary tree

51 Pseudo LRU Algorithm (-way SA) AB/CD bit (L) A/B bit (L) C/D bit (L) Way A Way B Way C Way D Tree-based O(N): bits for -way Cache ways are the leaves of the tree Combine ways as we proceed towards the root of the tree Way Way Way Way A B C D Pseudo LRU Algorithm A/B bit (L) AB/CD bit (L) C/D bit (L) Way A Way B Way C Way D LLL =, there is a hit in Way B, what is the new updated LLL? LRU update algorithm CD AB AB/CD Way hit L L L Way A --- Way B --- Way C --- Way D --- Less hardware than LRU Faster than LRU LLL =, a way needs to be replaced, which way would be chosen? Replacement Decision CD AB AB/CD L L L Way to replace X Way A X Way B X Way C X Way D

52 M i s s r a t e Not Recently Used (NRU) Use R(eferenced) and M(odified) bits (not referenced or not modified) (referenced or modified) Classify lines into C: R=, M= C: R=, M= C: R=, M= C: R=, M= Chose the victim from the lowest class (C > C > C > C) Periodically clear R and M bits Reducing Miss Rate Enlarge Cache If cache size is fixed Increase associativity Increase line size % Does this always work? Increasing cache pollution % % % K B 8 K B % % K B K B K B % % % B l o c k s iz e ( b y t e s )

53 Reduce Miss Rate/Penalty: Way Prediction Best of both worlds: Speed as that of a DM cache and reduced conflict misses as that of a SA cache Extra bits predict the way of the next access Alpha Way Prediction (next line predictor) If correct, -cycle I-cache latency If incorrect, -cycle latency from I-cache fetch/branch predictor Branch predictor can override the decision of the way predictor Alpha Way Prediction (offset) (-way) Note: Alpha advocates to align the branch targets on octaword ( bytes)

54 Reduce Miss Rate: Code Optimization Misses occur if sequentially accessed array elements come from different cache lines Code optimizations No hardware change Rely on programmers or compilers Examples: Loop interchange In nested loops: outer loop becomes inner loop and vice versa Loop blocking partition large array into smaller blocks, thus fitting the accessed array elements into cache size enhances cache reuse Loop Interchange Row-major ordering /* Before */ for (j=; j<; j++) for (i=; i<; i++) x[i][j] = *x[i][j] i= j= What is the worst that could happen? Hint: DM cache /* After */ for (i=; i<; i++) for (j=; j<; j++) x[i][j] = *x[i][j] Improved cache efficiency i= j= Is this always safe transformation? Does this always lead to higher efficiency? 8

55 Loop Blocking /* Before */ for (i=; i<n; i++) for (j=; j<n; j++) { r=; for (k=; k<n; k++) r += y[i][k]*z[k][j]; x[i][j] = r; } k y[i][k] j z[k][j] X[i][j] i k i Does not exploit locality 9 Loop Blocking Partition the loop s iteration space into many smaller chunks Ensure that the data stays in the cache until it is reused k y[i][k] j z[k][j] j X[i][j] i k i

56 /* After */ for (jj=; jj<n; jj=jj+b) for (kk = ; kk<n; kk=kk+b) for (i=; i<n; i=i++) for (j=jj; j<min(jj+b, N); j=++) { } r=; for (k=kk; k<min (kk+b,n); k++) r += y[i][k]*z[k][j]; x[i][j] = x[i][j] + r; Other Miss Penalty Reduction Techniques Critical value first and Restart early Send requested data in the leading edge transfer Trailing edge transfer continues in the background Give priority to read misses over writes Use write buffer (WT) and writeback buffer (WB) Combining writes combining write buffer Intel s WC (write-combining) memory type Victim caches Assist caches Non-blocking caches Data Prefetch mechanism

57 Write Combining Buffer Need to initiate separate writes back to lower level memory Wr. addr 8 V V Mem[] Mem[8] Mem[] Mem[] V V For WC buffer, combine neighbor addresses One single write back to lower level memory Wr. addr V V V V Mem[] Mem[8] Mem[] Mem[] WC memory type Intel (starting in P) supports USWC (or WC) memory type Uncacheable, speculative Write Combining Expensive (in terms of time) for individual write Combine several individual writes into a bursty write Effective for video memory data Algorithm writing byte at a time Combine of -byte data into one -byte write Ordering is not important