CMSC Computer Architecture Lecture 10: Caches. Prof. Yanjing Li University of Chicago

Transcription

1 CMSC Computer Architecture Lecture 10: Caches Prof. Yajig Li Uiversity of Chicago

2 Midterm Recap Overview ad fudametal cocepts ISA Uarch Datapath, cotrol Sigle cycle, multi cycle Pipeliig Basic idea, desig cosideratios ad tradeoffs, performace aalysis Depedecies ad how to hadle them Data depedecy Cotrol depedecy (brach predictio) Exceptios/iterrupts Superscalar, SIMD, VLIW OoO will ot be o Exam I 2

3 What is A Computer? Processig cotrol (seuecig) datapath Memory (program ad data) I/O 3

4 Lecture Outlie Memory Memory Hierarchy Caches 4

5 The Memory Hierarchy

6 Memory (Programmer s View) 6

7 Ideal Memory Zero access time (latecy) Ifiite capacity Zero cost Ifiite badwidth (to support multiple accesses i parallel) 7

8 The Problem Ideal memory s reuiremets oppose each other Bigger is slower Bigger à Takes loger to access the locatio Faster is more expesive Memory/storage techology: SRAM vs. DRAM vs. SSD Higher badwidth is more expesive Need more baks, more ports, higher freuecy, or faster techology We wat both fast ad large But we caot achieve both with a sigle level of memory 8

9 Solutio: Memory Hierarchy Idea: Have multiple levels of storage (progressively bigger ad slower as the levels are farther from the processor) ad esure most of the data the processor eeds is kept i the fast(er) level(s) 9

10 The Memory Hierarchy: Fast ad Large move what you use here fast small With good locality of referece, memory appears as fast as ad as large as backup everythig here big but slow faster per byte cheaper per byte 10

11 Memory Locality We wat to esure most of the data the processor eeds is kept i the fast(er) level(s) How do we kow what data the processor eeds? The aswer: memory locality Temporal: A program teds to referece the same memory locatio may times Example: loops Spatial: A program teds to referece a cluster of memory locatios at a time most otable examples: 1. istructio memory refereces 2. array/data structure refereces 11

12 A Moder Memory Hierarchy Memory Abstractio Register File (SRAM) Tes of words, sub-sec L1 cache (SRAM) Tes of KB, ~sec Maual / compiler register spillig L2 cache (SRAM) A few MB, may sec L3 cache (SRAM)... Automatic HW cache maagemet Mai memory (DRAM), GB, ~100 sec Swap Disk TB, ~10 msec automatic demad pagig 12

13 Cachig i a Pipelied Desig The cache eeds to be tightly itegrated ito the pipelie Access i 1-cycle so that depedet operatios do ot stall High freuecy pipelie à Caot make the cache large But, we wat a large cache AND a pipelied desig Idea: Cache hierarchy First-level (L1): latecy-criticalà small, lower associativity L2/L3/ : latecy ot as importatà larger, high associativity CPU RF Level1 Cache Level 2 Cache Mai Memory (DRAM) 13

14 Review: Basic Cache Cocepts

15 Caches Caches are fast memory structures (implemeted usig SRAM) that exploit locality of referece i memory Temporal locality Spatial locality Or both 15

16 Caches: Temporal Locality Idea: Store recetly accessed data i cache Aticipatio: the data will be accessed agai soo Temporal locality priciple Recetly accessed data will be agai accessed i the ear future 16

17 Caches: Spatial Locality Idea: Store addresses adjacet to the recetly accessed oe i cache Logically divide memory ito eual size blocks Fetch block to cache i its etirety Aticipatio: earby data will be accessed soo Spatial locality priciple Nearby data i memory will be accessed i the ear future 17

18 Blocks ad Addressig the Cache Memory is logically divided ito fixed-size blocks Each block maps to a locatio i the cache, determied by the idex bits i the address tag idex offset Cache access: address 1) idex ito the cache with idex bits i address 2) check valid bit 3) compare tag bits i address with the stored tag i the cache If a block is i the cache (cache hit), the stored tag should be valid ad match the tag of the block A memory access ca also miss i the cache, of course 18

19 Classificatio of Cache Misses Compulsory miss (or cold miss) first referece to a address (block) always results i a miss subseuet refereces should hit uless the cache block is displaced for the reasos below Capacity miss cache is too small to hold everythig eeded defied as the misses that would occur eve i a fullyassociative cache (with optimal replacemet) of the same capacity Coflict miss ay miss that is either a compulsory or a capacity miss 19

20 Direct-Mapped Caches Direct-mapped cache: Two blocks i memory that map to the same idex i the cache caot be preset i the cache at the same time Oe idex à oe etry Ca lead to 0% hit rate if more tha oe block accessed i a iterleaved maer map to the same idex Assume addresses A ad B have the same idex bits but differet tag bits A, B, A, B, A, B, A, B, à coflict i the cache idex All accesses are coflict misses 20

21 Direct-Mapped Cache tag idex byte i block (byte offset) Tag / Valid Data Address V tag =? MUX byte i block Hit? Data 21

22 Direct-Mapped Cache Example 8-bit address, 8-byte blocks Assume byte-addressable memory à 256 bytes, 32 blocks Assume cache à64 bytes, 8 blocks Block offset: 3 bits; idex: 3 bits; tag: 2 bits 22

23 Set Associativity Addresses 0 ad 8 always coflict i direct mapped cache Istead of havig oe colum of 8 blocks, have 2 colums of 4 blocks (ote: cache capacity is the same) SET Tag / valid Data V tag V tag =? =? MUX Address tag idex byte i block 3b 2 bits 3 bits Logic Hit? MUX byte i block Key idea: Associative memory withi the set + Accommodates coflicts better (fewer coflict misses) -- More complex, slower access; more bits i tag 23

24 Higher Associativity 4-way Tag / valid =? =? =? =? Logic Hit? Data MUX MUX byte i block + Likelihood of coflict misses eve lower -- More tag comparators ad wider data mux; larger tags 24

25 Full Associativity Fully associative cache A block ca be placed i ay cache locatio Tag / valid =? =? =? =? =? =? =? =? Logic Hit? Data MUX MUX byte i block 25

26 Cache Metrics Cache hit rate (# hits) / (# hits + # misses) = (# hits) / (# accesses) Average memory access time (AMAT) = ( hit-rate * hit-latecy ) + ( miss-rate * miss-latecy) = hit-latecy + miss-rate * miss-pealty Example CPU with 1s clock, hit time = 1 cycle, if miss, additioal 20 cycles o top of hit latecy, cache miss rate = 5% AMAT = 0.95* * (1 + 20) = 2s Note, AMAT does ot directly measure program executio time / system performace 26

27 Memory Hierarchy Performace Aalysis L1 cache AMAT: AMAT_L1= ( hit-rate-l1 * hit-latecy-l1 ) + ( miss-rate-l1 * miss-latecy-l1) Miss-pealty of L1 cache: AMAT of L2 cache Assumig that L2 is iclusive of L1 L2 cache AMAT: AMAT_L2= ( hit-rate-l2 * hit-latecy-l2 ) + ( miss-rate-l2 * miss-latecy-l2) Miss-pealty of L2 cache: AMAT of L3 cache Hit rate ad miss rate of L2 are calculated based o the accesses that miss i L1 oly Performace aalysis ca be doe usig a recursive formula 27

28 Cache Desig Decisios ad Tradeoffs

29 Cache Desig Cosideratios Orgaizatio: cache size, block size, associatively? Replacemet: what data to remove to make room i cache? Write policy: what do we do about writes? Istructios/data: Do we treat them separately? Iclusio: Low-level caches cotai data of high-level oes? Performace optimizatio Icrease hit rate, reduce miss rate Reduce hit time Reduce miss pealty 29

30 Cache Size Cache size: total data (ot icludig tag) capacity bigger ca exploit temporal locality better Too large a cache adversely affects hit ad miss latecy smaller is faster => bigger is slower access time may degrade critical path Too small a cache does t exploit temporal locality well useful data replaced ofte hit rate workig set size Workig set: the whole set of data the executig applicatio refereces Withi a time iterval cache size 30

31 Block Size Block size is the data that is associated with a address tag Too small blocks do t exploit spatial locality well have larger tag overhead Too large blocks Too few total # of blocks à less temporal locality exploitatio Waste cache space, badwidth ad eergy if spatial locality is ot high hit rate block size 31

32 Critical-Word First Large cache blocks ca take a log time to fill ito the cache fill cache lie critical word first restart cache access before complete fill Example Assume 8-byte words ad 8-word cache block Applicatio wats to access the 4 th word ad miss i cache Fetch the 4 th word first, the the rest 32

33 Associativity How may blocks ca map to the same idex (or set)? Higher associativity ++ Higher hit rate (reduce coflict misses) -- Slower cache access time (hit latecy ad data access latecy) -- More expesive hardware (more comparators) -- Dimiishig returs from higher associativity Smaller associativity lower cost hit rate lower hit latecy Especially importat for L1 caches associativity 33

34 Cache Desig Cosideratios Orgaizatio: cache size, block size, associatively? Replacemet: what data to remove to make room i cache? Write policy: what do we do about writes? Istructios/data: Do we treat them separately? Iclusio: Low-level caches cotai data of high-level oes? Performace optimizatio Icrease hit rate, reduce miss rate Reduce hit time Reduce miss pealty 34

35 Evictio/Replacemet Policy Which block i the set to replace o a cache miss? Ay ivalid block first If all are valid, cosult the replacemet policy 35

36 LRU (Least Recetly Used) Policy Idea: Evict the least recetly accessed block Problem: Need to keep track of access orderig of blocks Questio: 2-way set associative cache: What do you eed to implemet LRU perfectly? Questio: 4-way set associative cache: What do you eed to implemet LRU perfectly? How may differet orderigs possible for the 4 blocks i the set? How may bits eeded to ecode the LRU order of a block? What is the logic eeded to determie the LRU victim? 36

37 Approximatios of LRU Most moder processors do ot implemet true LRU (also called perfect LRU ) i highly-associative caches Why? True LRU is complex LRU is a approximatio to predict locality ayway (i.e., ot the best possible cache maagemet policy) Examples: Not MRU (ot most recetly used) Victim-NextVictim Replacemet: Oly keep track of the victim ad the ext victim 37

38 Victim/Next-Victim Policy Oly 2 blocks status tracked i each set: victim (V), ext victim (NV) all other blocks deoted as (O) Ordiary block O a cache miss O a cache hit to NV Replace V Radomly pick a O Demote NV to V block as NV Radomly pick a O as NV Tur NV to O O a cache hit to V O a cache hit to O Demote NV to V Do othig Radomly pick a O as NV Tur V to O 38

39 Radom Replacemet Policy LRU vs. Radom: Which oe is better? Example: 4-way cache, cyclic refereces to A, B, C, D, E 0% hit rate with LRU policy Set thrashig: Whe the program workig set i a set is larger tha set associativity Radom replacemet policy is better whe thrashig occurs I practice: Depeds o workload Average hit rate of LRU ad Radom are similar 39

40 What Is the Optimal Replacemet Policy? Belady s OPT Replace the block that is goig to be refereced furthest i the future by the program Belady, A study of replacemet algorithms for a virtualstorage computer, IBM Systems Joural, Optimal for miimizig miss rate 40

41 Cache Desig Cosideratios Orgaizatio: cache size, block size, associatively? Replacemet: what data to remove to make room i cache? Write policy: what do we do about writes? Istructios/data: Do we treat them separately? Iclusio: Low-level caches cotai data of high-level oes? Performace optimizatio Icrease hit rate, reduce miss rate Reduce hit time Reduce miss pealty 41

42 Write-Back vs. Write-Through Caches Whe do we write the modified data i a cache to the ext level? Write through: At the time the write happes Write back: Whe the block is evicted Write-back + Ca cosolidate multiple writes to the same block before evictio Potetially saves badwidth betwee cache levels + saves eergy -- Need a bit idicatig the block is dirty/modified Write-through + Simpler + All levels are up to date (simpler cache coherece because o eed to check lower-level caches) -- More badwidth itesive; o coalescig of writes 42