ECE 172 Digital Systems. Chapter 5 Uniprocessor Data Cache. Herbert G. Mayer, PSU Status 6/10/2018

Transcription

1 ECE 172 Digita Systems Chapter 5 Uniprocessor Data Cache Herbert G. Mayer, PSU Status 6/10/2018 1

2 Syabus UP Caches Cache Design Parameters Effective Time t eff Cache Performance Parameters Repacement Poicies Trace Cache Concusion Bibiography 2

3 Data Cache in UP Microprocessor (see aso ater presentation about MP Data Cache) 3

4 Syabus UP Caches Intro: Purpose, Design Parameters, Architecture Effective Time t eff Singe-Line Degenerate Cache Muti-Line, Singe-Set Cache Singe-Line, Muti-Set Cache, Bocked Mapping Singe-Line, Muti-Set, Cycic Mapping Muti-Line per Set (Associative), Muti-Set Cache Repacement Poicies LRU Sampe Compute Cache Size Trace Cache Characteristic Cache Curve Bibiography 4

5 Intro: Purpose of Cache Cache is ogicay part of Memory Subsystem, yet physicay often part of microprocessor: ocated on same siicon die Purpose: transform sow memory into fast one Possibe with minima cost despite high cost per cache bit, as tota cache size is just a few % of tota physica main store Works we, if ocaity is good; ese performance is same as sow memory access, or even worse, depending on architecture With poor ocaity, i.e. with random distribution of memory accesses, cache can actuay sow down if: t eff = t cache + (1-h) * t mem and not: t eff = max( t cache, (1-h) * t mem ) 5

6 Intro: Purpose of Cache With good ocaity, cache deivers avaiabe data in cose to unit cyce time: Now That Is Awesome! In MP systems, caches must cooperate with other processors caches, memory, some peripheras Even on a UP system there are mutipe agents that access memory and thus impact caches, e.g. the DMA agent and memory controer Cache must cooperate with VMM of memory subsystem to jointy render a physicay sma, sow memory into a virtuay arge, fast memory at sma cost of added HW (siicon), and system SW L1 cache access time is ideay contained within singe machine cyce; reaized on many CPUs 6

7 Intro: Trend of Speeds 7

8 Intro: Growing Caches Inte Haswe-E Die, Center Showing 20 MB Shared L3 Cache 8

9 Line From Definitions in Appendix Storage area in cache abe to hod a copy of a contiguous bock of memory ces, AKA paragraph The portion of memory stored in that ine is aigned on a memory address moduo ine size! For exampe, if a ine hods 64 bytes on a byteaddressabe architecture, the address of the first byte of a ine has 6 traiing zeros: i.e. it is eveny divisibe by 64, or we say it is 64-byte aigned Such known zeros don t need to be stored in the tag; they are impied, they are known a-priori! This shortens the tag, rendering a cache simper, and cheaper to manufacture: ess HW bits! 9

10 Set From Definitions in Appendix A ogicay connected region of memory, mapped to a specific area of cache (ine), is a set; memory is partitioned into N sets Eements of a set don t need to be physicay contiguous in memory; if contiguous, eftmost og 2 (N) bits are known a-priori and don t need to be stored; if cycic distribution, then the rightmost og 2 (N) are known a-priori The number of sets is conventionay abeed N A degenerate case maps a memory onto the whoe cache, thus ony a singe set exists: N = 1; one set; not meaningfu! Notion of set is meaningfu ony if there are mutipe sets. Again: A memory region beonging to one set can be a physicay contiguous bock, or a cycicay distributed part of memory Former case is caed bocked, the atter cycic. Cache area into which such a portion of memory is mapped to is aso caed set 10

11 Intro: Cache Design Parameters Number of ines in set: K Quick test: K is how arge in a direct-mapped cache? Number of units bytes in a ine is named L, AKA Length of ine L Number of sets in memory, and hence in the cache: N Poicy upon store miss: cache write poicy Poicy upon oad miss: cache read poicy What to do, when an empty ines is needed for the next paragraph to be streamed-in, but none is avaiabe? That action is the: repacement poicy 11

12 Intro: Cache Design Parameters Size here is the tota size of a cache; unit being discussed can be: bits or bytes; be carefu! Size = K * ( L + bits for tag and contro bits ) * N Ratio of cache size to physica memory is generay on order of very sma percentage, e.g. < 1 % Cache access time, typicay cose to 1 cyce for L1 cache Number of processors with cache: 1 in UP, M in MP architecture Leves of caches, L1, L2, L3 Last one referred to as LLC, for ast eve cache 12

13 Intro: Cache Architecture Cache-reated definitions used throughout are common, though not a manufacturers appy the same nomencature Initiay we discuss cache designs for singe-processor architectures In MP cache ecture we progress to more compex cache designs, covering the MESI protoco for a two-processor system with L1 and L2 cache Focus here: Soey L1 data cache on UP computer architecture 13

14 Effective Time t eff Starting with t eff = t cache + ( 1 - h ) * t mem we observe: No matter how many hits (H) we experience during repeated memory access, the effective cyce time is never ess than t cache No matter how many misses (M) we experience, the effective cyce time to access a datum is never more than t cache + t mem Desirabe to have t eff = t mem in case of cache miss Another way to compute effective access time is to add a memory-access times, and divide them by the tota number of accesses, and thus compute the effective, or average, time t eff 14

15 Effective Time t eff Average time per access: t eff = ( hits * t cache + misses * ( t cache + t mem ) ) / tota_accesses t eff = h * t cache + m * ( t cache + t mem ) if memory accessed immediatey: t eff = ( h + m ) * t cache + m * t mem = t cache + m * t mem Assume an access time of 1 cyce to reference data in the cache; best case, at times feasibe Assume an access time of 10 cyces for data in memory; yet is unreaisticay fast!! Assume that a memory access is initiated after a cache miss; then: 15

16 Effective Time t eff 16

17 Effective Time t eff Symb. Name Expanation H Hits Number of successfu cache accesses M Misses Number of faied cache accesses A A A accesses A = H + M T Tota time Time for A memory accesses t cache Cache time Time to access data once via the data cache t mem Mem time Time to access data via memory once t eff Effective tm. Average time over a memory accesses h Hit rate H / A = h = 1 m m Miss rate M / A = m = 1 h h + m Tota rate = 1 Tota rate, either hit or miss, probabiity is 1 17

18 Effective Time t eff 18

19 Effective Time t eff Compare t eff, the effective memory access time in L1 data cache at 99% hit rate vs. 97% hit rate Time for hit t hit = 1 cyce, time for miss t miss = 100 cyces; then compare 99 and 97 percent hit rates: Given a 99% hit rate: 1 miss costs 100 cyces 99 hits cost 99 cyces tota t eff = ( ) / 100 = cyces per average access Given a 97% hit rate: Students compute t eff here! 19

20 Effective Time t eff Compare t eff, the effective memory access time in L1 data cache at 99% hit rate vs. 97% hit rate Time for hit t hit = 1 cyce, time for miss t miss = 100 cyces; then compare 99 and 97 percent hit rates: Given a 99% hit rate: 1 miss costs 100 cyces 99 hits cost 99 cyces tota t eff = ( ) / 100 = cyces per average access Given a 97% hit rate: 3 misses costs 300 cyces 97 hits cost 97 cyces tota t eff = ( ) / 100 = 397 / 100 = cyces per average access Or 100% additiona cyces for oss of 2% hit accuracy! 20

21 Actua Cache Data Inte Core i7 with 3 eves of cache, L1 access > 1 cyce, L3 access costing dozens of cyces, sti way faster than memory access! 21

22 Compexity of Opteron Data Cache -Taken from Wikipedia 22

23 Cache Performance Parameters 23

24 Singe-Line Degenerate Cache 24

25 Singe-Line Degenerate Cache Quick test: what is the minimum size (in number of bits) of the tag for this degenerate cache? (assume 32-bit architecture, and 64-byte ines) The singe-ine cache, shown here, stores mutipe words Can improve memory access if extremey good ocaity exists within a very narrow address range Upon miss cache initiates a stream-in operation Is a direct mapped cache: a memory ocations know a priori where they reside in cache; there is but one ine, one option for them Is a singe-set cache 25

26 Singe-Line Degenerate Cache As data cache: expoits ony ocaity of near-by addresses in the same paragraph As instruction cache: Expoits ocaity of tight oops that competey fit inside the address range of a singe cache ine However, there wi be a cache-miss as soon as an address makes reference outside of ine s range For exampe, tight oop with a function ca wi cause cache miss Stream-in time is time to oad a ine of data from memory Tota overhead: tag bits + vaid bit + dirty bit (if write-back) Not advisabe to buid this type of a cache subsystem 26

27 Dua-Line, Singe-Set Cache 27

28 Dua-Line, Singe-Set Cache Next cache has 1 set, mutipe ines; here 2 ines shown Quick sanity check: minimum size of tag on byte-addressabe, 32-bit architecture with 2 ines, 1 set, ine size of 16 bytes? Each ine hods mutipe, contiguous addressing units, 4 words, 16 bytes shown Thus 2 disparate areas of memory can be cached at the same time Is associative cache; a ines (i.e. 2 ines) in singe set must be searched to determine, whether a memory eement is present in cache Is singe-set associative cache, since a of memory (singeton set) is mapped onto the same cache ines 28

29 Dua-Line, Singe-Set Cache Some tight oops with a function ca can be competey cached in an I-cache, assuming oop body fits into ine and caée fits into the other ine Aso woud aow one arger oop to be cached, whose tota body does not fit into a singe ine, but woud fit into 2 ines Appies to more reaistic programs But if number of ines K >> 1, the time to search a tags (in set) can grow beyond unit cyce time Again not advisabe to buid this kind of cache subsystem 29

30 Singe-Line, Dua-Set Cache 30

31 Singe-Line, Dua-Set Cache This cache architecture has mutipe sets, 2 shown, 2 distinct areas of memory, each being mapped onto separate cache ines: N = 2, K = 1 Quick test: minimum size of the tag on 4-byte per word, 32-bit architecture with 16-byte ines? Each set has a singe ine, in this case 4 memory words; AKA paragraph in memory Thus 2 disparate areas of memory can be cached at the same time But these areas must reside in separate memory sets, each contiguous, each having ony 1 option Is direct mapped; a memory ocations know a priori where they reside in cache Is muti-set cache, since parts of memory have their own portion of cache 31

32 Singe-Line, Dua-Set Cache Aows one arger oop to be cached, whose tota body does not fit into a singe ine of an I-cache, but woud fit into two ines But ony if by some great coincidence both parts of that oop reside in different memory sets If used as instruction cache, a programs consuming haf of memory or ess never use the second ine in the second set. Hence this cache architecture is a bad idea! If used as data cache, a data areas that fit into first bock wi never popuate the second set Probem specific to bocked mapping; so et s try cycic instead Aso not advisabe to buid this type cache 32

33 Dua-Set, Singe-Line, Cycic 33

34 Dua-Set, Singe-Line, Cycic Cache architecture beow aso has 2 sets, N = 2 Each set has a singe ine, each hoding 4 contiguous memory units, 4 words, 16 bytes, K = 1 So at east 2 disparate areas of memory can be cached at the same time Quick test: tag size on 32-bit, 4-byte architecture? Disparate areas (of ine size, equa to paragraph size) are scattered cycicay throughout memory Cycicay distributed memory areas associated with each respective set Is direct mapped; a memory ocations know a priori where they reside in cache, as each set has a singe ine Is muti-set cache: different ocations of memory are mapped onto different cache ines, the sets 34

35 Dua-Set, Singe-Line, Cycic Aso aows one arger oop to be cached, whose tota body does not fit into a singe ine, but woud fit into two ines Even if parts of oop beong to different sets If used as instruction cache, sma code section can use the tota cache If used as data cache, sma data areas can utiize compete cache Cycic mapping of memory areas to sets is generay superior to bocked mapping Sti not advisabe to buid this type of cache 35

36 Muti-Line, Muti-Set, Cache 36

37 Muti-Line, Muti-Set, Cache Reminder: Tag is that minima number of address bits to store, in order to ID the ine Quick test: minimum size (in bits) of the tag? Use: 32-bit architecture, byte addressabe, 2 sets cycic, ine ength 16 bytes, 2-way set associative Two sets, memory wi be mapped cycicay, AKA in a round-robin fashion Each set has two ines, each ine hoding 16 bytes; i.e. paragraph ength of memory is 16 bytes in this exampe! Note: direct mapped caches, i.e. caches with one ine per set, are aso common; AKA non-associative 37

38 Muti-Line, Muti-Set, Cache Associative cache: once set is known, search a tags for the memory address in a ines of that set In earier exampe p. 34, ine 2 of set 2 is unused, AKA invaid in MESI terminoogy By now you know: sets, ines, associate, nonassociative, direct mapped, and more terms! 38

39 Repacement Poicy The repacement poicy is the rue that determines, when a ines are vaid (i.e. aready busy with other, good data), and a new ine must be streamed in: Which of the vaid ines in a cache is to be repaced, AKA removed? Remova can be ow cost, if the modified bit (AKA dirty bit) is cear = 0; this means: data in memory and cache ine are identica! Otherwise remova may be costy: If the dirty bit is set = 1, data have to be written back into memory, costing a memory access! We ca this copying to memory: stream out 39

40 Repacement Poicy # Name Summary 1 LRU Repaces Least Recenty Used cache ine; requires keeping track of reative ages of ines. Retire ine that has remained unused for the ongest time of a candidate ines. Specuate that that ine wi remain unused for the ongest time in the future. 2 LFU Repaces Least Frequenty Used cache ine; requires keeping track of the number m of times this ine was used over the ast n>=m uses. Depending on how ong we track the usage, this may require many bits. 3 FIFO First In First Out: The first of the ines in the set that was streamed in is the first to be retired, when it comes time to find a candidate. Has the advantage that no further update is needed, whie a ines are in use. 4 Random Pick a random ine from candidate set for retirement; is not as bad as this irrationa agorithm might suggest. Reason: The other methods are not too good either J 5 Optima If a cache were omniscient, it coud predict, which ine wi remain unused for the ongest time in the future. Of course, that is not computabe. However, for creating the perfect reference point, we can do this with past memory access patterns, and use the optima access pattern for comparison, how we our chosen poicy rates vs. the optima strategy! 40

41 LRU Sampe 1 Assume the foowing cache architecture: N = 16 sets, cycic distribution K = 4 ines per set 32-bit architecture, byte-addressabe write back (dirty bit) vaid ine indicator (vaid bit) L = 64 bytes per ine; AKA ine ength LRU repacement; uses 2 bits (4 ines per set), to store reative ages This resuts in tag size of???? bits 41

42 LRU Sampe 1 Assume the foowing cache architecture: This resuts in a tag size of 22 bits What is the overhead size per cache ine, measured in bits? 42

43 LRU Sampe 1 Assume the foowing cache architecture: Tag size = 22 bits 2 LRU bits (4 ines per set), to store reative ages of the 4 ines in each set Dirty bit needed, AKA Modified bit = 1 Vaid bit needed = 1 Overhead per ine: = 26 bits 43

44 LRU Sampe 2 Sampe 2 focuses on one particuar Set: Let the 4 ines be numbered 0..3 Set is accessed in the order: ine 0 miss, ine 1 miss, ine 0 hit, ine 2 miss, ine 0 hit again, ine 3 miss, ine 0 hit again, and another miss Now cache is fu, now find an avaiabe ine by eviction, to have a ine for another miss! Assume initiay a cod cache, a ines in the cache were free before these accesses Probem: Once a ines are fied (Vaid bit is 1 for a 4 ines) some ine must be retired to make room for the new access that missed, but which? Answer is based on LRU poicy (Least Recenty Used ine), which here is ine 1 44

45 LRU Sampe 2 The access order, assuming a memory accesses are just reads (oads), no writes (no stores), i.e. dirty bit is aways cear: Read miss, a ines invaid, stream paragraph in ine 0 Read miss (impies new address), stream paragraph in ine 1 Read hit on ine 0 Read miss to a new address, stream paragraph into ine 2 Read hit, access ine 0 Read miss, stream paragraph into ine 3 Read hit, access ine 0 Now another Read Miss, a ines vaid, find ine to retire, AKA to evict Note that LRU age 00 2 is youngest for cache ine 0, and 11 2 is the odest ine (AKA the east recenty used ine) for cache ine 1, of the 4 reative ages out of 4 tota ines 45

46 LRU Sampe 2 46

47 LRU Sampe 2 1. Initiay, in a party cod cache, if we experience a miss, there wi be an empty ine (party cod cache), the paragraph is streamed into the empty ine, its reative age is set to 0, and a other ages are incremented by 1 2. In a warm cache (a ines are used) when a ine of age X experiences a hit, its new age becomes 0. Ages of a other ines whose ages were < X are incremented by 1 3. Of course the oder ones remain oder 47

48 Compute Cache Size Typica Cache Design Parameters: 1. Number of ines in every set: K 2. Number of bytes in a ine, i.e. the Length of ine: L 3. Number of sets in memory, and hence in cache: N 4. Poicy upon memory write (cache write poicy) 5. Poicy upon read miss (cache read poicy) 6. Repacement poicy (e.g. LRU, random, FIFO, etc.) 7. Size (bits) = K * ( 8 * L + tag + contro bits ) * N 48

49 Compute Cache Size Compute minimum number of bits for 8-way, setassociative cache with 64 sets, using cycic aocation of sets, ine ength L = 32 bytes, using LRU and write-back. Memory is byte addressabe, with 32-bit addresses: Tag = = 21 bits LRU 8-ways = 3 bits Dirty bit = 1 bit Vaid bit = 1 bit Overhead per ine = = 26 bits # of ines = K*N = 64 * 8 = 2 9 ines Data bits per cache ine= 32*8 = 2 8 bits Tota cache size = 2 9 *( )= 144,384 bits Size in bytes approx. = ~17.6 k Bytes 49

50 Trace Cache Trace Cache is a specia-purpose cache that does not hod (raw) instruction bits, but instead stores predecoded operations AKA micro-ops The od AMD K5 uses a Trace Cache; see [1] Inte s Pentium P4 uses a 12 k micro-op Trace Cache Advantages: faster access to executabe bits at every cached instruction Disadvantage: ess dense storage, i.e. wasted cache bits, when compared to a reguar I-cache Note that cache bits are way more costy than memory bits; severa decima orders of magnitude! Trace caches are faing out of favor since eary

51 Trace Cache 51

52 Characteristic Cache Curve In the graph beow we a use reative number of cache misses [RM] to avoid infinitey high abscissa RM = 0 is idea case: No misses, a hits RM = 1 is worst case: A memory accesses are cache misses If a program exhibits good ocaity, reative cache size of 1 resuts in good performance; we use this as the reference point: Very coarsey, in some ranges, doubing the cache s size resuts in 30% ess cache misses In other ranges of the characteristic curve, doubing the cache resuts in just a few % of reduced misses: beyond the sweet spot! 52

53 Characteristic Cache Curve 53

54 Cache vs. Core on 22 nm Die 54

55 UP Cache Summary Cache is a specia HW storage, aowing fast access to sma areas of memory, copied into cache ines Buit with expensive technoogy, hence the size of a cache reative to memory size is sma; cache hods ony a sma subset of memory, typicay < 1 % Frequenty used data (or instructions in an I-cache) are copied to cache, with the hope that the data present in the cache are accessed frequenty Miracuousy that is generay true, so caches in genera do speed up execution despite sow memories: Expoiting what is known as ocaity Caches are organized into sets, with each set having 1 or more ines; mutipe ines require searching Defined portions of memory get mapped into any one of these sets 55

56 Bibiography 1. Shen, John Pau, and Mikko H. Lipasti: Modern Processor Design, Fundamentas of Superscaar Processors, McGraw Hi, catid=11&threadid=29382&enterthread=y 3. Lam, M., E. E. Rothberg, and M. E. Wof [1991]. "The Cache Performance and Optimizations of Bocked Agorithms," ACM /91, p MESI: 6. Kiburn, T., et a: One-eve storage systems, IRE Transactions, EC-11, 2, 1962, p

57 Bibiography 7. Don Anderson and Shaney, T., MindShare [1995]. Pentium TM Processor System Architecture, Addison-Wesey Pubishing Company, Reading MA, PC System Architecture Series. ISBN Pentium Pro Deveoper s Manua, Voume 1: Specifications, 1996, one of a set of 3 voumes 9. Pentium Pro Deveoper s Manua, Voume 2: Programmer's Reference Manua, Inte document, 1996, one of a set of 3 voumes 10. Pentium Pro Deveoper s Manua, Voume 3: Operating Systems Writer s Manua, Inte document, 1996, one of a set of 3 voumes 11. Y. Sheffer: ectures/mesi.pdf 12. MOESI protoco: MOESI_protoco 13. MESIF protoco: 57

58 Definitions For Caches 58

59 Aging Definitions A cache ine s age is tracked; ony in associative cache, doesn t appy for direct-mapped cache Aging tracks, when a cache ine was accessed, reative to the other ines in this set This impies that ages are compared Generay, the reative ages are of interest, such as: am I oder than you? Rather than the absoute age, e.g.: I was accessed at cyce such and such Think about the minimum number of bits needed to store the reative ages of, say, 8 cache ines! Memory access addresses ony one ine, hence a ines in a set have distinct (reative) ages 59

60 Definitions Aignment Aignment is a spacing requirement, i.e. the restriction that an address adhere to a specific pacement condition For exampe, even-aignment means that an address is even, that it be divisibe by 2 E.g. address 3 is not even-aigned, but address 1000 is; thus the rightmost address bit wi be 0 In VMM, page addresses are aigned on pageboundaries. If a page-frame has size 4k, then page addresses that adhere to page-aignment are eveny divisibe by 4k As a resut, the ow-order (rightmost) 12 bits are 0. Knowedge of aignment can be expoited to save storing address bits in VMM, caching, etc. 60

61 Aocate-on-Write Definitions If a store instruction experiences a cache miss, and as a resut a cache ine is fied, then the aocate-on-write cache poicy is used If the write miss causes the paragraph from memory to be streamed into a data cache ine, we say the cache uses aocate-on-write Pentium processors, for exampe, do not use aocate-on-write Antonym: write-by 61

62 Associativity Definitions If a cache has mutipe ines per set, we ca it k-way associative; k stands for number of ines in a set Having a cache with mutipe ines (i.e. k > 1) does require searching, or address comparing; search checks, whether some referenced object is in fact present Another way of saying this is: In an associative cache any memory object has more cache ines than just one, where it might ive Antonym: direct mapped; if ony a singe ine (per set) exists, the search is reduced to a simpe, singe tag comparison 62

63 Back-Off Definitions If processor P1 issues a store to a data address shared with another processor P2, and P2 has cached and modified the same data, a chance for data inconsistency arises To avoid this, P2 with the modified cache ine must snoop for other processors accesses, to guarantee deivery of the newest data Once the snoop detects the access request from P1, P1 must be prevented from getting ownership of the data; accompished by temporariy preventing P1 bus access This bus denia for the sake of preserving data integrity is caed back-off 63

64 Bocking Cache Definitions Let a cache miss resut in streaming-in a ine If during that stream-in no further accesses can be made to this cache unti the data transfer is compete, this cache is caed bocking Antonym: non-bocking Generay, a bocking cache yieds ower performance than a non-bocking 64

65 Bus Master Definitions Ony one of the devices connected to a system bus has the right to send signas across the bus; this ownership is caed being the bus master Initiay Memory & IO Controer (MIOC) is bus master; chipset may incude specia-purpose bus arbiter Over time, a processors or their caches may request to become bus master for some number of bus cyces The MIOC can grant this right; yet each of the processors p i (more specificay: its cache) can request a back-off for p j, even if otherwise p j woud be bus master 65

66 Critica Chunk First Definitions The number of bytes in a ine is generay arger than the number of bytes that can be brought to the cache across the bus in 1 step, requiring mutipe bus transfers to fi a ine competey Woud be efficient, if the actuay needed bytes resided in the first chunk brought across the bus Deiberate poicy that accompishes just that is the Critica Chunk First poicy This aows the cache to be unbocked after the first transfer, though ine is not competey oaded Other parts of the ine may be used ater, but the critica byte can thus be accessed right away 66

67 Direct Mapped Definitions If each memory address has just one possibe ocation (i.e. one singe ine, of K = 1) in the cache where it coud possiby reside, then that cache is caed direct mapped Antonym: associative, or fuy associative Synonym: non-associative 67

68 Directory Definitions The coection of a tags is referred to as the cache directory In addition to the directory and the actua data there may be further overhead bits in a data cache Dirty Bit Dirty bit is a data structure associated with a cache ine. This bit expresses whether a write hit has occurred on a system appying write-back Synonym: Modified bit There may be further overhead bits in a data cache 68

69 Definitions Effective Cyce Time t eff Let the cache hit rate h be the number of hits divided by the number of a memory accesses, with an idea hit rate being 1; m being the miss rate = 1-h; thus: t eff = t cache + (1-h) * t mem = t cache + m * t mem Aternativey, the effective cyce time might be t eff = max( t cache, m * t mem ) The atter hods, if a memory access to retrieve the data is initiated simutaneousy to the cache access t cache = time to access a datum in the cache, ideay 1 cyce, whie t mem is the time to access a data item in memory; generay not a constant vaue The hit rate h varies from 0.0 to

70 Excusive Definitions State in MESI protoco. The E state indicates that the current cache is not aware of any other cache sharing the same information, and that the ine is unmodified E aows that in the future another ine may contain a copy of the same information, in which case the E must transition to another state Possibe that a higher-eve cache (L1 for exampe viewed from an L2) may actuay have a shared copy of the ine in excusive state; however that eve of sharing is transparent to other potentiay sharing agents outside the current processor 70

71 Definitions Fuy Associative Cache Possibe to not partition cache into sets In that case, a ines need to be searched for a cache hit or miss We ca this a fuy associative cache Generay works for sma caches, since the search may become costy in time or HW if the cache were arge 71

72 Hit Rate h Definitions The hit rate h is the number of memory accesses (read/writes, or oad/stores) that hit the cache, over the tota number of memory accesses By contrast H is the tota number of hits A hit rate h = 1 means: a accesses are from the cache, whie h = 0 means, a are from memory, i.e. none hit the cache Conventiona notations are: h r and h w for read and write hits See aso miss rate 72

73 Invaid Definitions State in the MESI protoco State I indicates that its cache ine is invaid, and consequenty hods no vaid data; it is ready for use It is desirabe to have I ines: Aows the stream-in of a paragraph without evicting another cache ine Invaid (I) state is aways set for any cache ine after a system reset 73

74 Line Definitions Storage area in cache abe to hod a copy of a contiguous bock of memory ces, i.e. a paragraph The portion of memory stored in that ine is aigned on an address moduo the ine size For exampe, if a ine hods 64 bytes on a byteaddressabe architecture, the address of the first byte has 6 traiing zeros: eveny divisibe by 64, it is 64-byte aigned Such known zeros don t need to be stored in the tag, the address bits stored in the cache; they are impied This shortens the tag, rendering cache cheaper to manufacture: ess HW bits! 74

75 LLC Definitions Last Leve Cache is the argest cache in the memory hierarchy, the one cosest to physica memory, or furthest from the processor Typica on muti-core architectures Typica cash sizes: 4 MB to 32 MB Common to have one LLC be shared between a cores of an MCP (Muti-Core Processor), but have option of separating (by fusing) and creating dedicated LLC caches, with identica tota size 75

76 LRU Definitions Acronym for Least Recenty Used Cache repacement poicy (aso page repacement poicy discussed under VMM) that requires aging information for the ines in a set Each time a cache ine is accessed, that ine become the youngest one touched Other ines of the same set do age by one unit, i.e. get oder by 1 event: event is a memory access Reative ages are sufficient for LRU tracking; no need to track exact ages! Antonym: ast recenty used! 76

77 Locaity of Data Definitions A surprising, beneficia attribute of memory access patterns: when an address is referenced, there is a good chance that in the near future another access wi happen at or near that same address I.e. memory accesses tend to custer, aso observabe in hashing functions and memory page accesses Antonym: Randomy distributed, or normay distributed 77

78 MESI Definitions Acronym for Modified, Excusive, Shared and Invaid This is an ancient protoco to ensure cache coherence on the famiy of Pentium processors. A protoco is necessary, if mutipe processors have copy of common data with right to modify Through the MESI protoco data coherence is ensured no matter which of the processors performs writes AKA as Iinois protoco due to its origin at the University of Iinois at Urbana-Champaign 78

79 Definitions Miss Rate Miss rate is the number of memory (read/write) accesses that miss the cache over tota number of accesses, denoted m Ceary the miss rate, ike the hit rate, varies between The miss rate m = 1 - h Antonym: hit rate h 79

80 Modified State in MESI protoco Definitions M state impies that the cache ine found by a write hit was excusive, and that the current processor has modified the data The modified state expresses: Currenty not shared, excusivey owned data have been modified In a UP system, this is generay expressed by the dirty bit 80

81 Paragraph Definitions Conceptua, aigned, fixed-size area of the ogica address space that can be streamed into cache Area in the cache of paragraph-size is caed a ine In addition to the actua data, a ine in cache has further information, incuding the dirty and vaid bit (in UP systems), the tag, LRU information, and in MP systems the MESI bits The MESI M state corresponds to the dirty bit in a UP system 81

82 Repacement Poicy Definitions A repacement poicy is a defined convention that defines which ine is to be retired in case a new ine must be oaded, none is free in a set, so one has to be evicted Ideay, the ine that woud remain unused for the ongest time in the future shoud be repaced and its contents overwritten with new data Generay we do not know which ine wi stay unreferenced for the ongest time in the future In a direct-mapped cache, the repacement poicy is trivia: it is moot, as there wi be just 1 ine 82

83 Set Definitions A ogicay connected region of memory, to be mapped onto a specific area of cache (ine), is a set; there are N sets in memory Eements of a set don t need to be physicay contiguous in memory; if contiguous, eftmost og 2 (N) bits are 0; if cycic distribution, then the rightmost og 2 (N) after aignment bits are 0 The number of sets is conventionay abeed N A degenerate case is to map a memory onto the whoe cache, in which case ony a singe set exists: N = 1; i.e. one set Notion of set is meaningfu ony if there are mutipe sets. A memory region beonging to one set can be physicay contiguous or distributed cycicay In the former case the distribution is caed bocked, the atter cycic. Cache area into which a portion of memory is mapped to is aso caed set 83

84 Definitions Set-Associative A cached system in which each set has mutipe cache ines is caed set-associative For exampe, 4-way set associative means that there are mutipe sets (coud be 4 sets, 256 sets, 1024 sets, or any other number of sets) and each of those sets has 4 ines Integra powers of 2 are good to use That s what the 4 refers to in a 4-way cache Antonym: non-associative, AKA direct-mapped 84

85 Shared Definitions State in the MESI protoco S state expresses that the hit ine is present in more than one cache. Moreover, the current cache (with the shared state) has not modified the ine after stream-in Another cache of the same processor may be such a sharing agent. For exampe, in a two eve cache, the L2 cache wi hod a data present in the L1 cache Simiary, another processor s L2 cache may share data with the current processor s L2 cache 85

86 Definitions Stae Memory A vaid cache ine may be overwritten with new data The write-back poicy records such over writing At the moment of a cache write with write-back, cache and memory are out of synch; we say memory is stae Poses no danger, since the dirty bit (or modified bit) refects that memory eventuay must be updated But unti this happens, memory is stae Note that if two processors caches share memory and one cache renders memory stae, the other processor shoud no onger have access to that portion of shared memory 86

87 Definitions Stream-Out Streaming out a ine refers to the movement of one ine of modified data, out of the cache and back into a memory paragraph Stream-In The movement of one paragraph of data from memory into a cache ine. Since ine ength generay exceeds the bus width (i.e. exceeds the number of bytes that can be move in a singe bus transaction), a stream-in process requires mutipe bus transactions in a row Possibe that the byte actuay needed wi arrive ast in a cache ine during a sequence of bus transactions; can be avoided with the critica chunk first poicy 87

88 Snooping Definitions After a ine write hit in a cache using write-back, the data in cache and memory are no onger identica. In accordance with the write-back poicy, memory wi be written eventuay, but unti then memory is stae The modifier (the cache that wrote) must pay attention to other bus masters trying to access the same ine. If this is detected, action must be taken to ensure data integrity This paying attention is caed snooping. The right action may be forcing a back-off, or snarfing, or yet something ese that ensures data coherence Snooping starts with the owest-order cache, here the L2 cache. If appropriate, L2 ets L1 snoop for the same address, because L1 may have further modified the ine 88

89 Squashing Starting with a read-miss: Definitions In a non-bocking cache, a subsequent memory access may be issued after a read-miss, even if that previous miss resuts in a stream-in that is sti under way That subsequent memory access wi be a miss again, which is being queued. Whenever an access references an address for which a request is aready outstanding, the dupicate request to stream-in can be skipped Not entering this in the queue is caed squashing The second and any further outstanding memory access can be resoved, once the first stream-in resuts in the ine being present in the cache 89

90 Strong Write Order Definitions A poicy ensuring that memory writes occur in the same order as the store operations in the executing object code Antonym: Weak order The advantage of weak ordering can be speed gain, aowing a compier or cache poicy to schedue instructions out of order; this requires some other poicy to ensure data integrity 90

91 Definitions Stream-In The movement of a paragraph from memory into a cache ine Since ine ength generay exceeds the bus width (i.e. exceeds the number of bytes that can be move in a singe bus transaction), a stream-in process requires mutipe bus transactions It is possibe that the byte actuay needed arrives ast (or first) in a cache ine during a sequence of bus transactions Antonym: Stream-out 91

92 Stream-Out Definitions The movement of one ine of modified data from cache into a memory paragraph Antonym: Stream-in Note that unmodified data don t need to be streamed-out from cache to memory; they are aready present in memory 92

93 Definitions Trace Cache Specia-purpose cache that hods predecoded instructions, AKA micro-ops Advantage: Repeated decoding for instructions is not needed Trace caches have faen out of favor in the 2000s 93

94 Vaid Bit Definitions Singe-bit data structure per cache ine, indicating, whether or not the ine is free; free means invaid If a ine is not vaid (i.e. if vaid bit is 0), it can be fied with a new paragraph upon a cache miss Ese, (vaid bit 1), the ine hods vaid information After a system reset, a vaid bits of the whoe cache are set to 0 The I bit in the MESI protoco takes on that roe on an MP cache subsystem To be discussed in MP-cache coherence topic 94

95 Weak Write Order Definitions A memory-write poicy aowing (a compier or cache) that memory writes may occur in a different order than their originating store operations Antonym: Strong Write Order The advantage of weak ordering is potentia speed gain 95

96 Write-Back Definitions Cache write poicy that keeps a ine of data (a paragraph) in the cache even after a write, i.e. after a modification The changed state must be remembered via the dirty bit, AKA Modified state, or modified bit Memory is temporariy stae in such a case Upon retirement, any dirty ine must be copied back into memory; caed write-back Advantage: ony one stream-out, no matter how many write hits did occur to that same ine 96

97 Write-By Definitions Cache write poicy, in which the cache is not accessed on a write miss, even if there are cache ines in I state A cache using write-by hopes that soon there may be a oad, which wi resut in a miss and then stream-in the appropriate ine; if not, it was not necessary to stream-in the ine in the first pace Antonym: aocate-on-write 97

98 Write-Once Definitions Cache write poicy that starts out as write-through and changes to write-back after the first write hit to a ine Typica poicy imposed onto a higher eve L1 cache by the L2 cache Advantage: The L1 cache paces no unnecessary traffic onto the system bus upon a cache-write hit Lower eve L2 cache can remember that a write has occurred by setting the MESI state to modified 98

99 Definitions Write-Through Cache write poicy that writes data to memory upon a write hit. Thus, cache and main memory are in synch Disadvantage: repeated memory access traffic on the bus 99