CS 61C: Great Ideas in Computer Architecture (Machine Structures) More Cache: Set Associa0vity. Smart Phone. Today s Lecture. Core.

Transcription

1 CS 6C: Great Ideas in Computer Architecture (Machine Structures) More Cache: Set Associavity Instructors: Randy H Katz David A PaGerson Guest Lecture: Krste Asanovic hgp://insteecsberkeleyedu/~cs6c/fa Spring - - Lecture # Spring - - Lecture # New- School Machine Structures (It s a bit more complicated!) Parallel Requests Assigned to computer eg, Search Katz Parallel Threads Assigned to core eg, Lookup, Ads So3ware Parallel Instruc\ons > one \me eg, 5 pipelined instruc\ons Parallel Data > data one \me eg, Add of 4 pairs of words Hardware descrip\ons All one \me Harness Parallelism & Achieve High Performance Hardware Warehouse Scale Computer Core Input/Output Instruc\on Unit(s) Main Computer (Cache) Core Core Func\onal Unit(s) A +B A +B A +B A 3 +B 3 Smart Phone Today s Lecture Logic Gates Spring - - Lecture # 3 Review Big Ideas of Instruc\on- Level Parallelism Pipelining, Hazards, and Stalls Forwarding, Specula\on to overcome Hazards Mul\ple issue to increase performance IPC instead of CPI Dynamic Execu\on: Superscalar in- order issue, branch predic\on, register renaming, out- of- order execu\on, in- order commit unroll loops in HW, hide cache misses Spring - - Lecture # 4 Agenda Recap: Components of a Computer Cache Recap Administrivia Set- Associa\ve Caches Technology Break AMAT and Mul\level Cache Review Processor Control Datapath Devices Input Output Secondary (Disk) Main Cache Spring - - Lecture # 5 Spring - - Lecture # 6

2 Recap: Typical Hierarchy Take advantage of the principle of locality to present the user with as much memory as is available in the cheapest technology at the speed offered by the fastest technology On- Chip Components Control Datapath RegFile ITLB DTLB Instr Data Cache Cache Second Level Cache (SRAM) Main (DRAM) Speed (%cycles): ½ s s s s, s Size (bytes): s K s M s G s T s Cost: highest lowest Secondary (Disk) Spring - - Lecture # 7 Recap: Cache Performance and Average Access Time (AMAT) CPU \me = IC CPI CC = IC (CPI ideal + - stall cycles) CC CPI stall - stall cycles = Read- stall cycles + Write- stall cycles Read- stall cycles = reads/program read miss rate read miss penalty Write- stall cycles = (writes/program write miss rate write miss penalty) + write buffer stalls AMAT is the average \me to access memory considering both hits and misses AMAT = Time for a hit + Miss rate x Miss penalty Spring - - Lecture # 8 Improving Cache Performance Reduce the \me to hit in the cache Eg, Smaller cache, direct- mapped cache, special tricks for handling writes Reduce the miss rate Eg, Bigger cache, larger blocks More flexible placement (increase associavity) Reduce the miss penalty Eg, Smaller blocks or cri\cal word first in large blocks, special tricks for handling writes, faster/higher bandwidth memories Use mul\ple cache levels Sources of Cache Misses: The 3Cs Compulsory (cold start or process migra\on, st reference): First access to block impossible to avoid; small effect for long running programs Solu\on: increase block size (increases miss penalty; very large blocks could increase miss rate) Capacity: Cache cannot contain all blocks accessed by the program Solu\on: increase cache size (may increase access \me) Conflict (collision): Mul\ple memory loca\ons mapped to the same cache loca\on Solu\on : increase cache size Solu\on : increase associa\vity (may increase access \me) Spring - - Lecture # 9 Spring - - Lecture # Reducing Cache Misses Allow more flexible block placement Direct mapped $: memory block maps to exactly one cache block Fully associave $: allow a memory block to be mapped to any cache block Compromise: divide $ into sets, each of which consists of n ways (n- way set associave) to place memory block block maps to unique set determined by index field and is placed in any of the n- ways of that set Calcula\on: (block address) modulo (# sets in the cache) Spring - - Lecture # Alterna\ve Block Placement Schemes DM placement: mem block in 8 block cache: only one cache block where mem block can be found ( modulo 8) = 4 SA placement: four sets x - ways (8 cache blocks), memory block in set ( mod 4) = ; either element of the set FA placement: mem block can appear in any cache blocks Spring - - Lecture #

3 Agenda Cache Recap Administrivia Set- Associa\ve Caches Technology Break AMAT and Mul\level Cache Review Administrivia Project 4, Part due Sunday 4/7 Design a 6- bit pipelined computer in Logisim Extra Credit due 4/4 Fastest Matrix Mul\ply Final Review: Mon 5/ 5 8PM, 5 VLSB Final Exam: Mon 5/9 :3- :3PM Haas Pavilion Spring - - Lecture # 3 Spring - - Lecture # 4 Last survivor of group of women programmers of ENIAC, the first all- electronic digital calculator Used for calcula\ng ar\llery tables, but completed in 946 too late for WWII 6c in the News: Jean Bar\k, dies age 86 A Pioneer Programmer When the Eniac was shown off at the University of Pennsylvania in February 946, it generated headlines in newspapers across the country But the agen\on was all on the men and the machine The women were not even introduced at the event Spring - - Lecture # 5 [NY Times, 4/7/] Geyng to Know Your Prof Grains are mashed ~45-6 minutes to convert starch to sugar Missing the ales I grew up with in England, I learnt how to make beer Start from grains, about 5- gallons/batch Brewed over 5 batches so far Need thirsty friends to help consume experiments! Boil extracted wort for 9 minutes, add hops, then cool to add yeast ESB II ~6% ABV ESB I ~6% ABV Ordinary BiGer 4%ABV Belgian Pale II ~5% ABV Belgian Pale I ~5% ABV Mixed Cider 7%ABV In cellar: Ferments for 5-7 days Gravenstein Cider 7%ABV Bri\sh Barleywine 8% ABV Spring - - Lecture # 6 Condi\on for week to years Belgian Trappist % ABV Example: 4- Word Direct- Mapped $ Worst- Case Reference String Consider the main memory word reference string Start with an empty cache - all blocks ini\ally marked as not valid 4 4 Example: 4- Word Direct- Mapped $ Worst- Case Reference String Consider the main memory word reference string Start with an empty cache - all blocks ini\ally marked as not valid miss 4 miss miss 4 miss 4 4 Mem() Mem() Mem(4) Mem() 4 4 miss 4 miss 4 miss 4 miss 4 Mem(4) Mem() Mem(4) Mem() Spring - - Lecture # 7 8 requests, 8 misses Ping pong effect due to conflict misses - two memory loca\ons that map into the same cache block Spring - - Lecture # 8 3

4 Example: - Way Set Associa\ve $ Cache Way Set V (4 words = sets x ways per set) Tag Q: Is it there? Data Compare all the cache tags in the set to the high order 3 memory address bits to tell if the memory block is in the cache xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx Main One word blocks Two low order bits define the byte in the word (3b words) Q: How do we find it? Use next low order memory address bit to determine which cache set (ie, modulo the number of sets in the cache) Spring - - Lecture # 9 Example: 4 Word - Way SA $ Same Reference String Consider the main memory word reference string Start with an empty cache - all blocks ini\ally marked as not valid 4 4 Spring - - Lecture # Example: 4- Word - Way SA $ Same Reference String Consider the main memory word reference string Start with an empty cache - all blocks ini\ally marked as not valid Example: Eight- Block Cache with Different Organiza\ons miss 4 miss hit 4 hit Mem() Mem() Mem() Mem() Mem(4) Mem(4) Mem(4) 8 requests, misses Solves the ping pong effect in a direct mapped cache due to conflict misses since now two memory loca\ons that map into the same cache set can co- exist! Spring - - Lecture # Total size of $ in blocks is equal to number of sets x associavity For fixed $ size, increasing associa\vity decreases number of sets while increasing number of elements per set With eight blocks, an 8- way set- associa\ve $ is same as a fully associa\ve $ Spring - - Lecture # Four- Way Set- Associa\ve Cache 8 = 56 sets each with four ways (each with one block) Byte offset Tag Index Index V Tag Data V Tag Data V Tag Data V Tag Data Way Way Way Way Range of Set- Associa\ve Caches For a fixed- size cache, each increase by a factor of two in associa\vity doubles the number of blocks per set (ie, the number or ways) and halves the number of sets decreases the size of the index by bit and increases the size of the tag by bit Tag Index Block offset Byte offset 3 4x select Spring - - Lecture # 3 Hit Data Spring - - Lecture # 4 4

5 Range of Set- Associa\ve Caches For a fixed- size cache, each increase by a factor of two in associa\vity doubles the number of blocks per set (ie, the number or ways) and halves the number of sets decreases the size of the index by bit and increases the size of the tag by bit Used for tag compare Tag Decreasing associa\vity Direct mapped (only one way) Smaller tags, only a single comparator Selects the set Index Increasing associa\vity Selects the word in the block Block offset Byte offset Fully associa\ve (only one set) Tag is all the bits except block and byte offset Costs of Set- Associa\ve Caches When miss occurs, which way s block selected for replacement? Least Recently Used (LRU): one that has been unused the longest Must track when each way s block was used rela\ve to other blocks in the set For - way SA $, one bit per set set to when a block is referenced; reset the other way s bit (ie, last used ) N- way set- associa\ve cache costs N comparators (delay and area) MUX delay (set selec\on) before data is available Data available a er set selec\on (and Hit/Miss decision) DM $: block is available before the Hit/Miss decision In Set- Associa\ve, not possible to just assume a hit and con\nue and recover later if it was a miss Spring - - Lecture # 5 Spring - - Lecture # 6 Cache Block Replacement Policies Random Replacement Hardware randomly selects a cache item and throw it out Least Recently Used Hardware keeps track of access history Replace the entry that has not been used for the longest \me For - way set- associa\ve cache, need one bit for LRU replacement Example of a Simple Pseudo LRU Implementa\on Assume 64 Fully Associa\ve entries Hardware replacement pointer points to one cache entry Whenever access is made to the entry the pointer points to: Move the pointer to the next entry Otherwise: do not move the pointer Replacement Pointer Entry Entry Spring - - Lecture # 7 Entry 63 : Benefits of Set- Associa\ve Caches Choice of DM $ or SA $ depends on the cost of a miss versus the cost of implementa\on Largest gains are in going from direct mapped to - way (%+ reduc\on in miss rate) Spring - - Lecture # 8 How to Calculate 3C s using Cache Simulator Compulsory: set cache size to infinity and fully associa\ve, and count number of misses Capacity: Chance cache size from infinity, usually in powers of, and count misses for each reduc\on in size 6 MB, 8 MB, 4 MB, 8 KB, 64 KB, 6 KB 3 Conflict: Change from fully associa\ve to n- way set associa\ve while coun\ng misses Fully associa\ve, 6- way, 8- way, 4- way, - way, - way Spring - - Lecture # 9 3Cs Revisted Three sources of misses (SPEC integer and floa\ng- point benchmarks) Compulsory misses 6%; not visible Capacity misses, func\on of cache size Conflict por\on depends on associa\vity and cache size Spring - - Lecture # 3 5

6 Reduce AMAT Use mul\ple levels of cache As technology advances, more room on IC die for larger L$ or for addi\onal levels of cache (eg, L$ and L3$) Normally the higher cache levels are unified, holding both instruc\ons and data Spring - - Lecture # 3 AMAT Revisited For a nd- level cache, L Equa\ons: AMAT = Hit Time L + Miss Rate L x Miss Penalty L Miss Penalty L = Hit Time L + Miss Rate L x Miss Penalty L AMAT = Hit Time L + Miss Rate L x (Hit Time L + Miss Rate L Miss Penalty L ) Defini\ons: Local miss rate: misses in this $ divided by the total number of memory accesses to this $ (Miss rate L ) Global miss rate: misses in this $ divided by the total number of memory accesses generated by the CPU (Miss Rate L x Miss Rate L ) Global miss rate is what magers to overall performance Local miss rate is factor in evalua\ng the effec\veness of L cache Spring - - Lecture # 3 CPI stalls Calcula\on Assume CPI ideal of cycle miss penalty to main memory 5 cycle miss penalty to Unified L$ 36% of instruc\ons are load/stores % L I$ miss rate; 4% L D$ miss rate 5% U(nified)L$ miss rate CPI stalls = + x x = 354 (vs 544 with no L$) Spring - - Lecture # 33 IFetch Ld/St L L Hierarchy with Two Cache Levels mem refs 4 mem refs mem refs CPU L$ L$ MM cycle cycles cycles For every CPU to memory references 4 will miss in L$; what is the miss rate? will miss in L$; what is the miss rate? Global vs local miss rate? If 5 mem refs per instruc\on, how do we normalize these numbers to # instruc\ons? Ave Mem Stalls/Instruc\on Spring - - Lecture # 34 AMAT Calcula\ons Local vs Global Miss Rates Example: For memory refs: 4 misses in L$ (miss rate 4%) misses in L$ (miss rate %) L$ hits cycle, L$ hits in cycles Miss to MM costs cycles 5 memory references per instruc\on (ie, 5% ld/st) mem refs = 667 instrs OR instrs = 5 mem refs Ask: Local miss rate AMAT Stall cycles per instruc\on with and without L$ With L$ Local miss rate = AMAT = Ave Mem Stalls/Ref = Ave Mem Stalls/Instr= Without L$ AMAT= Ave Mem Stalls/Ref= Ave Mem Stalls/Instr= Assume ideal CPI=, performance improvement = AMAT Calcula\ons Local vs Global Miss Rates Example: For memory refs: 4 misses in L$ (miss rate 4%) misses in L$ (miss rate %) L$ hits cycle, L$ hits in cycles Miss to MM costs cycles 5 memory references per instruc\on (ie, 5% ld/st) mem refs = 667 instrs OR instrs = 5 mem refs Ask: Local miss rate AMAT Stall cycles per instruc\on with and without L$ With L$ Local miss rate = 5% (/4) AMAT = +4%x(+5%x)=34 Ave Mem Stalls/Ref = (34- )=4 Ave Mem Stalls/Instr=4x5=36 Without L$ AMAT=+4%x=5 Ave Mem Stalls/Ref=(5- )=4 Ave Mem Stalls/Instr=4x5=6 Assume ideal CPI=, performance improvement = (6+)/(36+)=5% Spring - - Lecture # 35 Spring - - Lecture # 36 6

7 CPI/Miss Rates/DRAM Access SpecInt6 Design Considera\ons Different design considera\ons for L$ and L$ L$ focuses on fast access: minimize hit \me to achieve shorter clock cycle, eg, smaller $ L$, L3$ focus on low miss rate: reduce penalty of long main memory access \mes: eg, Larger $ with larger block sizes/ higher levels of associa\vity Miss penalty of L$ is significantly reduced by presence of L$, so can be smaller/faster even with higher miss rate For the L$, fast hit \me is less important than low miss rate L$ hit \me determines L$ s miss penalty L$ local miss rate >> than the global miss rate Spring - - Lecture # 37 Spring - - Lecture # 38 Improving Cache Performance Reduce the \me to hit in the cache Eg, Smaller cache, direct- mapped cache, special tricks for handling writes Reduce the miss rate (in L$, L3$) Eg, Bigger cache, larger blocks More flexible placement (increase associavity) Reduce the miss penalty (in L$) Eg, Smaller blocks or crical word first in large blocks, special tricks for handling for writes, faster/ higher bandwidth memories Use mulple cache levels Sources of Cache Misses: 3Cs for L$, L3$ Compulsory (cold start or process migra\on, st reference): First access to block impossible to avoid; small effect for long running programs Solu\on: increase block size (increases miss penalty; very large blocks could increase miss rate) Capacity: Cache cannot contain all blocks accessed by the program Soluon: increase cache size (may increase access me) Conflict (collision): Mul\ple memory loca\ons mapped to the same cache loca\on Solu\on : increase cache size Soluon : increase associavity (may increase access me) Spring - - Lecture # 39 Spring - - Lecture # 4 Two Machines Cache Parameters L cache organization & size L associativity Intel Nehalem Split I$ and D$; 3KB for each per core; 64B blocks 4-way (I), 8-way (D) set assoc; ~LRU replacement AMD Barcelona Split I$ and D$; 64KB for each per core; 64B blocks -way set assoc; LRU replacement L write policy write-back, write-allocate write-back, write-allocate L cache organization & size Unified; 56KB (5MB) per core; 64B blocks Unified; 5KB (5MB) per core; 64B blocks L associativity 8-way set assoc; ~LRU 6-way set assoc; ~LRU L write policy write-back write-back L write policy write-back, write-allocate write-back, write-allocate L3 cache organization & size Unified; 89KB (8MB) shared by cores; 64B blocks Unified; 48KB (MB) shared by cores; 64B blocks L3 associativity 6-way set assoc 3-way set assoc; evict block shared by fewest cores L3 write policy write-back, write-allocate write-back; write-allocate Spring - - Lecture # 4 Intel Nehalem Die Photo 4 cores, 3KB I$/3- KB D$, 5KB L$ Share one 8- MB L3$ Spring - - Lecture # 4 7

8 Cache Design Space Summary Several interac\ng dimensions Cache size Block size Associa\vity Replacement policy Write- through vs write- back Write alloca\on Op\mal choice is a compromise Depends on access characteris\cs Workload Use (I- cache, D- cache) Depends on technology / cost Simplicity o en wins Cache Size AssociaIvity Block Size Bad Good Factor A Factor B Less More Name of the Game: Reduce Cache Misses memory blocks mapping to same block knock each other out as program bounces from memory loca\on to next One way to do it: set- associa\vity block maps into more than cache block N- way: n possible places in cache to hold a memory block N- way Cache of N+M blocks: N ways x M sets Mul\- level caches Op\mize first level to be fast! Op\mize nd and 3 rd levels to minimize the memory access penalty Spring - - Lecture # 43 Spring - - Lecture # 44 8