14:332:331. Week 13 Basics of Cache

Transcription

1 14:332:331 Computer Architecture and Assembly Language Spring 2006 Week 13 Basics of Cache [Adapted from Dave Patterson s UCB CS152 slides and Mary Jane Irwin s PSU CSE331 slides] 331 Week131 Spring 2006

2 A question to think about Given a pipelined datapath, which instruction may slow down the pipeline the most: R-type beq j lw sw 331 Week132 Spring 2006

3 Review: Major Components of a Computer Processor Devices Control Memory Input Datapath Output 331 Week133 Spring 2006

4 A Typical Memory Hierarchy By taking advantage of the principle of locality: Present the user with as much memory as is available in the cheapest technology Provide access at the speed offered by the fastest technology On-Chip Components Control edram Datapath RegFile ITLB DTLB Instr Data Cache Cache Second Level Cache (SRAM) Main Memory (DRAM) Secondary Memory (Disk) Speed (ns): 1 s 1 s 10 s 100 s 1,000 s Size (bytes): 100 s K s 10K s M s T s Cost: highest lowest 331 Week134 Spring 2006

5 Characteristics of the Memory Hierarchy Increasing distance from the processor in access time Processor 4-8 bytes (word) L1$ 8-32 bytes (block) L2$ 1 block Main Memory Inclusive what is in L1$ is a subset of what is in L2$ is a subset of what is in MM that is a subset of is in SM Secondary Memory 1,023+ bytes (disk sector = page) (Relative) size of the memory at each level 331 Week135 Spring 2006

6 Performance Why Care About the Memory Hierarchy? Processor-DRAM Memory Gap Moore s Law Time Week136 Spring 2006 CPU Processor-Memory Performance Gap: (grows 50% / year) DRAM µproc 60%/year (2X/15yr) DRAM 9%/year (2X/10yrs)

7 Memory Hierarchy: Goals Fact: Large memories are slow, fast memories are small How do we create a memory that gives the illusion of being large, cheap and fast (most of the time)? by taking advantage of The Principle of Locality: Programs access a relatively small portion of the address space at any instant of time Probability of reference 0 Address Space 2 n Week137 Spring 2006

8 Memory Hierarchy: Why Does it Work? Temporal Locality (Locality in Time): => Keep most recently accessed data items closer to the processor Spatial Locality (Locality in Space): => Move blocks consists of contiguous words to the upper levels To Processor From Processor Upper Level Memory Blk X Lower Level Memory Blk Y 331 Week138 Spring 2006

9 Memory Hierarchy: Terminology Hit: data appears in some block in the upper level (Block X) Hit Rate: the fraction of memory accesses found in the upper level Hit Time: Time to access the upper level which consists of RAM access time + Time to determine hit/miss To Processor From Processor Upper Level Memory Blk X Lower Level Memory Blk Y Miss: data needs to be retrieve from a block in the lower level (Block Y) Miss Rate = 1 - (Hit Rate) Miss Penalty: Time to replace a block in the upper level + Time to deliver the block the processor Hit Time << Miss Penalty 331 Week139 Spring 2006

10 How is the Hierarchy Managed? registers <-> memory by compiler (programmer?) cache <-> main memory by the hardware main memory <-> disks by the hardware and operating system (virtual memory) by the programmer (files) 331 Week1310 Spring 2006

11 Cache Two questions to answer (in hardware): Q1: How do we know if a data item is in the cache? Q2: If it is, how do we find it? First method: Direct mapped - For each item of data at the lower level, there is exactly one location in the cache where it might be (ie, lots of items at the lower level share locations in the upper level) Block size is one word of data Mapping: (word address) modulo (# of words in the cache) 331 Week1311 Spring 2006

12 Caching: A Simple First Example Cache Valid Tag Q1: Is it there? Data Compare the cache tag to the high order 2 memory address bits to tell if the memory block is in the cache Main Memory Q2: How do we find it? Use low order 2 memory address bits to determine which cache block (ie, modulo the number of blocks in the cache) 331 Week1312 Spring 2006

13 Direct Mapped Cache Consider the main memory word reference string Start with an empty cache - all blocks initially marked as not valid Week1313 Spring 2006

14 Another Reference String Mapping Consider the main memory word reference string Start with an empty cache - all blocks initially marked as not valid Week1314 Spring 2006

15 Sources of Cache Misses Compulsory (cold start or process migration, first reference): first access to a block Cold fact of life, not a whole lot you can do about it If you are going to run billions of instruction, Compulsory Misses are insignificant Conflict (collision): Multiple memory locations mapped to the same cache location Solution 1: increase cache size Solution 2: increase associativity Capacity: Cache cannot contain all blocks accessed by the program Solution: increase cache size 331 Week1315 Spring 2006

16 MIPS Direct Mapped Cache Example One word/block, cache size = 1K words Byte offset Hit Tag Index Data Index Valid Tag Data Week1316 Spring 2006

17 Multiword Block Direct Mapped Cache Four words/block, cache size = 1K words Hit Byte offset Data Tag 20 Index 8 Block offset Index Valid Tag 20 Data What kind of locality are we taking advantage of? 331 Week1317 Spring

18 Taking Advantage of Spatial Locality Let cache block hold more than one word Start with an empty cache - all blocks initially marked as not valid Week1318 Spring 2006

19 Reducing Cache Miss Rates #1 1 Allow more flexible block placement In a direct mapped cache a memory block maps to exactly one cache block At the other extreme, could allow a memory block to be mapped to any cache block fully associative cache A compromise is to divide the cache into sets each of which consists of n ways (n-way set associative) A memory block maps to a unique set (specified by the index field) and can be placed in any way of that set (so there are n choices) (block address) modulo (# sets in the cache) 331 Week1319 Spring 2006

20 Set Associative Cache Example Cache Way 0 1 Set V Tag Q1: Is it there? Data Compare all the cache tags in the set to the high order 3 memory address bits to tell if the memory block is in the cache 0000xx 0001xx 0010xx 0011xx 0100xx 0101xx 0110xx 0111xx 1000xx 1001xx 1010xx 1011xx 1100xx 1101xx 1110xx 1111xx Main Memory Two low order bits define the byte in the word (32-b words) One word blocks Q2: How do we find it? Use next 1 low order memory address bit to determine which cache set (ie, modulo the number of sets in the cache) 331 Week1320 Spring 2006

21 Another Reference String Mapping Consider the main memory word reference string Start with an empty cache - all blocks initially marked as not valid Week1321 Spring 2006

22 Four-Way Set Associative Cache 2 8 = 256 sets each with four ways (each with one block) Byte offset Tag 22 Index 8 Index V Tag Data V Tag Data V Tag Data V Tag Data Hit 4x1 select Data 331 Week1322 Spring 2006

23 Range of Set Associative Caches For a fixed size cache, each increase by a factor of two in associativity doubles the number of blocks per set (ie, the number or ways) and halves the number of sets decreases the size of the index by 1 bit and increases the size of the tag by 1 bit Used for tag compare Selects the set Selects the word in the block Tag Index Block offset Byte offset Decreasing associativity Direct mapped (only one way) Smaller tags Increasing associativity Fully associative (only one set) Tag is all the bits except block and byte offset 331 Week1323 Spring 2006

24 Handling Cache Misses Handling hit is trivial Handling misses needs to stall the processor Upon an instruction cache miss Send the original PC value (current PC 4) to the memory Instruct main memory to perform a read and wait for the memory to complete its access Write the cache entry, putting the data from memory in the data portion of the entry, writing the upper bits of the address (from the ALU) into the tag field, and turning the valid bit on Restart the instruction execution at the first step, which will re-fetch the instruction, this time finding it in the cache Similar for data cache miss 331 Week1324 Spring 2006

25 Handling Writes The cache and memory are inconsistent when their values (of the same data) are different A simple solution: write through Write to both the cache and the memory at the same time Poor performance Every store instruction needs to stall the processor (a memory access can take 100 CPU cycles) Alternative: write back Write to the cache; write to the memory when the cache block is replaced later 331 Week1325 Spring 2006

26 Cache Summary The Principle of Locality: Program likely to access a relatively small portion of the address space at any instant of time - Temporal Locality: Locality in Time - Spatial Locality: Locality in Space Three Major Categories of Cache Misses: Compulsory Misses: sad facts of life Example: cold start misses Conflict Misses: increase cache size and/or associativity Nightmare Scenario: ping pong effect! Capacity Misses: increase cache size Cache Design Space total size, block size, associativity (replacement policy) write-hit policy (write-through, write-back) write-miss policy (write allocate, write buffers) 331 Week1326 Spring 2006

27 Memory Systems that Support Caches The off-chip interconnect and memory architecture can affect overall system performance in dramatic ways on-chip 32-bit data & 32-bit addr per cycle CPU Cache bus Memory One word wide organization (one word wide bus and one word wide memory) Assume 1 1 clock cycle (2 ns) to send the address 2 25 clock cycles (50 ns) for DRAM cycle time, 8 clock cycles (16 ns) access time 3 1 clock cycle (2ns) to return a word of data Memory-Bus to Cache bandwidth number of bytes accessed from memory and transferred to cache/cpu per clock cycle 331 Week1327 Spring 2006

28 One Word Wide Memory Organization on-chip CPU Cache bus Memory If the block size is one word, then for a memory access due to a cache miss, the pipeline will have to stall the number of cycles required to return one data word from memory cycle to send address cycles to read DRAM cycle to return data total clock cycles miss penalty Number of bytes transferred per clock cycle (bandwidth) for a single miss is 4/27 = 0148 bytes per clock 331 Week1328 Spring 2006

29 One Word Wide Memory Organization, con t What if the block size is four words? on-chip 1 cycle to send 1 st address CPU 4 x 25 = cycles to read DRAM cycles to return last data word Cache 102 total clock cycles miss penalty bus Memory 25 cycles 25 cycles 25 cycles 25 cycles Number of bytes transferred per clock cycle (bandwidth) for a single miss is (4 x 4)/102 = 0157 bytes per clock 331 Week1329 Spring 2006

30 Interleaved Memory Organization on-chip CPU Cache For a block size of four words 1 cycle to send 1 st address = 28 cycles to read DRAM 1 cycles to return last data word 30 total clock cycles miss penalty bus Memory Memory Memory Memory bank 0 bank 1 bank 2 bank 3 25 cycles 25 cycles 25 cycles 25 cycles Number of bytes transferred per clock cycle (bandwidth) for a single miss is (4 x 4)/30 = 0533 bytes per clock 331 Week1330 Spring 2006

31 DRAM Memory System Summary Its important to match the cache characteristics caches access one block at a time (usually more than one word) with the DRAM characteristics use DRAMs that support fast multiple word accesses, preferably ones that match the block size of the cache with the memory-bus characteristics make sure the memory-bus can support the DRAM access rates and patterns with the goal of increasing the Memory-Bus to Cache bandwidth 331 Week1331 Spring 2006