Eastern Mediterranean University School of Computing and Technology CACHE MEMORY. Computer memory is organized into a hierarchy.

Transcription

1 Eastern Mediterranean University School of Computing and Technology ITEC255 Computer Organization & Architecture CACHE MEMORY Introduction Computer memory is organized into a hierarchy. At the highest level (closest to processor), are the processor registers. Next comes one or more levels (L1, L2, L3) of cache. Next comes main memory, which is usually made out of dynamic random-access memory (DRAM). All of these are considered internal to computer system (directly accessible by processor). 2 1

2 Introduction Hierarchy continues with external memory (accessible by processor via an I/O module), with next level typically being a fixed hard disk. One or more level below that consisting of removable media such as ZIP cartridges, optical disks and tape. 3 Introduction As one goes down memory hierarchy, one finds decreasing cost per bit, increasing capacity and slower access time. It would be nice to use only fastest memory, but because that is most expensive memory, we trade off access time for cost by using more of slower memory. The design challenge is to organize data and programs in memory so that the accessed memory are usually in faster memory. 4 2

3 Introduction In general, it is likely that most future accesses to main memory by the processor will be to locations recently accessed. So the cache automatically retains a copy of some of the recently used words from the DRAM. If the cache is designed properly, then most of the time the processor will request memory words that are already in the cache. 5 Computer Memory System Overview The complex subject of computer memory is made more manageable if we classify memory systems according to their key characteristics. Most important of these are: Location: Refers to whether memory is internal or external to the computer. Internal memory is often equated with main memory but there are other forms of internal memory such as registers and caches. External memory consists of peripheral storage devices such as disk and tape that are accessible to processor via I/O controllers. 6 3

4 Computer Memory System Overview Capacity: For internal and external memory, capacity is typically expressed in terms of bytes. Consider three related concepts for internal memory: Unit of transfer Word size Number of words 7 Computer Memory System Overview Method of accessing units of data Sequential access: Memory is organized into units of data, called records. Access must be made in a sequential linear sequence. Time to access an arbitrary record is highly variable. Tape units are sequential access. Direct access: Individual blocks or records have unique address based on physical location. Access is accomplished by direct access to general area of desired information, then some search for final location. Access time is variable, but not as much as sequential access. Disk units are direct access. 8 4

5 Computer Memory System Overview Random access: Each addressable location has a unique physical location. Access is direct access to desired location. Access time is constant and independent of prior accesses. Main memory and some cache systems are random access. Associative: Data is located by a comparison with contents of a portion of store. Access time is constant and independent of prior accesses. Cache memories may employ associative access. 9 Computer Memory System Overview Performance: Three performance parameters are used: Access time (latency): For random-access memory, this is time it takes to perform a read or write operation, that is, the time from the instant that an address is presented to the memory to the instant that data have been stored or made available for use. For non-random-access memory, access time is time it takes to position read-write mechanism at desired location. Memory cycle time: This concept is primarily applied to random-access memory and consists of the access time plus any additional time required before a second access can commence. Transfer rate: Rate at which data can be transferred into or out of a memory unit. Generally measured in bits/second. 10 5

6 Computer Memory System Overview A variety of physical types of memory have been employed. The most common today are semiconductor memory (RAM), magnetic surface memory (disk, tape) and optical (CD, DVD). Several physical characteristics of data storage are important: Volatile: Information is lost when power is switched off (RAM). Non-volatile: Information remains without deterioration until changed, no electrical power is needed (disk). Non-erasable memory: Cannot be altered, except by destroying storage unit (ROM). 11 Computer Memory System Overview There are trade-offs between three key characteristics of memory: cost, capacity, and access time. The following relationship holds: Faster access time greater cost per bit Greater capacity less cost per bit Greater capacity slower access time The dilemma facing the designer is clear. The designer would like to use memory technologies that provide for large-capacity memory, both because the capacity is needed and because the cost per bit is low. 12 6

7 Computer Memory System Overview However, to meet performance requirements, the designer needs to use expensive, relatively lower-capacity memories with short access times. The way out of this dilemma is not to rely on a single memory component or technology, but to employ a memory hierarchy. 13 Computer Memory System Overview A typical hierarchy is illustrated in the figure. 14 7

8 Computer Memory System Overview Success of the memory hierarchy scheme depends upon locality of reference principle. During course of execution of a program, memory references, both instructions and data, tend to cluster. Temporal locality. If a location is referenced, it is likely to be referenced again in near future. Positional locality. When a location is referenced, it is probably close to last location referenced. 15 Cache Memory Principles Cache memory is intended to give memory speed approaching that of the fastest memories available, and at the same time provide a large memory size at the price of less expensive types of semiconductor memories. There is a relatively large and slow main memory together with a smaller, faster cache memory. Cache is in middle of CPU and main memory. 16 8

9 Cache Memory Principles Cache contains a copy of portions of main memory. When processor attempts to read a word of memory, a check is made to determine if word is in cache. If so, word is delivered to processor. If not, a block of main memory is read into cache and then word is delivered to processor. Because of locality of reference, when a block of data is fetched into cache to satisfy a single memory reference, it is likely that there will be future references to that memory location or to other words in block. 17 Cache Memory Principles Figure shows the use of multiple levels of cache. The L2 cache is slower and typically larger than the L1 cache, and the L3 cache is slower and typically larger than the L2 cache. 18 9

10 Cache Memory Principles Main memory consists of up to 2 n addressable words, with each word having a unique n-bit address. This memory is considered to consists of a number of fixedlength blocks of K words each. That is, there are M=2 n /K blocks. 19 Cache Memory Principles Cache consists of C lines of K words each (C<<M). At any time, some subset of blocks of memory resides in lines in cache. If a word in a block of memory is read, that block is transferred to one of lines of cache. Cache includes tags to identify which block of main memory is in each cache slot

11 Cache Memory Principles Flowchart for cache read operation is shown below. RA=Read Address 21 Cache Memory Principles Cache connects to processor via data, control and address lines. Data and address lines are also attached to data and address buffers which attached to a system bus from which main memory is reached. When a cache hit occurs, data and address buffers are disabled and communication is only between processor and cache without system bus. When a cache miss occurs, desired address is loaded onto system bus and data are returned through data buffer to both cache and processor

12 There are a few basic design elements that serve to classify and differentiate cache architectures: 1. Cache address: Almost all processors, support virtual memory. If virtual addresses are used, the system designer may choose to place the cache between the processor and the memory management unit (MMU) or between the MMU and main memory. A logical cache, also known as a virtual cache, stores data using virtual addresses. The processor accesses the cache directly, without going through the MMU. A physical cache stores data using main memory physical addresses. 23 Logical cache Physical cache 24 12

13 2. Cache size: We would like size of cache to be small enough so that overall average cost per bit is close to that of main memory alone large enough so that overall average access time is close to that of cache alone Large caches tend to be slightly slower than small ones since they include more number of gates for cache addressing. Available chip and board area also limits cache size. Performance is sensitive to nature of workload, so there is no single optimum size Mapping function: Because there are fewer cache lines than main memory blocks, an algorithm is needed for mapping main memory blocks into cache lines. Further, a means is needed for determining which main memory block currently occupies a cache line. The choice of the mapping function dictates how the cache is organized. Three techniques can be used: Direct mapping Associative mapping Set associative mapping 26 13

14 Direct mapping Maps each block of main memory into only one possible cache line. Mapping is expressed as i = j modulo m where i = cache line number j = main memory block number m = number of lines in cache 27 In direct mapping, 24 bit memory address contains: 2 bit word identifier 22 bit block identifier 8 bit tag 14 bit line no 28 14

15 Example 1: Find the tag, line and word for the address = Word = 00 2 = 0 16 Tag = = Line no = = Example 2: Find the tag, line and word for the address FFFFFC 16 FFFFFC 16 = Word = 00 2 = 0 16 Tag = = FF 16 Line no = = 3FFF Direct mapping is simple, inexpensive and uses fixed location for given block. If a program happens to reference words repeatedly from two different blocks that map into same line, then blocks will be continually swapped in cache and hit ratio will be low. This phenomenon is known as thrashing

16 Associative mapping Allows each memory block to be loaded into any line of cache. In this case, the cache control logic interprets a memory address simply as a tag and a word field. Tag uniquely identifies a block of main memory. To determine whether a block is in cache, cache control logic must simultaneously examine every line s tag for a match. It requires fully associative memory. Implementation is expensive and has very complex circuitry. 31 In associative mapping, a 24 bit memory address consists of 2 bit word 22 bit tag Example: Find the tag and word for a 24 bit memory address 16339C C 16 = Word = 00 2 = 0 16 Tag = = 058CE

17 Set associative mapping Compromise between direct and associative mappings to reduce their disadvantages. Cache is divided into v sets, each of which consists k lines. m = v k i = j modulo v where i = cache set number j = main memory block number m = number of lines in cache So a given block will map directly to a particular set, but can occupy any line in that set (associative mapping is used within set). 33 Most common set associative mapping is 2 lines per set and is called two-way set associative. It significantly improves hit ratio over direct mapping and associative hardware is not too expensive. Tag in a memory address must be compared to tag of every line in cache

18 In set associative mapping, a 24 bit memory address consists of: 9 bit tag 13 bit set (identifies a unique set of lines within cache) 2 bit word 35 Example 1: Find tag, set and word for memory address FF7FFC 16 FF7FFC = Word = 00 2 = 0 16 Tag = = 1FE 16 Set = = 1FFF 16 Example 2: Find tag, set and word for memory address 2C339C 16 2C339C = Word = 00 2 = 0 16 Tag = = Set = = 0CE

19 4. Replacement algorithms: When all lines are occupied, bringing in a new block requires that an existing line be overwritten. For direct mapping, there is only one possible line for any particular block and no choice is possible. For associative and set associative techniques, a replacement algorithm is required. Algorithms must be implemented in hardware for speed. Four most common algorithms are: Least-recently-used (LRU) First-in-first-out (FIFO) Least-frequently-used (LFU) Random 37 Least-recently-used (LRU): The idea is to replace that block in set which has been in cache longest with no reference to it. Probably most effective method. First-in-first-out (FIFO): The idea is to replace that block in set which has been in cache longest. The implementation uses a round-robin or circular buffer technique. Least-frequently-used (LFU): The idea is to replace that block in set which has experienced fewest references. For implementation, associate a counter with each slot and increment when used. Random: The idea is to replace a random block in set. Interesting because it provides only slightly inferior performance to an algorithms based on usage

20 5. Write policy: If a block has been altered in cache, it is necessary to write it back out to main memory before replacing it with another block. There are two problems: More than one device may have access to main memory. I/O modules may be able to read/write directly to memory. If a word has been altered only in cache, then corresponding memory word is invalid or vice versa. Multiple CPUs may be attached to same bus, each with their own cache. Then, if a word is altered in one cache, it could possibly invalidate a word in other caches. 39 There are some techniques to overcome these problems: Write through All write operations are made to main memory as well as to cache, so main memory is always valid. Other CPUs monitor traffic to main memory to update their caches when needed. This generates large memory traffic and may create a traffic jam

21 Write back Minimizes memory writes. Updates are done only in cache. When an update occurs, an UPDATE bit associated with line is set, so when a block is replaced it is written back to memory if and only if UPDATE bits is set. Portions of main memory are invalid, so accesses by I/O modules must occur through cache. Multiple caches still can become invalidated, unless some cache coherency system is used Line size: As block size increases, more useful data is brought into cache. But larger blocks reduce number of blocks that fit into cache and a small number of blocks results in data being overwritten shortly after it is fetched. As a block becomes larger, each additional word is farther from requested word, therefore less likely to be needed in near future. A size from 8 to 64 bytes seems close to optimum

22 7. Number of caches: When caches were originally introduced, the typical system had a single cache. More recently, the use of multiple caches has become the norm. Two aspects of this design issue concern the number of levels of caches and the use of unified versus split caches. 43 Multilevel Caches As logic density has increased, it has become possible to have a cache on the same chip as the processor: the onchip cache. Most contemporary designs include both on-chip and external caches. The simplest such organization is known as a two-level cache, with the internal cache designated as level 1 (L1) and the external cache designated as level 2 (L2). With the increasing availability of on-chip area available for cache, most contemporary microprocessors have moved the L2 cache onto the processor chip and added a level 3 (L3) cache. More recently, most microprocessors have incorporated an on-chip L3 cache

23 Unified versus split caches Unified cache: A single cache stores data and instruction. It has higher hit rate than split cache, because it automatically balances load between data and instructions. Only one cache need to be designed and implemented. Split cache: One cache is dedicated to instructions and one cache is dedicated to data. These two caches both exist at the same level, typically as two L1 caches. Trend is toward split caches particularly for superscalar machines (Pentium). The key advantage of the split cache design is that it eliminates contention for the cache between the instruction fetch/decode unit and the execution unit