Lecture 11. Virtual Memory Review: Memory Hierarchy

Transcription

1 Lecture 11 Virtual Memory Review: Memory Hierarchy 1

2 Administration Homework 4 -Due 12/21 HW 4 Use your favorite language to write a cache simulator. Input: address trace, cache size, block size, associativity Output: miss rate 2

3 Review:Main Memory Simple, Interleaved and Wider Memory Interleaved Memory: for sequential or independent accesses Avoiding bank conflicts: SW & HW DRAM specific optimizations: page mode & Specialty DRAM 3

4 Main Memory Background Performance of Main Memory: Latency: time to finish a request Access Time: time between request and word arrives Cycle Time: time between requests Cycle time > Access Time Bandwidth: Bytes/per second Main Memory is DRAM: Dynamic Random Access Memory Dynamic since needs to be refreshed periodically (8 ms) Addresses divided into 2 halves (Memory as a 2D matrix): RAS or Row Access Strobe CAS or Column Access Strobe Cache uses SRAM: Static Random Access Memory No refresh (6 transistors/bit vs. 1 transistor /bit) Address not divided Size: DRAM/SRAM - 4-8, Cost/Cycle time: SRAM/DRAM

5 Virtual Memory: Motivation virtual memory A B C D physical memory C B A Permit applications to grow larger than main memory size 32, 64 bits v.s. 28 bits (256MB) Automatic management Multiple process management Sharing Protection Relocation D Disk 5

6 Virtual Memory Terminology Page or segment A block transferred from the disk to main memory Page: fixed size Segment: variable size Page fault The requested page is not in main memory (miss) Address translation or memory mapping Virtual to physical address 6

7 Cache vs. VM Difference What controls replacement? Cache miss: HW Page fault: often handled by the OS Size VM space is determined by the address size of the CPU Cache size is independent of the CPU address size Lower level use Cache: main memory is not shared by anything els VM: most of the disk contains the file system 7

8 Virtual Memory 4Qs for VM? Q1: Where can a block be placed in the upper level? Fully Associative Q2: How is a block found if it is in the upper level? Pages: use a page table Segments: segment table Q3: Which block should be replaced on a miss? LRU Q4: What happens on a write? Write Back 8

9 Page Table Virtual-to-physical address mapping via page table virtual page number page offset PTE Main Memory page table What is the size of the page table given a 28-bit virtual address, 4 KB pages, and 4 bytes per page table entry? 2 ^ (28-12) x 2^2 = 256 KB 9

10 Inverted Page Table (HP, IBM) virtual page number hash page offset Inverted page table VA PA One PTE per page frame Pros & Cons: The size of table is the number of physical pages Must search for virtual address (using hash) Hash Another Table (HAT) 10

11 Fast Translation: Translation Look-aside Buffer (TLB) Cache of translated addresses Alpha TLB: 32 entry fully associative page frame address <30> page offset <13> 1 2 V R W Tag <30> Physical address <21> :::: Problem: combine caches with virtual memory <13> bit <21> physica 32:1 Mux address 11

12 TLB & Caches CPU CPU CPU TB $ VA PA VA Tags $ TB VA VA PA Tags VA $ TB PA L2 $ PA PA MEM MEM Conventional Organization MEM Virtually Addressed Cache Translate only on miss Overlap $ access with VA translation: requires $ index to remain invariant 12 across translation

13 Virtual Cache Avoid address translation before accessing cache Faster hit time Context switch Flush: time to flush + compulsory misses from empty cache Add processor id (PID) to TLB I/O (physical address) must interact with cache Physical -> virtual address translation Aliases (Synonyms) Two virtual addresses map to the same physical addresses Two identical copies in the cache 13

14 Solutions for Aliases HW: anti-aliasing Guarantee every cache block a unique physical address OS : page coloring Guarantee that the virtual and physical addresses match in the last n bits Avoid duplicate physical addresses for block if using a directmapped cache, size < 2^n virtual address < 18 > Physical address y) aliasing virtual address x virtual address y index bit block offset index bit block offset (x,y) map to the same set 14

15 Virtually indexed & physically tagged cache Use the part of addresses that is not affected by address translation to index a cache Page offset Overlap the time to read the tags with address translation Page address Page offset Address tag index block offset Limit cache size to cache size for direct-mapped cache how to get a bigger cache Higher associativity Page coloring Page address Page offset 15

16 Selecting a Page Size Reasons for larger page size Page table size is inversely proportional to the page size; therefore memory saved Fast cache hit time easy when cache <= page size (VA caches); bigger page makes it feasible as cache size grows Transferring larger pages to or from secondary storage, possibly over a network, is more efficient Number of TLB entries are restricted by clock cycle time, so a larger page size maps more memory, thereby reducing TLB misses Reasons for a smaller page size Fragmentation: don t waste storage; data must be contiguous within page Quicker process start for small processes Hybrid solution: multiple page sizes Alpha: 8KB, 16KB, 32 KB, 64 KB pages (43, 47, 51, 55 virt addr bits) 16

17

18 Virtual Memory Summary Why virtual memory? Fast address translation Page table, TLB TLB & Cache Virtual cache vs. physical cache Virtually indexed and physically tagged 18

19 Cross Cutting Issues Superscalar CPU & Number Cache Ports Speculative Execution and non-faulting option on memory Parallel Execution vs. Cache locality Want far separation to find independent operations vs. want reuse of data accesses to avoid misses For ( i=0; i<512; i= i+1) for (j= 0; j< 512; j = j+1) x[i][j] = 2 * x[i][j-1] For ( j=0; j<512; j= j+1) for (i = 0; i< 512; i = i+1) x[i][j] = 2 * x[i][j-1] For (i = 0; i<512 ; i=i+1) for (j=1; j< 512; j= j+4) { x[i][j] = 2 * x[i][j-1]; x[i][j+1] = 2 * x[i][j]; x[i][j+2] = 2 * x[i][j+1]; x[i][j+3] = 2 * x[i][j+2]; }; For (j= 0; j<512 ; j=j+1) for (i=1; i< 512; i= i+4) { x[i][j] = 2 * x[i][j-1]; x[i+1][j] = 2 * x[i+1][j-1]; x[i+2][j] = 2 * x[i+2][j-1]; x[i+3][j] = 2 * x[i+3][j-1]; }; 19

20 Cross Cutting Issues I/O and consistency of data between cache and memory I/O always see the latest data interfere with CPU CPU not interfering with the CPU might see the stale data Output: write-through Input: noncacheable SW: flush the cache HW : check I/O address on input CPU Cache I/O Bridge Cache Main Memory Main Memory DMA I/O Bridge 20

21 itfall: Predicting Cache Performance from Different Prgrm (ISA, compiler,...) 35% 4KB Data cache miss rate 8%,12%, or 28%? 1KB Instr cache miss rate 0%,3%, or 10%? Alpha vs. MIPS for 8KB Data: 17% vs. 10% Miss Rate 30% 25% 20% 15% 10% D: tomcatv D: gcc D: espresso I: gcc I: espresso I: tomcatv 5% 0% Cache Size (KB) 21

22 Pitfall: Simulating Too Small an Address Trace Cummlati ve Average Memory Access Time Instructions Executed (billions) 22

23 Review: Memory Hierarchy Definition & principle of memory hierarchy Cache Organization :Cache ABC Improve cache performance Main memory Organization Improve memory bandwidth Virtual memory Fast address translation TLB & Cache 23

24 Capacity Access Time Cost CPU Registers 100s Bytes <1s ns Cache 10s-100s K Bytes 1-10 ns $10/ MByte Main Memory M Bytes 100ns- 300ns $1/ MByte Disk 10s G Bytes, 10 ms (10,000,000 ns) $0.0031/ MByte Tape infinite sec-min $0.0014/ MByte Levels of the Memory Hierarchy Registers Cache Memory Disk Tape Instr. Operands Blocks Pages Files Staging Xfer Unit prog./compiler 1-8 bytes cache cntl bytes OS 512-4K bytes user/operator Mbytes Upper Level faster Larger Lower Level 24

25 General Principle The Principle of Locality: Program access a relatively small portion of the address space at any instant of time. Two Different Types of Locality: Temporal Locality (Locality in Time): If an item is referenced, it will tend to be referenced again soon (e.g., loops, reuse) Spatial Locality (Locality in Space): If an item is referenced, items whose addresses are close by tend to be referenced soon (e.g., straightline code, array access) ABC of Cache: Associativity Block size Capacity Cache organization Direct-mapped cache : A = 1, S = C/B N-way set-associative: A = N, S =C /(BA) Fully assicaitivity S =1 25

26 4 Questions for Memory Hierarchy Q1: Where can a block be placed in the upper level? (Block placement) Q2: How is a block found if it is in the upper level? (Block identification) Q3: Which block should be replaced on a miss? (Block replacement) Q4: What happens on a write? (Write strategy) 26

27 Q1: Where can a block be placed in the upper level? Block 12 placed in 8 block cache: Fully associative, direct mapped, 2-way set associative S.A. Mapping = Block Number Modulo Number Sets Full Mapped Direct Mapped (12 mod 8) = 4 2-Way Assoc (12 mod 4) = Cache set 0 set 1 set 2 set Memory 27

28 1 KB Direct Mapped Cache, 32B blocks For a 2 ** N byte cache: The uppermost (32 - N) bits are always the Cache Tag The lowest M bits are the Byte Select (Block Size = 2 ** M) 31 Cache Tag block address Example: 0x50 Stored as part of the cache state 9 Cache Index Ex: 0x01 4 Byte Select Ex: 0x00 0 Valid Bit Cache Tag Cache Data Byte 31 : Byte 1 Byte 0 0 0x50 Byte 63 : Byte 33 Byte : : : Byte 1023 : Byte

29 Q2: How is a block found if it is in the upper level? Tag on each block No need to check index or block offset Block Address Tag Index Block Offset 29

30 Q3: Which block should be replaced on a miss? Easy for Direct Mapped Set Associative or Fully Associative: Random LRU (Least Recently Used) Hardware keeps track of the access history Replace the entry that has not been used for the longest time 30

31 Q4: What happens on a write? Write through The information is written to both the block in the cache and to the block in the lower-level memory. Write back The information is written only to the block in the cache. The modified cache block is written to main memory only when it is replaced. is block clean or dirty? Pros and Cons of each? WT: Good: read misses cannot result in writes Bad: write stall WB: no repeated writes to same location Read misses could result in writes WT always combined with write buffers so that don t wait for lower level memory 31

32 Write Miss Policy Write allocate (fetch on write) The block is loaded on a write miss No-write allocate (write-around) The block is modified in the lower level and not loaded into the cache Write through Write back Write allocate hit: write to cache/memory miss: load block into cache; write to cache/memory hit: write to cache, set dirty bit. miss: load block into cache; write to cache Write around hit: write to cache/memory miss: write to memory hit: write to cache, set dirty bit. miss: write to memory 32

33 Improving Cache Performance 1. Reduce the miss rate, 2. Reduce the miss penalty, or 3. Reduce the time to hit in the cache. Average memory access time = hit time + miss-rate X miss-penalty 33

34 Reducing Misses Classifying Misses: 3 Cs Compulsory The first access to a block is not in the cache, so the block must be brought into the cache. These are also called cold start misses or first reference misses. (Misses in Infinite Cache) Capacity If the cache cannot contain all the blocks needed during execution of a program, capacity misses will occur due to blocks being discarded and later retrieved. (Misses in Size X Cache) Conflict If the block-placement strategy is set associative or direct mapped, conflict misses (in addition to compulsory and capacity misses) will occur because a block can be discarded and later retrieved if too many blocks map to its set. These are also called collision misses or interference misses. (Misses in N-way Associative, Size X Cache) 34

35 1. Reduce Misses via Larger Block Size Block size Compulsory misses Spatial locality Example: access patter 0x0000,0x0004,0x0008,0x0012,.. Block size = 2 Word 0x0000 (miss) 0x0004 (hit) 0x0008 (miss) 0x0012 (hit) Block size = 4 Word 0x0000 (miss) 0x0004 (hit) 0x0008 (hit) 0x0012 (hit) 35

36 2. Reduce Misses via Higher Associativity 2:1 Cache Rule: Miss Rate DM cache size N - Miss Rate 2-way cache size N/2 Beware: Execution time is only final measure! Will Clock Cycle time increase? Hill [1988] suggested hit time external cache +10%, internal + 2% for 2-way vs. 1-way 36

37 3. Reducing Misses via Victim Cache How to combine fast hit time of Direct Mapped yet still avoid conflict misses? victim cache Add buffer to place data discarded from cache Jouppi [1990]: 4-entry victim cache removed 20% to 95% of conflicts for a 4 KB direct mapped data cache

38 4. Reducing Misses via Pseudo-Associativity How to combine fast hit time of Direct Mapped and have the lower conflict misses of 2-way SA cache? Divide cache: on a miss, check other half of cache to see if there, if so have a pseudo-hit (slow hit) Hit Time Pseudo Hit Time Miss Penalty Time Drawback: CPU pipeline is hard if hit takes 1 or 2 cycles Better for caches not tied directly to processor 38

39 5. Reducing Misses by HW Prefetching of Instruction & Data Bring a cache block up the memory hierarchy before it is requested by the processor Example: Stream Buffer for instruction prefetching (alpha 21064) Alpha fetches 2 blocks on a miss Extra block placed in stream buffer On miss check stream buffer stream buffer Issue prefetch request 6 CPU memory 39

40 6. Reducing Misses by SW Prefetching Data Data Prefetch Compiler insert prefetch instructions to the request the data before they are needed Load data into register (HP PA-RISC loads) Cache Prefetch: load into cache (MIPS IV, PowerPC, SPARC v. 9) Special prefetching instructions cannot cause faults; a form of speculative execution Need a non-blocking cache Issuing Prefetch Instructions takes time Is cost of prefetch issues < savings in reduced misses? 40

41 7. Reducing Misses by Compiler Optimizations Instructions Reorder procedures in memory so as to reduce misses Profiling to look at conflicts McFarling [1989] reduced caches misses by 75% on 8KB direct mapped cache with 4 byte blocks Data Merging Arrays: improve spatial locality by single array of compound elements vs. 2 arrays Loop Interchange: change nesting of loops to access data in order stored in memory Loop Fusion: Combine 2 independent loops that have same looping and some variables overlap Blocking: Improve temporal locality by accessing blocks of data repeatedly vs. going down whole columns or rows 41

42 1. Reducing Miss Penalty: Read Priority over Write on Miss Write-through: check write buffer contents before read; if no conflicts, let the memory access continue How to reduce read-miss penalty for a write-back cache? Read miss replacing dirty block Normal: Write dirty block to memory, and then do the read Instead copy the dirty block to a write buffer, then do the read, and then do the write CPU stall less since restarts as soon as do read Write buffer cache write Read memory 42

43 2. Subblock Placement to Reduce Miss Penalty Don t have to load full block on a miss Have bits per subblock to indicate valid (Originally invented to reduce tag storage) Sub-blocks Valid Bits 43

44 3. Early Restart and Critical Word First Don t wait for full block to be loaded before restarting CPU Early restart As soon as the requested word of the block arrives, send it to the CPU and let the CPU continue execution Critical Word First Request the missed word first from memory and send it to the CPU as soon as it arrives; let the CPU continue execution while filling the rest of the words in the block. Also called wrapped fetch and requested word first Generally useful only in large blocks, Spatial locality a problem; tend to want next sequential word, so not clear if benefit by early restart 44

45 4. Non-blocking Caches to reduce stalls on misses Non-blocking cache or lockup-free cache allowing the data cache to continue to supply cache hits during a miss hit under miss reduces the effective miss penalty by being helpful during a miss instead of ignoring the requests of the CPU hit under multiple miss or miss under miss may further lower the effective miss penalty by overlapping multiple misses Significantly increases the complexity of the cache controller as there can be multiple outstanding memory accesses 45

46 5th Miss Penalty Reduction: Second Level Cache L2 Equations AMAT = Hit Time L1 + Miss Rate L1 x Miss Penalty L1 Miss Penalty L1 = Hit Time L2 + Miss Rate L2 x Miss Penalty L2 L1 L2 AMAT = Hit Time L1 + Miss Rate L1 x (Hit Time L2 + Miss Rate L2 + Miss Penalty L2 ) Definitions: Local miss rate misses in this cache divided by the total number of memory accesses to this cache (Miss rate L2 ) Global miss rate misses in this cache divided by the total number of memory accesses generated by the CPU (Miss Rate L1 x Miss Rate L2 ) Main Memory 46

47 1. Fast Hit times via Small and Simple Caches Why Alpha has 8KB Instruction and 8KB data cache + 96KB second level cache Direct Mapped, on chip 47

48 2. Fast hits by Avoiding Address Translation Address translation from virtual to physical addresses Physical cache: 1. Virtual -> physical 2. Use physical address to index cache => longer hit time Virtual cache: Use virtual cache to index cache => shorter hit time problem: aliasing More on this after covering virtual memory issues! 48

49 2. Fast hits by Avoiding Address Translation CPU CPU CPU TB $ VA PA VA Tags $ TB VA VA PA Tags VA $ TB PA L2 $ PA PA MEM MEM Conventional Organization MEM Virtually Addressed Cache Translate only on miss Overlap $ access with VA translation: requires $ index to remain invariant 49 across translation

50 Virtual Cache Avoid address translation before accessing cache Faster hit time Context switch Flush: time to flush + compulsory misses from empty cache Add processor id (PID) to TLB I/O (physical address) must interact with cache Physical -> virtual address translation Aliases (Synonyms) Two virtual addresses map to the same physical addresses Two identical copies in the cache Solution: HW: anti-aliasing SW: page coloring 50

51 Virtually indexed & physically tagged cache Use the part of addresses that is not affected by address translation to index a cache Page offset Overlap the time to read the tags with address translation Page address Page offset Address tag index block offset Limit cache size to cache size for direct-mapped cache how to get a bigger cache Higher associativity Page coloring Page address Page offset 51

52 3. Fast Hit Times Via Pipelined Writes Pipeline Tag Check and Update Cache as separate stages; current write tag check & previous write cache update Only Write in the pipeline; empty during a miss write x1 write x2 tag check x1 write data tag check x2 write data 52

53 4. Fast Writes on Misses Via Small Subblocks If most writes are 1 word, subblock size is 1 word, & write through then always write subblock & valid bit immediately Tag match and valid bit already set: Writing the block was proper, & nothing lost b setting valid bit on again. Tag match and valid bit not set: The tag match means that this is the proper block; writing the data into the subblock makes it appropriate to turn the valid bit on. Tag mismatch: This is a miss and will modify the data portion of the block. As this is a write-through cache, however, no harm was done; memory still has an up-to-date copy of the old value. Only the tag to the address of the write and the valid bits of the other subblock need be changed because the valid bit for this subblock has already been set Doesn t work with write back due to last case 53

54 Main Memory Background Performance of Main Memory: Latency: time to finish a request Access Time: time between request and word arrives Cycle Time: time between requests Cycle time > Access Time Bandwidth: Bytes/per second Main Memory is DRAM: Dynamic Random Access Memory Dynamic since needs to be refreshed periodically (8 ms) Addresses divided into 2 halves (Memory as a 2D matrix): RAS or Row Access Strobe CAS or Column Access Strobe Cache uses SRAM: Static Random Access Memory No refresh (6 transistors/bit vs. 1 transistor /bit) Address not divided Size: DRAM/SRAM - 4-8, Cost/Cycle time: SRAM/DRAM

55 Main Memory Performance CPU CPU Simple: CPU, Cache, Bus, Memory same width (32 bits) Wide: CPU/Mux 1 word; Mux/Cache, Bus, Memory N words (Alpha: 64 bits & 256 bits) Interleaved: CPU, Cache, Bus 1 word: Memory N Modules (4 Modules); example is word interleaved One word Cache One word Memory Simple Wide CPU Bank 0 `multiplexor Cache Cache Bank 1 One word One word Bank 2 Interleaved Bank 3 Memory 55

56 Independent Memory Banks Memory banks for independent accesses vs. faster sequential accesses Multiprocessor I/O Miss under Miss, Non-blocking Cache Super bank ::: bank Superbank number bank bank Superbank offset Bank number Bank offset Independent banks 56

57 Avoiding Bank Conflicts Bank Conflicts Memory references map to the same bank Problem: cannot take advantage of multiple banks (supporting multiple independent request) Example: all elements of a column are in the same memory bank with 128 memory banks, interleaved on a word basis int x[256][512]; for (j = 0; j < 512; j = j+1) for (i = 0; i < 256; i = i+1) x[i][j] = 2 * x[i][j]; SW: loop interchange or declaring array not power of 2 HW: Prime number of banks Problem: more complex calculation per memory access 57

58 Fast Memory Systems: DRAM specific Multiple RAS accesses: several names (page mode) 64 Mbit DRAM: cycle time = 100 ns, page mode = 20 ns New DRAMs to address gap; what will they cost, will they survive? Synchronous DRAM: Provide a clock signal to DRAM, transfer synchronous to system clock RAMBUS: reinvent DRAM interface Each Chip a module vs. slice of memory Short bus between CPU and chips Does own refresh Variable amount of data returned 1 byte / 2 ns (500 MB/s per chip) Niche memory e.g., Video RAM for frame buffers, DRAM + fast serial output 58

59 Virtual Memory: Motivation virtual memory A B C D physical memory C B A Permit applications to grow larger than main memory size 32, 64 bits v.s. 28 bits (256MB) Automatic management Multiple process management Sharing Protection Relocation D Disk 59

60 Virtual Memory 4Qs for VM? Q1: Where can a block be placed in the upper level? Fully Associative Q2: How is a block found if it is in the upper level? Pages: use a page table Segments: segment table Q3: Which block should be replaced on a miss? LRU Q4: What happens on a write? Write Back 60

61 Page Table Virtual-to-physical address mapping via page table virtual page number page offset PTE Main Memory page table What is the size of the page table given a 28-bit virtual address, 4 KB pages, and 4 bytes per page table entry? 2 ^ (28-12) x 2^2 = 256 KB 61

62 Inverted Page Table (HP, IBM) virtual page number hash page offset Inverted page table VA PA One PTE per page frame Pros & Cons: The size of table is the number of physical pages Must search for virtual address (using hash) Hash Another Table (HAT) 62

63

64 Fast Translation: Translation Look-aside Buffer (TLB) Cache of translated addresses Alpha TLB: 32 entry fully associative page frame address <30> page offset <13> 1 2 V R W Tag <30> Physical address <21> :::: Problem: combine caches with virtual memory <13> bit <21> physica 32:1 Mux address 64