Computer Structure. X86 Virtual Memory and TLB

Transcription

1 Computer Structure X86 Virtual Memory and TLB Franck Sala Slides from Lihu and Adi s Lecture 1

2 Virtual Memory Provides the illusion of a large memory Different machines have different amount of physical memory Allows programs to run regardless of actual physical memory size The amount of memory consumed by each process is dynamic Allow adding memory as needed Many processes can run on a single machine Provide each process its own memory space Prevents a process from accessing the memory of other processes running on the same machine Allows the sum of memory spaces of all process to be larger than physical memory Basic terminology Virtual Address Space: address space used by the programmer Physical Address: actual physical memory address space 2

3 Virtual Memory: Basic Idea Divide memory (virtual and physical) into fixed size blocks s in Virtual space, Frames in Physical space size = Frame size size is a power of 2: page size = 2 k All pages in the virtual address space are contiguous s can be mapped into physical Frames in any order Some of the pages are in main memory (DRAM), some of the pages are on disk All programs are written using Virtual Memory Address Space The hardware does on-the-fly translation between virtual and physical address spaces Use a Table to translate between Virtual and Physical addresses 3

4 Virtual to Physical Address translation 47 Virtual Number Virtual Address offset 0 Dirty bit table base reg V D AC Phy. page # 1 0 Access Control - Memory type (WB, WT, UC, WP ) - User / Supervisor Valid bit 38 Physical Number Physical Address offset size: 2 12 byte =4K byte 4

5 Translation Look aside Buffer (TLB) table resides in memory each translation requires an extra memory access Virtual Address TLB caches recently used PTEs TLB Access speed up translation typically 128 to 256 entries, 4 to 8 way associative TLB Indexing Access Table In memory No TLB Hit? Virtual page number Offset Yes On A TLB miss Tag Set Miss Handler (HW PMH) gets PTE from memory Physical Addresses 5

6 Virtual Memory And Cache Virtual Address Access TLB Access Cache Walk: get PTE from memory Hierarchy No STLB Hit? No TLB Hit? Yes L1 Cache Hit? Yes No L2 Cache Hit? No Access Memory Physical Addresses Data TLB access is serial with cache access table entries are cached in L1 D$, L2$ and L3 $ as data 6

7 Fault is not in main memory Context switch Demand load policy Glossary Access control / Protection fault Context switch Aliasing DLL, Shared code, large Malloc swap out snoop-invalidate for each byte in the cache (both L1 and L2) All data in the caches which belongs to that page is invalidated The page in the disk is up-to-date If the TLB hits for the swapped-out page, TLB entry is invalidated In the page table The valid bit in the PTE entry of the swapped-out pages set to 0 All the rest of the bits in the PTE entry may be used by the operating system for keeping the location of the page in the disk 7

8 32bit Mode: 4KB / 4MB Mapping 2-level hierarchical mapping: Directories and tables 4KB aligned PDE Present (0 = page fault) size (4KB or 4 MB) Linear Address Space (4K ) DIR TABLE OFFSET CR4.PSE=1 both 4MB & 4KB pages supported Separate TLBs Linear Address Space (4MB ) DIR OFFSET K entry Directory 1K entry Table 4K data MByte data PTE 20 Directory PDE =32 (4K aligned) PDE =32 (4K aligned) CR3 (PDBR) 8 CR3 (PDBR)

9 32bit Mode: PDE and PTE Format 20 bit pointer to a 4K Aligned address Virtual memory Present Accessed Dirty (in PTE only) size (in PDE only) Global Protection Writable (R#/W) User / Supervisor # Caching 2 levels/type only WT Cache Disabled PAT PT Attribute Index 3 bits available for OS usage Present Writable User / Supervisor Write-Through Cache Disable Accessed Size (0: 4 Kbyte) Global Available for OS Use Directory Entry (4KB page table) Frame Address 31:12 AVAIL G 0 A A Present Writable User / Supervisor Write-Through Cache Disable Accessed Dirty PAT Global Available for OS Use Frame Address 31:12 Table Entry AVAIL G P A T D A P C D P W U W P T P C D P W U W P T

10 4KB Mapping with PAE Physical addresses is extended to M bits / Linear address remains 32 bit 1995: Pentium Pro Linear Address Space (4K ) DIR TABLE OFFSET Linear Address Space (4K ) Dir ptr DIR TABLE OFFSET K KByte 1K entry Directory 32 PDE 20 1K entry Table PTE =32 (4K aligned) 20 data 4 entry Directory Pointer Table Dir ptr entry M entry Directory 64 PDE M entry Table PTE 64 M-12 data 32 (32B aligned) CR3 (PDBR) CR3 (PDPTR) 10

11 2MB Mapping with PAE Physical addresses is extended to M bits / Linear address remains 32 bit Linear Address Space (4MB ) 31 DIR 21 OFFSET Directory 32 PDE =32 (4K aligned) 0 4MByte data 32 Linear Address Space (2MB ) Dir ptr 2 29 Directory Pointer Table Dir ptr entry DIR 32 (32B aligned) Directory PDE OFFSET 9 21 M M MByte data 64 CR3 (PDBR) CR3 (PDPTR) 11

12 4KB Mapping in 64 bit Mode 2003: AMD Opteron 63 sign ext. Linear Address Space (4K ) PML4 PDP DIR TABLE OFFSET PML4: Map Level 4 PDP: Directory Pointer KByte 512 entry PML4 Table PML4 entry M (4KB aligned) CR3 (PDPTR) 512 entry Directory Pointer Table 64 PDP entry M entry Directory 64 PDE M entry Table PTE 64 M-12 data TB of virtual memory (2 48 ) 1 TB of physical memory (2 40 ) 12

13 2MB Mapping in 64 bit Mode Linear Address Space (2M ) sign ext. PML4 PDP DIR OFFSET entry PML4 Table 512 entry Directory Pointer Table 512 entry Directory 2MByte data PML4 entry M-12 PDP entry M-12 PDE M (4KB aligned) CR3 (PDPTR) 13

14 1GB Mapping in 64 bit Mode 63 sign ext. Linear Address Space (1G ) PML4 PDP OFFSET entry PML4 Table 512 entry Directory Pointer Table 1GByte data PML4 entry M (4KB aligned) PDP entry M-30 CR3 (PDPTR) 14

15 TLBs The processor saves most recently used PDEs and PTEs in TLBs Separate TLB for data and instruction caches Separate TLBs for 4KB and 2/4MB page sizes 15

16 We have a core similar to X86 64 bit mode Question 1 Support Small s (PTE) and Large s (DIR) table size in each hierarchy is the size of a small page Entry size in the Table is 16 byte, in all the hierarchies 63 N4 N3 N2 N sign ext. PML4 PDP DIR TABLE OFFSET What is the size of a small page? 12 bits in the offset field 2 12 B = 4KB How many entries are in each Table? Table size = Size = 4KB PTE = 16B 4KB / 16B = 2 12 / 2 4 = 2 8 = 256 entries in each Table 16

17 - 64 bit (large & small) - PT size = size = 4KB - PTE = 16B - Table: 256 entries Question 1 63 N4 N3 N2 N sign ext. PML4 PDP DIR TABLE OFFSET What are the values of N1, N2, N3 and N4? Since we have 256 entries in each table, we need 8 bits to address them - Table [19:12] N1 = 19 - DIR [27:20] N2 = 27 - PDP [35:28] N3 = 35 - PML4 [43:36] N4 = 43 What is the size of a large page? Large pages are pointed by DIR So the large pages offset is 20 bits [19:0] large pages size: 2 20 = 1MB We can also say: DIR can point to 256 pages of 4KB = 1MB 17

18 - 64 bit (large & small) - PT size = size = 4KB - Large page size = 1 MB - PTE = 16B - Table: 256 entries Question sign ext. PML4 PDP DIR TABLE OFFSET We access a sequence of virtual addresses For each address, what is the minimal number of tables that were added in all the hierarchies? See next foil in presentation mode 18

19 Question 1: sequence of allocations sign ext. PML4 PDP DIR TABLE OFFSET 8 bits instead of 9 for example purposes only D3 71 PML4 Table 8 82 PDP Table 8 FF 28 Dir E2 68 Table B KB PML4 PDP DIR PTE offset D3 82 FF E2 B25 3 new tables + 1 page D3 82 FF E new table D3 82 FF new table + 1 page CR C D C00 1 new tables + 1 page B B new tables + 1 page 19

20 Caches and Translation Structures Core Instruction bytes On-die Platform L1 Inst. cache L1 data cache L2 L3 Memory translation Inst. TLB Data. TLB PTE PTE Load entry STLB PTE VA[47:12] PMH PDE cache PDP cache PML4 cache PDE entry VA[47:21] PDP entry VA[47:30] PML4 entry VA[47:39] Walk Logic 20

21 SIGN EXT. PML4 PDP DIR TABLE OFFSET cache Accessed with virtual address bits If hits, returns PDE cache [47:21] PDE PDP cache [47:30] PDP entry PML4 cache [47:39] PML4 entry 21

22 Question 2 Processor similar to X86 64 bits s of 4KB The processor has a TLB TLB Hit: we get the translation with no need to access the translation tables TLB Miss: the processor has to do a Walk The hardware that does the Walk (PMH) contains a cache for each of the translation tables All Caches and TLB are empty on Reset sign ext. PML4 PDP DIR TABLE OFFSET For the sequence of memory access below, how many accesses are needed for the translations? Address H memory access Explanations H H 22

23 Question 2 Processor similar to X86 64 bits s of 4KB The processor has a TLB TLB Hit: we get the translation with no need to access the translation tables TLB Miss: the processor has to do a Walk The hardware that does the Walk (PMH) contains a cache for each of the translation tables All Caches and TLB are empty on Reset sign ext. PML4 PDP DIR TABLE OFFSET For the sequence of memory access below, how many accesses are needed for the translations? Address H memory access Explanations H H 23

24 Question 2 Processor similar to X86 64 bits s of 4KB The processor has a TLB TLB Hit: we get the translation with no need to access the translation tables TLB Miss: the processor has to do a Walk The hardware that does the Walk (PMH) contains a cache for each of the translation tables All Caches and TLB are empty on Reset sign ext. PML4 PDP DIR TABLE OFFSET For the sequence of memory access below, how many accesses are needed for the translations? Address memory access Explanations H 4 We need to access the memory for each of the 4 translation tables H H 24

25 Question 2 Processor similar to X86 64 bits s of 4KB The processor has a TLB TLB Hit: we get the translation with no need to access the translation tables TLB Miss: the processor has to do a Walk The hardware that does the Walk (PMH) contains a cache for each of the translation tables All Caches and TLB are empty on Reset sign ext. PML4 PDP DIR TABLE OFFSET For the sequence of memory access below, how many accesses are needed for the translations? Address memory access Explanations H 4 We need to access the memory for each of the 4 translation tables H 0 Same pages as above TLB hit No memory access H 25

26 Question 2 Processor similar to X86 64 bits s of 4KB The processor has a TLB TLB Hit: we get the translation with no need to access the translation tables TLB Miss: the processor has to do a Walk The hardware that does the Walk (PMH) contains a cache for each of the translation tables All Caches and TLB are empty on Reset sign ext. PML4 PDP DIR TABLE OFFSET For the sequence of memory access below, how many accesses are needed for the translations? Address memory access Explanations H 4 We need to access the memory for each of the 4 translation tables H H 3 26 Same pages as above TLB hit No memory access We hit in PML4 cache. Then we miss in the PDP and so we need to access the memory 3 times: PDP, DIR, PTE

27 Question 2 L1 data Cache: 32KB 2 ways of 64B each How can we access this cache before we get the physical address? 64B 6 bits offset bits [5:0] 32KB = 2 15 / (2 ways * 2 6 bytes) = 2 8 = 256 sets [13:6] 12 bits are not translated: [11:0] we lack 2 bits [13:12] to get the set address So we do a lookup using 2 un-translated bits for the set address Those bits can be different from the PFN obtained after translation, therefore we need to compare the whole PFN to the tag stored in the Cache Tag array 27

28 Question 2: Read Acces Translated Not Translated VPN Set Offset PFN Tag [40:12] Tag [40:12] Way 0 Way 1 40:12 = = Tag Match Tag Match 28

29 Translated VPN Set Not Translated Offset Question 2: example Write Virtual Address A VPN Offset PFN Offset VA: PA: Set: [13:6] = 0 Tag: 0 Read Virtual Address B VB: PB: Set: [13:6] = 0 Tag: 3 (0 if we don t take [13:12]) 29

30 L1 data Cache: 32KB 2 ways of 64B each What will happen when we access with a given offset the virtual page A and after this, there is an access with the same offset in the virtual page B, which is mapped by the OS to the same physical page as A? Question 2: Virtual Alias 47 Translated VPN Set PFN Not Translated 6 5 Offset 0 VPN xxxx01 VPN yyyy00 PFN zzzz Set[11:6] Not translated Max: 64 sets Physical Addresses Cache zzzz 01.set[11:6] 00.set[11:6] Physical Addresses Cache zzzz zzzz - 2 virtual pages map to the same frame xxxx01.set.ofset and yyyy00.set.ofset 01 and 00 are bits 13:12 Phys. addressed cache: The data exist only once Virtual addressed cache: The data may exist twice AVOID THIS!!! 30

31 Question 2: Virtual Alias L1 data Cache: 32KB 2 ways of 64B each What will happen when we access with a given offset the virtual page A and after this, there is an access with the same offset in the virtual page B, which is mapped by the OS to the same physical page as A? 47 Translated VPN Set PFN Not Translated Offset Avoid having the same data twice in the cache xxxx01.set.ofset and yyyy00.set.ofset 01.set[11:6] Physical Addresses Cache zzzz Check 4 sets when we allocate a new entry and see if the same tag appears If yes, evict the second occurrence of the data (the alias) set[11:6] zzzz Virtual addressed cache: The data may exist twice AVOID THIS!!!

32 Question 2: Snoop L1 data Cache: 32KB 2 ways of 64B each What happens in case of snoop in the cache? 47 Translated VPN Set Not Translated Offset PFN The cache is snooped with a physical address [40:0] Since the 2 MSB bits of the set address are virtual, a given physical address can map to 4 different sets in the cache (depending on the virtual page that is mapped to it) So we must snoop 4 sets * 2 ways in the cache 01.set[11:6] 00.set[11:6] Physical Addresses Cache zzzz zzzz Virtual addressed cache: The data may exist twice AVOID THIS!!! 32

33 Core similar to X86 in 64 bit mode Question sign ext. PML4 PDP DIR TABLE OFFSET Supports small pages (pointed by PTE) and large pages (pointed by DIR). Size of an entry in all the different page tables is 8 Bytes PMH Caches at all the levels 4 entries direct mapped Access time on hit: 2 cycles Miss known after 1 cycle PMH caches are accessed at all the levels in parallel In each level, when there is a HIT, the PMH cache provides the relevant entry in the page table in the relevant level In each level, when there is a miss: the core accesses the relevant page table in the main memory. Access time to the main memory is 100 cycles, not including the time needed to get the PMH cache miss. 33

34 Question sign ext. PML4 PDP DIR TABLE OFFSET What is the size of the large pages? The large page is pointed by DIR, therefore, all the bits under are offset inside the large page: 2 24 = 16 MB How many entries in each Table? PTE: 2 12 DIR: 2 12 PDP: 2 12 PML4:

35 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD FF ABCD FF ABCD FF ABCD FF A0CD 35

36 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 401 Miss and memory access at each level: 1+ 4 *100 = 401 cycles FF ABCD FF ABCD FF ABCD FF A0CD 36

37 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 401 Miss and memory access at each level: 1+ 4 *100 = 401 cycles FF ABCD 202 (PML4, PDP, DIR, stlb) = (H,H,M,M) 1cyc PDP TLB read 1 more cycle DIR TLB: Miss 100 cycles PTE TLB: Miss 100 cycles FF ABCD FF ABCD FF A0CD 37

38 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 401 Miss and memory access at each level: 1+ 4 *100 = 401 cycles FF ABCD 202 (PML4, PDP, DIR, stlb) = (H,H,M,M) 1cyc PDP TLB read 1 more cycle DIR TLB: Miss 100 cycles PTE TLB: Miss 100 cycles FF ABCD 401 PMH cache miss in all the levels: 1 + 4*100 = 401 cycles FF ABCD FF A0CD 38

39 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 401 Miss and memory access at each level: 1+ 4 *100 = 401 cycles FF ABCD 202 (PML4, PDP, DIR, stlb) = (H,H,M,M) 1cyc PDP TLB read 1 more cycle DIR TLB: Miss 100 cycles PTE TLB: Miss 100 cycles FF ABCD 401 PMH cache miss in all the levels: 1 + 4*100 = 401 cycles FF ABCD 302 (PML4, PDP, DIR, stlb) = (H,M,M,M) 1 cyc PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set miss 2 + (3 100) = 302 FF A0CD 39

40 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 401 Miss and memory access at each level: 1+ 4 *100 = 401 cycles FF ABCD 202 (PML4, PDP, DIR, stlb) = (H,H,M,M) 1cyc PDP TLB read 1 more cycle DIR TLB: Miss 100 cycles PTE TLB: Miss 100 cycles FF ABCD 401 PMH cache miss in all the levels: 1 + 4*100 = 401 cycles FF ABCD 302 (PML4, PDP, DIR, stlb) = (H,M,M,M) 1 cyc PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set miss 2 + (3 100) = 302 FF A0CD 2 Hit in TLB no need to go to PMH: 2 cycles 40

41 Backup Slides 41

42 Translated not translated Virtual Address Set Offset VPN Tag field 40 : PFN Way 0 Way 3 = = Tag Match Tag Match 42

43 Translated not translated Virtual Address Set Offset VPN Tag field : 40 PFN Way 0 Way 3 = = Tag Match Tag Match 43

44 Question sign ext. PML4 PDP DIR TABLE OFFSET TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle We have a sequence of accesses to virtual addresses. For each address, how many cycles are needed to translate the address. Assume that we use small pages only and that the TLBs are empty upon reset Virtual Addr. Cycles Comment FF ABCD FF ABCD FF ABCD FF ABCD FF A0CD 44

45 sign ext. PML4 PDP DIR TABLE OFFSET TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle Virtual Addr. Cycles Comment FF ABCD 404 Miss and memory access at each level: 4 *101 = 404 cycles FF ABCD FF ABCD FF ABCD FF A0CD 45

46 sign ext. PML4 PDP DIR TABLE OFFSET TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle Virtual Addr. Cycles Comment FF ABCD 404 Miss and memory access at each level: 4 *101 = 404 cycles FF ABCD 206 PML4 TLB: Hit 2 cycles PDP TLB: Hit 2 cycles DIR TLB: Miss 101 cycles PTE TLB: Miss 101 cycles FF ABCD FF ABCD FF A0CD 46

47 sign ext. PML4 PDP DIR TABLE OFFSET TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle Virtual Addr. Cycles Comment FF ABCD 404 Miss and memory access at each level: 4 *101 = 404 cycles FF ABCD 206 PML4 TLB: Hit 2 cycles PDP TLB: Hit 2 cycles DIR TLB: Hit 101 cycles PTE TLB: Hit 101 cycles FF ABCD 404 FF ABCD FF A0CD TLB miss in all the levels: 4 *101 = 404 cycles 47

48 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 404 Miss and memory access at each level: 4 *101 = 404 cycles FF ABCD 206 PML4 TLB: Hit 2 cycles PDP TLB: Hit 2 cycles DIR TLB: Hit 101 cycles PTE TLB: Hit 101 cycles FF ABCD 404 TLB miss in all the levels: 4 *101 = 404 cycles FF ABCD 305 PML4 TLB: 2 cycles PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set miss DIR & PTE: miss (1 2) + (3 101) = 305 FF A0CD 48

49 sign ext. PML4 PDP DIR TABLE OFFSET Virtual Addr. Cycles Comment TLB 4entries Direct mapped TLB hit: 2 cycles TLB miss: 1 cycle Memory access: 100 cycle FF ABCD 404 Miss and memory access at each level: 4 *101 = 404 cycles FF ABCD 206 PML4 TLB: Hit 2 cycles PDP TLB: Hit 2 cycles DIR TLB: Hit 101 cycles PTE TLB: Hit 101 cycles FF ABCD 404 TLB miss in all the levels: 4 *101 = 404 cycles FF ABCD 305 PML4 TLB: 2 cycles PDP: the entry 234 that was filled for the second access was replaced by the entry that was filled in access 3, as it is in the same set miss DIR & PTE: miss (1 2) + (3 101) = 305 FF A0CD 8 Hit in all TLBs: 4 * 2 = 8 49