Persistent Storage - Datastructures and Algorithms

Transcription

1 Persistent Storage - Datastructures and Algorithms 1 / 21

2 L 03: Virtual Memory and Caches 2 / 21

3 Questions How to access data, when sequential access is too slow? Direct access (random access) file, how does it work? Everything possible with DA files? (dis-) advantages? Tree structured access (ISAM), how does it work? Hash access, how does it work? Various methods to handle / avoid overflows? Bitmap index, structure and how does it work? 3 / 21

4 Management of Space Filesystems Structures inside of the filesystem layout Usually one volume/disk/lun per filesystem Exceptions (recently): ZFS, WAFL, VxFS, Cluster FS... Database Configuration file (f.e. init.ora) System Tablespace Contains all further dataspace file configurations 4 / 21

5 Filesystem Buffer Management Buffer is part of the main memory OS has no knowledge about the use of the data Page Replacement Algorithm of OS frees buffers Database Buffer is in the address space of the database process Is often a shared segment (several databases) Database knows about the use of the datra Page Replacement Algorithm runs in database Hints can optimize the page replacment Actual data use can enhance the page replacement decisions 5 / 21

6 Shared Segments Segments are visible in multiple processes Rights are similar to rights of Unix files rwx for user/group/others (frequent) Used for Memory mapped files opened by multiple processes Libraries used by multiple processses Data segments used by multiple processes Database with multiple processes Accessing the same data 6 / 21

7 Virtual Adresses Process address space contains segments Different access rights (read, write, execute) CPU divides address (according to page size) Virtual page number (number of page in address space) Address inside of the page (aka displacement) Each address is specific for the address space Current CPUs have one address space per process 7 / 21

8 Access to a virtual memory location CPU checks (with MMU, TLB, TSB) Virtual page associated to real memory? No (not in TLB, TSB entry not in page table): Paging interrupt, process stops at current instruction OS takes page from free list and creates the association Read data from secondary storage (if mapped to data) Now the page is associated to a main memory page Either with mapped data or as new usable space Restart the process at the current instruction HW needs to be able to restart the instruction from beginning HW uses the real page and the displacement to find data 8 / 21

9 Virtual Memory Helpers TLB: Translation lookaside buffer Fully associative cache for virtual-to-real translation Returns very fast the real memory page of a virtual address (1-2 CPU cycles) Is small (usually 64 entries on Intel CPUs) => 64 * 4KB = 256 KB usable without additional translations Memory better usable with large pages (64KB, 2MB, 4 GB) Available in Unix (Solaris, AIX,..) for some time, new in Linux Built into CPU or MMU TSB: Software TLB Extension of the TLB using a handful of locked pages CPU needs some instructions for lookup 9 / 21

10 Virtual Memory Helpers (2) MMU: contains the memory management hardware Handles the page tables Page table stored only partly Sometimes with help of the CPU Handles the access bits Set bit when page is read Set bit when page is written (dirty) Access bits stored in or parallel to page tables HW implementation of bit setting usually limited to TLB Copy to page table by software when page address leaves TLB 10 / 21

11 Page Table CPU has access to a page table Access of data requires virtual real conversion Address space segment table page table Size of tables can be large (need to be paged also...) Current Solutions Solution 1: hierarchical page tables Multiple accesses to find page entry Solution 2: inverted page table One entry per physical page Organized as B tree or hash table or associative 11 / 21

12 Virtual addresses: Example Example: Intel x86 CPUs have 4KB pages Adress Decomposition Last 12 bits (address 4096 bytes = 1 page) = 52 Bits virtual page address Each process has one address space Virtual addresses are listed in a hierarchical page table Splitted in pieces of the size of one page (4Kbyte) Currently: 4 Bytes for virtual page addresses => 1024 adresses per page = 10 bits This means max 16 TBytes per Address space ( 2 32 * 4KB = 2 44 Bytes) More than 16 TB in an address space would need other structure 12 / 21

13 Page Replacement Algorithm (1) OS needs access to free pages Otherwise allocation needs too much time 2-5% are kept free all the time (free list) A lot of internal processes need free pages, like disk I/O, network traffic buffers, process/thread start, graphics,... HW support in CPU / MMU is necessary read-bit is set, when a page was read dirty-bit is set, when a page is changed (dirtied...) CPU can access these bits in privileged mode Sometimes more bits are necessary (for algorithms) Perhaps HW support for page table lookup 13 / 21

14 Page Replacement Algorithm (2) Several Algorithms Optimal free page, which will not be used in the near future (?) FIFO (First In / First Out) NRU (Not Recently Used) LRU (least recently used) Second Chance Clock Replacement ARC ( 2003, by Megiddo, Modha) 14 / 21

15 FIFO Algorithm Use the next free page in the list Used pages are put into a used list When all pages are used free the pages from the used list Disadvantages: Frequently used pages are also affected (=> slowdown) How to handle changed pages? Delay until page is written? Modify FIFO and get more complex? How to handle locked pages? More efficiency needs a more complex scheme 15 / 21

16 Clock Algorithm Algorithm Pages are ordered like the numbers on a wall clock A hand rotates like the hour hand on a clock If a page was written (dirty bit set), it is put onto write queue Else if a page was read, the read bit is reset Else if the page is locked, it is ignored Else (not written, read and locked): it is put onto the free list Result: This generates 2 sets of pages recently read recently not read 16 / 21

17 Algorithm Second Chance Algorithm Basis is the Clock algorithm Additionally uses an indicator bit (second chance bit) If the page is not read and the second chance bit is 1 Then the page is put onto the free list Otherwise the second chance bit is set to 1 Often used with 2 hands in a certain distance Todays memory is too large to wait for 3 rotations for a free page Result: 3 sets of pages are created recently read, having a second chance, has had the chance 17 / 21

18 Least Recently Used Algorithm (LRU) Page numbers are arranged sorted by access Access to the page moves the page number to front Implementation: Linked list, Heap, Access Timestamp,... Result: Best behaviour, very fine grained A lot of work to do at each memory access Used as the back end of other algorithms Re-ordering is rare, software implementation feasible 18 / 21

19 ARC - Adaptive Replacement Cache Adapts to real world applications Pages used very frequently Pages used only once Data structures L1 cache: T1: recent B1: recent ghosts L2 cache: T2: frequent B2: frequent ghosts T1 + T2 are the cache B1 and B2: only page addresses Implementation New pages T1 Hit in T1 page is moved to T2 Hit in B1: T1 grows, T2 shrinks Hit in B2: T2 grows, T1 shrinks 19 / 21

20 Pagetable and TLB: Bit Writing of data Writing to the page sets the bit to 1 Implementation: contain the modified (dirty) Scanner moves the page to the write queue Page daemon moves page to the write queue fflush() or sync() add page to the write queue 20 / 21

21 Like with a filesystem Database Buffer Database knows logical structure Reading initiated through query Writing is done with a consistency mechanism Consistency mechanisms Shadow copy (order 0) Log (order 1) Copy-on-Write (order 2) 21 / 21