INTELLIGENT DATABASE GROUP. Foundations of Information Systems. 5 DBMS Architecture. Prof. Dr.-Ing. Wolfgang Lehner

Transcription

1 Prof. Dr.-Ing. Wolfgang Lehner INTELLIGENT DATABASE GROUP 5 DBMS Architecture

2 What is in the Lecture?. Database Usage Query Programming Design 2. Database Architecture Indexes Transactions Query Processing 264

3 How is Database System build? SELECT s.firstname, s.lastname, COUNT(l.name) FROM Student s INNER JOIN Program p ON s.programid = p.id INNER JOIN Attendance a ON a.studentid = s.studentid INNER JOIN Lecture l ON a.lectureid = l.id GROUP BY s.firstname, s.lastname WHERE p.name= DSE byte[] b = read(file f, int pos, int length) 265

4 Architectural Blue Print SQL, JDBC, ODBC, Query processing - Parsing - Plan generation - Plan optimization - Plan execution SELECT s.firstname, s.lastname, COUNT(l.name) FROM Student s INNER JOIN Program p ON s.programid = p.id INNER JOIN Attendance a ON a.studentid = s.studentid INNER JOIN Lecture l ON a.lectureid = l.id GROUP BY s.firstname, s.lastname WHERE p.name= DSE Run Application Data System Database System Data model semantics - System catalog - Record format - Logical access paths Storage Structures - Record management - Free space management - Physical access paths Table Person: id INT, name VARCHAR, birthday DATE TID : TID : TID : Smith TID : TID : TID : Index P_id_IX on Person.id Access System Storage System Database System Buffered Pages - Page replacement strategy - Materialization strategy - Logging, Backup, Recovery Buffer Paged files File System Disks, Flash, RAID, SAN, Hardware 266

5 Architectural Trends 267

6 Different Access Characteristics OLTP (On-line Transaction Processing) Mix between read-only and update queries Minor analysis tasks Used for data preservation and lookup Read typically only a few records at a time High performance by storing contiguous records in disk pages OLAP (On-line Analytical Processing) Query-intensive DBMS applications Infrequent batch-oriented updates Complex analysis on large data volumes Read typically only a few attributes of large amounts of historical data in order to partition them and compute aggregates High performance by storing contiguous values of a single attribute 268

7 Hardware Developments Hardware improvements not equally distributed Advances in CPU speed have outpaced advances in RAM latency Main-memory access has become a performance bottleneck for many computer applications Bandwidth Latency Address translation (TLB) Memory Wall Cache memories can reduce the memory latency when the requested data is found in the cache. Vertically fragmented data structures optimize memory cache usage 269

8 Row Storage vs. Column Storage Row Storage Column Storage + easy to add/modify a record - might read unnecessary data + only need to read in relevant data - tuple writes require multiple accesses -> suitable for read-mostly, readintensive, large data repositories 27

9 Processing Models [Marcin Zukowski, Peter A. Boncz, Niels Nes, Sándor Héman: MonetDB/X - A DBMS In The CPU Cache. IEEE Data Eng. Bull. 28(2), p7-22, 25] 27

10 Transaction Management Principle of a transaction Sequence of successive DB operations that transform a database from a consistent state into another consistent state surrounded by: BOT... EOT (Commit / Abort) consistent database DB possibly inconsistent database consistent database DB BOT (begin of transaction) EOT (end of transaction) DML operations Properties ACID: Atomicity, Consistency, Isolation, Durability A transaction will always come to an end Normal (commit): changes are permanently stored within the DB Abnormal (abort / rollback): already composed changes are taken back Note: EOT state must not be different from BOT state 272

11 ACID Properties of Transactions Atomicity Indivisibility due to the transaction definition (Begin - End) All-or-nothing principle, i.e., the DBS guarantees Either the complete execution of a transaction or the ineffectiveness of the whole transaction (and of all associated operations) Consistency A successful transaction guarantees that all consistency requirements (integrity requirements) have been met Isolation Multiple transactions run isolated from each other and do not use (inconsistent) intermediate results from other transactions Durability All results of successful transactions have to be made persistent 273

12 Motivation Atomicity UNDO Recovery Part of the transaction is done, but we want to cancel it ABORT/ROLLBACK System crashes during transaction, some changes made it to the disk, some did not Durability REDO Recovery Transaction finished, user notified COMMIT System crashes before changes sent successfully to disk (asynchronous write) Consistency UNDO Recovery for consistency-related rollbacks Physical consistency Correctness of the storage and access structures Completely executed modification operations preserve the consistency Logical consistency Correctness of data contents correspond to a (possible) state of the real world Completely executed transactions preserve the logical consistency - All modifications of finished transactions are included - No modifications of open transactions are included Remember: Logical consistency requires physical consistency in the first place! 274

13 Reasons for crashes Transaction error Violation of system restrictions Violation of security regulations Excessive resource requirements / deadlocks Application-related errors, e.g. wrong operations and values ROLLBACK System error System crash with loss of main-memory contents Database system / operating system / hardware / power failure /... Device error (especially storage-medium error) Destruction of secondary storage systems Catastrophes Destruction of the computing center 275

14 Guarantee Atomicity & Durability Assumptions System may crash, but the disk is durable The only atomicity guarantee is that a disk block write is atomic Materialization strategy Preferred Policy: Steal/No Force This combination is most complicated but allows for highest performance No Force complicates enforcing Durability What if system crashes before a modified page written by a committed transaction makes it to disk? Write as little as possible, in a convenient place, at commit time, to support REDOing modifications Steal complicates enforcing Atomicity What if the transaction that performed udpates aborts? What if system crashes before transaction is finished? Must remember the old value of P (to support UNDOing the write to page P) 276

15 SQL, JDBC, ODBC, Query processing - Parsing - Plan generation - Plan optimization - Plan execution SELECT s.firstname, s.lastname, COUNT(l.name) FROM Student s INNER JOIN Program p ON s.programid = p.id INNER JOIN Attendance a ON a.studentid = s.studentid INNER JOIN Lecture l ON a.lectureid = l.id GROUP BY s.firstname, s.lastname WHERE p.name= DSE Run Application Data System Database System Data model semantics - System catalog - Record format - Logical access paths Storage Structures - Record management - Free space management - Physical access paths Table Person: id INT, name VARCHAR, birthday DATE TID : TID : TID : Smith Record Management TID : TID : TID : Index P_id_IX on Person.id Access System Storage System Database System Buffered Pages - Page replacement strategy - Materialization strategy - Logging, Backup, Recovery Buffer Paged files File System Disks, Flash, RAID, SAN, Hardware 277

16 Record Record Package of fields that together describe a thing, a person, a fact, etc. Each fields represents on property of the entity described by the record Similar to a struct in C Variable length (in contrast to pages) Record Manager Organizes physical storage of records in pages Operations: Get, Insert, Update, Delete, Scan Agnostic to record structure and semantic; records considered as byte strings of variable length Structure and content of record is defined be Access System and application Challenges Record addressing Free space management 278

17 Record Addressing Record address Identifier for records, used to address records, e.g., in indexes or query processing Assigned during insert of a record Goals Stability of identifier Fast and direct access Less organizational overhead Direct addressing Byte address or position number in file or page Instable Byte address: If record grows in length, following records would get new address Position number: Insert and delete operations change series or records Indirect addressing Surrogate with mapping table (complete indirection) Tuple Identifier (TID concept) 279

18 Surrogate with Mapping Table Surrogate Record type + serial number Serial number remains constant during record s life time Mapping table Maps surrogate to page Mapping Table Surrogate Page ID Problems Where to store mapping table? How can it be extended? How to search mapping table efficiently? H2 use B-Tree to store mapping table 28

19 TID Concept Record addressing with indirection inside the page Each page contains an array with record positions TID of a record consist of page id and index in position array Pros Access with one page access (two pages in case of overflow) Stable No mapping table required Operations Insert: Reuse unused position or add position Delete: Mark position as unused in array Update: Update all positions in array Update with overflow: Store record Record as overflow record and store TID of overflow record at original position (No double overflow: Update TID at original position) Overflow Record 28

20 Free Space Management Problem In which page is enough space for new record? Solution Free space table lists for all pages how much space is left Free space value Precise value: Ceil(Log 2 (page size)) => 2 bytes for common page size of 4K Rough value: use less bytes, free space = (value / page size)*2^(bits per value) Free space table With direct page addressing Assuming a single page can take n free space entries First page and each (n+)-th page takes free space entries With indirect page addressing Free space information stored in page table 282

21 SQL, JDBC, ODBC, Query processing - Parsing - Plan generation - Plan optimization - Plan execution SELECT s.firstname, s.lastname, COUNT(l.name) FROM Student s INNER JOIN Program p ON s.programid = p.id INNER JOIN Attendance a ON a.studentid = s.studentid INNER JOIN Lecture l ON a.lectureid = l.id GROUP BY s.firstname, s.lastname WHERE p.name= DSE Run Application Data System Database System Data model semantics - System catalog - Record format - Logical access paths Storage Structures - Record management - Free space management - Physical access paths Table Person: id INT, name VARCHAR, birthday DATE TID : TID : TID : Smith TID : TID : TID : Index P_id_IX on Person.id Access System Physical Access Paths Index Structures Storage System Database System Buffered Pages - Page replacement strategy - Materialization strategy - Logging, Backup, Recovery Paged files Buffer File System Disks, Flash, RAID, SAN, Hardware 283

22 Overview Indexes Table scan Read all pages and for each record evaluate the search criteria Pre-fetching Pers(PID, NAME, AGE, SALARY) Age Index Scan Salary Use index for search criteria on one or more attributes Fast access to single values or value ranges of index attributes Logical/physical sorting of values of key attributes (depending on index structure) Enforcing uniqueness Types if indexes Primary (Clustered) Index, determines physical organization; use for PK Secondary (Non-Clustered) Index, redundant access path Primary Index Secondary Index 284

23 Overview Indexes (2) Choice of Access Paths Index scan Only useful for low selectivity (low number of result tuples) Break even-point according to the output ratio of the number of tuples (usually max. 5%) Requires statistics about data Additional costs for index storage and updating access time Index Scan Table Scan Table Scan adequate/efficient for small tables (e.g., 5 pages) Queries with high selectivity (large result sets) -2MB/s sequential read ~ disk seeks/s hit rate 285

24 Classification of Index Structures Classification Onedimensional Index Structures Key Comparison Key Transformation Sequential Tree-Based Hash-Based Seq. Lists (phys. seq) Linked Lists (log. seq) Binary Search Trees Multiway Trees Example: B-Tree Prefix Trees (Tries) Static Dynamic Multiway Trees Tree structure with multiple children per node Idea: chose fan out so that node size suits page size 286

25 B-Tree (K i, D i, P i ) = entry min P = k+ max P = 2k+ free space keys < K K i < keys < K i+ keys > K p 287

26 B-Tree (2) Example B-Tree with k = 2, h = 3 Keys Agnostic to specific key semantic Only defined complete order required Could be of fixed or variable length Operations Search for data for given key value Insertion and deletion of key-data pair Payload Agnostic to specific data semantic Can be record or reference (TID) or mix 288

27 Search in the B-Tree Starting at the root node, each node is searched from left to right ) if K i matches the desired key value, the data record has been found (further records with the same key value might be located in a sub-tree to which P i- points) 2) if K i is smaller than the desired value, the search will be continued in the root of the sub-tree identified by P i- 3) if K i is larger than the desired value, the comparison with K i+ is repeated 4) if K 2k is also smaller than the desired value, the search will be continued in the sub-tree of P 2k If it s impossible to descend further into a sub-tree (2. or 4.) (leaf node): The search is aborted, no record with the desired key value is found Search for 38, 2, 6 289

28 Insertions in the B-Tree () Insertion Rule: insert only into leaf nodes! At Non-Leaf Nodes: descend down the tree as for the search S K i : follow P i- S > K i : check K i+ S > K 2k : follow P 2k At Leaf Node Insert the data record according to the sorting order Special case: leaf node is full (2k records) split the leaf node Splitting Generate a new leaf node Split the 2k+ entries (in order) into two leaf nodes first k entries left node last k entries right node middle entry (k+-th) is used as new discriminator (branching) and inserted into the parent node 29

29 Insertions in the B-Tree (2) Node Splitting during Insertion Two possible situations after a split The parent node is full repeat split on this level Enough space FINISHED Special case: root split Split of the root node New root with two successor nodes Height of a tree grows by The tree has been split from the bottom to the top Dynamic reorganization (self-balancing) No unloading or loading necessary Tree is always balanced But: In case of many insertions / deletions reorganization can be beneficial 29

30 Insertions in the B-Tree (3) Insertion Example Order k =, n=2k Keys:, 5, 2, 6, 7, 4, 8, 3 Finally, h=3 292

31 Insertion and Deletion in the B-Tree Problem Insertion can create overflow Deletion can create underflow and overflow Example: Insertion of key 22 Overflow Split Insert 22 Deletion of key 22? Underflow, need to access all four nodes, finally same as input 293

32 Deletion in the B-Tree Example Order k =, n=2k Delete key 3 Underflow Merge 294

33 Deletion in the B-Tree (2) Example Order k =, n=2k Delete key 3 Remember: Each path from the root to the leaf has the same length h Underflow Merge 295

34 Deletion in the B-Tree (3) Example Order k =, n=2k Delete key 3 Underflow Merge Overflow Split 296

35 Deletion in the B-Tree (4) Example Order k =, n=2k Delete key 3 Overflow Split Root Split! 297

36 Deletion Algorithm Example there are different algorithms Search the node that contains the key K to be deleted If key K is in a leaf node, delete the key in the leaf node and handle potentially resulting underflow, by merging with sibling If key K is in an inner node, pull up new discriminator from one of the successors Analyze which successor node of K has more elements: left or right one; If both have the same number of elements, decide for one Replace the key K to be deleted with the direct successor K from the left successor node or with the direct successor K from the right successor node, respectively Delete K or K from the respective successor node (recursively) Note: Major variants Merge (tis lecture) Re-distribution (instead of split/merge in case of overflow/underflow, the entries are re-distributed under consideration of one or multiple adjacent nodes) 298

37 B-Trees, B + -Trees, and B*-Trees B + -Trees and B*-Trees Data is only in leaf nodes Key redundancy, but higher fan-out lower tree high, less I/O Simpler delete procedure requires only merging of nodes Double linked list of all leaf nodes B*-Trees Modified valid node sizes: from [k,2k] to [4/3k,2k] better node utilization, but more splits/merges B*-Tree with k = 2, h = 3 Example Secondary index Non unique 299

38 Indexing Low Cardinality Columns Problem Example: B-tree on the sex of customers for a table with,, tuples results in two lists with approximately 5, tuples each F M TID TID TID TID TID TID TID TID Query for all female customers requires 5, random page accesses (secondary index!) Table scan would be much faster Conclusion B-trees (and also hashing) are useful for predicates with low selectivity (output/input cardinality ratio) Rule of thumb: margin hit rate is approx. 5% higher hit rates do not justify the efforts for an index access 3

39 Bitmap Index Idea (Long history since the 96s) Create a bitmap/bitlist for each attribute value Each tuple in the table is assigned to one bit in the bitmap (by position/ sequential TID) Bit values attribute value set attribute value not set Necessary condition: Sequential numbering of the tuples (TIDs) Name Sex Region Race Carol f n white Harold m e black Anne f e asian Iris f ne white m se hisp f e white f sw asian f w black f n asian m e hisp m se black f s white m nw black f s white f w black F Sex M 3

40 Querying Bitmap Indexes Main advantage of bitmap indexes Simple and efficient logical join possible Read only data that is relevant for predicates Example: σ Sex= f ᴧ Region= n R Bitmaps B and B2 in conjunction: for (i=; i<b.length; i++) B = B[i] & B2[i]; Example I/O Costs Estimation σ Sex= f ᴧ Region= n ᴧ Race= Asian R ( Asian women of region North ) Selectivity: /2 /8 /4 = /64 N=, tuples, with length of 4 bytes each (~ tuples per page for 4kB pages) Table scan: pages Bitmap access: /64 56 pages (worst case: each tuple in a different page), plus page for bitmaps F AND N AND A = * 32