CS317 File and Database Systems http://commons.wikimedia.org/wiki/category:r-tree#mediaviewer/file:r-tree_with_guttman%27s_quadratic_split.png Lecture 10 Physical DBMS Design October 23, 2017 Sam Siewert
Reminders Assignment #4 Grading in Progress Assignment #5 (Available on Canvas) Physical DB Design Practice Propose your Final DBMS Project Assignment #6, DBMS Project of Your Interest FINAL ORAL PRESENTATION Design Schema for DBMS project in a small team Logical design focus Normalization Physical is MySQL on PRClab Combine Network Applications with DBMS in C/C++, JDBC, or Python - http://www.mysql.com/products/connector/ Add Stored Programs and Triggers Add Views Create Transactions where needed Sam Siewert 2
Goals for This Week Wrap-Up DBMS Conceptual Design Wrap-Up DBMS Logical Design Introduce DBMS Physical Design DBMS Indexing [B-Tree & B+-Tree, Hash, R-Tree] Indexing Fundamentals ISAM Indexed Sequential Access Method [Records in Files] MyISAM Index, Format, Data Files for Each Table InnoDB Cached Indexes and Data, Several Files or Raw Disk Partitions Memory (Heap) Unordered Set of Records, Hash Index, B-Tree Options Introduction to B and B+ Trees (Bayer or perhaps Boeing or Balance Tree) Hash - Fast Table Lookup, Collisions handled with List or Tree (Similar, but different than data digest or cryptographic - e.g. MD5 Hash) Exploration beyond B-Trees - R-Tree, X-Tree (Not covered) Mapping to DBMS to Storage Memory Nand Flash and SSD HDD File Systems and Block Storage Sam Siewert 3
DBMS Design in 10 Steps DBMS Design - Decomposition 1. Conceptual Design Analysis, Centralized or View Integration 2. Build and Validate Logical Data Model ER/EER, Normalization, Schema Prototype (Workbench) 3. Translate Logical Data Model for Target DBMS (MySQL) 4. Design File Organization (or Block) and Indexing 5. Triggers and SQL/PSM for Referential Integrity 6. User View Design (Lossless Join Verification) 7. Security 8. Transactions for Complex Updates 9. Redundancy (RTO/RPO and RAID) 10. Connectors for App Interfaces Covered, To be covered Sam Siewert 4
Logical DBMS Engineering Ideally use a CASE tool like Workbench EER DB SQL EER Forward Reverse Sam Siewert 5
Workbench Manual Model Editor Name your Model to match DB deployment Target name e.g. siewertsdhv1_3, siewertsdb, etc. Test and Interaction Tab (Queries, Schemas, Sam Siewert 6
Workbench Model High Level View of Logical Model and Test SQL Scripts Deployed DB from EER Logical Design Sam Siewert 7
Logical to Physical Mappings Branch Table Logical Design Details Primary Key Designation, Not- NULL, Default, etc. Foreign Keys, Triggers, etc. Branch Table Physical Design Details Indexes Engine [InnoDB, MyISAM, MEMORY], Columns to Index, Storage Type [BTREE, HASH] Indexing for Optimal Query Engine = MEMORY Index Type = BTREE Sam Siewert 8
Physical Design - Objectives Purpose of physical database design. How to map the logical database design to a physical database design. How to design base relations for target DBMS. How to design general constraints for target DBMS. 9
Physical Design - Objectives How to select appropriate file organizations based on analysis of transactions. When to use secondary indexes to improve performance. How to estimate the size of the database. How to design user views. How to design security mechanisms to satisfy user requirements. 10
Logical v. Physical Database Design Sources of information for physical design process includes logical data model and documentation that describes model. Logical database design is concerned with the what, physical database design is concerned with the how. 11
Physical Database Design Process of producing a description of the implementation of the database on secondary storage. It describes the base relations, file organizations, and indexes used to achieve efficient access to the data, and any associated integrity constraints and security measures. 12
Overview of Physical Database Design Methodology Step 1 Conceptual Design (Chapters 10-12, Lifecycle, Analysis Centralized, View Integration) Step 2 Build and Validate Logical Data Model (Chapters 12 14 ER/EER, Normalization) Step 3 Translate logical data model for target DBMS Step 3.1 Design base relations Step 3.2 Design representation of derived data Step 3.3 Design general constraints 13
Overview of Physical Database Design Methodology Step 4 Design file organizations and indexes Step 4.1 Analyze transactions Step 4.2 Choose file organizations Step 4.3 Choose indexes Step 4.4 Estimate disk space requirements 14
Overview of Physical Database Design Methodology Step 5 Design user views (Lossless Join) Step 6 Design security mechanisms Step 7 Consider the introduction of controlled redundancy Step 8 Monitor and tune operational system Step 9 RAID and DR (Disaster Recovery), Scaling Step 10 Applications and Connectors 15
Step 3 Translate Logical Data Model for Target DBMS To produce a relational database schema from the logical data model that can be implemented in the target DBMS. Need to know functionality of target DBMS such as how to create base relations and whether the system supports the definition of: PKs, FKs, and AKs; required data i.e. whether system supports NOT NULL; domains; relational integrity constraints; general constraints. 16
Step 3.1 Design base relations To decide how to represent base relations identified in logical model in target DBMS. For each relation, need to define: the name of the relation; a list of simple attributes in brackets; the PK and, where appropriate, AKs and FKs. referential integrity constraints for any FKs identified. 17
Step 3.1 Design base relations From data dictionary, we have for each attribute: its domain, consisting of a data type, length, and any constraints on the domain; an optional default value for the attribute; whether it can hold nulls; whether it is derived, and if so, how it should be computed. 18
DBDL for the PropertyForRent Relation 19
Step 3.2 Design representation of derived data To decide how to represent any derived data present in logical data model in target DBMS. Examine logical data model and data dictionary, and produce list of all derived attributes. Derived attribute can be stored in database or calculated every time it is needed. 20
Step 3.2 Design representation of derived data Option selected is based on: additional cost to store the derived data and keep it consistent with operational data from which it is derived; cost to calculate it each time it is required. Less expensive option is chosen subject to performance constraints. 21
PropertyforRent Relation and Staff Relation with Derived Attribute noofproperties 22
Step 3.3 Design general constraints To design the general constraints for target DBMS. Some DBMS provide more facilities than others for defining enterprise constraints. Example: CONSTRAINT StaffNotHandlingTooMuch CHECK (NOT EXISTS (SELECT staffno FROM PropertyForRent GROUP BY staffno HAVING COUNT(*) > 100)) 23
Step 4 Design File Organizations and Indexes To determine optimal file organizations to store the base relations and the indexes that are required to achieve acceptable performance; that is, the way in which relations and tuples will be held on secondary storage. Must understand the typical workload that database must support. 24
Cross-referencing transactions and relations 25
Example Transaction Usage Map 26
Example Transaction Analysis Form 27
Step 4.2 Choose file organizations To determine an efficient file organization for each base relation. File organizations include Heap [Unordered Set of Records], Hash, Indexed Sequential Access Method (ISAM), B+-Tree, and Clusters. Some DBMSs may not allow selection of file organizations. 28
Step 4.3 Choose indexes To determine whether adding indexes will improve the performance of the system. One approach is to keep tuples unordered and create as many secondary indexes as necessary. 29
Step 4.3 Choose indexes Another approach is to order tuples in the relation by specifying a primary or clustering index. In this case, choose the attribute for ordering or clustering the tuples as: attribute that is used most often for join operations - this makes join operation more efficient, or attribute that is used most often to access the tuples in a relation in order of that attribute. 30
Step 4.3 Choose indexes If ordering attribute chosen is key of relation, index will be a primary index; otherwise, index will be a clustering index. Each relation can only have either a primary index or a clustering index. Secondary indexes provide a mechanism for specifying an additional key for a base relation that can be used to retrieve data more efficiently. 31
Step 4.3 Choose indexes Have to balance overhead involved in maintenance and use of secondary indexes against performance improvement gained when retrieving data. This includes: adding an index record to every secondary index whenever tuple is inserted; updating secondary index when corresponding tuple updated; increase in disk space needed to store secondary index; possible performance degradation during query optimization to consider all secondary indexes. 32
Step 4.3 Choose indexes Guidelines for choosing wish-list 1. Do not index small relations. 2. Index PK of a relation if it is not a key of the file organization. 3. Add secondary index to a FK if it is frequently accessed. 4. Add secondary index to any attribute heavily used as a secondary key. 5. Add secondary index on attributes involved in: selection or join criteria; ORDER BY; GROUP BY; and other operations involving sorting (such as UNION or DISTINCT). 33
Step 4.3 Choose indexes Guidelines for choosing wish-list 6. Add secondary index on attributes involved in builtin functions. 7. Add secondary index on attributes that could result in an index-only plan. 8. Avoid indexing an attribute or relation that is frequently updated. 9. Avoid indexing an attribute if the query will retrieve a significant proportion of the relation. 10. Avoid indexing attributes that consist of long character strings. 34
Note on MySQL Indexes MySQL Create Index Defaults - Common MySQL 5.7 Provides Indexes to Speed Query for Query Engines as Follows Spatial R-Tree Indexing Possible for MyISAM Options Advanced Indexing Using Spatial Data Extensions MEMORY Storage Engine Hash Option Storage Engine MyISAM InnoDB Memory (Heap) Supported Index Types B-tree, R-Tree Spatial on POINT, GEOMETRY Columns B-tree Hash, B-Tree Sam Siewert 35
B-Tree Rudolf Bayer, Edward McCreight Boeing Optimized for Read/Write Large Block Data DBMS and File Systems B+-Tree is Variant with Keys [Indexes] Only for Internal Nodes and all Keys and Data Found in Leaves Keys [Indexes] Repeated in Tree Keys [Indexes] + Data in Sequential Order in Leaves Goal is to Increase the Size of Nodes to Match File System Block Size [E.g. 4K 64K] with N Keys [Data] Stored in Each Tree Node [8x512 128x512 Sectors] Sam Siewert 36
B-Tree Example staffno Key, staff tuples with multiple attributes in stafftable Order N=4, 4 Pointers, 3 Keys per Node, 2 Pointers, 1 Key Minimum [Splits and Combines on Insert and Delete] Tuple Pointer for All Keys [Implicit] Insert Tuples for staffno S005, S003, S021, S009, S001, S013 Ptr1 S003 Ptr2 S005 Ptr3 S0021 Ptr4 First 3 inserts, fill root node Ptr1 S009 Ptr2 S009 insert, splits root node S001 and S013 fill out children Ptr1 S001 Ptr2 S003 Ptr3 S005 Ptr4 Ptr1 S013 Ptr2 S021 Ptr3 Sam Siewert 37
B-Tree Inserts Continued Splits Continue on Staff Table Inserts Insert S002, Ptr1 S009 Ptr2 S002 insert splits left child and promotes S003 to root node Ptr1 S001 Ptr2 S003 Ptr3 S005 Ptr4 Ptr1 S013 Ptr2 S021 Ptr3 Ptr1 S003 Ptr2 S009 Ptr3 Ptr1 S001 Ptr2 S002 Ptr3 Ptr1 S005 Ptr2 Ptr1 S013 Ptr2 S021 Ptr3 Sam Siewert 38
B-Tree Provides Nice WHERE Clause Support Index [Key] Appears ONCE in Each Node Downside SELECT * with Tuples in staffno ASC or DSC Order is Time Consuming Split and Combines Required on Insert/Delete Key, Pointers, and Tuple Data in Nodes Limit Bushiness B+-Tree Stores Keys, Pointers ONLY on Internal Nodes, Increasing Bushiness [Key for File systems], Record/File Data ONLY on Leaf Nodes Sam Siewert 39
B+-Tree Replication of Keys [Indexes] Tuple Data ONLY in Leaves Useful for Key [Index] Cache and Storage of Leaves I/O CACHE Ptr1 S003 Ptr2 S009 Ptr3 DISK DRIVE Ptr1 S001 Ptr2 S002 Ptr3 Ptr1 S003 Ptr2 S005 Ptr3 Ptr1 S009 Ptr2 S013 Ptr3 S021 Ptr4 Sam Siewert 40
B-Tree Details Covered in Data Structures B-tree Example in C The Ubiquitous B-Tree, by Douglas Comer Insertion Balancing Deletion Balancing Cost of Operations Recall Disk Drive Performance 100 to 200 Random I/Os per Second [Seek + Rotate is 5 to 10 msec] Sequential Rates 100 to 200 Mbytes per Second MySQL Memory (Heap) B-Tree for Small Tables Fit in Memory Pages [Cache Lines] Sam Siewert 41
Hash Table Index = Function (Candidate Key) E.g. Index = MD5(staffNo, staffname) % Table_Size Perfect Hash Guarantees No 2 Staff will Map to Same Index Hash Collision Requires Secondary Collision Structure E.g. List Great for Small Tables and Data Sets Typically in Memory Performs like LUT [Look Up Table or Array] Sam Siewert 42
Multi-Value Query B-Tree Great for Selection and Theta-Join Finding Tuples Satisfying WHERE clauses What About Complex Query for Multiple Conditions? E.g. WHERE clause and Distance from Current Location < 20 miles Satisfy B-Tree WHERE clause Check X,Y Distance Constraint as Post Filter? Possible to Combine Multiple Dimensions for Query in the Tree Index? R-Trees or Rectangular Trees, X-Tree, etc. [Not Covering, but Interesting] Wrap-Up B-Trees and RAID Next Week Sam Siewert 43