CS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #4: Constraints, Storing and Indexes

Transcription

1 CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #4: Constraints, Storing and Indexes

2 Constraints in Rela;onal Algebra and SQL

3 Maintaining Integrity of Data Data is dirty. How does an applica>on ensure that a database modifica>on does not corrupt the tables? Two approaches: Applica>on programs check that database modifica>ons are consistent. Use the features provided by SQL. 3

4 Maintaining Integrity of Data Data is dirty. How does an applica>on ensure that a database modifica>on does not corrupt the tables? Two approaches: Applica>on programs check that database modifica>ons are consistent. Use the features provided by SQL. 4

5 Integrity Checking in SQL Mainly: PRIMARY KEY and UNIQUE constraints. FOREIGN KEY constraints. Also: (we will skip them) Constraints on a_ributes and tuples. Triggers (schema-level constraints). How do we express these constraints? How do we check these constraints? What do we do when a constraint is violated? 5

6 Keys in SQL A set of a_ributes S is a key for a rela>on R if every pair of tuples in R disagree on at least one a_ribute in S. Select one key to be the PRIMARY KEY; declare other keys using UNIQUE. 6

7 Primary Keys in SQL Modify the schema of Students to declare PID to be the key. CREATE TABLE Students( PID VARCHAR(8) PRIMARY KEY, Name CHAR(20), Address VARCHAR(255)); What about Courses, which has two a_ributes in its key? CREATE TABLE Courses(Number integer, DeptName: VARCHAR(8), CourseName VARCHAR(255), Classroom VARCHAR(30), Enrollment integer, PRIMARY KEY (Number, DeptName) ); 7

8 Effect of Declaring PRIMARY KEYs Two tuples in a rela>on cannot agree on all the a_ributes in the key. DBMS will reject any ac>on that inserts or updates a tuple in viola>on of this rule. A tuple cannot have a NULL value in a key a_ribute. 8

9 Other Keys in SQL If a rela>on has other keys, declare them using the UNIQUE keyword. Use UNIQUE in exactly the same places as PRIMARY KEY. There are two differences between PRIMARY KEY and UNIQUE: A table may have only one PRIMARY KEY but more than one set of a_ributes declared UNIQUE. A tuple may have NULL values in UNIQUE a_ributes. 9

10 Enforcing Key Constraints Upon which ac>ons should an RDBMS enforce a key constraint? Only tuple update and inser>on. RDMBS searches the tuples in the table to find if any tuple exists that agrees with the new tuple on all a_ributes in the primary key. To speed this process, an RDBMS automa>cally creates an efficient search index on the primary key. User can instruct the RDBMS to create an index on one or more a_ributes 10

11 Foreign Key Constraints Referen>al integrity constraint: in the rela>on Teach (that connects Courses and Professors), if Teach relates a course to a professor, then a tuple corresponding to the professor must exist in Professors. How do we express such constraints in Rela>onal Algebra? Consider the Teach(ProfessorPID, Number, DeptName) rela>on. every non-null value of ProfessorPID inteach must be a valid ProfessorPID in Professors. RA πprofessorpid(teach) π PID(Professors). 11

12 Foreign Key Constraints in SQL every non-null value of ProfessorPID inteach must be a valid ProfessorPID in Professors. In Teach, declare ProfessorPID to be a foreign key. CREATE TABLE Teach(ProfessorPID VARCHAR(8) REFERENCES Professor(PID), Name VARCHAR(30)...); CREATE TABLE Teach(ProfessorPID VARCHAR(8), Name VARCHAR(30)..., FOREIGN KEY ProfessorPID REFERENCES Professor(PID)); If the foreign key has mul>ple a_ributes, use the second type of declara>on. 12

13 Requirements for FOREIGN KEYs If a rela>on R declares that some of its a_ributes refer to foreign keys in another rela>on S, then these a_ributes must be declared UNIQUE or PRIMARY KEY in S. Values of the foreign key in R must appear in the referenced a_ributes of some tuple in S. 13

14 Enforcing Referen;al Integrity Three policies for maintaining referen>al integrity. Default policy: reject viola>ng modifica>ons. Cascade policy: mimic changes to the referenced a_ributes at the foreign key. Set-NULL policy: set appropriate a_ributes to NULL. 14

15 Specifying Referen;al Integrity Policies in SQL SQL allows the database designer to specify the policy for deletes and updates independently. Op>onally follow the declara>on of the foreign key with ON DELETE and/or ON UPDATE followed by the policy: SET NULL or CASCADE. Constraints can be circular, e.g., if there is a one-one mapping between two rela>ons. In this case, SQL allows us to defer the checking of constraints. 15

16 SKIP! Constraining ATributes and Tuples SQL also allows us to specify constraints on a_ributes in a rela>on and on tuples in a rela>on. Disallow courses with a maximum enrollment greater than 100. A chairperson of a department must teach at most one course every semester. How do we express such constraints in SQL? How can we change our minds about constraints? A simple constraint: NOT NULL Declare an a_ribute to be NOT NULL ajer its type in a CREATE TABLE statement. Effect is to disallow tuples in which this a_ribute is NULL. 16

17 STORING DATA 17

18 DBMS Layers: Queries Query Optimization and Execution Relational Operators Files and Access Methods TODAY à Buffer Management Disk Space Management DB 18

19 Leverage OS for disk/file management? Layers of abstrac>on are good but: 19

20 Leverage OS for disk/file management? Layers of abstrac>on are good but: Unfortunately, OS ojen gets in the way of DBMS 20

21 Leverage OS for disk/file management? DBMS wants/needs to do things its own way Specialized prefetching Control over buffer replacement policy LRU not always best (some>mes worst!!) Control over thread/process scheduling Convoy problem Arises when OS scheduling conflicts with DBMS locking Control over flushing data to disk WAL protocol requires flushing log entries to disk 21

22 Disks and Files DBMS stores informa>on on disks. but: disks are (rela>vely) VERY slow! Major implica>ons for DBMS design! 22

23 Disks and Files Major implica>ons for DBMS design: READ: disk -> main memory (RAM). WRITE: reverse Both are high-cost opera>ons, rela>ve to in-memory opera>ons, so must be planned carefully! 23

24 Why Not Store It All in Main Memory? 24

25 Why Not Store It All in Main Memory? Costs too much. disk: ~$1/Gb; memory: ~$100/Gb High-end Databases today in the TB range. Approx 60% of the cost of a produc>on system is in the disks. Main memory is volable. Note: some specialized systems do store en>re database in main memory. 25

26 The Storage Hierarchy Smaller, Faster Bigger, Slower 26

27 Main memory (RAM) for currently used data. Disk for the main database (secondary storage). Tapes for archiving older versions of the data (tertiary storage). The Storage Hierarchy Registers L1 Cache. Main Memory Magnetic Disk Magnetic Tape Smaller, Faster Bigger, Slower 27

28 Jim Gray s Storage Latency Analogy: How Far Away is the Data? 10 9 Tape Andromeda The image cannot be displayed. Your computer The image may not have enough memory to cannot open the be image, or the image may have been displayed. corrupted. Your Restart your computer, and computer then open may the file again. If the red x still not appears, have you may have to delete the image and then insert it The again. image cannot be displayed. Your computer may not have enough 2,000 Years 10 6 Disk Pluto 2 Years 100 Memory Boston 1.5 hr On Board Cache On Chip Cache Registers This Building This Room My Head 10 min 1 min 28

29 Disks Secondary storage device of choice. Main advantage over tapes: random access vs. sequenbal. Data is stored and retrieved in units called disk blocks or pages. Unlike RAM, >me to retrieve a disk page varies depending upon loca>on on disk. rela>ve placement of pages on disk is important! 29

30 Anatomy of a Disk Disk head Spindle Tracks Sector Track Cylinder Platter Block size = multiple of sector size (which is fixed) Arm assembly Arm movement Platters Sector #30

31 Accessing a Disk Page Time to access (read/write) a disk block:... 31

32 Accessing a Disk Page Time to access (read/write) a disk block: seek Bme: moving arms to posi>on disk head on track rotabonal delay: wai>ng for block to rotate under head transfer Bme: actually moving data to/from disk surface 32

33 Accessing a Disk Page Rela>ve >mes? seek Bme: rotabonal delay: transfer Bme: 33

34 Accessing a Disk Page Rela>ve >mes? seek Bme: about 1 to 20msec rotabonal delay: 0 to 10msec transfer Bme: < 1msec per 4KB page Seek Rotate transfer Transfer 34

35 Seek ;me & rota;onal delay dominate Key to lower I/O cost: reduce seek/rota>on delays! Also note: For shared disks, much >me spent wai>ng in queue for access to arm/controller Seek Rotate transfer 35

36 Arranging Pages on Disk Next block concept: blocks on same track, followed by blocks on same cylinder, followed by blocks on adjacent cylinder Accesing next block is cheap A useful op>miza>on: pre-fetching See textbook page

37 Rules of thumb 1. Memory access much faster than disk I/O (~ 1000x) Sequen>al I/O faster than random I/O (~ 10x) 37

38 Disk Arrays: RAID Just FYI Logical Physical Benefits: Higher throughput (via data striping ) Longer MTTF (via redundancy) 38

39 Conclusions---Storing Memory hierarchy Disks: (>1000x slower) - thus pack info in blocks try to fetch nearby blocks (sequen>ally) 39

40 TREE INDEXES 40

41 Declaring Indexes No standard! Typical syntax: CREATE INDEX StudentsInd ON Students(ID); CREATE INDEX CoursesInd ON Courses(Number, DeptName); 41

42 Types of Indexes Primary: index on a key Used to enforce constraints Secondary: index on non-key a_ribute Clustering: order of the rows in the data pages correspond to the order of the rows in the index Only one clustered index can exist in a given table Useful for range predicates Non-clustering: physical order not the same as index order 42

43 Using Indexes (1): Equality Searches Given a value v, the index takes us to only those tuples that have v in the a_ribute(s) of the index. E.g. (use CourseInd index) SELECT Enrollment FROM Courses WHERE Number = 4604 and DeptName = CS 43

44 Using Indexes (1): Equality Searches Given a value v, the index takes us to only those tuples that have v in the a_ribute(s) of the index. Can use Hashes, but see next 44

45 Using Indexes (2): Range Searches ``Find all students with gpa > 3.0 may be slow, even on sorted file Hashes not a good idea! What to do? Page 1 Page 2 Page 3 Page N Data File 45

46 Range Searches ``Find all students with gpa > 3.0 may be slow, even on sorted file Solu>on: Create an `index file. k1 k2 kn Index File Page 1 Page 2 Page 3 Page N Data File 46

47 Range Searches More details: if index file is small, do binary search there Otherwise?? k1 k2 kn Index File Page 1 Page 2 Page 3 Page N Data File 47

48 B-trees the most successful family of index schemes (B-trees, B+-trees, B*-trees) Can be used for primary/secondary, clustering/non-clustering index. balanced n-way search trees Original Paper: Rudolf Bayer and McCreight, E. M. Organiza>on and Maintenance of Large Ordered Indexes. Acta Informa>ca 1, ,

49 B-trees Eg., B-tree of order d=1: <6 6 >6 <9 9 >

50 B - tree proper;es: each node, in a B-tree of order d: Key order at most n=2d keys at least d keys (except root, which may have just 1 key) all leaves at the same level if number of pointers is k, then node has exactly p1 pn k-1 keys (leaves are empty) v1 v2 v n-1 50

51 Proper;es block aware nodes: each node is a disk page O(log (N)) for everything! (ins/del/search) typically, if d = , then 2-3 levels u>liza>on >= 50%, guaranteed; on average 69% 51

52 Queries Algo for exact match query? (eg., ssn=8?) <6 6 >6 <9 9 >

53 JAVA anima;on h_p://slady.net/java/bt/ 53

54 Queries Algo for exact match query? (eg., ssn=8?) <6 6 >6 <9 9 >

55 Queries Algo for exact match query? (eg., ssn=8?) <6 6 >6 <9 9 >

56 Queries Algo for exact match query? (eg., ssn=8?) <6 6 >6 <9 9 >

57 Queries Algo for exact match query? (eg., ssn=8?) <6 6 >6 <9 9 >9 H steps (= disk accesses)

58 Queries what about range queries? (eg., 5<salary<8) Proximity/ nearest neighbor searches? (eg., salary ~ 8 ) 58

59 Queries what about range queries? (eg., 5<salary<8) Proximity/ nearest neighbor searches? (eg., salary ~ 8 ) <6 6 9 >6 <9 >

60 Queries what about range queries? (eg., 5<salary<8) Proximity/ nearest neighbor searches? (eg., salary ~ 8 ) <6 6 9 >6 <9 >

61 Queries what about range queries? (eg., 5<salary<8) Proximity/ nearest neighbor searches? (eg., salary ~ 8 ) <6 6 9 >6 <9 >

62 Queries what about range queries? (eg., 5<salary<8) Proximity/ nearest neighbor searches? (eg., salary ~ 8 ) <6 6 9 >6 <9 >

63 Varia;ons How could we do even be_er than the B-trees above? 63

64 B+ trees - Mo;va;on B-tree print keys in sorted order: <6 6 >6 <9 9 >

65 B+ trees - Mo;va;on B-tree needs back-tracking how to avoid it? <6 6 >6 <9 9 >

66 B+ trees - Mo;va;on Stronger reason: for clustering index, data records are sca_ered: <6 6 >6 <9 9 >

67 Solu;on: B+ - trees facilitate sequen>al ops They string all leaf nodes together AND replicate keys from non-leaf nodes, to make sure every key appears at the leaf level (vital, for clustering index!) 67

68 B+ trees <6 6 >=6 <9 9 >=

69 B+ trees <6 6 9 Index Pages >=6 <9 >= Data Pages 69

70 B+ trees More details: next (and textbook) In short: on split at leaf level: COPY middle key upstairs at non-leaf level: push middle key upstairs (as in plain B-tree) 70

71 Example B+ Tree Search begins at root, and key comparisons direct it to a leaf Search for 5*, 15*, all data entries >= 24*... Root * 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39* Based on the search for 15*, we know it is not in the tree! 71

72 Inser;ng a Data Entry into a B+ Tree Find correct leaf L. Put data entry onto L. If L has enough space, done! Else, must split L (into L and a new node L2) Redistribute entries evenly, copy up middle key. parent node may overflow but then: push up middle key. Splits grow tree; root split increases height. 72

73 Example B+ Tree Inser;ng 30* Root * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* 73

74 Example B+ Tree Inser;ng 30* Root * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* 30* 74

75 Example B+ Tree - Inser;ng 8* Root * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* 75

76 Example B+ Tree - Inser;ng 8* Root * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* No Space 76

77 Example B+ Tree - Inser;ng 8* Root So Split! * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* * 3* 5* 14* 16* 19* 20* 22* 23* 24* 27* 5* 7* 8* 29* 77

78 Example B+ Tree - Inser;ng 8* Root So Split! * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* And then push middle UP * 3* 5* 14* 16* 19* 20* 22* 23* 24* 27* 5* 7* 8* 29* 78

79 Example B+ Tree - Inser;ng 8* Root * 3* 5* 7* 14* 16* 19* 20* 22* 23* 24* 27* 29* Final State <5 >= * 3* 14* 16* 19* 20* 22* 23* 24* 27* 5* 7* 8* 29* 79

80 Example B+ Tree - Inser;ng 21* Root * 3* 5* 7* 8* 14* 16* 19* 20* 22* 23* 24* 27* 29* 2* 3* 5* 7* 8* 14* 16* 19* 20* 22* 23* 24* 27* 29* 80

81 Example B+ Tree - Inser;ng 21* Root * 3* 5* 7* 8* 14* 16* 19* 20* 22* 23* 24* 27* 29* Root is Full, so split recursively 2* 3* 5* 7* 8* 14* 16* 19* 20* 21* 22* 23* 24* 27* 29* 81

82 Example B+ Tree: Recursive split Root * 3* 5* 7* 8* 14* 16* 19* 20* 21* 22* 23* 24* 27* 29* Notice that root was also split, increasing height. 82

83 Example: Data vs. Index Page Split leaf: copy non-leaf: push Data Page Split 2* 3* 5* 7* 8* 5 2* 3* 5* 7* 8* why not non-leaves? Index Page Split #83

84 Same Inser;ng 21*: The Deferred Root Split * 3* 5* 7* 8* 14* 16* 19* 20* 22* 23* 24* 27* 29* Note this has free space. So 84

85 Inser;ng 21*: The Deferred Split Root * 3* 5* 7* 8* 14* 16* 19* 20* 22* 23* 24* 27* 29* Root LEND keys to sibling, through PARENT! 2* 3* 5* 7* 8* 14* 16* 19* 20* 21* 22* 23* 24* 27* 29* 85

86 Inser;ng 21*: The Deferred Split Root * 3* 5* 7* 8* 14* 16* 19* 20* 22* 23* 24* 27* 29* Root Shorter, more packed, faster tree * 3* 5* 7* 8* 14* 16* 19* 20* 21* 22* 23* 24* 27* 29* 86

87 Inser;on examples for you to try Root (not shown) 2* 3* 5* 7* 8* 11* 14* 16* 21* 22* 23* Insert the following data entries (in order): 28*, 6*, 25* 87

88 Answer After inserting 28*, 6* * 3* 5* 6* 7* 8* 11* 14* 16* 21* 22* 23* 28* After inserting 25* 88

89 Answer After inserting 25* * 3* 5* 6* 7* 8* 11* 14* 16* 21* 22* 23* 25* 28* 89

90 Dele;ng a Data Entry from a B+ Tree Start at root, find leaf L where entry belongs. Remove the entry. If L is at least half-full, done! If L underflows Try to re-distribute, borrowing from sibling (adjacent node with same parent as L). If re-distribu>on fails, merge L and sibling. update parent and possibly merge, recursively 90

91 Dele;on from B+Tree 1 Root * 3* 5* 7* 8* 14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39* 91

92 Example: Delete 19* & 20* Root Dele>ng 19* is easy: * 3* 5* 7* 8* 14* 16* 19* 20* 20* 22* 22* 24* 27* 29* 33* 34* 38* 39* 3 Root * 3* 5* 7* 8* 14* 16* 22* 24* 27* 29* 33* 34* 38* 39* Dele>ng 20* -> re-distribu>on (no>ce: 27 copied up) 92

93 ... And Then Dele;ng 24* 3 Root * 3* 5* 7* 8* 14* 16* 22* 24* 27* 29* 33* 34* 38* 39* 4 Root * 3* 5* 7* 8* 14* 16* 22* 27* 29* 33* 34* 38* 39* Must merge leaves: OPPOSITE of insert 93

94 ... And Then Dele;ng 24* 3 Root * 3* 5* 7* 8* 14* 16* 22* 24* 27* 29* 33* 34* 38* 39* 4 Root 17 but are we done?? * 3* 5* 7* 8* 14* 16* 22* 27* 29* 33* 34* 38* 39* Must merge leaves: OPPOSITE of insert 94

95 ... Merge Non-Leaf Nodes, Shrink Tree 4 Root * 3* 5* 7* 8* 14* 16* 22* 27* 29* 33* 34* 38* 39* 5 Root * 3* 5* 7* 8* 14* 16* 22* 27* 29* 33* 34* 38* 39* 95

96 Example of Non-leaf Re-distribu;on Tree is shown below during dele>on of 24*. Now, we can re-distribute keys Root * 3* 5* 7* 8* 14* 16* 17* 18* 20* 21* 22* 27* 29* 33* 34* 38* 39* 96

97 Aper Re-distribu;on need only re-distribute 20 ; did 17, too why would we want to re-distribute more keys? Ans: reduces likelihood of split (see Root Book, pg. 356) * 3* 5* 7* 8* 14* 16* 17* 18* 20* 21* 22* 27* 29* 33* 34* 38* 39* 97

98 Main observa;ons for dele;on If a key value appears twice (leaf + nonleaf), the above algorithms delete it from the leaf, only why not non-leaf, too? 98

99 Main observa;ons for dele;on If a key value appears twice (leaf + nonleaf), the above algorithms delete it from the leaf, only why not non-leaf, too? lazy dele>ons - in fact, some vendors just mark entries as deleted (~ underflow), and reorganize/compact later 99

100 Recap: main ideas on overflow, split (and push, or copy ) or consider deferred split on underflow, borrow keys; or merge or let it underflow

101 B+ Trees in Prac;ce Typical order: 100. Typical fill-factor: 67%. average fanout = 2*100*0.67 = 134 Typical capaci>es: Height 4: 1334 = 312,900,721 entries Height 3: 1333 = 2,406,104 entries 101

102 B+ Trees in Prac;ce Can ojen keep top levels in buffer pool: Level 1 = 1 page = 8 KB Level 2 = 134 pages = 1 MB Level 3 = 17,956 pages = 140 MB 102

103 Conclusions B+tree is the prevailing indexing method Excellent, O(logN) worst-case performance for ins/del/search; (~3-4 disk accesses in prac>ce) guaranteed 50% space u>liza>on; avg 69% 103

104 Conclusions Can be used for any type of index: primary/ secondary, sparse (clustering), or dense (nonclustering) Several fine-extensions on the basic algorithm deferred split; (underflows) check textbook for these, if interested: prefix compression bulk-loading duplicate handling 104