CS 461: Database Systems. Final Review. Julia Stoyanovich

CS 461: Database Systems Final Review (stoyanovich@drexel.edu)

Final exam logistics When: June 6, in class The same format as the midterm: open book, open notes 2 hours in length The exam is cumulative, it will include material from the first half of the term. However, we will likely have more of a focus on the material from the second half of the term. To study: go over lecture notes, midterm, homework / solutions 2

Normalization: important to know Closures, keys compute the closure of a set of attributes compute candidate keys of a relation check whether an FD follows from a set of FDs compute a minimal basis of a set of FDs Normal forms check whether a relation is in BCNF check whether a relation is in 3NF I will not ask you to produce a BCNF or a 3NF decomposition on the final exam 3

Closure of a set of attributes Suppose A = {A1,, An} is a set of attributes and S is a set of FDs. The closure of A under the FDs in S is the set of attributes B s.t. every relation that satisfies all the FDs in S also satisfies A B We denote the closure of {A 1, A 2,, A n } by {A 1, A 2,, A n } + Note that {A 1, A 2,, A n } {A 1, A 2,, A n } + 4

Computing the closure of a set of attributes Algorithm AttributeClosure Input: a set of attributes {A 1, A 2,, A n } and a set of FDs S Output: the closure {A 1, A 2,, A n } + 1. Split the FDs of S using the splitting rule, so that each FD has one attribute on the right 2.Initialize {A 1, A 2,, A n } + {A 1, A 2,, A n } 3. Repeatedly search for some FD B 1, B 2,, B m C such that {B 1, B 2,, B m } {A 1, A 2,, A n } + C {A 1, A 2,, A n } + 4.Stop when no more attributes can be added to {A 1, A 2,, A n } + 5

Closures and keys Q: How can we tell if a set of attributes A 1 A 2 A n is a candidate key or a superkey of a relation R? A: If {A 1, A 2,..., A n } + = all the attributes in R Q: How can we compute the candidate keys for R? A: Find all sets of attributes that functionally determine all other attributes and make sure these sets are minimal. 6

inimal basis of a set of FDs For a given relation R, there may exist several sets of FDs that are equivalent: - they give rise to the same closures of all subsets of R s attributes - the same sets of FDs follow from them - all such equivalent sets of FDs are called bases for S in R A minimal basis B is a set of FDs that satisfies 3 conditions 1. All FDs in B have 1 attribute on the right 2. If any FD is removed from B, the result is no longer a basis 3. If for any FD in B we remove 1 attribute on the left, the result is no longer a basis 7

Example R(ABCD) C B BC A A C BD A Find all candidate keys Check whether the following are minimal bases of the set of FDs. {AC D, D B} {D A, D B, D C} 8

Example Compute a projection of the set of FDs when R (ABCD) is projected onto ACD. R(ABCD) A B ; B C ; C D π ACD (R) 9

Example Compute a projection of the set of FDs when R (ABCD) is projected onto ACD. R(ABCD) A B ; B C ; C D π ACD (R) Compute closures of all subsets of attributes in the projected relation. {A} + = {A, B,C, D} {C} + = {C, D} = {C, D} + {D} + = {D} we stop here, since any set that includes A will have the same closure as A alone Compute FDs from these closures that involve only A,C,D on either side, remove redundant FDs (keeping only the minimal basis) T = {A C, C D} done! 10

Boyce-Codd Normal Form (BCNF) Let R be a relation schema, S be the set of FDs given to hold over R. R is in BCNF if, for every FD A 1 A 2 A n B 1 B 2 B m one of the following statements is true: 1. The FD is trivial: {B 1, B 2,, B m } {A 1, A 2,, A n } A 1 A 2 A n 2. is a candidate key of R A 1 A 2 A n 3. is a superkey of R In a BCNF relation, the only set of attributes that determines values for other attributes is a superkey! 11

Third Normal Form (3NF) Let R be a relation schema, S be the set of FDs given to hold over R. R is in 3NF if, for every FD A 1 A 2 A n B 1 B 2 B m one of the following statements is true: 1. The FD is trivial: {B 1, B 2,, B m } {A 1, A 2,, A n } same for BCNF 2. A 1 A 2 A n is a candidate key of R as{ 3. is a superkey of R A 1 A 2 A n B i 4. Each is part of some candidate key of R In contrast to BCNF, some redundancy is possible with 3NF. This normal form is a compromise, needed when no dependency-preserving decomposition into BCNF exists. 12

Example Find all candidate keys of the given set of FDs. Check whether R is in BCNF, 3NF R( ABCD) ABD C ; A B ; AB C ; B A {A} + = {ABC} {AD} + = {ABCD} {BD} + = {ABCD} {B} + = {ABC} 13

Example R(ABCD) AB C ; D B ; AC D (a) list candidate keys of R AB, AC, AD (b) does this FD follow from the set of FDs above? AD B Yes. To check this, we must check whether B is in the closure of {AD}. We know that this is the case because, as we saw in (a), {AD} is a candidate key of R, and so all attributes are in the closure of {AD}. (c) is R in BCNF? is it in 3NF? In 3NF but not in BCNF 14

2-way merge-sort example with N=7 pages 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 2,3 4,4 6,7 8,9 4,7 8,9 1,3 5,6 2 1,2 3,5 6 Input file PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 1,2 2,3 3,4 4,5 6,6 7,8 9 8-page runs 15

2-way merge-sort suppose the input occupies N = 2 k disk pages What is the cost of this algorithm? In each pass, we read each page process it, and write it out: 2 disk I/Os per page, per pass There are k = log 2N + 1 passes The over-all cost is 2N (log 2N + 1) I/Os INPUT 1 INPUT 2 OUTPUT Disk ain memory buffers Disk 16

2-way external merge sort A file with 10,000 records, each record is 1KB. Size of a page/block is 64KB (i.e., 64 records / block). What is the number of passes, the cost of 2-way external merge-sort? In this dataset, there are ceil(10,000 / 64) = 157 pages that must be sorted. In two-way external merge-sort, we use 1 memory block in pass 0 (each 64-record block is sorted), and 3 memory blocks in subsequent passes (pairs of adjacent sorted runs are merged). To sort 157 pages, we will need 1 + ceil(log2157) = 9 passes. Each page is read and written once on each pass (2 I/Os per page per pass). Thus, the total cost of two-way external merge-sort on this dataset is 2 * 157 * 9 = 2,826 I/Os. 17

Generalization: external merge-sort B: block size : main memory size N: input size (blocks) R: size of 1 record N records, divided into NR / sorted runs of / R records each final sorted result 18

External merge-sort example A file with 10,000 records, each record is 1KB. Size of a page/ block is 64KB (i.e., 64 records / block). With memory size of 320KB, how many passes for generalized external merge sort? What is the cost? emory (its 320 / 64 = 5 pages. All are used for sorting in pass 0. All but 1 are used for sorting in subsequent runs, the remaining page is used for output. In phase 0 of generalized external merge-sort, we read in and sort 320KB (5 blocks worth) at a time, creating ceil(157/5) = 32 sorted runs of 5 blocks each. Then in subsequent passes we merge 5-1=4 neighboring runs. We need ceil(log432)=3 passes to complete sorting. That s a total of 3 passes, with 2 I/Os per page per pass, for a total of 2 * 157 * 4 = 1256 I/Os. 19

Basic file organization Heap files: good for full file scans or frequent updates unordered files insert at the end of file assumes equality selection on key, exactly one match (why?) Sorted files: good for range queries on sort field(s) need external sort to keep sorted compacted after deletion assumes selection on sort field(s) Hashed files: good for selection on equality collection of buckets with primary & overflow pages hashing function h(r) = bucket for record r each bucket is a heap file 20

Cost of operations p(t) - number of data pages in table T r(t) - number of records in table T D - time to read or write a disk page Heap File Sorted File Hashed File Scan all recs p(t) D p(t) D 1.25 p(t) D * Equality Search p(t) D / 2 D log 2 p(t) D Range Search p(t) D D log 2 p(t) + (# pages with matches) 1.25 p(t) D Insert 2D Search + p(t) D 2D Delete Search + D Search + p(t) D 2D * assuming no overflow bucket, 80% page occupancy 21

Access paths An access path is a method of retrieving tuples: file scan, or index that matches a selection in the query An index matches a conjunction of terms if it can be used to retrieve all data values that match this conjunction of terms. A tree index matches a conjunction of terms that involve only attributes in a prefix of the search key. e.g., tree index <a,b,c> matches the selection a=5 AND b=3; it also matches a=5 AND b>4; it does not match b=3. A hash index matches a conjunction of terms that has a term attribute=value for every attribute in the search key of the index. e.g., hash index on <a,b,c> matches a=b AND b=3 AND c=5; it does not match b=3; or a=5 and b=5; or a>5 AND b=3 and c=5 22

Clustered vs. unclustered index Data entries (Index File) (Data file) Data entries CLUSTERED Data Records Data Records UNCLUSTERED 23

Using an index for selection Sailors (sid:int, sname: string, rating:int, age:real) Reserves (sid:int, bid:int, day:date, rname:string) SELECT * FRO Reserves R WHERE R.rname < C% Reserves (R): each tuple us 40 bytes long, 100 tuples per page, 1000 pages Sailors (S): each tuple is 50 bytes long, 80 tuples per page, 500 pages Cost of finding qualifying data entries (typically small) plus cost of retrieving records (could be large) Example: assuming uniform distribution of names, about 10% of tuples qualify (100 pages, 10,000 tuples). with a clustered index, cost is little more than 100 I/Os with an unclustered index, cost is up to 10,000 I/Os! 24

Access paths: example Employees (eid, name, salary, age, did); Departments (did, budget, floor, manager_eid); Salaries from $10K to $100K ages from 20 to 80; 5 employees per department; 10 floors; budgets from $10K to $1. Uniform, uncorrelated values. Q1. Print name, age, salary for all employees A1: clustered hash index on (name, age, salary) of Employees A2: unclustered hash index on (name, age, salary) of Employees A3: clustered B+-tree index on (name, age, salary) of Employees A4: unclustered hash index on (eid, did) of Employees A5: no index 25

Access paths: example Employees (eid, name, salary, age, did); Departments (did, budget, floor, manager_eid); Salaries from $10K to $100K ages from 20 to 80; 5 employees per department; 10 floors; budgets from $10K to $1. Uniform, uncorrelated values. Q2. Find dids of departments on the 10th floor with budget < $15K A1: clustered hash index on (floor) of Departments A2: clustered hash index on (floor, budget) of Departments A3: clustered B+-tree index on (floor, budget) of Departments A4: clustered B+-tree index on (budget) of Departments A5: no index 26

Access paths: example Sailors (sid, name, rating, age); Sids from 1 to 100K, ratings from 1 to 10, ages from 20 to 80. Uniform, uncorrelated values. Q1. Print name, age, rating of all sailors A1: sequential scan of sorted file, sorted on (id) A2: clustered hash index on (rating) A3: unclustered hash index on (id) A4: unclustered hash index on (age, rating) A5: unclustered hash index on (name, age) A6: clustered B+-tree index on (name, age) A7: unclustered B+-tree index on (age, rating) 27

Access paths: example Sailors (sid, name, rating, age); Sids from 1 to 100K, ratings from 1 to 10, ages from 20 to 80. Uniform, uncorrelated values. Q2. Print name, age, rating of the sailor with sid 123 A1: sequential scan of sorted file, sorted on (id) A2: clustered hash index on (rating) A3: unclustered hash index on (id) A4: unclustered hash index on (age, rating) A5: unclustered hash index on (name, age) A6: clustered B+-tree index on (name, age) A7: unclustered B+-tree index on (age, rating) 28

Access paths: example Sailors (sid, name, rating, age); Sids from 1 to 100K, ratings from 1 to 10, ages from 20 to 80. Uniform, uncorrelated values. Q3. Count sailors with rating = 5 and age = 40 A1: sequential scan of sorted file, sorted on (id) A2: clustered hash index on (rating) A3: unclustered hash index on (id) A4: unclustered hash index on (age, rating) A5: unclustered hash index on (name, age) A6: clustered B+-tree index on (name, age) A7: unclustered B+-tree index on (age, rating) 29

Access paths: another example Employees (ssn, name, salary, age, did); 100,000 employees, 10 employee records per disk page. Stored on disk in a sorted file (alternative 1), with did as the sort key. Salaries from 0 to $100K; ages from 20 to 80; 50 employees per department. Uniform, uncorrelated values. Q1. Compute the number of employees whose salary is $35K and who work in department 177. Q2. List name, age, salary of employee with eid=12357. Q3. Compute the number of employees who are between 30 and 35 years old. For each query: (1) List indexes that would match the query. (2) What index would you build? (3) What is the cost of using that index to answer this query? 30

Relational algebra and SQL Flights (flno: int, origin: string, destintation: string, dist: int, departs:date, arrives: date) Aircraft (aid: int, aname: string, range: int) Employees (eid: int, ename: string, salary: int) Certified (eid: int, aid: int) (a) List eids of pilots certified to fly Boeing. π eid ((σ name='boeing' Aircraft) aid Certified) (b) List names of pilots certified to fly Boeing. π ename (Employees eid ((σ name='boeing' Aircraft) aid Certified)) 32

Relational algebra and SQL Flights (flno: int, origin: string, destintation: string, dist: int, departs:date, arrives: date) Aircraft (aid: int, aname: string, range: int) Employees (eid: int, ename: string, salary: int) Certified (eid: int, aid: int) (d) Find names of pilots who can operate planes with a range greater than 3,000 miles but are not certified on any Boeing aircraft. π ename (Employees eid (π eid ((σ range>3000 Aircraft) aid Certified) π eid ((σ name='boeing' Aircraft) aid Certified))) not the same as: π ename (Employees eid (π eid ((σ range>3000 name 'Boeing' Aircraft) aid Certified))) 36

SQL Flights (flno: int, origin: string, destintation: string, dist: int, departs:date, arrives: date) Aircraft (aid: int, aname: string, range: int) Employees (eid: int, ename: string, salary: int) Certified (eid: int, aid: int) (e) List eids of pilots certified to fly exactly 3 aircraft. select eid from Certified group by eid having count(*) = 3 38

SQL Flights (flno: int, origin: string, destintation: string, dist: int, departs:date, arrives: date) Aircraft (aid: int, aname: string, range: int) Employees (eid: int, ename: string, salary: int) Certified (eid: int, aid: int) (f) List aids of aircraft that can be used on flight AF007, along with an average salary of pilots who are certified to operate these aircraft. select avg(e.salary), A.aid from Flights F, Aircraft A, Certified C, Employees E where F.flno = 'AF007' and F.dist <= A.range and A.aid = C.aid and C.eid = E.eid group by A.aid 40

When writing queries For relational algebra, do worry about efficiency: avoid Cartesian product whenever possible, push selections For SQL, do worry about efficiency and readability: avoid nested queries if your query can be expressed with a join use group by / having as appropriate, not a subquery and a where clause in the outer use standard notation, like we covered in class, e.g., no need to write inner join, and do write your queries by hand For both SQL and relational algebra: do not join with relations unnecessarily. You should have exactly the right number of tables in the from clause of a SQL query, no more no less 41

ER modeling Draw an ER diagram that encodes the following business rules. Clearly mark all key and participation constraints. Chefs work at restaurants. A chef is uniquely identified by an SSN, and is also described by a name and a cuisine in which she specialized. A restaurant is uniquely identified by a combination of name and city. Each chef works in at least one restaurant, and each restaurant must have at least one chef working at it. Some chefs own restaurants, and if a chef owns a restaurant - she is its sole owner. 42

ER to relational 44

ER to relational create table Teams ( name varchar(32) primary key); create table Athletes ( name varchar(32), dob date, team_name varchar(32), primary key (name, dob), foreign key (team_name) references Teams (name)); create table Sports ( name varchar(32) primary key, olympic char(3) ); create table Athletes_play_Sports ( athlete_name, sport_name, primary key (athlete_name, sport_name), foreign key (athlete_name) references Athletes(name), foreign key (sport_name) references Sports(name) ); 45

ER to relational name) party) name) PRESIDENTS) running_mate) VICE_PRESIDENTS) 46

ER to relational name) party) name) PRESIDENTS) running_mate) VICE_PRESIDENTS) create table Presidents_VPs ( president_name varchar(32) primary key, vp_name varchar(32) unique not null, party varchar(32) ); 47

Binary vs. ternary relationship sets name) party) name) PRESIDENT) running_mate) VICE_PRESIDENT) name) name) PRESIDENT) running_mate) VICE_PRESIDENT) name) Party) 48

And now with constraints name) name) PRESIDENT) running_mate) VICE_PRESIDENT) name) Party) 49

And now with constraints name) name) PRESIDENT) running_mate) VICE_PRESIDENT) name) Party) create table Parties ( name varchar(32) primary key ); create table Presidents_VPs ( president_name varchar(32) primary key, vp_name varchar(32) unique not null, party varchar(32) not null, foreign key party references Parties(name) ); 50

Candidate keys, superkeys Consider a relation schema and business rules below. Dancers (name: string, dob: date, stage_name: string, company: string) No two dancers have the same combination of name and date of birth (dob). No two dancers have the same combination of stage name and company. A name, a dob and a stage name have to be specified for each dancer, but not all dancers belong to a company. What are the candidate keys? Which of these would be appropriate for a primary key? Which are not appropriate for a primary key? What are the superkeys? Write a valid create table statement. 51