CSE 232A Graduate Database Systems

Transcription

1 CSE 232A Graduate Database Systems Fall 2018 Arun Kumar 1

2 What is this course about? Why take it? 2

3 1. IBM s Watson beats humans in Jeapordy!

4 How did Watson achieve that? 4

5 Watson devoured LOTS of data! 5

6 2. Structured data with Google search results 6

7 How does Google know that? 7

8 Google also devours LOTS of data! 8

9 3. Amazon s spot-on recommendations 9

10 How does Amazon know that? 10

11 You guessed it! LOTS and LOTS of data! Analysis 11

12 And innumerable traditional applications 12

13 13

14 Large-scale data management systems are the cornerstone of many digital applications, both modern and traditional 14

15 The Age of Big Data / Data Science 15

16 Data data everywhere, All the wallets did shrink! Data data everywhere, Nor any moment to think? 16

17 CSE 232A will get you thinking about the fundamentals of database systems 1. How are large structured datasets stored and organized? 2. How are queries handled? 3. How to make the system faster? 4. Deeper and more recent issues 17

18 And now for the (boring) logistics 18

19 Course Administrivia Lectures: MWF 1:00-1:50pm, CNTR 214 Attending ALL lectures is mandatory! Instructor: Arun Kumar; Office hours: Wed 2-3pm, 3218 EBU3b (CSE building) TAs: Chand Anand and Gurkanwal Singh Batra Piazza:

20 Prerequisites CSE 132A (or equivalent) is essential CSE 120 is also helpful For all other cases, the instructor with proper justification. A waiver can be considered. 20

21 Grading Midterm Exam 1: 20% Date: Friday, October 26, in-class Midterm Exam 1: 20% Date: Wednesday, November 21, in-class Final Exam: 60% (cumulative) Date: TBD, Room TBD (Optional/Extra Credit) 8 Paper Reviews: 8% 21

22 Course Outline 1. Storage How is large and file structured layout data stored? 2. Indexing, How are queries sorting, relational handled? operator implementation, and query processing 3. Query How to optimization make the system and parallelism faster? 4. Transactions; How to support data data integration integrity? 5. ML Recent for DB issues and DB and for trends ML 22

23 The primary focus will be the relational data model and Relational Database Management Systems (RDBMS) 23

24 Relational model in a nutshell Basically, Relation:Table :: Pilot:Driver (okay, a bit more) RatingID Rating Date UserID MovieID /27/ /20/ /02/ The model formalizes operations to manipulate relations 24

25 Relational model in a nutshell Invented by E. F. Codd in 1970s at IBM Research in CA 25

26 Relational DBMS in a nutshell A software system to implement the relational model, i.e., enable users to manage data stored as relations 26

27 Relational DBMS in a nutshell First RDBMSs: System R (IBM) and Ingres (Berkeley) in 1970s A rare photo of the original System R manual Mike Stonebraker won the Turing Award in 2015! 27

28 Relational DBMS in a nutshell RDBMS software is now a USD 40+ billions/year industry! Numerous open source RDBMSs also popular People still start companies about what are basically RDBMSs! 28

29 Course Textbook Prescribed: Database Management Systems 3 rd Edition Raghu Ramakrishnan and Johannes Gehrke Aka The Cow Book Which cow are you? Optional: Database Systems: The Complete Book by Garcia-Molina, Widom, and Ullman Big Data Integration by Dong and Srivastava 29

30 Course Schedule Week Topic 1 Introduction; Data Storage (Disks; Files; New Hardware) 2-3 Indexing (B+ Tree; Hash Indexes; Learned Indexes); Sorting 3-4 Relational Operator Implementations; Query Processing 4 Midterm Exam 1 on Friday, 10/26 5 Query Optimization; Materialized Views 6 Parallel RDBMSs and Dataflow Systems 7-8 Transaction Management; Concurrency Control 8 Midterm Exam 2 on Wednesday, 11/ Semi-structured Data; Data Integration and Cleaning 10 Advanced Topics: ML for DB and DB for ML 11 Final Exam on TBD 30

31 General Dos and Do NOTs Do: Raise your hand before asking questions during lectures Discuss papers in the reading list with peers and others Participate in class discussions; and on Piazza, if you like Use CSE232A: as subject prefix for all s to me/tas Do NOT: Plagiarize or share content for your paper reviews Harass, cut off, or be disrespectful to others in class Use as primary communication mechanism for doubts/questions instead of Office Hours Record or quote the instructor s anecdotes out of class! 31

32 Questions? Recap of Relational Model and SQL not covered in class; slides made available 32

33 Example Relational DB: Netflix! Ratings RatingID NumStars Timestamp UserID MovieID /27/ UserID Name Age JoinDate 79 Alice 23 01/10/13 Users MovieID Name ReleaseDate Director Movies 20 Inception 07/13/2010 Christopher Nolan 33

34 Surprise Review Questions! Q1. Which of the following is not a basic operation in relational algebra? A B C D [./ 34

35 Surprise Review Questions! R U M RID NumStars RateDate UID MID UID UName Age JoinDate MID MName Year Director Q2. What type of join is the following NOT an example of? R./ U A B C D Inner join Outer join Natural join Theta join 35

36 Surprise Review Questions! R U M RID NumStars RateDate UID MID UID UName Age JoinDate MID MName Year Director Q3. How many movies did Jim Cameron direct? A B C D Director= Jim Cameron ( COUNT ( ) (M)) ReleaseYear,COUNT ( )( Director= Jim Cameron (M)) COUNT ( )( Director= Jim Cameron (M./ R)) COUNT ( )( Director= Jim Cameron (M)) 36

37 Surprise Review Questions! R U M RID NumStars RateDate UID MID UID UName Age JoinDate MID MName Year Director Q4. Which of the following attributes is a primary key? A B C D U.UID R.UID R.MID M.Year 37

38 Surprise Review Questions! Q5. Which of the following SQL features do not have a counterpart operation in extended relational algebra? A B C D SELECT DISTINCT WHERE GROUP BY LIMIT 38

39 Surprise Review Questions! R RID NumStars RateDate UID MID /27/ /20/ /02/ /05/ /09/ Q6. What is the cardinality of this query s output? SELECT COUNT(DISTINCT MID) FROM R A 1 C 3 B 2 D 4 39

40 Surprise Review Questions! M MID MName Year Director Q7. Get the year and director of all movies from the last decade that have the term Avengers in their title. A B C D SELECT Year, Director from M WHERE Year >= 2008 AND MName LIKE %Avengers% SELECT Year, Director from M WHERE Year >= 2008 AND MName LIKE %Avengers SELECT Year, Director from M WHERE Year >= 2008 AND MName LIKE Avengers% SELECT Year, Director from M WHERE Year >= 2008 AND MName = Avengers 40

41 Surprise Review Questions! Ratings (R) RatingID Stars UserID MovieID Movies (M) MovieID Name Year Director Q8. Write an SQL query to get the number of 5-star ratings for each movie directed by Christopher Nolan SELECT R.MovieID, COUNT(R.Stars) AS NumHighRatings FROM Ratings R, Movies M WHERE M.Director = Christopher Nolan AND R.MovieID = M.MovieID AND R.Stars = 5 GROUP BY R.MovieID 41

42 Recap: Relational Model (Not covered in class) 42

43 Relational Model: Basic Terms What What are is a is Attributes? Domains? Relation? Arity? Ratings RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ Integers Real The mathematical Number A These glorified domains of things attributes table! for the attributes 43

44 Relational Model: Basic Terms Ratings What are Tuples? What is Cardinality? RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ Number These of things tuples 44

45 Referring to tuples : Two notations 1. Without using attribute names (positional/sequence) 2. Using attribute names (named/set) Ratings (R) RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ t[1] = 3.5 t.numstars =

46 Relational Model: Basic Terms What is Schema? Ratings RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ The relation name, and the name and logical descriptions of the attributes (including domains) Aka metadata 46

47 Relational Model: Basic Terms What is an Instance? Ratings RatingID NumStars Timestamp UserID MovieID RatingID 1 NumStars /27/15 Timestamp 79 UserID MovieID /20/15 06/27/ /02/15 07/10/ /08/ Instance 1 Instance 2 A given relation populated with a set of tuples (loose analogy: schema:instance::type:value in PL) 47

48 Relational Model: Basic Terms Ratings What is a Relational Database? RatingID NumStars Timestamp UserID MovieID /27/ UserID Name Age JoinDate 79 Alice 23 01/10/13 Users MovieID Name ReleaseDate Director Movies 20 Inception 07/13/2010 Christopher Nolan A collection of relations; similarly, schema vs. instance 48

49 Write Operations on a Relation Insert Add tuples to a relation Delete Remove tuples from a relation (typically based on predicate matches, e.g., NumStars <= 4.5 Modify Logically, deletes + inserts, but typically implemented as in-place updates to a relation instance 49

50 Read Operations on a Relation Select Select all tuples from Ratings with UserID == 19 Project Select only Director attribute from Movies Aggregate Select Average of all NumStars in Ratings And a few more formal algebraic operations 50

51 Recap: Relational Algebra 51

52 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 52

53 Select Ratings (R) RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ Example: Get all ratings with 4.0 or more stars Select Operator Selection condition/predicate 53

54 Select R RatingID NumStars Timestamp UserID MovieID R /27/ /20/ /02/ /05/ Schema preserved Subset of tuples (satisfying selection condition) RatingID NumStars Timestamp UserID MovieID /20/ /05/

55 Complex Selection Conditions R RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ R RatingID NumStars Timestamp UserID MovieID /27/

56 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 56

57 Project Ratings (R) RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ Example: Get all MovieIDs Projection list 57

58 Project R RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ R Schema reduced (to projection list) MovieID Tuple values deduplicated 20 (slightly different semantics in SQL) 16 58

59 Composition of Relational Ops R RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ Example: Get UserID and NumStars of ratings with less than 3 stars R UserID NumStars RelOp(Relation) gives a Relation 59

60 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 60

61 Rename Ratings (R) RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ Example: Rename Timestamp to RateDate RatingID,NumStars,RateDate,UserID,MovieID (R) C(2 >RateDate) (R) 61

62 Rename R R RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ C(2 >RateDate) (R) Instance preserved Schema modified RatingID NumStars RateDate UserID MovieID /27/ /20/ /02/ /05/

63 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 63

64 Cross Product UserID Name Age JoinDate 79 Alice 23 01/10/13 80 Bob 41 05/10/13 Users (U) Movies (M) MovieID Name ReleaseYear Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron Cartesian product (construct all pairs of tuples across tables) Schema of output concatenates the input schemas Be careful with attribute name conflicts! Use Rename op 64

65 Cross Product UserID Name Age JoinDate 79 Alice 23 01/10/13 80 Bob 41 05/10/13 Users (U) MovieID Name ReleaseYear Director Movies (M) 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron C(1 >U.Name (U) C(1 >M.Name (M) UserID U.Name Age JoinDate MovieID M.Name Release Year Director 79 Alice 23 01/10/13 20 Inception 2010 Christopher Nolan 79 Alice 23 01/10/13 16 Avatar 2009 Jim Cameron 80 Bob 41 05/10/13 20 Inception 2010 Christopher Nolan 80 Bob 41 05/10/13 16 Avatar 2009 Jim Cameron 65

66 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 66

67 Union R1 RatingID NumStars Timestamp UserID MovieID R2 Union of sets of tuples (instances) /27/ /20/ /02/ Inputs must have identical schema: Union-compatible RatingID NumStars Timestamp UserID MovieID /02/ /05/ /09/

68 Union RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ /09/

69 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 69

70 Set Difference R1 RatingID NumStars Timestamp UserID MovieID R2 Set difference of sets of tuples (instances) /27/ /20/ /02/ Inputs must be Union-compatible RatingID NumStars Timestamp UserID MovieID /02/ /05/ /09/

71 Set Difference R1 R2 RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ RatingID NumStars Timestamp UserID MovieID /02/ /05/ /09/ R = R1 R2 RatingID NumStars Timestamp UserID MovieID /27/ /20/

72 Basic Relational Operations Select Project Rename Cross Product (aka Cartesian Product) Set Operations: Union Set Difference 72

73 Derived and Other Relational Ops Set Operation: Intersection Join Group By Aggregate 73

74 Intersection R1 RatingID NumStars Timestamp UserID MovieID R2 Set intersection of sets of tuples (instances) /27/ /20/ /02/ Inputs must be Union-compatible RatingID NumStars Timestamp UserID MovieID /02/ /05/ /09/

75 Intersection R1 R2 RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ RatingID NumStars Timestamp UserID MovieID /02/ /05/ /09/ RatingID NumStars Timestamp UserID MovieID /02/

76 Derived and Other Relational Ops Set Operation: Intersection Join Group By Aggregate 76

77 Join R./ JoinCondition M Equivalent Select on Cross Product, but bypasses full X JoinCondition(R M) Perhaps the most intensively studied Rel Op! Several types of Joins: Natural Join and Equi-Join Condition Join (aka Theta Join) Semi-Join, Inner Join, Outer Join, Anti-Join, etc. 77

78 Natural Join Ratings (R) RatingID NumStars Timestamp UserID MovieID /27/ /20/ /02/ /05/ Movies (M) MovieID Name ReleaseYear Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron R./ M 78

79 Natural Join Rating ID R./ M NumS tars Timesta mp UserI D MovieI D Name Release Year Director /27/ Inception 2010 Christopher Nolan /20/ Inception 2010 Christopher Nolan /02/ Avatar 2009 Jim Cameron /05/ Avatar 2009 Jim Cameron Join attributes : attributes that determine matching tuples Have same name in both inputs! MovieID in R and M Implicit equality condition on join attributes If > 1 pair, implicit logical and of all equality terms Output schema concatenates input schemas But join attributes appear only once in output (Project) 79

80 Equi-Join Generalization of the Natural Join R./ EqualityCondition M Attribute names of join attributes need not be the same EqualityCondition is a general boolean expression (logical and, and/or or ) of terms with equality predicates only Join attributes from both R and M in output (no Project) Perhaps the most important and common type of Join! Lots of R&D on efficient implementations! 80

81 Equi-Join: Example T(J, K, P, Q) =R 1./ K=P R 2 R 1 (J,K) J K 10 x 20 y 30 x R 2 (P,Q) P Q x 4 y 9 x 8 T(J,K,P,Q) J K P Q 10 x x 4 20 y y 9 30 x x 8 10 x x 8 30 x x 4 81

82 (Primary) Key-Foreign Key Join A special kind of equi-join One of the join attributes is the (Primary) Key of an input relation; the other is a Foreign Key in the other relation Ratings RatingID NumStars Timestamp UID MovieID UserID Name Age JoinDate Users Ratings./ UID=UserID Users Also a common and important (sub) type of join with even more specialized efficient implementations! 82

83 Condition Join (aka Theta Join) Generalization of the Equi-Join R./ JoinCondition M 83

84 Condition Join: Example T(J, K, P, Q) =R 1./ J/2>Q R 2 R 1 (J,K) R 2 (P,Q) T(J,K,P,Q) J K P Q 10 x x 4 20 y y 9 30 x x 12 Perhaps the most difficult type of Join to implement efficiently! J K P Q 10 x x 4 20 y x 4 20 y y 9 30 x x 4 30 x y 9 30 x x 12 84

85 Join Expressions Can compose many joins into a single complex expression Ratings RatingID NumStars Timestamp UserID MovieID UserID UName Age JoinDate Users MovieID Name ReleaseYear Director Movies AllStuff Q. What do we get as the output? UserI D UNa me Ag e JoinDat e RatingI D NumSta rs Timesta mp MovieI D Name ReleaseY Ear Directo r 85

86 Taxonomy of Joins All kinds of joins Outer joins Semi joins Anti joins Inner joins Theta joins Equi joins Key-Foreign Key Joins Snowflake joins Star joins Natural joins 86

87 Derived and Other Relational Ops Set Operation: Intersection Join Group By Aggregate 87

88 Group By Aggregate NOT a part of relational algebra, but Extended RA! Useful for analytics queries that aggregate numerical data Ratings RatingID NumStars Timestamp UserID MovieID What is the average rating for each movie? How many movies has each user rated? Standard 5 numerical aggregations supported in SQL: Count, Sum, Average, Maximum, and Minimum Extra: Median, Mode, Variance, Standard Deviation, etc. 88

89 Group By Aggregate X,Agg(Y )(R) Grouping Attributes A numerical attribute in R (Subset of R s attributes) Aggregate Function (SUM, COUNT, AVG, MAX, MIN) Output schema will have X and an extra numerical attribute (result of the aggregate function) Can list multiple aggregate functions in the same operation 89

90 Group By Aggregate R RatingID NumStars Timestamp UserID MovieID R /27/ /20/ /02/ /05/ What is the average rating for each movie? MovieID,AV G(NumStars)(R) MovieID AVG(NumStars)

91 Group By Aggregate The set of Grouping Attributes can be empty too! UserID Name Age JoinDate 79 Alice 23 01/10/13 80 Bob 41 05/10/ Carol 19 08/09/ Dan 20 03/01/15 Users (U) U What is the average age of the users? AV G(Age)(U) AVG(Age)

92 Derived and Other Relational Ops Set Operation: Intersection Join Group By Aggregate 92

93 Recap: SQL (Not covered in class) 93

94 Basic Form of an SQL Query Optional List of attributes to project SELECT [DISTINCT] target-list FROM relation-list [WHERE condition] Selection/join condition (optional) List of relations (possibly with aliases ) 94

95 What does it mean logically? SELECT [DISTINCT] target-list FROM relation-list [WHERE condition] 1. Cross-product of relations in relation-list 2. If condition given, apply it to filter out tuples 3. Remove attributes not present in target-list 4. If DISTINCT given, deduplicate tuples in result The above is only a logical interpretation. It is NOT a plan an RDBMS would use in general to run an SQL query! 95

96 Example SQL Query Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen Example: Get the names of movies released in 2013 SELECT M.Name FROM Movies M WHERE M.Year = 2013 Name ( Y ear=2013 (M)) 96

97 Example SQL Query Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT M.Name FROM Movies M WHERE M.Year = 2013 Name Gravity Blue Jasmine 97

98 Example SQL Query Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT * FROM Movies M WHERE M.Year = 2013 MovieID Name Year Director 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen 98

99 Example SQL Query Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen Example: Get the names of movies from years other than 2013 SELECT M.Name FROM Movies M WHERE M.Year <> 2013 Name ( Y ear6=2013 (M)) 99

100 Example SQL Query Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen Example: For which years do we have movie data? SELECT M.Year FROM Movies M Y ear (M) 100

101 Example SQL Query Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT M.Year FROM Movies M Year SQL allows repetitions of tuples in a relation! Not the same semantics as RA s Project Called bag semantics vs. RA s set semantics 101

102 DISTINCT in SQL Movies (M) MovieID Name Year Director Year Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT DISTINCT M.Year FROM Movies M DISTINCT needed to achieve set semantics of RA s Project in SQL 102

103 Aliases in SQL Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT M.Name FROM Movies M WHERE M.Year = 2013 SELECT Name FROM Movies WHERE Year = 2013 Why bother with the alias? Not needed here! 103

104 Aliases in SQL Useful for Joins! Movies (M) MovieID Name Year DirectorID Directors (D) DID Name Age Example: Get names of movies directed by Jim Cameron SELECT M.Name FROM Movies M, Directors D WHERE D.Name = Jim Cameron AND M.DirectorID = D.DID Aliases help disambiguate attributes with the same name from multiple relations (or even a self-join!) 104

105 More SQL Examples Movies (M) MovieID Name Year DirectorID Directors (D) DID Name Age Example: Get names of movies released in 2013 by Woody Allen or some other director 50 years or older SELECT M.Name FROM Movies M, Directors D WHERE (D.Name = Woody Allen OR D.Age >= 50) AND M.Year = 2013 AND M.DirectorID = D.DID 105

106 LIKE in SQL Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen Example: Get the directors of movies that start with Blue SELECT DISTINCT M.Director FROM Movies M WHERE M.Name LIKE Blue% % matches any number of characters; _ matches one 106

107 ORDER BY in SQL Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT M.Name FROM Movies M WHERE M.Year = 2013 ORDER BY M.Name Name Blue Gravity Jasmine Blue Gravity Jasmine Useful for data readability Ordering defined by domain semantics Can specify DESC; multiple attributes 107

108 LIMIT in SQL MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 74 Blue Jasmine 2013 Woody Allen SELECT M.Name FROM Movies M WHERE M.Year >= 2010 ORDER BY M.Year LIMIT 2 Year Inception Gravity Blue Jasmine Also useful for data readability Prevents flooding of screen with data Be wary of using it without ORDER BY! 108

109 Aggregate Functions in SQL Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 91 Interstellar 2014 Christopher Nolan How many movies came out after 2010? SELECT COUNT(*) FROM Movies M WHERE M.Year >

110 Aggregate Functions in SQL Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 91 Interstellar 2014 Christopher Nolan COUNT(*) How many movies came out after 2010? SELECT COUNT(*) FROM Movies M WHERE M.Year >

111 5 Native Aggregate Functions in SQL COUNT ([DISTINCT] attribute) AVG ([DISTINCT] attribute) SUM ([DISTINCT] attribute) MAX (attribute) MIN (attribute) 111

112 Aggregate Functions in SQL Movies (M) MovieID Name Year Director 20 Inception 2010 Christopher Nolan 16 Avatar 2009 Jim Cameron 53 Gravity 2013 Alfonso Cuaron 91 Interstellar 2014 Christopher Nolan SELECT FROM How many directors do we have? COUNT(DISTINCT M.Director) Movies M 112

113 Aggregate Functions in SQL Ratings (R) RatingID Stars UserID MovieID Which MovieID(s) have the highest rating? SELECT FROM R.MovieID, MAX(R.Stars) Ratings R Other attributes NOT allowed in the target-list as such! 113

114 Aggregate Functions in SQL SELECT FROM WHERE Ratings (R) RatingID Stars UserID MovieID Which MovieID(s) have the highest rating? DISTINCT R.MovieID Ratings R R.Stars = (SELECT MAX(R2.Stars) FROM Ratings R2) 114

115 Group By Aggregate in SQL X,Agg(Y )(R) SELECT [DISTINCT] target-list FROM relation-list [WHERE condition] GROUP BY grouping-list X HAVING group-condition target-list must be in this form: X, Agg(Y) Subset of X Condition on each group in aggregate 115

116 Group By Aggregate in SQL Ratings (R) RatingID Stars UserID MovieID What is the average rating for each movie? MovieID,AVG (Stars)(R) SELECT R.MovieID, AVG(R.Stars) AS AvgRating FROM Ratings R GROUP BY R.MovieID 116

117 Group By Aggregate in SQL Ratings (R) RatingID Stars UserID MovieID SELECT R.MovieID, AVG(R.Stars) AS AvgRating FROM Ratings R GROUP BY R.MovieID MovieID AvgRating One tuple in output per unique value of R.MovieID (aka group ) 117

118 Advanced SQL Operations (Optional) 118

119 UNION in SQL Ratings (R) RID Stars UID MID Movies (M) MID Name Year Director Get the IDs of users that have rated a movie directed by Ang Lee or a movie that released in 2013 SELECT R.UID FROM Ratings R, Movies M WHERE R.MID = M.MID AND M.Director = Ang Lee UNION SELECT R.UID Union-compatible! FROM Ratings R, Movies M WHERE R.MID = M.MID AND M.Year =

120 Semantics of UNION in SQL RID Stars UID MID MID Name Year Director 20 Inception 2010 Christopher Nolan 42 Life of Pi 2012 Ang Lee 53 Gravity 2013 Alfonso Cuaron R M UID SELECT R.UID 79 FROM Ratings R, Movies M 123 WHERE R.MID = M.MID AND M.Director = Ang Lee UNION UID SELECT R.UID FROM Ratings R, Movies M 79 WHERE R.MID = M.MID AND M.Year =

121 Semantics of UNION in SQL UID 79 UID 79 UNION UID UNION implicitly deduplicates tuples (unlike SELECT)! Q. How to retain duplicates with UNION? UID 79 UID 79 UNION ALL UID

122 INTERSECT in SQL RID Stars UID MID MID Name Year Director 20 Inception 2010 Christopher Nolan 42 Life of Pi 2012 Ang Lee 53 Gravity 2013 Alfonso Cuaron R M UID SELECT R.UID INTERSECT FROM Ratings R, Movies M 123 WHERE R.MID = M.MID AND M.Director = Ang Lee INTERSECT UID SELECT R.UID FROM Ratings R, Movies M 79 WHERE R.MID = M.MID AND M.Year = UID

123 EXCEPT (Set Difference) in SQL RID Stars UID MID MID Name Year Director 20 Inception 2010 Christopher Nolan 42 Life of Pi 2012 Ang Lee 53 Gravity 2013 Alfonso Cuaron R M UID SELECT R.UID 79 EXCEPT FROM Ratings R, Movies M WHERE R.MID = M.MID AND M.Director = Ang Lee EXCEPT UID SELECT R.UID FROM Ratings R, Movies M 79 WHERE R.MID = M.MID AND M.Year = UID

124 The Contentious Bag Semantics! UID UID UNION ALL UID Add the number of repetitions

125 The Contentious Bag Semantics! UID UID EXCEPT ALL UID Subtract the number of repetitions

126 The Contentious Bag Semantics! UID UID INTERSECT ALL UID Minimum of the number of repetitions