SQL & intro to db architectures

Transcription

1 class 3 SQL & intro to db architectures prof. Stratos Idreos

2 welcome brave cs165 students! Stratos Idreos 2 /55

3 guest lecture Laura Haas Data Systems Researcher Director of IBM Research s Accelerated Discovery Lab & Harvard alumna October 5 a 1 hour discussion with students will follow after class 1 hour wics meeting 3:30-4:30 The Power Behind the Throne: Information Integration in the Age of Data-Driven Discovery Stratos Idreos 3 /55

4 guest lecture Peter Haas Data Systems Researcher IBM Almaden Research Center & Harvard alumnus, class 78 October 5, 3:30pm Balancing Recency and Continuity in Massive Scale Dynamic Interaction Graphs Stratos Idreos 4 /55

5 guest lecture Nga Tran Data Systems Researcher Head of Optimizer Group at HP Vertica Nov 9 Stratos Idreos 5 /55

6 when should I start my project? next week in the meantime: 1) play with code base & tools 2) linked list, binary tree 3) MonetDB, PostgreSQL Stratos Idreos 6 /55

7 which tests should I pass? we care about DSL tests by Wed it should be obvious why leaderboard will test all DSL tests so you will know exactly your status at any time more tests soon Stratos Idreos 7 /55

8 how much should I optimize? enough e.g., see m3 we expect a generally elegant design & implementation often OH & sections rule of thumb: at least 2 areas Stratos Idreos 8 /55

9 starting today we will do collaborative note taking: ping for Wasay for access slides are not notes! How: 1) register for a class or two 2) take notes during class and put them in later on 3) edit notes your name: note Stratos Idreos 9 /55

10 from last time assume an array of N integers: find all positions where value>x qualifying positions select operator exists in all systems: sql, nosql, newsql data not as simple as it looks Stratos Idreos 10/55

11 logical design physical design today system design Stratos Idreos 11/55

12 essential steps in using a database system experts/system admins clean schema load tune query user/apps Stratos Idreos 12/55

13 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) Stratos Idreos 13/55

14 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) Stratos Idreos 13/55

15 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, ) Stratos Idreos 13/55

16 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, ) give me the names of all students: select name from students Stratos Idreos 13/55

17 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, ) give me the names of all students: select name from students where GPA>3.0 Stratos Idreos 13/55

18 employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) data schema (1, name1, office1, tel1, city1, salary1) (2, name2, office2, tel2, city2, salary2) (3, name3, office3, tel3, city3, salary3) (4, name4, office4, tel4, city4, salary4) (5, name5, office5, tel5, city5, salary5) (6, name6, office6, tel6, city6, salary6) (7, name7, office7, tel7, city7, salary7) (8, name8, office8, tel8, city8, salary8) (9, name9, office9, NULL, city9, salary9) SQL:insert into employee (1, name1, office1, tel1, city1, salary1) cardinality=9 value does not exist Stratos Idreos 14/55

19 relational model+sql database professors (id,name, ) courses (id,name, profid, ) students (id,name, ) give me all students enrolled in cs165 Stratos Idreos 15/55

20 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) students (id,name, ) give me all students enrolled in cs165 Stratos Idreos 15/55

21 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) foreign key students (id,name, ) give me all students enrolled in cs165 Stratos Idreos 15/55

22 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) foreign key students (id,name, ) give me all students enrolled in cs165 select student.name from students, enrolled, courses where courses.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid Stratos Idreos 15/55

23 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) foreign key students (id,name, ) give me all students enrolled in cs165 select student.name from students, enrolled, courses where courses.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid join Stratos Idreos 15/55

24 enrolled (studentid,courseid, ) students (id,name, ) how do we join Stratos Idreos 16/55

25 normalization say schema about university db contains one table AllData(student ID,student name,student address, course name, grade, professor name, professor ID, professor telephone, ) good Stratos Idreos 17/55

26 normalization say schema about university db contains one table AllData(student ID,student name,student address, course name, grade, professor name, professor ID, professor telephone, ) good duplicates - tons of data - updates - but no joins Stratos Idreos 17/55

27 star schema dimension table 1 (id1, ) fact table (id1,id2, ) dimension table 2 (id2, ) Stratos Idreos 18/55

28 snowflake schema Stratos Idreos 19/55

29 constraints create table employee (id:integer, name:varchar(50) not null, must have a value office:char(5), at most 5 chars telephone:char(10), city:varchar(30), salary:integer, primary key (id) must be unique check (salary<100000)) must not become rich Stratos Idreos 20/55

30 constraints create table employee (id:integer, name:varchar(50) not null, must have a value office:char(5), at most 5 chars telephone:char(10), city:varchar(30), salary:integer, primary key (id) must be unique check (salary<100000)) must not become rich when and how do we enforce constraints Stratos Idreos 20/55

31 more SQL examples aggregations select max(gpa),avg(gpa),min(gpa) from students math select R.a - R.b + R.c from R nested select * from R where R.a IN (select b from S where C<10) set ops select * from R where a =10 UNION select * from B where b =20 Stratos Idreos 21/55

32 select avg(gpa), class, major from students where GPA>3.0 and class>1990 group by class, major order by class Stratos Idreos 22/55

33 base table Employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) view to be used by managers in Berlin Employee-Berlin-Manager select * from employee where city= berlin view to be used by all employees in Berlin Employee-Berlin-All select id,name,city,office from employee where city= berlin Stratos Idreos 23/55

34 base table Employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) view to be used by managers in Berlin Employee-Berlin-Manager select * from employee where city= berlin how should we store views view to be used by all employees in Berlin Employee-Berlin-All select id,name,city,office from employee where city= berlin Stratos Idreos 23/55

35 design logical design physical/logical independence physical design app/user: no need to know how data is stored/accessed system design we can safely change lower layers Stratos Idreos 24/55

36 quiz 1 ~15 minutes - groups of 2+ it is summer now you know all about data systems you are building an augmented reality startup using Google Glass people wearing Google Glass can tag places/objects - voice/image recognition works fine tagging means assigning values, comments, etc to an object you can then query this data - again assume voice recognition works fine and a black box translates natural language to SQL how does the schema of your app look like? (tables, attributes, keys, relationships) (assume a limited working environment/features, say walking around Harvard square/yard) describe 2 interesting queries in SQL Stratos Idreos 25/55

37 a possible example q1: get all places where jenny said awesome q2: get all users that like what I like and are close by comment (id,user_id,oject_id,text, ) likes_comment (user_id,comment_id) object (id,name,location,telephone,date,url,color,taste, many more) user (id,name,location,device, ) likes_object (user_id,oject_id,) trust (user_id,user_id) Stratos Idreos 26/55

38 a possible example q1: get all places where jenny said awesome q2: get all users that like what I like and are close by comment (id,user_id,oject_id,text, ) likes_comment (user_id,comment_id) object (id,name,location,telephone,date,url,color,taste, many more) user (id,name,location,device, ) likes_object (user_id,oject_id,) trust (user_id,user_id) select object.location from object, user where user.name = jenny and comment.user_id=user.id and comment.text LIKE %awesome% Stratos Idreos 26/55

39 a possible example q1: get all places where jenny said awesome q2: get all users that like what I like and are close by comment (id,user_id,oject_id,text, ) likes_comment (user_id,comment_id) object (id,name,location,telephone,date,url,color,taste, many more) user (id,name,location,device, ) likes_object (user_id,oject_id,) trust (user_id,user_id) Stratos Idreos 26/55

40 a possible example q1: get all places where jenny said awesome q2: get all users that like what I like and are close by comment (id,user_id,oject_id,text, ) likes_comment (user_id,comment_id) object (id,name,location,telephone,date,url,color,taste, many more) user (id,name,location,device, ) likes_object (user_id,oject_id,) trust (user_id,user_id) select user.name, user.location from user, likes_object as L1, likes_object as L2 where L1.user_id=my_id and L1.object_id=L2.object_id and L2.user_id!=my_id and user.id=l2.user_id and close(user.location,mylocation)=true Stratos Idreos 26/55

41 how do we store the object table? what if we want to add another kind of object? object (id,name,location,telephone,date,url,color,taste, many more) Stratos Idreos 27/55

42 design logical design physical design system design Stratos Idreos 28/55

43 declarative interface ask what you want so do db systems just work? db system Stratos Idreos 29/55

44 declarative interface ask what you want DBA indexes/views/tuning knobs db system Stratos Idreos 30/55

45 declarative interface ask what you want indexes/views/tuning knobs DBA but db cracking, adaptive* ideas db system Stratos Idreos 30/55

46 essential steps in using a database system experts/system admins clean schema load tune query user/apps Stratos Idreos 31/55

47 design logical design physical design system design next up: db architectures 101 Stratos Idreos 32/55

48 declarative interface ask what you want the db system decides how to best store and access data db system Stratos Idreos 33/55

49 applications sql algorithms/operators database kernel design/implement numerous possible algorithms + data representations choose the best data source, algorithms and path for each query data data data Stratos Idreos 34/55

50 select min(a) from R where B<10 and C<80 algorithms/operators database kernel data data data parser optimizer execution storage Stratos Idreos 35/55

51 applications sql parser optimizer in/out admission execution storage database kernel Stratos Idreos 36/55

52 applications sql client programs thread 1 thread 2 thread 3 thread 4 thread 5 db program thread pool database kernel Stratos Idreos 37/55

53 applications sql parser in/out database kernel cpu optimizer thread pool memory execution transactions disk storage buffer pool Stratos Idreos 38/55

54 applications sql parser in/out database kernel cpu optimizer thread pool memory execution transactions disk storage buffer pool is it good to have modules Stratos Idreos 38/55

55 algorithms/operators database kernel query plan data data data Stratos Idreos 39/55

56 result select name from student where GPA>3.0 project name scan all the data? logical plan select GPA>3.0 student(id,name,gpa,address,class, ) Stratos Idreos 40/55

57 result result select name from student where GPA>3.0 project name project name physical plans scan GPA>3.0 index scan GPA>3.0 students students Stratos Idreos 41/55

58 result avg GPA select avg(gpa) from student where class=2017 project GPA select year=2017 student(id,name,gpa,address,class, ) Stratos Idreos 42/55

59 professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) students (id,name, ) give me all students enrolled in cs165 select student.name from students, enrolled, courses where courses.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid Stratos Idreos 43/55

60 project student.name join student.id=enrolled.studentid select student.name from students, enrolled, courses where courses.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid select courses.name= cs165 join enrolled.courseid=course.id students enrolled courses good plan Stratos Idreos 44/55

61 project student.name join student.id=enrolled.studentid select student.name from students, enrolled, courses where courses.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid join enrolled.courseid=course.id pushing selects down select courses.name= cs165 students enrolled courses Stratos Idreos 45/55

62 select min(a) from R where B<10 and C<80 logical plan internal language optimizer rules/cost model/statistics internal language physical plan execution Stratos Idreos 46/55

63 concurrency how many queries should a db run in parallel and how reads - writes Stratos Idreos 47/55

64 concurrency transactions ACID properties locks Jim Gray, IBM, Tandem, DEC, Microsoft ACM Turing award ACM SIGMOD Edgar F. Codd Inovations award 1993 Stratos Idreos 48/55

65 recovery what should happen if something fails during a query? reads - writes Stratos Idreos 49/55

66 recovery classic example joe owes mike 100$ both joe and mike have a Bank of Bla account possible actions 1) read mike; 2) mike+100; 3) write new mike; joe -100 mike mike joe Stratos Idreos 50/55

67 recovery logs write ahead replay checkpoints C Mohan, IBM ACM SIGMOD Edgar F. Codd Inovations award 1993 Stratos Idreos 51/55

68 can DBAs make wrong decisions? tuning can optimizers make wrong decisions? optimizer execution storage db kernel Stratos Idreos 52/55

69 Architecture of a Database System (Sections 1,2,3,4) by J. Hellerstein, M. Stonebraker and J. Hamilton Stratos Idreos 53/55

70 readings for next few classes The Design and Implementation of Modern Column-store Database Systems (Sections: all -4.6 & 4.8) by D. Abadi, P. Boncz, S. Harizopoulos, S. Idreos, S. Madden IEEE Data Engineering Bulletin, 35(1), March 2012 Special Issue on Column-stores (9 short overview papers) next class we start discussing data layouts and column-stores Stratos Idreos 54/55

71 class 3 SQL & intro to db architectures DATA SYSTEMS prof. Stratos Idreos