Models & Intro to DB Architectures

Transcription

1 class 3 Models & Intro to DB Architectures prof. Stratos Idreos

2 welcome brave cs165 students! Stratos Idreos 2 /55

3 NO LAPTOP/PHONE POLICY class is based on participation! we will bring a copy of the slides for every one in each time class so you can follow and keep notes + there is enough evidence that laptops and phones slow you down (check syllabus for more info) Stratos Idreos 3 /55

4 slides are not notes! slides are mainly there to trigger discussion note keeping is your task: starting class 3 we will do collaborative note taking Stratos Idreos 4 /55

5 Will cs165 help me with using (big, nosql, relational) data systems Yes it will but if that is your only goal, there are easier ways to do this Stratos Idreos 5 /55

6 applications sql database kernel algorithms/operators cpu memory data data data disk Stratos Idreos 6 /55

7 what should I be doing? do P0 register to Piazza and stay logged in check syllabus/website carefully check project timeline and plan around it keep up with reading (goes fast!) register for notes today (tmr we will do assignments) come to Labs and OH frequently Stratos Idreos 7 /55

8 for the semester project: install MonetDB & PostgreSQL/MySQL and play with SQL explain + SQL query to see query plans in MonetDB (use read-only mode) repeat throughout the semester compare with your system in terms of performance this is part of final deliverable & several logistics and tools (code.seas, gdb, perf, valgrind, testing infrastructure) do not underestimate this Stratos Idreos 8 /55

9 Read completely and more than once Read Browse Extra Read intro/related work, browse main part Read only if interested Stratos Idreos 9 /55

10 From the syllabus: In this class many of the reading assignments are recent research papers. Unless mentioned otherwise in class, you are not expected to read and understand all these research papers in extreme detail. The main purpose is for you to get exposed to recent ideas and concepts and get inspiration about new opportunities and what is coming in the future. We expect you to read and understand well the abstract, introduction and related work parts of all papers. For the rest of the material in a paper, i.e., the main technical part and the analysis we expect you to have a high level idea of what this does unless we explicitly cover in detail the exact techniques in class. Our goal is that by the end of the semester you will have enough background to be able to pick up any of these papers again and understand it fully! Of course if you want to discuss any of those papers in more detail we will happily do so during office hours. Stratos Idreos 10/55

11 how to read research papers 1) abstract-intro-related work-conclusions what is the problem why is it important why past solutions do not work what is the core idea what is success 2) core part-analysis basic idea what matters any gaps? 3) follow a few citations and repeat goal: by the end of the semester understand these papers fully Stratos Idreos 11/55

12 we want you to have fun! data systems is an exciting field! tell us how you are keeping up tell us what you need to better follow the class tell us your suggestions about how to improve the class Stratos Idreos 12/55

13 next few weeks: (1) data models and languages (today) (2) db architectures (today & next few classes) (3) column-stores & hardware-conscious designs Stratos Idreos 13/55

14 thousands of users - tons of data - frequent updates - tons of queries analytics in the background for game intelligence = the ultimate data-driven challenge Extra: Scaling Games to Epic Proportions By W. White, A. Demers, C. Koch, J. Gehrke and R. Rajagopalan ACM SIGMOD International Conference on Management of Data, 2007 Stratos Idreos 14/55

15 a simple example assume an array of N integers: find all positions where value>x qualifying positions select operator exists in all systems: sql, nosql, newsql data even the simplest tasks are actually far from trivial no obvious solutions; just a taste of what to come Stratos Idreos 15/55

16 philosophy/ sciences ~1960s declarative/ physical/logical independence recovery/ consistency ~2017 papyrus/ paper dbs dbs dbs vs OSs dbs dbs vs nosql dbs vs hand code history/timeline Stratos Idreos 16/55

17 db complex legacy tuning expensive nosql simple clean just enough abstractions declarative processing it is all the same as apps become more complex as apps need to be more scalable newsql->db memory hierarchy I/O vs CPU modeling optimization data layout concepts tradeoffs Stratos Idreos 17/55

18 photo of Anand here the evolution of data-driven applications Anand Rajaraman (3) use social data (social networks) use own data (1) (recommendations) use public data (2) (search) (4) all of the above (all of the above) (5) training data (machine learning) Stratos Idreos 18/55

19 food for (big data-driven) thought Anand Rajaraman amazon google Cyborg the cyborg knows & manages all your models linkedin bank X Stratos Idreos 19/55

20 food for (big data-driven) thought the biggest transportations companies have no cars the biggest data companies have no data the biggest hotel companies have no hotels more about this and other thoughts in our first brainstorming session (TBA) Stratos Idreos 20/55

21 logical design physical design system design Stratos Idreos 21/55

22 essential steps in using a database system experts/system admins clean schema load tune query user/apps Stratos Idreos 22/55

23 professors (id,name, ) key table/relation relational model+sql courses (id,name, profid, ) column/attribute database students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, ) give me the names of all students: select name from students where GPA>3.0 Stratos Idreos 23/55

24 employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) data schema (1, name1, office1, tel1, city1, salary1) (2, name2, office2, tel2, city2, salary2) (3, name3, office3, tel3, city3, salary3) (4, name4, office4, tel4, city4, salary4) (5, name5, office5, tel5, city5, salary5) (6, name6, office6, tel6, city6, salary6) (7, name7, office7, tel7, city7, salary7) (8, name8, office8, tel8, city8, salary8) (9, name9, office9, NULL, city9, salary9) SQL:insert into employee (1, name1, office1, tel1, city1, salary1) cardinality=9 value does not exist Stratos Idreos 24/55

25 professor (id,name, ) enrolled (studentid, courseid, ) relational model+sql course (id,name, profid, ) foreign key database student (id,name, ) give me all students enrolled in cs165 select student.name from student, enrolled, course where course.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid join Stratos Idreos 25/55

26 enrolled (studentid,courseid, ) student (id,name, ) how do we join Stratos Idreos 26/55

27 normalization say schema about university db contains one table AllData(student ID,student name,student address, course name, grade, professor name, professor ID, professor telephone, ) good duplicates - tons of data - updates - but no joins Stratos Idreos 27/55 query : no joins try to insert data now try to update professor data

28 star schema dimension table 1 (id1, ) fact table (id1,id2, ) dimension table 2 (id2, ) Stratos Idreos 28/55

29 snowflake schema Stratos Idreos 29/55

30 Alex Liu, class /265 project adaptive denormalization 1st prize in ACM SIGMOD undergrad research competition Special Interest Group on Management of Data Stratos Idreos 30/55

31 NORMALIZED DATA DENORMALIZED DATA only fast scans but expensive to create, storage & updates good for updates, storage but we need joins Stratos Idreos 31/55

32 adaptive denormalization continuously physically reorganize data based on incoming query patterns (joins) denormalized fragments queries only need to fast scan normalized data possible denormalized space Stratos Idreos 32/55

33 constraints create table employee (id:integer, name:varchar(50) not null, must have a value office:char(5), at most 5 chars telephone:char(10), city:varchar(30), salary:integer, primary key (id) must be unique check (salary<100000)) must not become rich when and how do we enforce constraints Stratos Idreos 33/55

34 more SQL examples aggregations select max(gpa),avg(gpa),min(gpa) from students math select R.a - R.b + R.c from R nested select * from R where R.a IN (select b from S where C<10) set ops select * from R where a =10 UNION select * from B where b =20 Stratos Idreos 34/55

35 select avg(gpa), class, major from students where GPA>3.0 and class>1990 group by class, major order by class Stratos Idreos 35/55

36 base table Employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) view to be used by managers in Berlin Employee-Berlin-Manager select * from employee where city= berlin Why is this useful How should we store views view to be used by all employees in Berlin Employee-Berlin-All select id,name,city,office from employee where city= berlin Stratos Idreos 36/55

37 why are data models great? Stratos Idreos 37/55 allow for abstraction

38 other models? Stratos Idreos 38/55 graph, RDF,

39 it is summer now you know all about data systems you are building an augmented reality startup using Google Glass people wearing Google Glass can tag places/objects - voice/image recognition works fine tagging means assigning values, comments, etc to an object you can then query this data - again assume voice recognition works fine and a black box translates natural language to SQL how does the schema of your app look like? (tables, attributes, keys, relationships) (assume a limited working environment/features, say walking around Harvard square) describe 2 interesting queries in SQL Stratos Idreos 39/55 user(id,first,last,dob,curlocation, ) comment(id,userid,objectid,text) object(id,location, all attributes of objects with nulls: telephone, year built, or only common attributes in object and store rest in extra tables with objid as both primary and foreign key) likesc(userid,commentid) likeso(userid,objid)

40 a possible example q1: get all places where jenny said awesome q2: get all users that like what I like and are close by comment (id,user_id,oject_id,text, ) likes_comment (user_id,comment_id) object (id,name,location,telephone,date,url,color,taste, many more) user (id,name,location,device, ) likes_object (user_id,oject_id,) trust (user_id,user_id) select user.name, user.location select object.location from user, likes_object as L1, likes_object as from L2 object, user where L1.user_id=my_id and L1.object_id=L2.object_id where user.name and L2.user_id = jenny!=my_id and and user.id=l2.user_id and close(user.location,mylocation)=true comment.user_id=user.id and comment.text LIKE %awesome% Stratos Idreos 40/55 tell them about dictionary compression - you do not do it in the schema - system does it underneath - better not let the designer worry about low level details - not always possible someone do an SQL query: select o.location from O,U where user.name = jenny and comment.user_id=user.id and comment.text LIKE %awesome% what the plan should look like? first the selection on comment or the join? select names,location from U, likes_o as L1, likes_o as L2 where L1.user_id=my_id and L1.object_id=L2.object_id and L2.user_id!=my_id and user.id=l2.user_id and close(user.location,mylocation)=true

41 how do we store the object table? what if we want to add another kind of object? object (id,name,location,telephone,date,url,color,taste, many more) open research and business problem sql vs nosql Stratos Idreos 41/55 we are going to have TONS of objects! Talk about nodal

42 design logical design physical design system design Stratos Idreos 42/55

43 declarative interface ask what you want so do db systems just work? db system Stratos Idreos 43/55

44 declarative interface ask what you want DBA indexes/views/tuning knobs but db cracking, adaptive* ideas db system Stratos Idreos 44/55

45 essential steps in using a database system experts/system admins clean schema load tune query user/apps Stratos Idreos 45/55

46 design logical design physical design system design next up: db architectures 101 Stratos Idreos 46/55

47 applications algorithms/operators sql database kernel design/implement numerous possible algorithms + data representations choose the best data source, algorithms and path for each query data data data Stratos Idreos 47/55 design best possible storage/access level with a specific API decision level whenever we make decisions - things can go wrong

48 select min(a) from R where B<10 and C<80 algorithms/operators database kernel data data data parser optimizer execution storage Stratos Idreos 48/55 parse like compiler flex, yacc, language grammar etc check catalog:maintain all meta, tables columns, access control : am I allowed to see this data etc who decides how to store the data? based on what?

49 applications sql parser optimizer in/out admission execution storage database kernel Stratos Idreos 49/55 simple listen send out with buffer admission control - memory load etc simple number restriction - assign priorities etc

50 applications sql client programs thread 1 thread 2 thread 3 thread 4 thread 5 db program thread pool database kernel Stratos Idreos 50/55 diff between program and thread - thread pool to avoid creating and destroying threads - databases had their own thread support - now use OS

51 applications sql parser in/out database kernel cpu memory disk optimizer execution storage thread pool transactions buffer pool is it good to have modules Stratos Idreos 51/55

52 Notes to remember models help create the right abstractions models help create >>1 applications over the same data we first need to clean, structure and load data data systems consist of software components Stratos Idreos 52/55

53 reading Read: textbook: chapters 1, 3 (-3.5), 5 (-5.8,-5.9) intro + relational model + SQL Browse: the Fourth Paradigm Stratos Idreos 53/55

54 readings for next few classes Read: Architecture of a Database System (Sections 1,2,3,4) by J. Hellerstein, M. Stonebraker and J. Hamilton Read: The Design and Implementation of Modern Columnstore Database Systems by D. Abadi, P. Boncz, S. Harizopoulos, S. Idreos, S. Madden Stratos Idreos 54/55

55 Models & Intro to DB Architectures class 3 DATA SYSTEMS prof. Stratos Idreos