column-stores basics

Transcription

1 class 3 column-stores basics prof.

2 project description is now online First background info will be given this Friday and detailed lecture on Feb 21

3 Basic Readings for Projects Modern B-Tree Techniques, by Goetz Graefe, Foundations and Trends in Databases, 2011 The Log-Structured Merge-Tree (LSM-Tree), by Patrick E. O'Neil, Edward Cheng, Dieter Gawlick, Elizabeth J. O Neil, Acta Inf. 33(4): ,1996 Monkey: Optimal Navigal Key-value Store, by Niv Data, Manos Athanassoulis and. ACM SIGMOD, 2017

4 C/C++ no libraries unless we explicitly allow it we expect you build everything from scratch so you can control storage and access 100%

5 Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation VectorH: Taking SQL-on-Hadoop to the Next Level H2O: A Hands-free Adaptive Store Fast Scans on Key-Value Stores The adaptive radix tree: ARTful indexing for main-memory databases Skipping-oriented Partitioning for Columnar Layouts The TileDB Array Data Storage Manager OrpheusDB: Bolt-on Versioning for Relational Databases. The End of a Myth: Distributed Transaction Can Scale Ground: A Data Context Service

6 How to read/review? P vs B 1) what is the problem? 2) why is it important? 3) why is it hard? 4) why existing solutions do not work? 5) what is the core intuition for the solution? 6) does the paper prove its claims? 7) what is the setup of analysis/experiments? is it sufficient? 8) are there any gaps in the logic/proof? 9) describe at least one possible next step

7 Action steps: 1) Read the syllabus & website carefully, 2) Register to Piazza, 3) Do P0 if you have not taken CS165 and check self-test, 4) Register for paper presentation (week 2), 5) Start submitting your paper reviews (week 3) web site: piazza: piazza.com/harvard/spring2018/cs265/home office hours: Stratos: Wed/Thur/Fri, 3-4pm, MD139 TF office hours: class website textbook: nope research papers will be available from the Harvard network

8 how can I prepare? 1) start browsing some basic texts Get familiar with the very basics of traditional database architectures: Architecture of a Database System. By J. Hellerstein, M. Stonebraker and J. Hamilton. Foundations and Trends in Databases, 2007 Get familiar with very basics of modern database architectures: The Design and Implementation of Modern Column-store Database Systems. By D. Abadi, P. Boncz, S. Harizopoulos, S. Idreos, S. Madden. Foundations and Trends in Databases, 2013 Get familiar with the very basics of modern large scale systems: Massively Parallel Databases and MapReduce Systems. By Shivnath Babu and Herodotos Herodotou. Foundations and Trends in Databases, ) play with basic data structures implementation in C (linked list/hash table/tree)

9 design logical design physical design system design

10 essential steps in using a database system experts/system admins clean schema load tune query user/apps

11 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, )

12 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), )

13 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, )

14 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, ) give me the names of all students: select name from students

15 relational model+sql database professors (id,name, ) key table/relation courses (id,name, profid, ) column/attribute students (id,name, ) create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), ) insert into professors ( , john smith, ) give me the names of all students: select name from students where GPA>3.0

16 employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) data schema (1, name1, office1, tel1, city1, salary1) (2, name2, office2, tel2, city2, salary2) (3, name3, office3, tel3, city3, salary3) (4, name4, office4, tel4, city4, salary4) (5, name5, office5, tel5, city5, salary5) (6, name6, office6, tel6, city6, salary6) (7, name7, office7, tel7, city7, salary7) (8, name8, office8, tel8, city8, salary8) (9, name9, office9, NULL, city9, salary9) SQL:insert into employee (1, name1, office1, tel1, city1, salary1) cardinality=9 value does not exist

17 relational model+sql database professors (id,name, ) courses (id,name, profid, ) students (id,name, ) give me all students enrolled in cs265

18 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) students (id,name, ) give me all students enrolled in cs265

19 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) foreign key students (id,name, ) give me all students enrolled in cs265

20 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) foreign key students (id,name, ) give me all students enrolled in cs265 select student.name from students, enrolled, courses where courses.name= cs265 and enrolled.courseid=course.id and student.id=enrolled.studentid

21 relational model+sql database professors (id,name, ) enrolled (studentid, courseid, ) courses (id,name, profid, ) foreign key students (id,name, ) give me all students enrolled in cs265 select student.name from students, enrolled, courses where courses.name= cs265 and enrolled.courseid=course.id and student.id=enrolled.studentid join

22 star schema dimension table 1 (id1, ) fact table (id1,id2, ) dimension table 2 (id2, )

23 key-value store vs relational

24 key-value store vs relational can we store a document collection in a relational systems can we store a relational database in a key-value store

25 design logical design physical design system design

26 essential steps in using a database system experts/system admins clean schema load tune query user/apps

27 declarative interface ask what you want so do db systems just work? db system

28 declarative interface ask what you want DBA indexes/views/tuning knobs db system

29 declarative interface ask what you want indexes/views/tuning knobs DBA but db cracking, adaptive* ideas db system

30 design logical design physical design system design

31 select min(a) from R where B<10 and C<80 algorithms/operators database kernel data data data parser optimizer execution storage

32 cheaper faster CPU registers on chip cache on board cache memory disk SRAM DRAM cache miss: looking for something which is not in the cache ~1ns ~10ns ~100ns speed memory wall memory miss: looking for something which is not in memory cpu mem time

33 registers 2x on chip cache 10x on board cache 100x memory my head ~0 this room 1 min this building 10 min New York 1.5 hours Jim Gray, IBM, Tandem, DEC, Microsoft ACM Turing award ACM SIGMOD Edgar F. Codd Innovations award 100Kx disk Pluto 2 years

34 random access & page-based access CPU need to only read x but have to read all of page 1 registers data value x on chip cache page1 page2 page3 data move on board cache memory disk

35 dbms block size os block size device block size os and db will typically refer to pages CS265, Spring 2017

36 employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) data storage blocks < pages < files (1, name1, office1, tel1, city1, salary1) (2, name2, office2, tel2, city2, salary2) (3, name3, office3, tel3, city3, salary3) (4, name4, office4, tel4, city4, salary4) (5, name5, office5, tel5, city5, salary5) (6, name6, office6, tel6, city6, salary6) (7, name7, office7, tel7, city7, salary7) (8, name8, office8, tel8, city8, salary8) (9, name9, office9, tel9, city9, salary9) file CS265, Spring 2017 remember: the way we store data defines the best possible way we can access it

37 slotted pages employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) page (1, name1, office1, tel1, city1, salary1) (2, name2, office2, tel2, city2, salary2) (3, name3, office3, tel3, city3, salary3) (4, name4, office4, tel4, city4, salary4) (5, name5, office5, tel5, city5, salary5) (6, name6, office6, tel6, city6, salary6) (7, name7, office7, tel7, city7, salary7) (8, name8, office8, tel8, city8, salary8) (9, name9, office9, tel9, city9, salary9) header row2 row1 row3 CS265, Spring 2017

38 slotted page free_offset, N, offset1-length1, offset2-lenght2, free space scan null update var length CS265, Spring 2017

39 row-store select A,B,C,D select A A BC D one page contains all fields of multiple attributes stored continuously file CS265, Spring 2017

40 select A,B,C,D select A row-store column-store A BC D A B C D one page contains fields of a single attribute stored continuously CS265, Spring 2017

41 the way we store data defines the possible (efficient) access methods

42 ~1960s 1970: column storage ideas start appearing monetdb ~2000: open source complete system rows rows rows rows rows rows rows history/timeline 1985: first rather complete column-store model 2005-now: more ideas and industry adoption of columnstore designs CS265, Spring 2017 c-store,vertica,vectorwise and then ibm,microsoft,oracle, and more

43 virtual ids/ positional alignment A B C columns do not need to have the same width tuple 1 tuple 2 tuple 3 tuple 4 tuple 5 tuple 6 a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6 c1 c2 c3 c4 c5 c6 fixed-width + dense positional lookups/joins A(i) = A + i * width(a) CS265, Spring 2017

44 ok so now we can selectively read columns but how do we process them? disk memory A A B C D option1 columnstore engine option2 A BC early tuple reconstruction/materialization row-store engine CS265, Spring 2017

45 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D CS265, Spring 2017

46 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk A B C D A<10 memory CS265, Spring 2017

47 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk A B C D A<10 memory 1: int *input=a 2: for (i=0;i<tuples;i++,input++) 3: if *input<10 4: *output=i 5: output++ A<10 CS265, Spring 2017

48 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs CS265, Spring 2017

49 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B CS265, Spring 2017

50 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 CS265, Spring 2017

51 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C CS265, Spring 2017

52 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc CS265, Spring 2017

53 late reconstruction/materialization select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc always sequential access patterns memory contains only what is needed at any point in time CS265, Spring 2017

54 working over fixed width & dense columns select for (i=0;i<size;i++) if column[i]>v res[j++]=i no function calls, no indirections, no auxiliary data, min ifs easy to prefetch next data values fetch for (i=0;i<size;i++) inter2[j++]=column[inter1[i]]

55 A<10 IDs B B<20 IDs C minc alt1) start with B alt2) scan A & B independently and merge alt3) store intermediates as bit vectors - not positions

56 A<10 IDs B B<20 IDs C minc late tuple reconstruction/materialization only reconstruct to present results no need to assemble tuples minimize memory footprint minimize data we are moving up the memory hierarchy but requires new processing engine

57 A<10 IDs B B<20 IDs C minc possible data flow patterns tuple at a time block/vector at a time column at a time

58 select min(c) from R where A<10 & B<20 A B C D A<10 IDs B B<20 IDs C minc column- A B C D A<10 IDs B B<20 IDs C minc vector-

59 essential column-stores features virtual ids late tuple reconstruction (if ever) vectorized execution compression fixed-width columns 45" 40" Performance'of'Column3Oriented'Op$miza$ons' Late" Materializa:on" Run$me'(sec)' 35" 30" 25" 20" 15" 10" 5" Compression" Join"Op:miza:on" Baseline" 0" Column"Store" Row"Store" Column-stores vs. row-stores: how different are they really? D. Abadi, S. Madden, and N. Hachem ACM SIGMOD Conference on Management of Data, 2008

60 but why now weren t all those design options obvious in the past as well? moving data from disk moving data from memory computation 1) big memories 2) cpu vs memory speed

61 class 3 column-stores basics BIG DATA SYSTEMS prof.