complex plans and hybrid layouts

Transcription

1 class 7 complex plans and hybrid layouts prof. Stratos Idreos

2 essential column-stores features virtual ids late tuple reconstruction (if ever) vectorized execution compression fixed-width columns 45" 40" Performance'of'Column3Oriented'Op$miza$ons' Late" Materializa:on" Run$me'(sec)' 35" 30" 25" 20" 15" 10" 5" Compression" Join"Op:miza:on" Baseline" 0" Column"Store" Row"Store" Column-stores vs. row-stores: how different are they really? D. Abadi, S. Madden, and N. Hachem ACM SIGMOD Conference on Management of Data, 2008 Stratos Idreos 2 /28

3 but why now weren t all those design options obvious in the past as well? moving data from disk moving data from memory computation 1) big memories 2) cpu vs memory speed Stratos Idreos 3 /28

4 main-memory systems optimized for the memory wall with or without persistent data Stratos Idreos 4 /28

5 other system categories nosql (cs265), newsql, matlab, etc.. column-stores = bad name modern systems Stratos Idreos /28

6 column-storage row-storage A B C D A B C D two extremes Stratos Idreos 6 /28

7 PAX: store all data about a row in a single page but organize data in a column major way inside each page Anastasia Ailamaki, EPFL Stratos Idreos 7 /28

8 column-groups: store data about a row in a single page but use any kind of column-group combination inside each page A B C D A B C D A B C D Stratos Idreos 8 /28

9 column-groups on separate files: group columns in separate pages each row is spread in >1 pages A B C D A B C D Stratos Idreos /28

10 how do we decide? A B C D A B C D Stratos Idreos 10 /28

11 how do we decide? A B C D A B C D first think what do we want to do with the data: access patterns query A,B, query A,B,C,D insert delete update A metric: data movement # pages Stratos Idreos 10 /28

12 offline? Stratos Idreos 11 /28

13 offline? online? Stratos Idreos 11 /28

14 offline? online? code generation? Stratos Idreos 11 /28

15 offline? online? code generation? Browse: H2O: A Hands-free Adaptive Store Ioannis Alagiannis, Stratos Idreos, and Anastassia Ailamaki In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2014 Stratos Idreos 11 /28

16 fixed-width dense unordered columns can we do any better? Stratos Idreos 12 /28

17 zone1: min value =m1 max value = k1 zone2: min value =m2 max value = k2 Stratos Idreos 13 /28

18 zone1: min value =m1 max value = k1 zone2: min value =m2 max value = k2 what if data is uniformly distributed over the column Stratos Idreos 14 /28

19 zone1: min value =m1 max value = k1 zone2: min value =m2 max value = k2 what if data is uniformly distributed over the column adaptively reorganize column to minimize # of zones a query has to scan cs165/265 project: finalist ACM SIGMOD undergrad research competition 2016 Stratos Idreos 14 /28

20 it is all about the bits column value1= value2= value3= Jignesh Patel, U of Wisconsin Stratos Idreos 15 /28

21 Browse: BitWeaving: fast scans for main memory data processing Yinan Li, Jignesh M. Patel In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2013 Extra: ByteSlice: Processing with a New Storage Layout Ziqiang Feng, Eric Lo, Ben Kao, Wenjian Xu In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2015 Stratos Idreos 16 /28

22 professor (id,name, ) enrolled (studentid, courseid, ) course (id,name, profid, ) foreign key database student (id,name, ) give me all students enrolled in cs165 select student.name from student, enrolled, course where course.name= cs165 and enrolled.courseid=course.id and student.id=enrolled.studentid Stratos Idreos 17 /28

23 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,50,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) Stratos Idreos 18 /28

24 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,50,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) select(ra,5,20) Ra inter inter reconstruct(rb,inter1) Rb inter select(inter2,40,50) select(inter2,30,40) inter inter3 Stratos Idreos 18 / inter3 4 5 reconstruct(rc,inter3) Rc join_input_r (1) (2) (3) (4) 4 5 2

25 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) Stratos Idreos /28

26 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) select(sa,55,65) select(sa,4,65) Sa inter inter reconstruct(sb,inter4) Sb inter inter reverse(inter5) join_input_s (5) (6) (7) Stratos Idreos /28

27 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) Stratos Idreos 20 /28

28 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) join_input_r join(join_input_r,join_input_s) join_input_s join_res_ R_S voidtail(join_res_r_s) join_res_ R_S inter6 (8) () 4 Stratos Idreos 20 /28

29 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) Stratos Idreos 21 /28

30 Ra Initial Status Relation R Relation S Rb Rc Sa Sb Query and Query Plan (MAL Algebra) select sum(r.a) from R, S where R.c = S.b and 5<R.a<20 and 40<R.b<50 and 4<S.a<65 30<S.a<40 1. inter1 = select(ra,5,20) 2. inter2 = reconstruct(rb,inter1) 3. inter3 = select(inter2,30,40) select(inter2,40,50) 4. join_input_r = reconstruct(rc,inter3) 5. inter4 = select(sa,55,65) select(sa,4,65) 6. inter5 = reconstruct(sb,inter4) 7. join_input_s = reverse(inter5) 8. join_res_r_s = join(join_input_r,join_input_s). inter6 = voidtail(join_res_r_s) 10. inter7 = reconstruct(ra,inter6) 11. result = sum(inter7) inter6 4 reconstruct(ra,inter6) Ra inter7 inter7 sum(inter7) result 28 (10) (11) Stratos Idreos 21 /28

31 select R.D+S.G from R,S where R.A=S.A and R.C<10 and S.F>30 Use a simple join algorithm: Build hash table on S.A and then we probe for each R.A value what happens after the join? access patterns select R.C fetch R.A fetch R.D join R.D+ S.G select S.F fetch S.A fetch S.G Goal: Describe each operator: input/output and access patterns watch out for random access! block operator Stratos Idreos 22 /28

32 select R.A, R.B, R.C, S.A, S.B, S.C from R, S where R.J=S.J and preparing the R join input R.* scan pos fetch R.J R.J pos we need the original positions so we can fetch other R columns after the join ordered sparse ordered sparse same for the S join input Stratos Idreos /28

33 select R.A, R.B, R.C, S.A, S.B, S.C from R, S where R.J=S.J and join join input R.J R.J + posr join input S.J S.J + poss partition (=reorder) both join inputs to join join result posr + poss ordered sparse ordered sparse one or both sides are unordered 1) We can sequentially scan R.J and probe S.J. Then result is in the order of R.J but S.J part is out of order Stratos Idreos 24 /28

34 select R.A, R.B, R.C, S.A, S.B, S.C from R, S where R.J=S.J and join join input R.J R.J + posr join input S.J S.J + poss partition (=reorder) both join inputs to join join result posr + poss ordered sparse ordered sparse one or both sides are unordered 1) We can sequentially scan R.J and probe S.J. Then result is in the order of R.J but S.J part is out of order 2) More advanced algorithms will partitions both inputs (we will see why and how in future classes) This means output is unordered with respect to both inputs Stratos Idreos 24 /28

35 cluster= partition just enough so that any random access is within a small range (e.g., a page) join result posr + poss both sides are unordered cluster on R ID + poss join result clustered on R posr + poss fetch R payload with sequential pattern clustered cluster on S ID + poss clustered sort on ID decluster fetch S payload with sequential pattern radix declustering all result columns aligned e.g., ID + S.A Stratos Idreos 25 /28

36 first part done: basic concepts in modern systems coming up: indexing and fast scans Stratos Idreos 26 /28

37 Browse: Cache-Conscious Radix Decluster Projections By S. Manegold, P. Boncz, N. Nes, and M. Kersten Very Large Databases Conference, 2004 Browse: H2O: A Hands-free Adaptive Store Ioannis Alagiannis, Stratos Idreos, and Anastassia Ailamaki In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2014 Browse: BitWeaving: fast scans for main memory data processing Yinan Li, Jignesh M. Patel In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2013 Stratos Idreos 27 /28

38 class 7 complex plans & hybrid layouts DATA SYSTEMS prof. Stratos Idreos