class 6 more about column-store plans and compression prof. Stratos Idreos

Transcription

1 class 6 more about column-store plans and compression prof. Stratos Idreos

2 query compilation an ancient yet new topic/research challenge query->sql->interpet to a generic query plan->c functions query->sql->generate and link tailored c code Stratos Idreos 2 /30

3 query compilation an ancient yet new topic/research challenge query->sql->interpet to a generic query plan->c functions query->sql->generate and link tailored c code why this can be better/faster Stratos Idreos 2 /30

4 query compilation an ancient yet new topic/research challenge query->sql->interpet to a generic query plan->c functions query->sql->generate and link tailored c code why this can be better/faster why it was not ideal in the past Stratos Idreos 2 /30

5 Generating code for holistic query evaluation Konstantinos Krikellas, Stratis Viglas, Marcelo Cintra: In Proc. of the Inter. Conference on Data Engineering (ICDE), 2010 Efficiently Compiling Efficient Query Plans for Modern Hardware Thomas Neumann In Proc. of the Very Large Databases Conference (VLDB), 2011 H2O: A Hands-free Adaptive Store Ioannis Alagiannis, Stratos Idreos, and Anastassia Ailamaki In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2014 Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation Harald Lang, Tobias Mühlbauer, Florian Funke, Peter A. Boncz, Thomas Neumann, Alfons Kemper In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2016 Stratos Idreos 3 /30

6 research night coming up during one of the labs next week (will announce in piazza) can we have bitmap indexing that works great for both reads and writes? manos & stratos UpBit: Scalable In-Memory Updatable Bitmap Indexing M. Athanassoulis, Z. Yan, and S. Idreos In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2016 Stratos Idreos 4 /30

7 working over fixed width & dense columns select for (i=0;i<size;i++) if column[i]>v res[j++]=i no function calls, no indirections, no auxiliary data, min ifs easy to prefetch next data values fetch for (i=0;i<size;i++) inter2[j++]=column[inter1[i]] Stratos Idreos 5 /30

8 select min(c) from R where A<10 & B<20 A B C D A<10 IDs B B<20 IDs C minc column- A B C D A<10 IDs B B<20 IDs C minc vector- Stratos Idreos 6 /30

9 update row7=(a=a,b=b,c=c,d=d) A B C D A B C D vs cost: 1 page cost: N pages, N=# of columns Stratos Idreos 7 /30

10 query A B C D update periodically A B C D base data pending updates Stratos Idreos 8 /30

11 A B C relational table tuple 1 tuple 2 tuple 3 tuple 4 tuple 5 tuple 6 a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6 c1 c2 c3 c4 c5 c6 inserts, deletes affect whole table Stratos Idreos 9 /30

12 A pending inserts pending deletes update= delete followed by insert what information do we need to remember Stratos Idreos 10 /30

13 CPU registers on chip cache compute compression= data & computation on board cache memory data speed but cpu disk mem time Stratos Idreos 11 /30

14 8 bytes width value1 value2 value3 value1 value1 value4 value2 value3 value5 3 bits width bytes width value1 value2 value3 value4 value5 how many bits do we need for the codes (width) Stratos Idreos 12 /30

15 A B C D A B C D which one gives better compression Stratos Idreos 13 /30

16 3 bits width bytes width value1 value2 value3 value4 value5 think about billions of entries can we do any better Stratos Idreos 14 /30

17 A A dictionary can we do something like huffman coding any side-effects (check: Business Analytics in (a) Blink from readings) Stratos Idreos 15 /30

18 3 bits width bytes width value1 value2 value3 value4 value5 and how do we process data e.g., select A<10 Stratos Idreos 16 /30

19 fixed-width is key ok and how do we store variable length data? dictionary of strings fixed width codes that point to dictionary entries why Stratos Idreos 17 /30

20 compression is a bonus task for the project Joins on Encoded and Partitioned Data Jae-Gil Lee, Gopi K. Attaluri, Ronald Barber, Naresh Chainani, Oliver Draese, Frederick Ho, Stratos Idreos, Min-Soo Kim, Sam Lightstone, Guy M. Lohman, Konstantinos Morfonios, Keshava Murthy, Ippokratis Pandis, Lin Qiao, Vijayshankar Raman, Vincent Kulandai Samy, Richard Sidle, Knut Stolze, Liping Zhang In Proc. of the Very Large Databases Conference (VLDB), 2014 IBM Blue Stratos Idreos 18 /30

21 A<10 IDs B B<20 IDs C minc late tuple reconstruction/materialization only reconstruct to present results no need to assemble tuples minimize memory footprint minimize data we are moving up the memory hierarchy but requires new processing engine Stratos Idreos 19 /30

22 early vs late tuple reconstruction Assume a column-store database with a table R(A,B,C,D,E). All attributes are integers. Our workload has two classes of queries: 1) select max(b),max(c),max(d),max(e) from R where A>v1 2) select B+C+D+E from R where A>v1 Should we use late or early tuple reconstruction plans? For each query, draw the 2 possible plans and the respective operators, explain which one is best and give the total cost. Stratos Idreos 20 /30

23 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

24 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

25 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

26 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

27 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

28 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

29 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

30 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

31 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

32 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

33 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

34 sel A IDs B max IDs C max IDs D max result late TR sel A IDs B C D E max(b), max(c) max(d), max(e) result hybrid Stratos Idreos 21 /30

35 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

36 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

37 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

38 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

39 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

40 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

41 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

42 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

43 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

44 sa IDs B r1 IDs C r2 +(r1,r2) r3 IDs D r4 +(r3,r4) res late TR sa IDs B C D E +(B,C,D,E) result hybrid Stratos Idreos 22 /30

45 default late tuple reconstruction (rather safe choice) open topic for optimization when and how to do selective early reconstruction issues to consider transformation overhead materialization overhead extra passes over the data it may be almost for free (sometimes in hash joins) dynamic code generation to fit data layouts DSM vs. NSM: CPU performance tradeoffs in block-oriented query processing Marcin Zukowski, Niels Nes, Peter A. Boncz International Workshop on Data Management on New Hardware (DaMoN) 2008 Stratos Idreos 23 /30

46 essential column-stores features virtual ids late tuple reconstruction (if ever) vectorized execution compression fixed-width columns 45" 40" Performance'of'Column3Oriented'Op$miza$ons' Late" Materializa:on" Run$me'(sec)' 35" 30" 25" 20" 15" 10" 5" Compression" Join"Op:miza:on" Baseline" 0" Column"Store" Row"Store" Column-stores vs. row-stores: how different are they really? D. Abadi, S. Madden, and N. Hachem ACM SIGMOD Conference on Management of Data, 2008 Stratos Idreos 24 /30

47 disk memory A A B C D option1 columnstore engine option2 A BC early tuple reconstruction/materialization row-store engine Stratos Idreos 25 /30

48 but why now weren t all those design options obvious in the past as well? moving data from disk moving data from memory computation 1) big memories 2) cpu vs memory speed Stratos Idreos 26 /30

49 main-memory systems optimized for the memory wall with or without persistent data Stratos Idreos 27 /30

50 other system categories nosql, newsql, key-value stores, matlab, etc.. column-stores = bad name modern systems Stratos Idreos 28 /30

51 research papers Column-stores vs. row-stores: how different are they really? D. Abadi, S. Madden, and N. Hachem In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2008 Positional update handling in column stores Sándor Héman, Marcin Zukowski, Niels J. Nes, Lefteris Sidirourgos, Peter A. Boncz In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2010 Updating a cracked database Stratos Idreos, Martin Kersten, Stefan Manegold In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2007 Integrating compression and execution in column-oriented database systems Daniel J. Abadi, Samuel Madden, Miguel Ferreira In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2006 Stratos Idreos 29 /30

52 class 6 more about column-store plans and compression DATA SYSTEMS prof. Stratos Idreos