class 5 column stores 2.0 prof. Stratos Idreos

Transcription

1 class 5 column stores 2.0 prof. Stratos Idreos

2 worth thinking about what just happened? where is my data? , cloud, social media, can we design systems that let us know what is going on? Stratos Idreos 2 /28

3 cool papers 2.0 The Case for RodentStore: An Adaptive, Declarative Storage System Philippe Cudré-Mauroux, Eugene Wu, Samuel Madden In Proc. of the Inter. Conference on Innovative Data Systems Research (CIDR), 2009 Abstraction Without Regret in Database Systems Building: a Manifesto Christoph Koch IEEE Data Eng. Bull. 37(1): (2014) dbtouch: Analytics at your Fingertips Stratos Idreos and Erietta Liarou In Proc. of the Inter. Conference on Innovative Data Systems Research (CIDR), 2013 Stratos Idreos 3 /28

4 design doc think, design, create 1-2 page PDF doc and ask for feedback mandatory M1-M3, optional afterwards submit through Canvas do not worry about perfection: fail fast wrong ideas ok if you eventually find out they are wrong :) (holds for midterms as well) Stratos Idreos 4 /28

5 registers 2x on chip cache 10x on board cache 100x memory my head ~0 this room 1 min this building 10 min New York 1.5 hours Jim Gray, IBM, Tandem, DEC, Microsoft ACM Turing award ACM SIGMOD Edgar F. Codd Innovations Award 100Kx disk Pluto 2 years Stratos Idreos 5 /28

6 the way we store data defines the possible (efficient) access methods Stratos Idreos 6 /28

7 slotted page free_offset, N, offset1-length1, offset2-lenght2, free space scan null update var length Stratos Idreos 7 /28

8 row-store A BC D column-store A B C D Stratos Idreos 8 /28

9 virtual ids/ positional alignment A B C columns do not need to have the same width tuple 1 tuple 2 tuple 3 tuple 4 tuple 5 tuple 6 a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6 c1 c2 c3 c4 c5 c6 fixed-width + dense positional lookups/joins A(i) = A + i * width(a) Stratos Idreos 9 /28

10 today column-stores 2.0 Stratos Idreos 10 /28

11 select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc query plan = select -> fetch -> select -> fetch - > min sequential access patterns, max 1 if Stratos Idreos 11 /28

12 select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc query plan = select -> fetch -> select -> fetch - > min sequential access patterns, max 1 if Stratos Idreos 11 /28

13 select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc query plan = select -> fetch -> select -> fetch - > min sequential access patterns, max 1 if Stratos Idreos 11 /28

14 select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc query plan = select -> fetch -> select -> fetch - > min sequential access patterns, max 1 if Stratos Idreos 11 /28

15 select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc query plan = select -> fetch -> select -> fetch - > min sequential access patterns, max 1 if Stratos Idreos 11 /28

16 select min(c) from R where A<10 & B<20 disk memory A B C D A<10 IDs B B<20 IDs C minc query plan = select -> fetch -> select -> fetch - > min sequential access patterns, max 1 if Stratos Idreos 11 /28

17 working over fixed width & dense columns select for (i=0;i<size;i++) if column[i]>v inter1[j++]=i no function calls, no indirections, no auxiliary data, min ifs easy to prefetch next data values fetch for (i=0;i<size;i++) inter2[j++]=column[inter1[i]] with data being memory resident these become significant cost components Stratos Idreos 12 /28

18 A<10 IDs B B<20 IDs C minc alt1) start with B alt2) scan A & B independently and merge alt3) store intermediates as bit vectors - not positions Stratos Idreos 13 /28

19 A<10 IDs B B<20 IDs C minc alt1) start with B alt2) scan A & B independently and merge alt3) store intermediates as bit vectors - not positions project: basic one + more if you decide to invest in this area midterm: basic one alternatives Stratos Idreos 13 /28

20 A<10 IDs B B<20 IDs C minc late tuple reconstruction/materialization only reconstruct to present results no need to assemble tuples minimize memory footprint minimize data we are moving up the memory hierarchy but requires new processing engine Stratos Idreos 14 /28

21 disk memory A A B C D option1 columnstore engine option2 A BC early tuple reconstruction/materialization row-store engine Stratos Idreos 15 /28

22 A<10 IDs B B<20 IDs C minc possible data flow patterns tuple at a time block/vector at a time column at a time Stratos Idreos 16 /28

23 select min(c) from R where A<10 & B<20 A B C D A<10 IDs B B<20 IDs C minc column- A B C D A<10 IDs B B<20 IDs C minc vector- Stratos Idreos 17 /28

24 the beer analogy Marcin Zukowski, PhD CEO/Co-founder of Vectorwise (now Actian) now: changing the world, one terabyte at a time co-founder of Snowflake Stratos Idreos 18 /28

25 A B A op1 op2 op3 faster CPU registers query plan on chip cache on board cache memory A B cheaper disk size of vector Stratos Idreos 19 /28

26 tuple at a time - good for minimizing memory footprint bulk processing - good minimizing functional overhead vectorized processing - somewhere in between Stratos Idreos 20 /28

27 ~1960s 1980s: ideas about block processing 2005: vectorwise tuple at a time tuple at a time tuple at a time history/timeline >2010: industry adoption Stratos Idreos 21 /28

28 project: column-at-a-time bonus: vectorized processing Stratos Idreos 22 /28

29 update row7=(a=a,b=b,c=c,d=d) row-store column-store A BC D A B C D vs which is better to update and why? how much does it cost to update a single row? (think about pages, data movement) how to update in column-stores? (query plan + algorithms) Stratos Idreos 23 /28

30 query A B C D update periodically A B C D base data pending updates Stratos Idreos 24 /28

31 query fractured mirrors optimizer A B C D A BC D columns copy rows copy A case for fractured mirrors Ravishankar Ramamurthy, David J. DeWitt, Qi Su Very Large Databases Journal, 12(2): , 2003 Stratos Idreos 25 /28

32 Notes to remember column-stores great for analytics row-stores great for transactions still basic concepts are the same hybrids possible keep access patterns sequential and simple (min ifs) Stratos Idreos 26 /28

33 reading The Design and Implementation of Modern Column-store Database Systems (Sections: all -4.6 & 4.8) by D. Abadi, P. Boncz, S. Harizopoulos, S. Idreos, S. Madden IEEE Data Engineering Bulletin, 35(1), March 2012 Special Issue on Column-stores (9 short overview papers) Stratos Idreos 27 /28

34 research papers Database Architecture Optimized for the New Bottleneck: Memory Access Peter Boncz, Stefan Manegold, Martin Kersten In Proc. of the Very Large Databases Conference (VLDB), 1999 MonetDB/X100: Hyper-Pipelining Query Execution Peter A. Boncz, Marcin Zukowski, Niels Nes In Proc. of the Inter. Conference on Innovative Data Systems Research (CIDR), 2005 Materialization Strategies in a Column-Oriented DBMS Daniel Abadi, Daniel Myers, David DeWitt, Samuel Madden In Proc. of the Inter. Conference on Data Engineering (ICDE), 2007 Self-organizing tuple reconstruction in column-stores Stratos Idreos, Martin Kersten, Stefan Manegold In Proc. of the ACM SIGMOD Inter. Conference on Management of Data, 2009 Stratos Idreos 28 /28

35 class 5 column-stores 2.0 DATA SYSTEMS prof. Stratos Idreos