How Achaeans Would Construct Columns in Troy. Alekh Jindal, Felix Martin Schuhknecht, Jens Dittrich, Karen Khachatryan, Alexander Bunte

Transcription

1 How Achaeans Would Construct Columns in Troy Alekh Jindal, Felix Martin Schuhknecht, Jens Dittrich, Karen Khachatryan, Alexander Bunte

2 Number of Visas Received 1 0,75 0,5 0,25 0 Alekh Jens

3 Health Level 5 days before CIDR 100 percentage Alekh Jens

4 Average Number of Slides per 20min talk Alekh Jens

5 Number of Slides Actually Prepared Alekh Jens

6

7 What is the problem?

8 Row-stores 8

9 Column-stores 9

10 OLTP OLAP 10

11 11

12 OLTP OLAP? Can we do efficient OLAP in Row-stores? 12

13 Any solutions out there?

14 C-Tables * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

15 C-Tables * Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

16 C-Tables * Physical Table Sorted Relation Customer name phone market segment smith 2134 automobile steve 2435 automobile mark 4312 building joe 9878 building kim 6756 furniture john 3425 household jim 5766 household ian 8789 household T market segment f v c 1 automobile 2 3 building 2 5 furniture 1 6 household 3 * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

17 C-Tables * Physical Table Sorted Relation Customer name phone market segment smith 2134 automobile steve 2435 automobile mark 4312 building joe 9878 building kim 6756 furniture john 3425 household jim 5766 household ian 8789 household T market segment f v c 1 automobile 2 3 building 2 5 furniture 1 6 household 3 * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

18 C-Tables * Physical Table Sorted Relation Customer name phone market segment smith 2134 automobile steve 2435 automobile mark 4312 building joe 9878 building kim 6756 furniture john 3425 household jim 5766 household ian 8789 household T market segment f v c 1 automobile 2 3 building 2 5 furniture 1 6 household 3 * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

19 C-Tables * Physical Table Sorted Relation Customer name phone market segment smith 2134 automobile steve 2435 automobile mark 4312 building joe 9878 building kim 6756 furniture john 3425 household jim 5766 household ian 8789 household T market segment f v c 1 automobile 2 3 building 2 5 furniture 1 6 household 3 * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

20 C-Tables * Physical Table T market segment f v c 1 automobile 2 3 building 2 5 furniture 1 6 household 3 T phone f v Sorted Relation Customer name phone market segment smith 2134 automobile steve 2435 automobile mark 4312 building joe 9878 building kim 6756 furniture john 3425 household jim 5766 household ian 8789 household T name f v 1 smith 2 steve 3 mark 4 joe 5 kim 6 john 7 jim 8 ian JOINS! * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

21 C-Tables * C-Table Standard Row C-Table Query Time (sec) Trojan Columns (SP) Row C-Tables Trojan Columns # referenced Attributes (a) Cardinality = 10 JOINS! Figure 7: Comparing query times of C * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

22 C-Tables * C-Table C-Table Standard RowStandard Row Query Time (sec) Query Time (sec) # referenced Attributes (b) Cardinality 1500 = 100 (c) ring query times of CTable and standard 1125 row for different attribut e (sec) * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

23 C-Tables * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n * Nicolas Bruno. Teaching an Old Elephant New Tricks. CIDR

24 Column Index * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD

25 Column Index * Physical Table Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household Customer_trojan segment_id attribute_id blob_data 1 name smith, john, kim, joe 1 phone 2134, 3425, 6756, market_segment automobile, household, furniture, building 2 name mark, steve, jim, ian 2 phone 4312, 2435, 5766, market_segment building, automobile, household, household * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD 2011 segment size = 4 25

26 Column Index * Physical Table Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household Customer_trojan segment_id attribute_id blob_data 1 name smith, john, kim, joe 1 phone 2134, 3425, 6756, market_segment automobile, household, furniture, building 2 name mark, steve, jim, ian 2 phone 4312, 2435, 5766, market_segment building, automobile, household, household * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD 2011 segment size = 4 26

27 Column Index * Physical Table Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household Customer_trojan segment_id attribute_id blob_data 1 name smith, john, kim, joe 1 phone 2134, 3425, 6756, market_segment automobile, household, furniture, building 2 name mark, steve, jim, ian 2 phone 4312, 2435, 5766, market_segment building, automobile, household, household * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD 2011 segment size = 4 27

28 Column Index * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n DEEP CHANGES! * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD

29 Column Index * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n LONG TIME! * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD

30 Column Index * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n SOURCE CODE! * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD

31 Column Index * Application User Database Query Processor Relations Physical Representation File 1 File 2 File 3... File n * P. Larson et. al. SQL Server Column Store Indexes. SIGMOD

32 What do we propose?

33 Trojan Columns User Database Application Query Processor Relations UDF Storage Layer Physical Representation File 1 File 2 File 3... File n 31

34 Trojan Columns Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household Physical Table Customer_trojan segment_id attribute_id blob_data 1 name smith, john, kim, joe 1 phone 2134, 3425, 6756, market_segment automobile, household, furniture, building 2 name mark, steve, jim, ian 2 phone 4312, 2435, 5766, market_segment building, automobile, household, household 32

35 write-udf Trojan Columns (a) Convert row tuples into blobs Data Parser Tuple Iterator (b) Store blob data (c) Get next row data Data Accesso r Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household Physical Table Customer_trojan segment_id attribute_id blob_data 1 name smith, john, kim, joe 1 phone 2134, 3425, 6756, market_segment automobile, household, furniture, building 2 name mark, steve, jim, ian 2 phone 4312, 2435, 5766, market_segment building, automobile, household, household 32

36 read-udf Trojan Columns (e) Reconstruct row tuples Data Parser Tuple Iterator (d) Parse blob data (g)end of table (f) Fetch blob data Data Accessor Relation Customer name phone market_segment smith 2134 automobile john 3425 household kim 6756 furniture joe 9878 building mark 4312 building steve 2435 automobile jim 5766 household ian 8789 household Physical Table Customer_trojan segment_id attribute_id blob_data 1 name smith, john, kim, joe 1 phone 2134, 3425, 6756, market_segment automobile, household, furniture, building 2 name mark, steve, jim, ian 2 phone 4312, 2435, 5766, market_segment building, automobile, household, household 33

37 Example: TPC-H Query 6 Result γ agg (extendedprice * discount) σ shipdate BETWEEN AND AND discount BETWEEN 0.05 AND 0.07 AND quantity < 24 π SCAN quantity, discount extendedprice, shipdate lineitem 34

38 Example: TPC-H Query 6 Result Result γ agg (extendedprice * discount) γ agg (extendedprice * discount) σ shipdate BETWEEN AND AND discount BETWEEN 0.05 AND 0.07 AND quantity < 24 π SCAN quantity, discount extendedprice, shipdate lineitem σ shipdate BETWEEN AND AND discount BETWEEN 0.05 AND 0.07 AND quantity < 24 π SCAN wn quantity, discount extendedprice, shipdate lineitem scanudf scanudf 35

39 Example: TPC-H Query 6 Result Result Result γ agg σ shipdate BETWEEN AND AND discount BETWEEN 0.05 AND 0.07 AND quantity < 24 π SCAN (extendedprice * discount) quantity, discount extendedprice, shipdate lineitem γ agg σ shipdate BETWEEN AND AND discount BETWEEN 0.05 AND 0.07 AND quantity < 24 π SCAN wn quantity, discount extendedprice, shipdate lineitem (extendedprice * discount) scanudf scanudf γ agg selectudf σ shipdate BETWEEN AND AND discount BETWEEN 0.05 AND 0.07 AND quantity < 24 π SCAN (extendedprice * discount) quantity, discount extendedprice, shipdate lineitem selectudf 36

40 Example: TPC-H Query 14 Result γ agg 100 * SUM(CASE WHEN type LIKE PROMO% THEN extendedprice*(1-discount) ELSE 0 END) / SUM(extendedprice*(1-discount)) π partkey type, partkey SCAN part shipdate BETWEEN σ AND π SCAN shipdate, discount extendedprice, partkey lineitem 37

41 Trojan Columns User Database Application Query Processor Relations UDF Storage Layer Physical Representation File 1 File 2 File 3... File n Plug-and-play 38

42 Trojan Columns User Database Application Query Processor Relations UDF Storage Layer Physical Representation File 1 File 2 File 3... File n Quick Deployment 38

43 Trojan Columns User Database Application Query Processor Relations UDF Storage Layer Physical Representation File 1 File 2 File 3... File n Closed-source 38

44 Trojan Columns User Database Application Query Processor Relations UDF Storage Layer Physical Representation File 1 File 2 File 3... File n 38

45 Will this work?

46 Experimental Setup Commercial closed-source Row-store (Standard Row) Trojan Columns in commercial closed-source Row-store (Trojan Columns) Three variants of TPC-H benchmark: 1. simplified queries, simplified dataset 2. simplified queries, original dataset 3. original queries, original dataset

47 10 partsupp supplier nation Simplified 1 Queries, Simplified Dataset 1* Q1 Q2 Q3 Q4 Q5 Q6 Q7 region *** *** *** *** 10 Query Time (sec) 30 Q1 Q6 20 Q12 Q14 Q3 10 Q5 Q10 Q19 Q2 Q4 500 Standard Row Trojan Columns Trojan Columns (SP) Standard Row Trojan Standard Row Standard Row Trojan Columns Trojan Columns Q1 Q2 Q1 Q3 Q2 Q4 Q3 Q5 Q4 Q6 Q5 Q7 Q6 Q7 (a) Simplified queries, Simplified dataset * Mike Stonebraker et. al. C-Store: A Column Oriented DBMS. VLDB 2005 Query Time (sec) (b)

48 10 partsupp supplier nation Simplified 1 Queries, Simplified Dataset 1* Q1 Q2 Q3 Q4 Q5 Q6 Q7 region *** *** *** *** 10 Query Time (sec) 30 Q1 Q6 20 Q12 Q14 Q3 10 Q5 Q10 Q19 Q2 Q4 5x 500 Standard Row Trojan Columns Trojan Columns (SP) Standard Row Trojan Standard Row Standard Row Trojan Columns Trojan Columns Q1 Q2 Q1 Q3 Q2 Q4 Q3 Q5 Q4 Q6 Q5 Q7 Q6 Q7 (a) Simplified queries, Simplified dataset * Mike Stonebraker et. al. C-Store: A Column Oriented DBMS. VLDB 2005 Query Time (sec) (b)

49 1 375 Simplified Queries, Original Dataset Q1 Q2 Q3 Q4 Q5 Q6 Q7 0 Quer 500 Trojan Columns Trojan Columns (O) Trojan Columns (SP) Factor Factor Query Time (sec) Standard Row Trojan Columns Query Time (sec) Q1 Q2 Q3 Q4 Q5 Q6 Q7 (b) Simplified queries, Unmodified dataset (c) 42

50 1 375 Simplified Queries, Original Dataset Q1 Q2 Q3 Q4 Q5 Q6 Q7 0 Quer 500 Trojan Columns Trojan Columns (O) Trojan Columns (SP) Factor Factor Query Time (sec) Standard Row Trojan Columns x Query Time (sec) Q1 Q2 Q3 Q4 Q5 Q6 Q7 (b) Simplified queries, Unmodified dataset (c) 42

51 Original Queries, Original Dataset * The Good Queries Query Time (sec) Standard Row Trojan Columns 0 Q1 Q6 Q12 Q14 * tpch.org/tpch 43

52 Original Queries, Original Dataset * The Good Queries Query Time (sec) Standard Row Trojan Columns 0 * tpch.org/tpch Q1 Q6 Q12 Q14 9x 43

53 What are the trade-offs?

54 Original Queries, Original Dataset * The Bad Queries Query Time (sec) Standard Row Trojan Columns 0 Q3 Q5 Q10 Q19 * tpch.org/tpch 45

55 Micro-Benchmark: Improvement over Row-store # referenced attributes (r) E-06 1E-05 1E-04 1E-03 1E-02 1E-01 1E+00 selectivity (fraction of tuples accessed) gure 7: Trojan Columns improvement factor in DBMS 46

56 Micro-Benchmark: Improvement over Row-store Not Affected # referenced attributes (r) E-06 1E-05 1E-04 1E-03 1E-02 1E-01 1E+00 selectivity (fraction of tuples accessed) gure 7: Trojan Columns improvement factor in DBMS 46

57 Micro-Benchmark: Improvement over Row-store Not Affected # referenced attributes (r) E-06 1E-05 1E-04 1E-03 1E-02 1E-01 1E+00 selectivity (fraction of tuples accessed) Affected gure 7: Trojan Columns improvement factor in DBMS 46

58 How far are we?

59 Four Systems Commercial Row-store (Standard Row) Trojan Columns in commercial Row-store Commercial Row-store with vendor support for column technology (DBMS-Y) Commercial Column-store (DBMS-Z) (a) default TPC-H schema (b) tuned schema

60 TPC-H Benchmark Query Time (sec) Standard Row DBMS-Y DBMS-Z (b) Trojan Columns DBMS-Z (a) 0 Q1 Q6 Q12 Q14 49

61 TPC-H Benchmark Comparable or Better! Query Time (sec) Standard Row DBMS-Y DBMS-Z (b) Trojan Columns DBMS-Z (a) 0 Q1 Q6 Q12 Q14 49

62 TPC-H Benchmark Comparable or Better! Query Time (sec) Standard Row DBMS-Y DBMS-Z (b) Trojan Columns DBMS-Z (a) 0 Q1 Q6 Q12 Q14 Still to catch-up! 49

63 What about query optimization?

64 Rules out query optimization?

65 Rules out query optimization? NO!

66 Rules out query optimization? NO! QO with aggregate UDFs [SIGMOD 06] Manimal [WebDB 10] HadoopToSQL [EuroSys 10] Black box QO [VLDB 12]

67

68 The UDF Business Model

69 UDFs Not just for application-specific code Integrate core database functionality after the fact Column layouts are just one example! Meet customer demands quickly Provide quick feedback before new product release

70 Summary row-store slow good-enough fast performance

71 Summary row-store native column-store slow good-enough performance fast

72 Summary row-store Trojan Columns native column-store slow good-enough performance fast