Columnstore and B+ tree. Are Hybrid Physical. Designs Important?

Transcription

1 Columnstore and B+ tree Are Hybrid Physical Designs Important? 1

2 B+ tree 2

3 C O L B+ tree 3

4 B+ tree & Columnstore on same table = Hybrid design 4? C O L C O L B+ tree B+ tree

5 ? C O L C O L B+ tree B+ tree Are Hybrid Designs important and which workloads can benefit? 5

6 Hybrid design = B+ tree & Columnstore C O L B+ tree 6

7 Hybrid design = B+ tree & Columnstore

8 Selectivity Sort order Updates Mix: Scans & Updates Concurrency B+ tree C O L B+ tree C O L 8

9 9

10 0 8.00E E Execution time (millisec) 100,000 10,000 1, SELECT sum(col1) FROM table WHERE col1 < {1} 10 GB, 1 int col B+ tree hot Selectivity (%) 10

11 0 8.00E E Execution time (millisec) 100,000 10,000 1, SELECT sum(col1) FROM table WHERE col1 < {1} 10 GB, 1 int col 0.2% B+ tree hot Selectivity (%) Skip data effectively & run single-threaded 11

12 0 8.00E E Execution time (millisec) 100,000 10,000 1, SELECT sum(col1) FROM table WHERE col1 < {1} 10 GB, 1 int col 0.03% Col hot B+ tree hot Selectivity (%) Superior performance of Columnstore scans 12

13 0 8.00E E Execution time (millisec) 100,000 10,000 1, SELECT sum(col1) FROM table WHERE col1 < {1} 10 GB, 1 int col 0.03% Col cold Col hot B+ tree cold B+ tree hot Selectivity (%) B+ tree helps for low selectivity & slower storage 13

14 0 8.00E E Execution time (millisec) 100,000 10,000 1, SELECT sum(col1) FROM table WHERE col1 < {1} 10 GB, 1 int col 0.03% 10% Col cold B+ tree cold Col hot B+ tree hot Selectivity (%) B+ tree helps for low selectivity & slower storage 14

15 Execution time (millisec) 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 SELECT col1, sum(col2) FROM table GROUP BY col1 20 GB, 2 int col, vary number of distinct values in col1 B+ tree sorted on col1 B+ tree Col ,000 1,000,000 # of groups 15

16 Execution time (millisec) 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 SELECT col1, sum(col2) FROM table GROUP BY col1 20 GB, 2 int col, vary number of distinct values in col1 B+ tree sorted on col1 B+ tree Col ,000 1,000,000 # of groups Scanning & hashing Col faster than reading sorted B+ tree 16

17 Execution time (millisec) 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 SELECT col1, sum(col2) FROM table GROUP BY col1 20 GB, 2 int col, vary number of distinct values in col1 B+ tree sorted on col1 B+ tree Col ,000 1,000,000 # of groups Sort order of B+ tree beneficial for scarce query memory 17

18 Primary Columnstores 18

19 Secondary Columnstores Key not in Delete buffer 19

20 Execution time (millisec) Update top 10 rows 1,000, ,000 10,000 1, Hybrid design Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col TPC-H GB, concurrent queries, Read Committed scan: 0,update: 100 scan: 1, update: 99 scan: 5, update: 95 Percentage (%) for scans and updates 20

21 Execution time (millisec) Update top 10 rows 1,000, ,000 10,000 1, Hybrid design Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col TPC-H GB, concurrent queries, Read Committed scan: 0,update: 100 scan: 1, update: 99 scan: 5, update: 95 Percentage (%) for scans and updates B+ trees cheaper than Columnstores for pure updates 21

22 Total execution time (millisec) Update top 10 rows, Select sum of quantity & price for a single shipdate from lineitem table 1,000, ,000 10,000 1, Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col TPC-H GB, concurrent queries, Read Committed scan: 0,update: 100 scan: 1, update: 99 scan: 5, update: 95 Percentage (%) for scans and updates Secondary Columnstore: balance small updates & large scans 22

23 Hybrid design = B+ tree & Columnstore 1. Micro-benchmarks

24 Database Server Workload, Constraints (e.g. storage budget) DTA what-if Query Optimizer Output Create Index Drop Index 24

25 Database Server Workload, Constraints (e.g. storage budget) DTA what-if Query Optimizer Output Create Index Drop Index 25

26 Database Server Workload, Constraints (e.g. storage budget) DTA what-if Query Optimizer Output Create Index Drop Index 26

27 Database Server Workload, Constraints (e.g. storage budget) DTA what-if Query Optimizer Output Create Index Drop Index 27

28 1. Costing queries on Hypothetical Columnstores 28

29 1. Costing queries on Hypothetical Columnstores 2. Per-column size estimation - stay within storage budget - per-column access cost - hard problem 29

30 1. Optimal designs searched over: combined space of Columnstores & B+ trees 30

31 1. Optimal designs searched over: combined space of Columnstores & B+ trees 2. Released as part of SQL Server 2017 CTP 31

32 Hybrid design = B+ tree & Columnstore 1. Micro-benchmarks 2. Auto Recommend Hybrid Designs

33 DTA 33

34 Percentage of query plans 100% 80% 60% 40% 20% 0% Cust1 Cust2 Cust3 Cust4 Cust5 TPC-DS Hybrid Query Plans 34

35 Percentage of query plans 100% 80% 60% 40% 20% 0% Cust1 Cust2 Cust3 Cust4 Cust5 TPC-DS Hybrid Query Plans Hybrid Query Plans (Same Table) 35

36 # of Queries TPC-DS benchmark 100 GB Hybrid Vs B+ tree only Hybrid Vs Columnstore only >10 Bins of Speedup (CPU time)

37 # of Queries TPC-DS benchmark 100 GB Hybrid Vs B+ tree only Hybrid Vs Columnstore only >10 Bins of Speedup (CPU time)

38 # of Queries TPC-DS benchmark 100 GB Hybrid Vs B+ tree only Hybrid Vs Columnstore only >10 Bins of Speedup (CPU time)

39 # of Queries >10 Bins of Speedup (CPU time) Hybrid designs significantly improve decision support workload 39

40 20X 10X hash join concatenate Columnstore-only design Col scan date_dim Col scan web_sales Col scan catalog_sales nestedloop join nested-loop join Hybrid design B+tree seek date_dim B+tree seek web_sales B+tree seek date_dim B+tree seek catalog_sales

41 No. of Queries Hybrid Vs B+ tree only (Snapshot Isolation) 11 1K warehouses >10 Bins of Speedup (wall-clock time) 41

42 No. of Queries Hybrid Vs B+ tree only (Snapshot Isolation) 11 1K warehouses >10 Bins of Speedup (wall-clock time) 2X slowdown of transactions & 10X speed-up of analytics 42

43

44 10X DTA 44

45

46

47 47

48 C O L VECTORIZED COMPRESSED LATE MATERIALIZATION Selectivity values Sortedness SORTED GLOBALLY FAST MODIFICATIONS MEMORY EFFICIENT B+ tree Concurrency 48

49 point lookups & short scans streaming via sortedness B+ tree balance scans & updates Leverage both B+ tree & Col C O L large & fast scans Batch-mode & compression 49

50 50

51 A B A B A B A 0, 1 1, 1 3, 4 B 0, 3 1, GEE estimator groups that occur once in the sample are scaled by total_size / sample_size (e.g. [0,1] and [1,1]), other groups are counted once in total 51

52 52

53 Workload DB Size (GB) # of tables Max table size (GB) Avg. # of cols per table # of queries Avg. # of joins Avg. # of ops per plan Cust Cust Cust Cust Cust TPC-DS TPC-CH

54 54

55 55

56 Key not in Delete buffer 57

57 B+ tree C O L SORTED B+ TREE PRIMARY B+ TREE & SECONDARY COLUMNSTORE NOT SORTED GLOBALLY COLUMNSTORE COLUMNSTORE PRIMARY STORAGE & SECONDARY B+ TREE B+ tree C O L 58

58 B+ tree C O L short range scans & lookups Balance updates & scans large scans & bulk updates scarce memory Mixed workload small storage footprint 59

59 B+ trees Secondary CSIs Hybrid physical designs C O L B+ tree 60

60 61

61 Large data & design space Sample based techniques Complex storage A A A A , , Input Encode Sort Compress 1) Build index on samples 2) Model full index 62

62 Build index on samples Model full index 63

63 64

64 Execution time (millisec) 1,000, ,000 10,000 1,000 Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col UPDATE N rows lineitem WHERE l_shipdate = {1} TPC-H 30 GB, single thread % of updated rows Primary & Secondary Columnstores comparable for large updates 65

65 Percentage of queries 100% 80% 60% 40% 20% 0% Cust1 Cust2 Cust3 Cust4 Cust5 TPC-DS Hybrid Plans (Same Query) Hybrid Plans (Same Table) 66

66 Execution time (millisec) 1,000, ,000 10,000 1, X Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col UPDATE N rows lineitem WHERE l_shipdate = {1} TPC-H 30 GB, single thread % of updated rows Primary Columnstores incur high cost for small updates 67

67 Execution time (millisec) 1,000, ,000 10,000 1,000 Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col UPDATE N rows lineitem WHERE l_shipdate = {1} TPC-H 30 GB, single thread % of updated rows Cheaper updates for B+ trees than for Columnstores 68

68 Large data & design space Sample based techniques Complex Columnar storage A A A A 2, 3 3, 2 Input Encode Sort Compress At least as hard as distinct value estimation 69

69 Percentage of query plans 100% 80% 60% 40% 20% 0% Cust1 Cust2 Cust3 Cust4 Cust5 TPC-DS Hybrid Query Plans Hybrid Qeury Plans (Same Table) 70

70 Large data & design space Sample based techniques Complex Columnar storage A A A A 2, 3 3, 2 Input Encode Sort Compress At least as hard as distinct value estimation 71

71 Execution time (millisec) Update top 10 rows, Select sum of quantity & price for a single shipdate from lineitem table 1,000, ,000 10,000 1, TPC-H 30 GB, 10 concurrent queries, Read Committed scan: 0,update: 100 Pri. B+ tree Pri. B+ tree with Sec. Col Pri. Col scan: 1, update: 99 scan: 2, update: 98 scan: 3, update: 97 Percentage (%) for scans and updates scan: 4, update: 96 scan: 5, update: 95 Secondary Columnstore: balance small updates & large scans 72

72 Concept Access Path Selection B+ tree Main memory optimized Hybrid designs General (diskbased) Scans Shared Non-shared DB engine Prototype SQL Server Main focus Model Concurrency DTA and many workloads Physical designs Columnstore and Secondary B+tree Hybrid Physical Designs 73

73 Primary B+ tree (base table) Secondary Columnstore Secondary B+tree on ship date Heap file base table Secondary Columnstore Secondary B+tree on ship date B+ tree clustered on order & linenumber Covers all columns HEAP FILE Covers all columns Ship date Ship date 74

74 Primary B+ tree (base table) Secondary Columnstore Secondary B+ tree on ship date Primary Columnstore Secondary B+ tree on order/line Secondary B+ tree on ship date B+ tree clustered on order & linenumber Covers all columns Covers all columns B+ tree on order & linenumber Ship date Ship date 75

75 Workload 1) Select candidates C what-if DB Engine Query Optimizer Constraints - Storage budget - 2) Merge Indexes 3) Enumerate DTA O S T Output Create Index Drop Index Create View 76

76 Workload 1) Select candidates C what-if DB Engine Query Optimizer Constraints - Storage budget - 2) Merge Indexes 3) Enumerate DTA O S T Output Create Index Drop Index Create View 77

77 Workload 1) Select candidates C what-if DB Engine Query Optimizer Constraints - Storage budget - 2) Merge Indexes 3) Enumerate DTA O S T Output Create Index Drop Index Create View 78

78 Workload 1) Select candidates C what-if DB Engine Query Optimizer Constraints - Storage budget - 2) Merge Indexes 3) Enumerate DTA O S T Output Create Index Drop Index Create View 79

79 Primary Columnstore & Secondary B+ tree index: - enforce constraints efficiently (e.g. Primary Key) Source: Real-Time Analytical Processing with SQL Server Paul Larson, Adrian Birka, Eric Hanson, Weiyun Huang, Michal Novakiewicz, Vassilis Papadimos (Microsoft) 80