σ (R.B = 1 v R.C > 3) (S.D = 2) Conjunctive normal form Topics for the Day Distributed Databases Query Processing Steps Decomposition

Transcription

1 Topics for the Day Distributed Databases Query processing in distributed databases Localization Distributed query operators Cost-based optimization C37 Lecture 1 May 30, Query Processing teps Decomposition Given QL query, generate one or more algebraic query trees Localization ewrite query trees, replacing relations by fragments Optimization Given cost model + one or more localized query trees Produce minimum cost query execution plan 3 Decomposition ame as in a centralized DBM Normalization (usually into relational algebra) elect,c From Natural Join Where (.B = 1 and.d = 2) or (.C > 3 and.d = 2) σ (.B = 1 v.c > 3) (.D = 2) Conjunctive normal form 1

2 Decomposition edundancy elimination (. = 1) (. > 5) False (. < 10) (. < 5). < 5 lgebraic ewriting Example: pushing conditions down σcond3 Localization teps 1. tart with query tree 2. eplace relations by fragments 3. Push up & π,σ down (C25 rules). implify eliminating unnecessary operations σcond Note: To denote fragments in query trees [: cond] σcond1 σcond2 T T 5 elation that fragment belongs to Condition its tuples satisfy 6 Example 1 Example 2 σe=3 σe=3 [: E<10] [: E 10] σe=3 σe=3 σe=3 [: E<10] [: E<10] [: E 10] 7 [: <5] [: 5 10] [: >10] [: <5] [: 5]

3 [:<5][:<5] [:5 10] [: 5] [:>10][: 5] 9 ules for Horiz. Fragmentation σ C1 [: C 2 ] [: C 1 C 2 ] [: False] Ø [: C 1 ] [: C 2 ] [ : C 1 C 2. =.] In Example 1: σ E=3 [ : E 10] [ : E=3 E 10] [ : False] Ø In Example 2: [: <5] [: 5] [ :. < =.] [ : False] Ø 10 Example 3 Derived Fragmentation s fragmentation is derived from that of [: <10] [: 10] [: =..<10] [: =.. 10] [: <10] [: =..<10] [: 10] [: =.. 10]

4 Example Vertical Fragmentation ule for Vertical Fragmentation Π Π (,,B) (,C,D) Π Given vertical fragmentation of (): i = Π i (), i For any B : Π B () = Π B [ i B i Ø] i Π (,,B) Π, Π, (,,B) (,C,D) 13 1 Parallel/Distributed Query Operations ort Basic sort ange-partitioning sort Parallel external sort-merge Join Partitioned join symmetric fragment and replicate join General fragment and replicate join emi-join programs ggregation and duplicate removal 15 Parallel/distributed sort Input: relation on single site/disk fragmented/partitioned by sort attribute fragmented/partitioned by some other attribute Output: sorted relation single site/disk individual sorted fragments/partitions 16

5 Basic sort Given (, ) range partitioned on attribute, sort on Each fragment is sorted independently esults shipped elsewhere if necessary 17 ange partitioning sort Given (,.) located at one or more sites, not fragmented on, sort on lgorithm: range partition on and then do basic sort a 0 a 1 3 Local sort s Local sort s Local sort 3s esult 18 electing a partitioning vector Possible centralized approach using a coordinator Each site sends statistics about its fragment to coordinator Coordinator decides # of sites to use for local sort Coordinator computes and distributes partitioning vector For example, tatistics could be (min sort key, max sort key, # of tuples) Coordinator tries to choose vector that equally partitions relation Example Coordinator receives: From site 1: Min 5, Max 9, 10 tuples From site 2: Min 7, Max 16, 10 tuples ssume sort keys distributed uniformly within [min,max] in each fragment Partition into two fragments What is k 0? k

6 Variations Different kinds of statistics Local partitioning vector Histogram ite # of tuples local vector Parallel external sort-merge Local sort Compute partition vector Merge sorted streams at final sites Multiple rounds between coordinator and sites ites send statistics Coordinator computes and distributes initial vector V ites tell coordinator the number of tuples that fall in each range of V Coordinator computes final partitioning vector V f Local sort Local sort s s In order a 0 a 1 3 esult Parallel/distributed join Partitioned Join Input: elations, May or may not be partitioned Output: esult at one or more sites Join attribute f() 3 Local join a b esult f() 23 Note: Works only for equi-joins 2 6

7 Partitioned Join ame partition function (f) for both relations f can be range or hash partitioning ny type of local join (nested-loop, hash, merge, etc.) can be used everal possible scheduling options. Example: partition ; partition ; join partition ; build local hash table for ; partition and join Good partition function important Distribute join load evenly among sites symmetric fragment + replicate join Join attribute Local join 3 f union Partition function esult a b 25 ny partition function f can be used (even round-robin) Can be used for any kind of join, not just equi-joins 26 General fragment + replicate join 1 m Partition n eplicate m copies n 1 ll n x m pairings of, fragments m a 1 1 n 1 n m b 2 eplicate n copies 2 esult symmetric F+ join is a special case of general F+. Partition m m 27 symmetric F+ is useful when is small. 28 7

8 emi-join programs Used to reduce communication traffic during join processing = ( ) = ( ) Example B C 2 a 10 b c d Compute Π () = [2,10,25,30] x y z w 32 x = ( ) ( ) ( ) = 10 y 25 w Using semi-join, communication cost = + 2 ( + C) + result Directly joining nd, communication cost = ( + B) + result Comparing communication costs ay is the smaller of the two relations nd ( ) is cheaper than if size (Π ) + size ( ) < size () imilar comparisons for other types of semi-joins Common implementation trick: Encode Π (or Π ) as a bit vector 1 bit per domain of attribute n-way joins To compute T emi-join program 1: T where = & = T emi-join program 2: T where = & = T everal other options In general, number of options is exponential in the number of relations

9 Other operations Example Duplicate elimination ort first (in parallel), then eliminate duplicates in the result Partition tuples (range or hash) and eliminate duplicates locally ggregates Partition by grouping attributes; compute aggregates locally at each site # dept sal 1 toy 10 2 toy 20 3 sales 15 # dept sal sales 5 5 toy 20 6 mgmt 15 7 sales 10 8 mgmt 30 # dept sal 1 toy 10 2 toy 20 5 toy 20 6 mgmt 15 8 mgmt 30 # dept sal 3 sales 15 sales 5 7 sales 10 sum sum dept sum toy 50 mgmt 5 dept sum sales 30 sum(sal) group by dept 33 3 Example Query Optimization # dept sal 1 toy 10 2 toy 20 3 sales 15 sum dept sum toy 30 toy 20 mgmt 5 sum dept sum toy 50 mgmt 5 Generate query execution plans (QEPs) Estimate cost of each QEP ($,time, ) Choose minimum cost QEP # dept sal sales 5 5 toy 20 6 mgmt 15 7 sales 10 8 mgmt 30 sum dept sum sales 15 sales 15 sum dept sum sales 30 Does this work for all kinds of aggregates? ggregate during partitioning to reduce communication cost What s different for distributed DB? New strategies for some operations (semi-join, range-partitioning sort, ) Many ways to assign and schedule processors ome factors besides number of IO s in the cost model

10 Cost estimation In centralized systems - estimate sizes of intermediate relations For distributed systems Transmission cost/time may dominate Work at site ccount for parallelism T1 Work at site T2 answer Plan B Plan 50 IOs 70 IOs 100 IOs 20 IOs Data distribution and result re-assembly cost/time 37 Optimization in distributed DBs Two levels of optimization Global optimization Given localized query and cost function Output optimized (min. cost) QEP that includes relational and communication operations on fragments Local optimization t each site involved in query execution Portion of the QEP at a given site optimized using techniques from centralized DB systems 38 earch strategies 1. Exhaustive (with pruning) 2. Hill climbing (greedy) 3. Query separation Exhaustive with Pruning fixed set of techniques for each relational operator earch space = all possible QEPs with this set of techniques Prune search space using heuristics Choose minimum cost QEP from rest of search space

11 > > T B T T T T T T ( ) T (T ) 1 2 hip to Example emi-join hip T to Prune because cross-product not necessary Prune because larger relation first emi-join 1 Hill Climbing 1 2 x Initial plan Begin with initial feasible QEP t each step, generate a set of new QEPs by applying transformations to current QEP Evaluate cost of each QEP in top if no improvement is possible Otherwise, replace current QEP by the minimum cost QEP from and iterate 2 Example Local search T V T V B C el. ite # of tuples T 3 30 V 0 Goal: minimize communication cost Initial plan: send all relations to one site To site 1: cost= = 90 To site 2: cost= = 80 To site 3: cost= = 70 To site : cost= = 60 Transformation: send a relation to its neighbor 3 Initial feasible plan P0: (1 ); (2 ); T (3 ) Compute join at site ssume following sizes: 20 T 5 T V 1 11

12 cost = No change Worse 20 cost = cost = cost = 50 T Improvement Improvement T T 5 20 T 2 3 cost = 35 cost = 25 6 Next iteration P1: (2 3); (1 ); α (3 ) where α = T Compute answer at site Now apply same transformation to nd α α 1 α 3 α 1 3 α esources Ozsu and Valduriez. Principles of Distributed Database ystems Chapters 7, 8, and