Query Processing. high level user query. low level data manipulation. query processor. commands

Transcription

1 Query Processing high level user query query processor low level data manipulation commands 1

2 Selecting Alternatives SELECT ENAME FROM EMP,ASG WHERE EMP.ENO = ASG.ENO AND DUR > 37 Strategy A ΠENAME(σDUR>37 EMP.ENO=ASG.ENO (EMP ASG)) Strategy B ΠENAME(EMP ENO (σdur>37 (ASG))) Strategy B avoids Cartesian product, so is better 2

3 What is the Problem? Site 1 Site 2 ASG1=σENO E3 (ASG) Site 3 ASG2=σENO> E3 (ASG) EMP1=σENO E3 (EMP) result = EMP1 EMP2 EMP1 Site 4 EMP2 =EMP2 ASG1 Site 1 ASG1 =σdur>37(asg1) EMP2=σENO> E3 (EMP) result2=(emp1 EMP2) EMP2 Site 3 ENOASG1 Site 5 Result Site 5 Site 5 EMP1 =EMP1 Site 4 Site 2 ENOASG2 ENOσDUR>37(ASG1 ASG1) ASG1 ASG2 EMP1 EMP2 Site 1 Site 2 Site 3 Site 4 ASG2 ASG2 =σdur>37(asg2) 3

4 Cost of Alternatives Assume: size(emp) = 400, size(asg) = 1000 uniform distribution of tuples tuple access cost = 1 unit; tuple transfer cost = 10 units there are 20 ASG tuples with DUR>37 4

5 Strategy 1 produce ASG': (10+10) tuple access cost 20 transfer ASG' to the sites of EMP: (10+10) tuple transfer cost 200 produce EMP': (10+10) tuple access cost 2 40 transfer EMP' to result site: (10+10) tuple transfer cost 200 Total cost 460 5

6 Strategy 2 transfer EMP to site 5: 400 tuple transfer cost 4000 transfer ASG to site 5: 1000 tuple transfer cost produce ASG':1000 tuple access cost 1,000 join EMP and ASG': tuple access cost 8,000 Total cost 23,000 6

7 Optimization Objectives Minimize a cost function I/O cost + CPU cost + communication cost These might have different weights in different distributed environments Wide area networks communication cost will dominate low bandwidth low speed high protocol overhead most algorithms ignore all other cost components Local area networks communication cost not that dominant total cost function should be considered 7

8 Query Optimization Issues: Optimization Granularity Single query at a time cannot use common intermediate results Multiple queries at a time efficient if many similar queries decision space is much larger 8

9 Query Optimization Issues Statistics Relation cardinality size of a tuple fraction of tuples participating in a join with another relation Attribute cardinality of domain actual number of distinct values Common assumptions independence between different attribute values uniform distribution of attribute values within their domain 9

10 Query Optimization Issues Decision Sites Centralized single site determines the best schedule simple need knowledge about the entire distributed database Distributed cooperation among sites to determine the schedule need only local information cost of cooperation Hybrid one site determines the global schedule each site optimizes the local subqueries 10

11 Query Optimization Issues Network Topology Wide area networks (WAN) Local area networks (LAN) System (cluster) area networks (SAN) 11

12 Distributed Query Processing Methodology Calculus Query on Distributed Relations Query Decomposition GLOBAL SCHEMA Algebraic Query on Distributed Relations CONTROL SITE Data Localization FRAGMENT SCHEMA Fragment Query Global Optimization STATS ON FRAGMENTS Optimized Fragment Query with Communication Operations LOCAL SITES Local Optimization LOCAL SCHEMAS Optimized Local Queries 12

13 Query Decomposition Input : Calculus query on global relations Normalization Analysis Simplification Translation/Restructuring Also called: NFST (Normalization, Factorization, Semantic Analysis, Translation) 13

14 Normalization Lexical and syntactic analysis check validity (similar to compilers) check for attributes and relations type checking on the qualification Put into normal form Conjunctive normal form (p 11 p 12 p 1n ) (p m1 p m2 p mn ) Disjunctive normal form (p 11 p 12 p 1n ) (p m1 p m2 p mn ) OR's mapped into union AND's mapped into join or selection 14

15 Analysis Annotation with type information Scoping for attributes Produce dependency graph 15

16 Simplification Why simplify? Avoid unnecessary operations (run-time and compile-time) How? Use transformation rules elimination of redundancy idempotency rules p1 ( p1) false, p1 (p1 p2) p1, p1 false p1 application of transitivity use of integrity rules 16

17 Simplification Example SELECT TITLE FROM EMP WHERE EMP.ENAME = J. Doe OR (NOT(EMP.TITLE = Programmer ) AND (EMP.TITLE = Programmer OR EMP.TITLE = Elect. Eng. ) AND NOT(EMP.TITLE = Elect. Eng. )) SELECT TITLE FROM EMP WHERE EMP.ENAME = J. Doe 17

18 Translation/Restructuring Convert relational calculus to relational algebra Example ΠENAME σdur=12 OR DUR=24 Find the names of employees other than J. Doe who worked on the CAD/CAM project for either 1 or 2 years. σpname= CAD/CAM SELECT ENAME σename J. DOE FROM EMP, ASG, PROJ WHERE EMP.ENO = ASG.ENO AND ASG.PNO = PROJ.PNO AND ENAME J. Doe AND PNAME = CAD/CAM AND (DUR = 12 OR DUR = 24) Project Select PNO Join ENO PROJ ASG EMP 18

19 Canonical Translation π A1,..., An σ P select distinct A1,..., An from R1,..., Rk where P Rk R3 R1 R2 19

20 Equivalences 1. (R ) ( (σcn(r )) )) σc1 c2... cn σc1(σc2 2. ((R )) (σc1((r )) σc1(σc2 σc2 3. πl1(π ( (π Ln(R )) )) (R ) L2 πl1 if L1 L2... Ln sch(r) 4. πa1,, (σc(r )) An σc (πa1,, An(R )) if C uses only A1,, An 5.,, and A are commutative 20

21 Equivalences 6. σc(r Aj S) σc(r) Aj S if c uses only attributes of R Combne with other rules for σ, e.g. σc1 c2(r A j S) σc1(r) A j (σc2(s)) 21

22 Equivalences 7. πl (R Ac S) πl sch(r) (R) Ac πl sch(s)(s) if C uses only attributes from L else (C set of attributes used in C) πl(racs) πl(π(l C) sch(r)(r) Ac π(l C) sch(s)(s)) (applicable to with C=true) 22

23 Equivalences 8. For Φ {A,,, }: (R Φ S) Φ T R Φ (S Φ T) (Associativity) 9. For Φ {,, }: σc(r Φ S) σc(r) Φ σc(s) 10. πc(r S) πc (R) πc (S) 23

24 Equivalences 11. (c1 c2) ( c1) ( c2) (c1 c2) ( c1) ( c2) 12. σc(r S ) R Ac S 24

25 Restructuring Heuristics 1. Split predicate use Rule 1 2. Use Rules 2, 4, 6, 9 to push selections down 3. Form joins using Rule Use 3, 4, 7, 10 to push projectons 5. Reduce number of operators using Rules 1,3 25

26 Example (Translation) ΠENAME SELECT σpname= CAD/CAM Recall the previous example: Find the names of employees other than J. Doe who worked on the CAD/CAM project for either one or two years. (DUR=12 DUR=24) ENAME J. DOE WHERE ASG.ENO=EMP.ENO SELECT ENAME ASG.PNO=PROJ.ENO FROM PROJ, ASG, EMP WHERE ASG.ENO=EMP.ENO AND ASG.PNO=PROJ.PNO AND ENAME J. Doe AND PROJ.PNAME= CAD/CAM AND (DUR=12 OR DUR=24) PROJ ASG FROM EMP 26

27 Restructuring Π ENAME PNO Π PNO,ENAME ENO Π PNO Π PNO,ENO Π PNO,ENAME σ PNAME = "CAD/CAM" σ DUR =12 DUR=24 σ ENAME "J. Doe" PROJ ASG EMP 27

28 Distributed Query Processing Methodology Calculus Query on Distributed Relations Query Decomposition GLOBAL SCHEMA Algebraic Query on Distributed Relations CONTROL SITE Data Localization FRAGMENT SCHEMA Fragment Query Global Optimization STATS ON FRAGMENTS Optimized Fragment Query with Communication Operations LOCAL SITES Local Optimization LOCAL SCHEMAS Optimized Local Queries 28

29 Step 2 Data Localization Input: Algebraic query on distributed relations Determine which fragments are involved Localization program substitute for each global relation its reconstruction expression optimize 29

30 Example Π Assume ENAME σdur=12 OR DUR=24 EMP is fragmented into EMP1, EMP2, EMP3 as follows: EMP1=σENO E3 (EMP) EMP2= σ E3 <ENO E6 (EMP) EMP3=σENO E6 (EMP) σpname= CAD/CAM σename J. DOE ASG fragmented into ASG1 and ASG2 as follows: PNO ASG1=σENO E3 (ASG) ENO ASG2=σENO> E3 (ASG) Replace EMP by (EMP1 EMP2 EMP3 ) and (ASG1 ASG2) in any query PROJ ASG by EMP1 EMP2 EMP3 ASG1 ASG2 30

31 Provides Parallellism... ENO EMP1 ASG1 ENO EMP2 ASG2 ENO EMP3 ASG1 ENO EMP3 ASG2 31

32 Eliminates Unnecessary Work ENO EMP1 ENO ENO ASG1 EMP2 ASG2 EMP3 ASG2 32

33 Reduction for PHF Reduction with selection Relation R and FR={R1, R2,, Rw} where Rj=σ pj(r) σ p (Rj)= if x in R: (pi(x) pj(x)) i Example σ ENO= E5 EMP1 EMP2 SELECT * FROM EMP σ ENO= E5 WHERE ENO= E5 EMP3 EMP2 33

34 Reduction for PHF Reduction with join Possible if fragmentation is done on join attribute Distribute join over union (R 1 R 2 ) S (R 1 S) (R 2 S) Given R i = σ pi (R) and R j = σ pj (R) R i R j = if x in R i, y in R j : (p i (x) p j (y)) 34

35 Reduction for PHF Reduction with join - Example Assume EMP is fragmented as before and ASG1: σeno "E3"(ASG) ASG2: σeno > "E3"(ASG) Consider the query SELECT* FROM EMP, ASG WHERE EMP.ENO=ASG.ENO ENO EMP1 EMP2 EMP3 ASG1 ASG2 35

36 Reduction for PHF Reduction with join - Example Distribute join over unions Apply the reduction rule ENO EMP1 ENO ASG1 EMP2 ENO ASG2 EMP3 ASG2 36

37 Reduction for VF Find useless (not empty) intermediate relations Relation R defined over attributes A = {A1,..., An} vertically fragmented as Ri = ΠA' (R) where A' A: ΠD,K(Ri) is useless if the set of projection attributes D is not in A' Example: EMP1= ΠENO,ENAME (EMP); EMP2= ΠENO,TITLE (EMP) SELECT ENAME FROM EMP ΠENAME ΠENAME ENO EMP1 EMP2 EMP1 37

38 Reduction for DHF Rule : Distribute joins over unions Apply the join reduction for horizontal fragmentation Example ASG1: ASG ENO EMP1 ASG2: ASG ENO EMP2 EMP1: σ TITLE= Programmer (EMP) EMP2: σ TITLE Programmer (EMP) Query SELECT * FROM EMP, ASG WHERE ASG.ENO = EMP.ENO AND EMP.TITLE = Mech. Eng. 38

39 Reduction for DHF Generic query ENO σ TITLE= Mech. Eng. ASG 1 Selections first ASG 2 EMP 1 EMP 2 ENO σ TITLE= Mech. Eng. ASG 1 ASG 2 EMP 2 39

40 Reduction for DHF Joins over unions ENO ENO σ TITLE= Mech. Eng. σ TITLE= Mech. Eng. ASG 1 EMP 2 ASG 2 EMP 2 Elimination of the empty intermediate relations (left sub-tree) ENO σ TITLE= Mech. Eng. ASG 2 EMP 2 40

41 Reduction for HF Combine the rules already specified: Remove empty relations generated by contradicting selections on horizontal fragments; Remove useless relations generated by projections on vertical fragments; Distribute joins over unions in order to isolate and remove useless joins. 41

42 Reduction for HF ΠENAME Example Consider the following hybrid fragmentation: EMP1=σENO "E4" (ΠENO,ENAME (EMP)) σ ENO= E5 EMP2=σENO>"E4" (ΠENO,ENAME (EMP)) ENO EMP3= ΠENO,TITLE (EMP) and the query SELECT FROM ΠENAME ENAME σ ENO= E5 EMP2 EMP WHERE ENO= E5 EMP1 EMP2 EMP3 42

43 Distributed Query Processing Methodology Calculus Query on Distributed Relations Query Decomposition GLOBAL SCHEMA Algebraic Query on Distributed Relations CONTROL SITE Data Localization FRAGMENT SCHEMA Fragment Query Global Optimization STATS ON FRAGMENTS Optimized Fragment Query with Communication Operations LOCAL SITES Local Optimization LOCAL SCHEMAS Optimized Local Queries 43

44 Step 3 Global Query Optimization Input: Fragment query Find the best (not necessarily optimal) global schedule Minimize a cost function Distributed join processing Bushy vs. linear trees Which relation to ship where? Ship-whole vs ship-as-needed Decide on the use of semijoins Semijoin saves on communication at the expense of more local processing. Join methods nested loop vs ordered joins (merge join or hash join) 44

45 Join Ordering Consider two relations only if size (R) < size (S) R S if size (R) > size (S) Multiple relations more difficult because too many alternatives. Compute the cost of all alternatives and select the best one. Necessary to compute the size of intermediate relations which is difficult. Use heuristics 45

46 Join Ordering Example Consider PROJ PNOASG ENOEMP Site 2 ASG ENO EMP Site 1 PNO PROJ Site 3 46

47 Join Ordering Example 1. EMP Site 2 2. ASG Site 1 Site 2 computes EMP'=EMP ASG Site 1 computes EMP'=EMP ASG EMP' Site 3 EMP' Site 3 Site 3 computes EMP PROJ Site 3 computes EMP PROJ 3. ASG Site 3 4. PROJ Site 2 Site 3 computes ASG'=ASG PROJ Site 2 computes PROJ'=PROJ ASG ASG' Site 1 PROJ' Site 1 Site 1 computes ASG' EMP Site 1 computes PROJ' EMP 5. EMP Site 2 PROJ Site 2 Site 2 computes EMP PROJ ASG 47

48 Semijoin Algorithms Consider the join of two relations: R[A] (located at site 1) S[A] (located at site 2) Alternatives: 1 Do the join R A S 2 Perform one of the semijoin equivalents R A S (R R (R A A S) (S A S) A A S R) A (S A R) 48

49 Semijoin Algorithms Perform the join send R to Site 2 Site 2 computes R S A Consider semijoin (R A S) A S S' A(S) S' Site 1 Site 1 computes R' = R A S' R' Site 2 Site 2 computes R' A S Semijoin is better if size(πa(s)) + size(r A S)) < size(r) 49

50 Step 4 Local Optimization Input: Best global execution schedule Select the best access path Use the centralized optimization techniques 50