Redundancy Awareness in SQL Queries

Transcription

1 Redundancy Awareness in QL Queries Bin ao and Antonio Badia omputer Engineering and omputer cience Department University of Louisville Abstract In tis paper, we study QL queries wit aggregate subqueries tat sare common tables and conditions wit te outer query. Wile several approaces can deal wit suc queries, tey ave limited applicability. We propose te redundancy awareness metod to detect te largest common part sared by query and subquery, compute it once, and determine wat operations are needed to finis evaluation of te original query. ur approac can deal wit redundancy in all types of subqueries. We offer te possibility for te optimizer to coose te most efficient plan for a given query. We ave implemented our approac on top of a commercial DBM; our experiments sow tat our approac compares favorably to existing optimization tecniques. 1. Introduction ne powerful feature of QL is te use of subqueries wit aggregation, so tat wenever possible, aggregates are computed and used witin te same query. However, one of te limitations is tat suc queries usually sow a great deal of redundancy, tat is, te outer query and te subquery sare common tables and conditions. It is important to point out tat redundancy is present because of te structure of QL. In tis paper, we study te general idea of optimizing queries aving redundancy by focusing on te problem of queries wit aggregate subqueries. Altoug oter approaces ave addressed suc queries [3, 9], tey ave some limitations tat we aim to overcome. In particular, suc approaces only apply under limited circumstances, wile we present a general and efficient approac, called te redundancy awareness metod. We attack te redundancy problem directly by identifying te largest common part sared by query and subquery and executing tis common part only once. However, tis does not, per se, guarantee improved performance. Tus, we propose a new operator, called te for-loop, tat allows efficient computation of aggregates and conditions involving tem wit one pass over te common part. ur approac can deal wit redundancy in WHERE clause subqueries witout restrictions, and applies also to subqueries in te HAVIG clause. It also can be extended to non-aggregate subqueries via a rewriting of te subquery and multiple subqueries. Te nested relational approac aims to detect te largest redundancy present in te outer query and te subquery. However, saring te largest common part can not always yield te most efficient plan. ometimes, executing none or part of redundancy may ave better performance. Wat we do ere is to offer te possibility for te optimizer to coose te best plan. We present experimental evidence tat our approac, wen applicable, performs better tan oter optimizations proposed in te literature. 2. Related Work and Motivation A typical query aving redundancy is sown below: Query 1 : select s_acctbal,s_name,n_name, p_partkey,p_mfgr,s_address, s_pone,s_comment from part,supplier,partsupp,nation,region were p_partkey=ps_partkey and s_suppkey=ps_suppkey and p_size=15 and p_type like %BRA and s_nationkey=n_nationkey and n_regionkey=r_regionkey and r_name= EURPE and ps_supplycost = (select min(ps_supplycost from partsupp,supplier,nation,region were p_partkey=ps_partkey and s_suppkey=ps_suppkey and s_nationkey=n_nationkey and n_regionkey=r_regionkey and r_name= EURPE Te most noticeable feature of te above query is redundancy: te tables and conditions in te subquery are totally

2 - - - < included in tose in te outer query. As a query aving a correlated aggregate subquery, Query 1 can be unnested by most approaces proposed in te literature (e.g. [6]. However, suc approaces would not recognize te redundancy. Terefore, common tables and common conditions ave to be accessed and computed more tan once. ptimization of nested queries as received significant attention since te 198 s. However, te redundancy problem as not received attention until recently [5, 3, 9]. Rao and Ross [5] proposed te invariant tecnique wic implements te nested iteration metod wile considering invariants in te subquery. However, te invariant tecnique only as better performance for te nested queries tat can not be unnested by traditional unnesting strategies. Galindo- Legaria and Josi [3] presented te decorrelation tecnique used in Microsoft QL erver wic solves te redundancy problem using te egmentapply operator. Zuzarte et al. [9] introduced te Win tecnique to evaluate queries aving redundancy by making use of extended window aggregation capabilities. Wile tese tecniques [3, 9] can ave muc better performance tan traditional unnesting approaces (e.g.[6], bot tecniques only consider queries wit some restricted redundancy. As a more complex example, consider te following query: Query 2 : select sum(l_extendedprice/7. from lineitem, part, orders were p_partkey=l_partkey and p_size=15 and p_type like %BRA and l_sipdate>= and l_sipdate<l_commitdate and l_orderkey=o_orderkey and o_orderdate>= and o_orderdate< and l_quantity < (select.2*avg(l_quantity from lineitem, partsupp were l_partkey=p_partkey and l_commitdate<l_receiptdate and l_sipdate<l_commitdate and l_suppkey=ps_suppkey and ps_availqty>5 In Query 2, te outer query and te subquery sare common tables and common conditions, but extra tables and conditions are present in bot te outer query and te subquery. Furtermore, te join operation between partsupp and lineitem in te subquery is not a lossless join, wic is not witin te scope of te Win tecnique. Redundancy may also appear in subqueries in te HAV- IG clause, in non-aggregate subqueries, or in queries aving multiple subqueries. In [9], te autors indicate tat Win can andle non-aggregate subqueries, but detailed tecniques are not provided. In fact, dealing wit te T I or ALL subquery needs careful consideration due to null values. To te best of our knowledge, no existing tecnique adequately considers te redundancy present in suc queries. In tis paper, we propose a general approac, te redundancy awareness metod, to avoid redundant computation in QL queries. 3. Redundancy Awareness Metod Te basic idea of te redundancy awareness metod is straigtforward: te conditions and tables in te outer query and te subquery are rougly divided into tree parts: one tat is common in bot te outer query and te subquery, one tat belongs only to te outer query, and one tat belongs only to te subquery. Based on tese tree parts, a query can be evaluated as follows: first, we create a base relation based on common tables and common conditions; second, starting from te base relation, we compute te aggregation in te subquery based on te tables and conditions belonging only to te subquery; finally, we generate te desired result based on te tables and conditions belonging only to te outer query and te subquery result. To acieve tis goal, te base relation must be correctly identified. Furtermore, we introduce te for-loop operator to compute te subquery and te outer query based on te base relation. Wit an efficient implementation, te for-loop operator needs only one pass over te base relation to compute te subquery and te outer query Generating base relations For te purpose of our approac, we use a query pattern to represent a query. A query pattern is a scematic representation of a query using keywords ELET, FRM, WHERE (and optional GRUP BY, HAVIG, and variables over tables, aggregate functions and conditions. A typical one-level nested query aving redundancy wit a correlated aggregate subquery in te WHERE clause can be expressed by te query pattern sown below (we assume tat te desired result is an aggregate function. ELET FRM, "!, $%& WHERE ' (*$+& AD ' &!, AD '$%& AD (.$%& AD (. /&! AD 1!23!+465 (ELET 1!2 7 1!23 7 FRM, 8!9+ WHERE ' (:!99 AD '!99 AD (.!;+9 AD " 7 Were = denotes aggregate functions, >*??@ denotes attributes, A denotes tables, B denotes selection conditions, and D denote join conditions. bviously, Query 2

3 7 Q m : = m matces tis query pattern. Due to lack of space, we omit te details of ow to matc a query to its query pattern. Query structure detection, especially te common part detection, is similar to matcing part or all of queries to materialized views [7, 4]. How to detect common subqueries and exploit tem in complex query optimization as been studied by Zu, Tao and Zuzarte [8]. Te algoritm proposed in [8] can be modified and reused to implement our matcing tecnique. Te key conditions to ceck wen using te redundancy awareness metod are: (1 te set of common tables (A is not empty, and (2 correlated predicates (D and te join conditions between common tables and correlation tables in te outer query ( are te same (for correlated subqueries. Even toug te above conditions are satisfied, wat conditions are exactly common to te outer query and te subquery still need furter consideration, because te conditions involving te common tables in te outer query (B and te conditions involving te common tables in te subquery (B! " migt contain different conditions. We define te base relation, denoted by, as te common part (tables and conditions between te outer query and te subquery. To obtain te base relation, we compute: B&('* B +-, B! ", B&(' +. B / B 12! 3, B&('12! 3 4 B 12! 3 B /. learly, B&(' is used to compute te base relation; but B &(' and B&56'12! 3 must be applied at te rigt time, to produce te rigt subquery result and te rigt final result. Tus, te base relation can be simply obtained by performing selection B &(' on A (for noncorrelated subqueries, or by performing a join of A and A on B &(', B and D (for correlated subqueries. If tere are extra tables in te subquery except common tables, i.e. A 2! 3 is not empty, we ave to extend te base relation to te extended base relation, denoted by 7. Generally, te extended base relation is obtained by performing a left outer join of and A8! " on 12! 3. ote tat B 2! 3 can be pused down. Performing an outer join of and A 2! 3 migt cause tuples in 9$ duplicated if 12! 3 is not a lossless join. uc duplicates are required to compute te aggregation in te subquery, but will cause an error wen computing te final result. Te solution to tis problem is to use 7 to compute te aggregation in te subquery, and ten reduce 7 to to compute te outer query M P by performing : DEFGIH+H+JLK, were : denotes group-by, and denotes te primary key. Wit respect to te QL syntax, te grouping attributes migt include te primary key of as well as oter attributes required for later processing Te for-loop operator nce obtaining te (extended base relation (denoted by 9$ 7PR, were te square bracket denotes optional, te subsequent computations of te aggregation in te subquery and te linking predicate between te outer query and te subquery can be done efficiently by only one pass over te (extended base relation. In order to do so, we define a new operator, for-loop, wic combines several relational operators into a new one (i.e. a macro-operator. Tis approac is based on te observation tat some basic operations appear frequently togeter and tey could be more efficiently implemented as a wole, tus saving considerable disk I/. In te following, : is used to indicate a group-by operation, MX$P indicates te aggregation = computed and T:P:VU EW over all values of te attribute of te relation $. Definition 1 Let $ $P be a relation, ]\ $P L$4 Y Z6[ Y Z6[, _^`Y+Z+[ condition on $ (i.e. involving $Pdcfe only attributes g of Y Z6[ and b a condition on Y Z6[ te scema of $,, = an aggregate function, a a $P (i.e. possibly involving =. Ten te for-loop operator is defined in two variants: -i/a MA jka ll$p = U EW l. Te meaning of te operator is $on <?> ikp8i qi/a MX j L$P given by m U EW, were te condition of te join is understood as te pairwise equality of eac attribute in. Tis is called a grouped for-loop. = U l EW by m for-loop. t MA jka l/l$p T:P:PU EW. Te meaning of te operator is given M( j L$P*rs$P. Tis is called a flat Altoug algebraic expression is an important issue, it is not te only reason to define te for-loop operator. Tere are also significant implementation issues. To acieve te objective of computing several results at once wit a single pass over te data, te operator can be implemented as an iterator tat loops over te input implementing a simple program. Te basic idea is twofold: first, selections and groupings (eiter grouping alone or togeter wit aggregate calculations can be effectively implemented in one algoritm; second, and more important, in some cases computing an aggregation and using te aggregate result in a selection can be done at te same time. Tis is due to te beavior of some aggregates and te semantics of te conditions involved. For instance, a comparison of te type attr1 = min(attr2, were bot attr1 and attr2 are attributes of some can be efficiently implemented by a sequential pass Furtermore, wen a subquery is correlated, we use te grouped for-loop and aggregate

4 = g computation is done per group. Tat means tat results (and temporary results are only accumulated by group, and terefore all temporary information needed is likely to fit in memory and performance may be good even for cases were te condition is ard to compute. Te for-loop operator is similar to te egmentapply operator [3] and te window aggregation [9] in tat all of tese operators are used to do computation based on te common part between te outer query and te subquery. Bot te forloop operator and te egmentapply operator are extended relational operators wic ave to be implemented inside DBM. Te window aggregation is one of te analytic features defined as part of te AI QL 1999 standard and as been implemented in some commercial DBM. Tus, te query can be rewritten in QL wit te redundancy considered by using te window aggregation and te WITH clause wic declares te common part. Te tecnique described in section 3.1 can be used to compose te WITH clause. From te performance point of view, te for-loop operator may perform more efficient tan te oter two due to its efficient implementation for computing te linked aggregation, te linking predicate, and te final result, wic needs only one pass over te common part Evaluation plan Based on te (extended base relation and te for-loop operator, te execution plan for queries aving redundancy can be created by te following tree steps: 1. We create te (extended base relation Q 9$ 7PR. ote tat standard relational optimization tecniques can be applied to tis part. 2. For queries wit correlated subqueries, we apply a ia MA jka l/ Q 9$P grouped for-loop operator, = U EW 74R, to Q 74R. For queries wit non-correlated subqueries, MA ja l/ Q we P apply a flat for-loop operator,. According to te U E2W 74R, to Q 74R for-loop operator definition, is >??@ X (te attributes of te outer query in te correlated predicates; if te correlating attribute is not te primary key of A X2, sould be replaced by te primary key. = is =I2 ( >*??@6 6, a is B&56'12! 3, b is B&('/ >*??@+I X =I ( >??@+I2 (. 3. If A is empty, te final result can be obtained by projection on te desired attributes. If A= + is not empty, te final result can be obtained by performing a join of Q 74R and A + on followed by projection of te desired attributes Extensions We consider tree main extensions of te redundancy awareness metod to cover subqueries in te HAVIG clause, non-aggregate subqueries, and multi-level queries. ubqueries in te HAVIG clause. In QL, te HAVIG clause usually occurs after te GRUP BY clause. If te subquery is correlated, te correlating attributes must be te same as te attributes in te GRUP BY clause. Tus, te aggregation in te outer query and te aggregation in te subquery can be computed simultaneously. If te subquery is not correlated, te aggregation in te subquery is computed over a wole base relation. Te only ting tat is needed is to continue te computations, witout resetting, across groups. After eac group is computed, we can compare te result of te global aggregate so far to te new group, and proceed as in te regular case. on-aggregate subqueries. A non-aggregate subquery is linked to its outer query by one of te following operators: EXIT, T EXIT, I, T I, ME/AY, and ALL, were ^ e. uc queries can be rewritten as queries wit aggregate subqueries, in particular wit te UT aggregate. uc rewrites must be carefully specified for te ALL or T I subquery, toug, since approaces usually tougt to work fail in te presence of null values. Let te subquery be attr1 ALL (select attr2... Ten te query can be rewritten as (select count(attr2...=(select (count(attr2..., were te first subquery is exactly as it was in te original, and te second one is also te same as te original but as te predicate attr1 attr2 added to te WHERE clause (tis approac is basically equivalent to tat of [1]. Multi-level queries. Wen considering queries of arbitrary dept, we distinguis between linear queries (were tere is at most one subquery in any given level and tree queries (were in some level tere are two or more subqueries. It is easy to see tat te redundancy awareness metod can be extended to linear queries of any dept: te metod is applied from te bottom up, starting at te innermost subquery. nce tis step is done, one can proceed up te query until te outermost block. Tree queries can also be taken care of, but require some additional care. Assuming tat query block as two subqueries, and, we consider two cases: (1 If as some overlap wit and does not, can be processed by a traditional approac and can be processed wit te redundancy awareness metod. (2 If bot and

5 = U U ave overlap wit, we furter distinguis two cases: if and are bot correlated and ave te same correlating attribute wit, bot subqueries could be computed by a for-loop operator for furter efficiency. terwise, it is better to treat and separately, creating a subtree for-loop operator for eac one. For eac case, te for-loop operator needs to be extended to deal wit several aggregations simultaneously; eac one wit its own conditions. Te extended notation of te for-loop operator is: i A E EW MA j MA E ï 2A EW MA j MA > > > A l Q P 74R, were te square bracket denotes optional. Tis definition is very similar to te multidimensional join (MD operator defined in [1]. Unlike tat work, toug, te for-loop operator is aware of redundancy and works on te (extended base relation, wile te MD operator does not consider redundancy and ave to access common tables more tan once. Tus, te MD operator relies on furter optimization to recognize redundancy on its operation, wile we create and control suc redundancy. 4. Experiments To verify te efficiency of te redundancy awareness metod, we ave implemented it on top of a leading commercial DBM, wic we call ystem A. We created TP-H databases [2] at scale factors 1 and 1 (size of 1GB and 1GB respectively in ystem A. We run Query 1 and Query 2 on a buffer cace of size 32MB and 128M respectively. Te execution results are sown in figure 1 and figure 2. From tese two figures, we can see tat bot te redundancy awareness metod and Win [9] perform muc better tan magic decorrelation [6] by avoiding redundant computations. Furtermore, te redundancy awareness metod performs sligtly better tan Win. Detailed performance analysis and te results of oter test queries will be included in our full paper Win Metod (a 32M Win Metod Figure 1. Query (b 128M Metod (a 32M 5. onclusions Metod 1.5 Figure 2. Query (b 128M In tis paper, we propose te redundancy awareness metod to efficiently evaluate queries aving redundancy. Te results of our experiments sow tat our approac indeed outperforms traditional optimizations. Furtermore, our approac is potentially applicable to a wide range of queries. We are currently expanding our experiments to examine te impact of different amounts of redundancy. References [1] M.. Akinde and M. H. Bolen. Efficient computation of subqueries in complex olap. In Proceedings of te IDE onference, pages , 23. [2] T. P. P. ouncil. Te tpc- bencmark. ttp:// [3]. A. Galindo-Legaria and M. M. Josi. rtogonal optimization of subqueries and aggregation. In Proceedings of te AM IGMD onference, pages , 21. [4] J. Goldstein and P.-A. Larson. ptimizing queries using materialized views: a practical, scalable solution. In Proceedings of te IGMD onference, pages , 21. [5] J. Rao and K. A. Ross. Reusing invariants: a new strategy for correlated queries. In Proceedings of te AM IGMD onference, pages 37 48, [6] P. esadri, H. Piraes, and T. Y.. Leung. omplex query decorrelation. In Proceedings of te IDE onference, pages , [7] M. Zaarioudakis, R. ocrane, G. Lapis, H. Piraes, and M. Urata. Answering complex sql queries using automatic summary tables. In Proceedings of te IGMD onference, pages , 2. [8] Q. Zu, Y. Tao, and. Zuzarte. ptimizing complex queries based on similarities of subqueries. Information ystems, 8(3:35 373, 25. [9]. Zuzarte, H. Piraes, W. Ma, Q. eng, L. Liu, and K. Wong. Winmagic: ubquery elimination using window aggregation. In Proceedings of te AM IGMD onference, pages ,