Automatic Physical Design Tuning: Workload as a Sequence Sanjay Agrawal Microsoft Research One Microsoft Way Redmond, WA, USA +1-(425)

Transcription

1 Automati Physial Design Tuning: Workload as a Sequene Sanjay Agrawal Mirosoft Researh One Mirosoft Way Redmond, WA, USA +1-(425) sagrawal@mirosoft.om Eri Chu * Computer Sienes Department University of Wisonsin-Madison Madison, WI, USA +1-(68) eri@s.wis.edu Vivek Narasayya Mirosoft Researh One Mirosoft Way Redmond, WA, USA +1-(425) viveknar@mirosoft.om ABSTRACT The area of automati seletion of physial database design to optimize the performane of a relational database system based on a workload of SQL queries and updates has gained prominene in reent years. Major database vendors have released automated physial database design tools with the goal of reduing the total ost of ownership. An important assumption underlying these tools is that the workload is a set of SQL statements. In this paper, we show that being able to treat the workload as a sequene, i.e., exploiting the ordering of statements an signifiantly broaden the usage of suh tools. We present senarios where exploiting sequene information in the workload is ruial for performane tuning. We also propose tehniques for addressing the tehnial hallenges arising from treating the workload as a sequene. We evaluate the effetiveness of our tehniques through experiments on Mirosoft SQL Server. 1. INTRODUCTION Database vendors suh as IBM, Mirosoft and Orale offer automated physial design tuning tools. Database Tuning Advisor (DTA) in SQL Server [1], Design Advisor [16] in IBM DB2 and SQL Aess Advisor [7] in Orale 1g automate the task of finding the best physial design strutures (e.g., indexes and materialized views) to optimize server performane. These tuning tools require a workload omprised of queries and updates to arrive at a physial design reommendation. All these tools are set-based the workload is viewed as a set of statements and no ordering of statements is assumed during tuning. The premise of this paper is that the ordering of statements an be important for performane tuning, speifially for physial database design. To illustrate this, we desribe three senarios in the ontext of physial database design where the set-based tuning approah falls short, and an alternative approah that exploits workload sequene information an lead to muh superior workload performane. Senario 1: Data warehousing: Query by day, update at Permission to make digital or hard opies of all or part of this work for personal or lassroom use is granted without fee provided that opies are not made or distributed for profit or ommerial advantage and that opies bear this notie and the full itation on the first page. To opy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speifi permission and/or a fee. Conferene 6, June 26 29, 26, Chiago, IL, USA. Copyright 26 ACM //4 $5.. night. During the day there are multiple appliations that issue omplex queries against the warehouse. At night there is a bath window during whih the warehouse data is updated, e.g., new data is inserted. Figure 1 below aptures the ordering information in data warehouses. Queries Updates Queries Figure 1. Data warehousing senario. If we tune the workload that inludes both queries and updates as a single set using a set-based approah, it is quite possible that we do not get any physial design struture to be reommended that benefit the workload as a whole. This is beause, although suh a tool may identify strutures that speed up the queries (in the day), the update ost inurred (in the night) for the strutures may far outweigh their benefit. On the other hand, if we treat the workload as a sequene, we may reommend the following: reate strutures before the queries arrive and drop suh strutures before the updates arrive. Suh a reommendation gives us the benefit of strutures for queries but without the update overhead. If the benefit of suh strutures is greater than their reation ost, the data warehouse senario an be optimized for performane as shown in Figure 2. Note that performane improvement arises from the fat that the strutures inur no maintenane ost sine they are dropped prior to updates. Create Indexes Drop Indexes Queries Updates Create Indexes Queries Figure 2. Optimized data warehousing senario. Observe that the obvious approah of breaking the workload into two workloads (the query workload and update workload respetively) and tuning eah without awareness of the other an lead to sub-optimal performane. This is due to the fat that physial design reommendations for eah workload an be very different and the ost of transitioning between the two reommendations an be signifiant. In the above example, indexes need to be reated for the queries and dropped for the *Work done when the author was visiting Mirosoft Researh.

2 updates. The reation and dropping of indexes an beome very expensive at times (e.g., the drop of lustered index on a large table may internally lead to rereation of all non-lustered indexes on that table). Therefore, the ost of physial design transitions must be inluded in the analysis for ahieving optimal performane. Similarly, a strategy that tunes physial design only for queries and ignores updates while tuning (thus there is no transition ost sine the physial design does not need to hange) an also be sub-optimal sine the ost of updating physial design strutures an be substantial. Senario 2: SQL appliations that use transient tables. // Step 1: reate table CREATE TABLE #t (multiple olumns TYPES) // Step 2: populate table INSERT INTO #t SELECT olumns FROM Y WHERE olumn = value //Step 3: use table in multiple queries SELECT X.*,#t.* FROM X INNER JOIN #t ON X.CUSTID = # t.custid WHERE X. PRODUCTID = value ORDER BY #t.price DESC //Step 4: drop table DROP TABLE #t Figure 3. Transient table usage senario Many appliations use transient or temporary tables in the manner as illustrated in Figure 3. Again one an reommend strutures on transient tables by leveraging the knowledge of how suh tables are used. In this senario, after Step 2 and before Step 3, one an reate an appropriate index on the transient table to improve the performane of the SELECT statement. This kind of tuning an not be ahieved by set-based tuning tools sine reognizing the sequene of CREATE TABLE (this marks the start of life of transient tables), followed by INSERT then SELECT and finally DROP (this marks the end of life of transient tables) is important. For example, reation of the index just before Step 2 would not be optimal sine the index would inur update ost but would yield no benefit in Step 2. Senario 3: Periodi data hange on prodution server. A ommon senario in appliations is where data pertaining to a ertain time period is available on the prodution server for a speified duration. For example, the sales table may ontain data for the urrent quarter. The data gets populated as follows. At the end of the urrent quarter, all rows from the table are deleted. Subsequently data for the new quarter is inserted into the table. New data gets added at the end of eah day from different sales soures. Meanwhile, there are queries that run against this data during the entire time period. Figure 4 aptures this senario. Create Index Trunate table Drop Index Insert Queries Insert Queries Data Data Figure 4. Periodi data hange Trunate table Note that even though the set of queries and updates an remain the same, the same physial design may be totally ineffetive for a signifiantly different data size and/or distribution. For example, non-lustered indexes may beome ineffetive for seeks if seletivity beomes large. Overheads of inserting rows and benefits of physial design strutures are inherently tied to table sizes. A sequene-based approah in onjuntion with additional database statistis that apture the dynami nature of data, an make the better trade offs for suggesting reate/drop of physial design strutures to optimize performane as ompared to a setbased approah. The above senarios highlight the fat that exploiting the order between statements an be ruial for improving performane. In pratie, rarely is the workload either a single set or a single sequene of statements. A more general model of a workload is a sequene of sets of statements. Let us see how eah of the above senarios fits this model. For senario 1 above, we may treat the workload as alternate sets of queries (during the day) and updates (during the night). Similarly for senario 3, sets of queries and inserts alternate in the workload. Note that in both ases the set of queries alternate with the set of updates and this defines a sequene. Likewise eah statement in senario 2 an be viewed as a single statement set, the sets being ordered naturally as the order of steps above. In the rest of this paper, for simpliity of exposition, we assume that workload is a sequene of sets where eah set is a single query/update. This makes it easier to understand the problem spae and reason about our solution. In Setion 7 we briefly disuss how our solution extends to the generalized model where the set ontains multiple statements. It is important to note that the output model of sequene-based tuning is different from set-based tuning solutions. Unlike a setbased approah where the output is a single SQL sript with reates/drops of strutures, for a sequene tuning tool the output ontains reate and drop of strutures interleaved with the input workload. For example, in Senario 2 above, the output would be reate index on transient table between the INSERT statement (Step 2) and the SELECT statement (Step 3). Thus, implementing the reommendations may require hanges in appliation ode. We summarize the key ontributions of this paper below: We motivate the physial design tuning opportunities that arise by treating the workload as a sequene. We formally define the problem of physial design tuning for workload sequenes (Setion 2). Our goal is to add reate and drop of physial design strutures to the input sequene suh that the overall performane of the generated sequene is maximized. We present an optimal solution to this problem by showing that the problem an modeled as finding the shortest path over a direted ayli graph (DAG) onstruted from the input (Setion 3). We present two tehniques ost-based pruning (Setion 4) and split and merge (Setion 5) that failitate pruning of the searh spae. We desribe a greedy heuristi (Setion 6) that sales well for large workloads and many physial design strutures. Finally, we note that the tehniques developed in the paper are general in the sense that they apply to any physial design struture (indexes, materialized views, et.) that may be supported by the underlying database.

3 2. PROBLEM DEFINITION 2.1 Preliminaries We model the workload as a sequene of SQL statements, i.e., SELECT, INSERT, DELETE and UPDATE statements. All the statements are ordered by monotonially inreasing ID (a timestamp is an example). We represent a statement in a sequene as S k where k denotes its ID. [S 1, S 2... S N ] denotes a sequene of N statements S 1 through S N. The workload an be gathered using traing tools (for example, Profiler tool in Mirosoft SQL Server) that are available on today s database systems. A physial design struture an be any aess path supported by the database server, e.g., index, materialized view, multidimensional lustering of tables, et.. A onfiguration is a valid set of physial design strutures that an be realized in a database. A struture is onsidered relevant for a statement if it ould potentially be used in an exeution plan for answering the statement (even if the struture is not atually hosen by the query optimizer in the final plan). We use the following notations in the paper. COST(S, C) denotes the ost of exeuting statement S for onfiguration C. We rely on optimizer estimated osts and what-if extensions that are available in several ommerially available database servers [1, 7, 16]. For the reasons desribed in [2], this allows our solution to be robust and salable; we an try out numerous alternatives during searh very effiiently without disrupting the normal database operations. TRANSITION-COST(C 1,C 2 ) denotes the minimum ost of realizing onfiguration C 2 in the database starting from onfiguration C 1, i.e. ost of reating and dropping strutures to get from onfiguration C 1 to onfiguration C 2. For example, suppose onfiguration C 1 ontains a single index {I 1 } and C 2 ontains a single index {I 2 }. C 2 an be realized from C 1 by exeuting the following statements - CREATE INDEX I 2 followed by the statement DROP INDEX I 1. TRANSITION- COST(C 1,C 2 ) would be the umulative ost of reating I 2 and dropping I 1. In general, this funtion ould be provided as an input. We represent the exeution of a sequene [S 1, S 2... S N ] as [C 1, S 1, C 2, S 2 C N, S N, C N+1 ] where C i is the onfiguration that is realized in the database prior to exeuting S i and C N+1 denotes the onfiguration after statement S N is exeuted. Note that there is an impliit ordering between the onfigurations: i C i is realized before C i+1. Let C denote the onfiguration prior to the sequene exeution. Note that C is an input and ould be impliit (the urrent onfiguration in the database) or optionally speified through what-if interfae [1]. We define the sequene exeution ost of [C 1, S 1, C 2, S 2 C N, S N, C N+1 ] as TRANSITION-COST (C N, C N+1 ) + N k=1 (COST (S k, C k ) + TRANSITION-COST (C k-1, C k )). Note that this inludes the ost of hanging the onfigurations during sequene exeution through the TRANSITION-COST omponent. 2.2 Physial Design Problem for a Workload Sequene Problem Statement: Given a database D, a sequene workload W=[S 1,S 2... S N ], initial onfiguration C and a storage bound M, find onfigurations C 1,C 2, C N+1 suh that storage requirement of C i (1 i N+1) does not exeed M, and sequene exeution ost of [C 1, S 1, C 2, S 2 C N, S N, C N+1 ] is minimized. There are few important points to observe about this problem formulation. First, the output of any solution to the above problem is itself another sequene where statements orresponding to reate and drop of physial design strutures (DDL statements) are inserted to the input sequene [S 1, S 2... S N ] suh that onfigurations C i for i=1 to N+1 are realized as above. Seond, if the workload ontains inserts/updates/deletes, the ost of updating the physial design strutures are aounted for automatially as part of our optimization problem this is aptured as part of COST(S i,c i ). Third, the ost of transitioning from one onfiguration to the next is also aounted for in the optimization problem this is done via TRANSITION-COST(C i- 1, C i ). Finally, observe that the storage bound M is required to hold at all points in the sequene. Our problem formulation is general enough to handle some other ommon onstraints. Some important onstraints inlude: Consider the ase where the sequenes are generated by individual appliations (for example, Senario 2 in the Introdution). An appliation s impat on the underlying databases physial design is limited to the duration when the orresponding workload sequene exeutes. This an be inorporated by onstraining C N+1 =C, whih ensures that the physial design is restored to the same state as it was prior to the workload sequene. We refer to this as transpareny onstraint. A variation of this inludes the ase where C N+1 is onstrained to be an expliitly provided onfiguration (need not be C ) by the user. Physial design hanges are allowed only at speifi points in the sequene. For example, in Senario 1 in the introdution, the user (e.g., DBA) may allow physial design hanges to happen only at two speifi points during the day. More generally, in the sequene [S 1..S p..s q..s N ] if we want to allow onfiguration hanges only between statements S p and S q, this an be speified via the onstraint: For 1 i<p, C i =C i- 1 and for q<i N+1, C i = C q. Thus we only need find onfigurations C p... C q. Only allow physial design hanges that omplete within a user speified ost bound. This an be represented as TRANSITION-COST(C i-1, C i ) t for all 1 i N+1. We now define the set of physial design strutures over whih we perform the sequene optimization desribed above. A naïve way to generate suh a set from the input sequene is to union the set of all relevant strutures for eah statement in the input sequene. However as detailed in [1, 2, 15, 16] the spae of relevant strutures for a single statement an beome prohibitively expensive to ompute, let alone for the entire sequene. There are a number of existing tehniques that allow us to effiiently generate a muh smaller set of strutures for the purposes of physial database design tuning. One example is the IBM DB2 [15] approah where the optimizer reommends its own strutures for a statement. An alternative approah is disussed in [2] where very good strutures are generated using andidate seletion and merging steps keeping the query optimizer in the loop. We note that the fous of this paper is orthogonal to the speifi method used for the purpose. We refer to the set of strutures generated using suh a tehnique as andidate strutures. For

4 the rest of paper, we assume that a set of andidate strutures an be obtained, given the input workload. Finally, we omment briefly on the searh spae for the optimization problem. If we are provided as input a sequene of N statements, and there are M andidate strutures, the number of possible onfigurations is 2 M, sine eah subset of strutures defines a unique onfiguration. Hene we have a total of 2 M*(N+1) hoies as we need to find N+1 onfigurations C 1,C 2, C N+1. We ontrast this with the set-based tuning problem where there are only 2 M possible onfigurations. Thus, we an view the setbased tuning problem as a onstrained version of sequene tuning problem, where physial design hanges are only allowed at the beginning of the sequene. 3. OPTIMAL ALGORITHM In this setion, we desribe an algorithm to generate an optimal solution to the physial design problem for workload sequenes (defined in Setion 2.2). The key observation is that given an instane of the problem, we an onstrut a graph suh that the shortest path in that graph is an optimal solution for that instane. We illustrate our solution through a simple example. The input workload is a sequene of N SQL statements. The set of andidate strutures is a single index (referred to as I). The goal is to find N+1 onfigurations (C 1 C N+1 ) as desribed in Setion 2.2. We observe the following. In this example, there are two possible onfigurations: (1) The empty onfiguration and (2) The onfiguration {I}. Thus, with any statement S i, there are two possible osts: COST (S i,) and COST (S i,{i}). SOURCE I S 1 {I} I I d S 2 {I} Figure 5 shows the graph that is generated for single index, N- statement ase. The graph is onstruted as follows: 1. For every statement in the input sequene and for every possible onfiguration generated from the input set of strutures we generate a node, i.e., a node n represents a (statement S, onfiguration C) pair with a node ost = COST (S,C). The two nodes SOURCE and DESTINATION representing the initial and final onfiguration are added to the graph and have a node ost of. SOURCE preedes the first statement in the input sequene and DESTINATION sueeds the last statement in the input sequene. Thus in our example, there are 2*N+2 nodes in the graph sine there are only two onfigurations or {I} possible for eah statement. 2. The graph has N+2 stages; a stage for eah statement, SOURCE and DESTINATION. We refer to SOURCE as - th stage and DESTINATION as (N+1)-th stage. 3. The edges in the graph are direted, and exist only between nodes in stage k and nodes in stage k+1 ( k N). Let edge S N {I} Id DESTI- NATION Figure 5. Graph for single index, N statements e=(n 1, n 2 ) represent the edge from node n 1 =(*,C) to node n 2 =(*,C ); * denotes that it ould be any statement. Then ost of e=transition-cost(c,c ). In our running example, there are 4*N edges and the osts of the edges an be: (i) when there is no hange in onfiguration between nodes that define the edge (ii) ost of reating the index (denoted in the figure by (I )) when transitioning from to {I} and (iii) ost of dropping the index (denoted by I d ) when transitioning from {I} to. 4. If the final onfiguration denoted by C N+1 is onstrained to be the same as initial onfiguration denoted by C or some other user provided onfiguration, we assign the appropriate edge ost between the nodes in stage N and DESTINATION node. However if we do not have any onstraints on the final onfiguration, we assign a ost of to all the edges between the nodes in the stage N and DESTINATION node. In our example, we assign edge osts of between all nodes in S N and DESTINATION and C N+1 will be the same as C N. One the graph is onstruted as above, any path from SOURCE to DESTINATION represents a valid sequene exeution. Note that the path ost inludes the ost of nodes as well as edges. The optimal output sequene is the shortest path in this graph. The equivalene between the shortest path in the graph above and sequene exeution ost is straightforward as the node osts represent COST (S k, C k ) and edge osts represent TRANSITION- COST (C k-1, C k ). We note the following properties of this graph. (1) The ost of nodes and edges are non negative. (2) The shortest path in this graph an be omputed very effiiently using single soure shortest path tehnique for DAGs desribed in [6] in linear omplexity as number of both edges and nodes are O (N); the intuition behind the linear omplexity is that eah edge needs to be examined exatly one to arrive at the solution. Generalizing the graph to N-statement sequene and M strutures to generate an optimal solution is oneptually straightforward. In eah stage 1 through N, there are 2 M nodes, eah representing a onfiguration. This is beause eah subset of input strutures defines a onfiguration. We refer to the solution that enumerates all 2 M onfigurations exhaustively at eah stage as EXHAUSTIVE. It is important to note that the graph has O (N*2 M ) nodes and O (N*2 2M ) edges, i.e., the graph is exponential in the number of input strutures. Though the shortest path an be solved in linear omplexity of number of nodes and edges as above, these however are exponential in number of strutures. Thus, if the input workload has only a small number of andidate strutures, we an apply EXHAUSTIVE to get an optimal solution. However, it beomes impratial for large workloads that an typially have hundreds or more andidate strutures. Note also that eah node represents a (statement, onfiguration) pair and has an assoiated node ost. Thus, we have to ompute COST(S,C) orresponding to that node. Using the osting tehnique desribed in Setion 2.1 may involve invoking the query optimizer, whih an also ontribute to making graph onstrution expensive. In the next two setions we disuss two tehniques that an improve the effiieny of the above algorithm by reduing the number of nodes in the graph signifiantly. The first tehnique (disussed in Setion 4) is an optimality preserving ost-based pruning. The seond tehnique (disussed in Setion 5) is

5 effetive for large workloads where different statements touh different tables/olumns of the database. 4. COST-BASED PRUNING This optimization allows us to prune onfigurations at a given stage in the input sequene while preserving optimality. It relies on the observation that in many ases we an effiiently ompute a lower bound on the ost of a statement for a given onfiguration. The intuition behind this tehnique is that we an leverage optimal solutions of individual strutures as desribed in Setion 3 to signifiantly prune nodes at various stages in the graph that would otherwise be generated by EXHAUSTIVE above. We first desribe the pruning tehnique under the assumption that the benefits of the strutures are independent. We subsequently desribe how the tehnique an be leveraged when the strutures interat, i.e., the benefits of the strutures are not independent; [2] provides a detailed desription of suh interations. One example of suh an interation is when both indexes used for a Merge Join are sorted on join key, join proeeds muh faster ompared to the ase when only one index is sorted on the join key. We define the following. SPS (s) for a physial design struture s refers to the shortest path solution that we get by optimizing the entire sequene for s alone using the algorithm presented in Setion 3. Let [b 1, S 1, b 2, S 2 b N, S N, b N+1 ] represent the solution SPS(s) where b i represents onfigurations. Let [C 1,S 1,C 2,S 2 C N,S N, C N+1 ] denote the solution where C i represents onfigurations that we get by running EXHAUSTIVE over the graph that has all the onfigurations (the power set of all strutures). If there are no interations aross strutures (i.e., for every statement, the benefits of strutures are independent of the presene of other strutures) then the following laim holds. Claim: s b k in SPS(s) s C k for every struture s and stage k (k = 1 to N). Proof: We prove it by ontradition. Assume that there exists a struture s where k is the first stage suh that s C k but s b k. All the osts are non negative and C =b (both SPS and EXHAUSTIVE are solved with the same initial onfiguration). We use the following notations: I as the reation ost for s. I d as the drop ost for s. p 1 = COST(S k, C k ) p 2 = COST(S k, C k {s}) where C k {s} is the onfiguration one gets by removing s from C k. q 1 = COST(S k, {s}) q 2 = COST(S k, ) Sine benefits are independent q 1 p 1 =q 2 p 2 q 1 q 2 =p 1 p 2. We enumerate all possible ases as following. s C k-1. Then p 1 <I d +p 2 must hold as EXHAUSTIVE would not pik C k otherwise. If s b k-1, then q 1 >I d +q 2 q 1 q 2 > I d p 1 p 2 >I d results in a ontradition. The other ase s b k-1 an not happen as k is the first stage where violation ours. s C k-1. Then I +p 1 <p 2 must hold. If s b k-1, then q 1 >I d +q 2 q 1 > q 2 p 1 > p 2 results in a ontradition. If s b k-1, then q 1 > q 2 p 1 > p 2 again results in a ontradition. We also note that C N+1 = C N in our original optimization problem (or equals C in transpareny onstrained problem) and the proof holds for stage N+1. This allows us to eliminate strutures (and onfigurations that ontain these) at a given stage as follows. We run the SPS for all strutures and analyze their respetive solutions at eah stage. For a given stage k, we onstrut a set of strutures R={s s b k in SPS(s)}. Note that every subset of R defines a unique onfiguration and is added to the graph as node (S k,) if onfiguration obeys the storage bound. Let s apply this to following example. Assume that the input sequene has 4 statements [S 1,S 2,S 3,S 4 ]. Also for statement S i index I i (i= 1, 2, 3 and 4) and no other index is relevant and the drop ost of every index is. Using EXHAUSTIVE we will enumerate all 2 4 =16 onfigurations to find the optimal solution. By looking at optimal solutions on a per struture basis, i.e., SPS(I 1 )=[{I 1 },S 1,,S 2,,S 3,,S 4,], SPS(I 2 )=[,S 1,{I 2 },S 2,, S 3,,S 4,] and so on, we know that only I i is present in stage i. Thus we an onstrut the redued graph with 5 onfigurations as shown in Figure 6 (the unlabelled edges represent a ost of ); the solution we get on this redued graph is optimal. SOURCE I 1 S 1 {I 1 } I 2 S 2 {I 2 } I 3 Figure 6. Graph with ost-based pruning DESTI- NATION When the benefits of strutures are not independent, in SPS(s) we use the lower bound ost of any onfiguration that ontains s for statement S k as COST(S k,{s}). We note that the laim above remains true when the assignment of the osts to individual strutures is done as desribed above. The argument is similar as for the independent struture ase. We reuse the notation from the proof above. We assume the optimizer is well behaved; for a query the addition of a struture to a onfiguration an never inrease the ost of the statement. For the ase where s C k and s b k-1, we get the following: p 1 p 2 < q 1 q 2 (p 1 q 1 ) + (q 2 p 2 )<. However p 1 q 1 due to the way we assign osts above and q 2 p 2 for queries. This leads to a ontradition. The proof for other ases is similar and is omitted. The effetiveness of ost-based pruning depends on how aurately and effiiently we an determine the lower bound of osts. This itself is a hard problem as we do not want to enumerate all onfigurations ontaining {s} to arrive at the lower bounds. A trivial lower bound is the minimal ost of a query under any physial design. This an be omputed very effiiently by probing the optimizer one [3]. There are other tehniques (e.g., [5] disusses how to derive osts of onfigurations using atomi onfigurations) that an be leveraged to get better lower bounds effiiently. The updates/inserts/deletes an be handled by splitting the update into a query part whih identifies the speifi rows that need to be updated and the atual update part. We omit details due to lak of spae. We expet the ost-based pruning tehnique to work well when s b k in SPS(s) for few values of k. This typially happens when there are a signifiant number of updates/inserts/deletes in the input sequene or in the presene of the transpareny onstraint. In our experiments, see Setion we demonstrate the S 3 {I 3 } I 4 S 4 {I 4 }

6 effetiveness of this pruning tehnique where it results in orders of magnitude redution in number of nodes in the graph. 5. EXPLOITING DISJOINT SEQUENCES As we saw in Setion 3, the effiieny of our solution depends on the number of input strutures whih in turn depends on the statements in the sequene. In general, workloads an be large. These ould be trae files olleted by traing the server ativity and ould be over a period of days with thousands of statements that touh different parts of the database. In this setion we present a tehnique that leverages the fat that groups of statements aess different parts of data, to redue the searh spae signifiantly. W: S 1 S 2 S 3 S 4 S 5 S 6 S 7 SRC S 1 S 3 d S I 1 I 4 1 {I 1 } {I 1 } SRC SRC I 2 W 1 = [S 1,S 3,S 4 ] S 2 S 5 {I 2 } {I 2 } W 2 = [S 2,S 5,S 6 ] I 3 W 3 = [S 7 ] Combine solutions of W 1, W 2 and W 3 to get solution for W Figure 7. Disjoint sequenes DEST SRC S 1 S 2 S 3 S 4 S 5 S 6 S 7 DEST {I 1 } {I 1,I 2 } {I 1, I 2 } {I 2 } {I 2 } {I 3 } {I 3 } We motivate the tehnique through an example. Figure 7 shows an input sequene workload W = [S 1, S 2... S 7 ]. Suppose struture I 1 is relevant for S 1, S 3 and S 4 only (these statements referene table A only), I 2 is relevant for S 2, S 5 and S 6 (these statements refer table B only) and I 3 is relevant for S 7 only (it is on table C). S 4 and S 6 are updates on the tables A and B respetively. Applying EXHAUSTIVE on S with input struture set {I 1,I 2,I 3 } would enumerate 2 3 = 8 onfigurations. If we apply ost-based redution as disussed in Setion 4 we an redue the number of onfigurations signifiantly. However at S 2 and S 3, we still need to onsider the onfigurations orresponding to all the subsets of {I 1,I 2 } a priori as both I 1 and I 2 are present in stages 2 and 3. Note that I 1 and I 2 are on different tables in the database and are relevant for different statements in the sequene. The interesting question is whether we an avoid generating suh onfigurations up front and if possible at all. It turns out that for the above sequene we an do muh better as desribed below. We define the notion of disjoint sequenes. Two sequenes X and Y are said to be disjoint if (1) X and Y do not share any statements and (b) no statement in X shares any relevant physial design struture with any statement in Y. This implies that given a physial design struture s, all the relevant statements orresponding to s are present in only one suh sequene, i.e., its impat is limited to one sequene. On applying this to our example in Figure 7, whether we reate or drop index I 2 on table I 2 d S 6 S 7 {I 3 } DEST DEST {I 3 } B annot impat the ost of statements in W 1 (reall that S 1, S 3 and S 4 are on table A). We note that disjoint sequenes may be interleaved with respet to how these our in the original sequene (W 1 and W 2 in Figure 7 are interleaved as S 2 in W 2 starts before W 1 ends). It may be tempting to suggest that we an break the input sequene into a set of disjoint sequenes, solve eah one independently as the hoie of onfigurations in one disjoint sequene does not impat hoies for other sequenes and ombine the results to get the globally optimal solution. In the absene of storage violations this strategy is indeed optimal and an lead to muh better searh performane than the alternative approah that tunes the input as a single sequene. The effiieny omes from signifiant redution in number of onfigurations (and hene nodes) that needs to be generated for the graph. In our urrent example, eah disjoint sequene in our example is solved for just one index (W i for I i ) and onfigurations like {I 1,I 2 } are never onsidered. However if there are storage violations, then the above strategy does not work as the resulting solution is not valid. In our example if we do not have suffiient storage for both I 1 and I 2, by using the above strategy we get {I 1,I 2 } at S 2 and S 3 as shown in Figure 7 whih is not valid. An interesting question is how an we generate a valid solution effiiently in the latter? We present two operators that mirror the ideas above. The first operator Split desribed in 5.1 takes an input a sequene and the relevant set of strutures and splits it into a set of disjoint sequenes. The seond operator Merge (in 5.2) takes as input a set of disjoint sequenes and their respetive solutions and ombines these to generate a valid solution for the sequene derived from superimposing input set of sequenes. The two operators an be ombined to effiiently generate lose to optimal solutions. 5.1 Split The algorithm to ahieve this is straightforward. For every struture we know the set of statements that are relevant by looking at the syntati struture of statements. Consequently, for every statement, the set of relevant strutures is known. Now, the split is ahieved by doing a transitive losure over the statements as follows. (1) Start with eah statement as a separate sequene. With every sequene assoiate its relevant set of strutures. (2) If two sequenes share any struture, ombine them into one sequene (union of the statements) and union their set of strutures. (3) Continue step 2 till no more sequenes an be ombined. At the end, our input sequene is split into a set of disjoint sequenes that neither share any statements nor any strutures. In the example above, we split the input sequene W into 3 disjoint sequenes W 1 = [S 1,S 3,S 4 ], W 2 = [S 2,S 5,S 6 ] and W 3 = [S 7 ] with relevant struture sets {I 1 }, {I 2 } and {I 3 } respetively. 5.2 Merge The input to Merge is a set of disjoint sequenes and their respetive solutions. The output of Merge is a solution that obeys storage onstraint and is defined over all the statements from input disjoint sequenes. Let P={p 1, p m } represent the set of solutions where p i orresponds to the solution for W i (1 i m) that is provided as input to Merge. Merge proeeds in two steps. Step 1: From the input set of solutions P, a solution P u is onstruted by performing a union of onfigurations of p i (1 i m)

7 at eah stage. In our urrent example as shown in Figure 8, we get P 1,3 by performing the union of solutions of W 1 and W 3. P u represents the union of solutions of disjoint sequenes W 1, W 2 and W 3. Note that the union involves the superimposition of the individual statements in the input sequenes as well as the union of onfigurations at eah stage. Step 2: If all the onfigurations that are part of P u obey storage bound then P u is optimal and is the output of Merge. However if there are onfigurations in P u (as shown in Figure 8) that violate storage bound, then P u is not valid. In that ase we make loal hanges to P u to make it a valid solution. The intuition behind loal optimization is based on the following observation. The solutions in the regions where storage bounds do not get violated remain optimal as long as the preeding and following onfigurations remains unhanged. This is guaranteed by the shortest path algorithm. If the ost of the violating ranges is small with respet to the entire sequene exeution ost, then applying loal optimizations to the violating ranges only to generate a valid solution an result in nearly optimal solutions. SRC S 1 S 3 S 4 S 7 DEST {I 1 } {I 1 } {I 3 } {I 3 } P 1,3 = Merge solutions of W 1 and W 3 SRC S 1 {I 1 } S 2 S 3 S 4 S 5 S 6 S 7 {I 1,I 2 } {I 1, I 2 } {I 2 } {I 2 } {I 3 } {I 3 } P u = Merge of solutions of W 1, W 2 and W 3 when no storage violation SRC S 1 S 2 S 3 S 4 S 5 S 6 {I 1 } {I 2 } {I 1 } {I 2 } {I 2 } {I 3 } {I 3 } P u = Merge of solutions of W 1, W 2 and W 3 when storage is violated Figure 8. Merge of disjoint sequenes Let us see how we apply this to our urrent example. Using the notation from 2.2, C 1 ={I 1 }, C 2 =C 3 = {I 1,I 2 }, C 4 ={I 2 } and so on in P u (obtained from step 1 above). Assume that we have suffiient storage for only one index. Then P u is not a valid solution as C 2 and C 3 (= {I 1,I 2 }) violate storage bound. By leveraging the observation above, we isolate the violating sequene whih in this ase is [S 2,S 3 ]. For the remaining parts of the sequene ([S 1 ] and [S 4,S 5,S 6,S 7 ]), there are no storage violations in their respetive solutions. If we loally optimize [S 2,S 3 ] to get a solution that respets the storage bound with SOURCE={I 1 } and DESTINATION={I 2 }, we preserve the optimality of solutions of [S 1 ] and [S 4,S 5,S 6, S 7 ]. Here, if the benefit of indexes I 1 and I 2 far outweigh their reate osts, then optimizing [S 2,S 3 ] with SOURCE={I 1 } and DESTINATION={I 2 } results in [{I 2 },S 2,{I 1 },S 3,{I 2 }]. Next we ombine the loally optimal solutions of [S 1 ] (from P u ), [S 2,S 3 ] (omputed loally as above) and [S 4,S 5,S 6,S 7 ] (from P u ) to get P u whih is a valid solution. The sequene exeution ost of P u is a lower bound on the ost that we an get for any solution that obeys the storage bound and therefore for the optimal solution. If the ost of the solution we get by applying Merge (=P u ) is lose to the sequene exeution ost of P u then we have a nearly optimal solution at hand. In general one may apply Merge in different ways over the disjoint sequenes to build alternative searh shemes (for example, one may Merge disjoint sequenes pair wise in a greedy manner). This is made possible as Merge preserves disjointedness, e.g., if W 1, W 2 and W 3 are mutually disjoint, then output sequene after Merge of W 1 and W 3 is disjoint with respet to W 2. In our experiments, see Setion 8.2.4, we used a simple strategy where we generated a valid solution for the entire sequene by first applying Split, and then Merge over the solutions of all disjoint sequenes. We observed that even this simple strategy often led to an order of magnitude speed up ompared to a strategy that blindly treated the input as a single sequene. Also the resulting solution after Merge was lose to optimal (within 3%) even under very limited storage bounds. We note that in data warehouse senarios (see for TPCH experiments that simulates this) the appliability of this beomes limited as almost all queries referene the fat table and we do not get multiple disjoint sequenes. However if the same server hosts multiple suh data warehouses (and workloads) then this tehnique an still be used very effetively. 6. GREEDY HEURISTIC As desribed in Setion 3, the number of onfigurations, and hene nodes and edges in our graph, is exponential in the number of andidate physial design strutures for the workload. The EXHAUSTIVE approah (Setion 3) an therefore infeasible in pratie where a workload an often have a large number of andidate strutures. Observe that the above problem exists even in the approah where the workload is treated as a set, e.g., [3,5, 15]. We note that previous work in the ontext of set-based workloads (e.g., [3, 5]) have developed tehniques to deal with the ombinatorial explosion that results in the number of onfigurations that need to be onsidered. These tehniques typially use a greedy heuristi to searh through the spae of onfigurations instead of looking at the entire (exponential) spae. In this setion, we desribe how to adapt suh a greedy heuristi for the ase of workload as a sequene. We refer to our algorithm as GREEDY-SEQ. The GREEDY-SEQ algorithm uses a funtion we refer to as UnionPair. We first desribe this funtion in Setion 6.1, and present the overall algorithm in Setion UnionPair UnionPair takes as input two solutions denoted by p 1 =[a 1,S 1,,a N,S N,a N+1 ] and p 2 =[b 1,S 1, b N,S N,b N+1 ]. Observe that the inputs are both solutions for the same sequene 1 [S 1,, S N ] but the onfigurations in eah solution are over a disjoint spae of physial design strutures. UnionPair generates a new solution for the sequene as desribed below. Initially, a graph is onstruted that has all the nodes (and edges) from the two input solutions p 1 and p 2. At eah stage k in the graph, additional onfigurations (as desribed below) are generated from onfigurations a k and b k and orresponding nodes and edges are added to the graph. The output of UnionPair is the shortest path solution in the graph thus generated. Next we disuss how to generate the additional onfigurations at eah stage. Intuitively, at eah stage k in the graph, i.e., (for S k ) we are looking for the best onfiguration (we refer to this as d k ) we an generate from the strutures in (a k b k ) that satisfies the 1 Observe also that unlike UnionPair, Merge (Setion 5.2) is defined over solutions to disjoint sequenes.

8 storage onstraint. In general, the number of onfigurations we need to onsider are exponential in (a k b k ). We observe however, that in many ommon ases the onfiguration (a k b k ) itself is optimal at stage k (e.g., if S k is a query and (a k b k ) obeys the storage onstraint) sine (a k b k ) preserves the benefits of both a k and b k. In general however, (a k b k ) an have worse update harateristis ompared to a k and b k. Note that sine we preserve the original onfigurations a k and b k in the generated graph, the update harateristis of (a k b k ) get aounted for automatially in our strategy (for example if (a k b k ) has a high update overhead, the shortest path output may ignore it and pik a k instead). Figure 9 represents the generated graph for the above input pair assuming d k = (a k b k ). SOURCE S 1 S 2 S k S N a 1 b 1 In general, it may be important to introdue other onfigurations in addition to (a k b k ) as well. In suh ases, we an take advantage of previous tehniques developed (e.g., in [3,5]) to generate nearly optimal onfigurations very effiiently. We omit further details due to lak of spae. 6.2 The GREEDY-SEQ Algorithm a 2 b 2 Figure 9. Generated graph for UnionPair 1. For every struture in the set S={s 1,..s M }, find the optimal solution using the graph formulation as desribed in Setion 3. At this point, we have a set of solutions P for individual strutures. Let P= {p 1, p M } and p i =[a i1,s 1 S N,a in+1 ]. 2. Let C be the set of all onfigurations over all p i s. 3. Run a greedy searh over P as follows. a. Let r = [ 1, S 1, N, S N, N+1 ] represent the least ost solution in P. P=P-{r}. Let C = C { 1, N+1 } b. Pik an element s from P suh that t=unionpair(r,s) has the minimal sequene exeution ost for among all elements of P and sequene exeution ost of t is less than that of r. If no suh element exists go to step 4. P=P-{s}. P=P {t}. Go to a. 4. Generate the graph with all the onfigurations in C at eah stage. Run the shortest path over this graph and return the solution. Figure 1. The GREEDY-SEQ algorithm a k b k a N b N a 1 b 1 a 2 b 2 a k b k a N b N DESTI- NATION We use UnionPair to build our greedy solution. The individual steps of GREEDY-SEQ are desribed in Figure 1. Let s take a look at how GREEDY-SEQ works in the following example. Assume that the input sequene has 8 statements [S 1,S 2,S 3,S 4, S 5,S 6,S 7,S 8 ], input set of strutures is {I 1,I 2,I 3,I 4 } and storage is infinite. Also for statements S i and S 4+i index I i (1 i 4) and no other index is relevant and that the benefit of every index for the relevant statement is greater than its reation (and drop) ost. Also assume that the ost using index I i <I i+1 (1 i 3). The exhaustive approah in onjuntion with ost-based pruning would still lead to 2 4 onfigurations at stage 4. Using GREEDY- SEQ in Step 4 above we have 8 onfigurations only and we still get the optimal solution as it lazily generates the onfigurations ({I 1,I 2 },{I 1,I 2,I 3 } and {I 1, I 2,I 3,I 4 }) that signifiantly impat the over all quality of reommendation. In Step 3, we look at a few extra onfigurations (beause of UnionPair) but the overall number of generated onfigurations is typially muh smaller than EXHAUSTIVE. In Setion 8.2.3, we present results that demonstrate the effetiveness of GREEDY-SEQ where it results in lose to optimal solutions and with signifiantly better performane ompared to EXHAUSTIVE. While GREEDY-SEQ appears to work well in pratie, it is important to note that it an in general be sub-optimal. The following example demonstrates suh a ase. Table 1 shows the ost of 4 statements in the sequene for various indexes. Assume that the storage available is 1 MB and that I 1 requires 8 MB, I 2 and I 3 require 4 MB eah. GREEDY-SEQ returns [{I 1 },S 1,{I 1 },S 2,{I 1 },S 3,{I 1 },S 4,{I 1 }] while optimal solution is [{I 2,I 3 },S 1,{I 2,I 3 },S 2,{I 2,I 3 },S 3,{I 2,I 3 },S 4,{I 2,I 3 }]. Table 1. Sub-optimality of GREEDY-SEQ Sequene S 1 S 2 S 3 S 4 Total Cost Index Initial Cost I I I The two main reasons for sub-optimality of GREEDY-SEQ are storage onstraints and interations aross various physial design strutures. We note there are already effetive greedy searh tehniques that result in very good and effiient solutions in the ontext of set-based workload to overome the limitations mentioned above. For example, one suh tehnique disussed in [5] advoates generating onfigurations with up to a ertain size exhaustively and proeeding greedily there after. The other [15] uses measures like benefit per unit store instead of pure benefit to mirror knapsak like approahes. We note that these tehniques an be integrated easily with GREEDY-SEQ. Let s analyze the omplexity of GREEDY-SEQ disussed above for a N-statement sequene and M strutures. The graph for eah struture is same as 3.1 and has O (N) edges and nodes. Step 1 requires O(N*M) time. Step 3 an be repeated at most M times (In eah invoation an element, i.e., two paths in P get merged and subsequently removed from the set and the merged path gets added). Sine we only retain shortest path solutions in P, an element in P always has O (N) edges and nodes. This allows us to solve step 3 in O (N*M 2 ) time. In step 4, C has O(M*N) onfigurations as we generate at most M solutions in step 3 and

9 eah solution has O(N) onfigurations; hene number of nodes in step 4 is O(M*N 2 ) and edges is O(M 2 *N 3 ). Therefore GREEDY- SEQ runs in O(M 2 *N 3 ). However in pratie, we found number of nodes in step 4 to be O(M*N) as step 3 led to O(M) onfigurations resulting in O(N*M 2 ) running time. We validate our analysis above through experiments in Setion DISCUSSION 7.1 End to End Solution We briefly outline a possible end to end solution based on the ideas disussed in the paper. Figure 11 shows the arhiteture of our solution. We Split the input sequenes into disjoint sequenes (Setion 5.1) based on the andidates (Setion 2.2 desribes how to get andidates). We apply the optimality preserving ost-based pruning (Setion 4) on eah sequene. Subsequently we tune eah sequene separately using EXHAUSTIVE (Setion 3) or GREEDY-SEQ (Setion 6). Finally we Merge (Setion 5.2) the results of disjoint sequenes to get the over all solution. We note that the various tehniques disussed in the paper an be put together in different ways based on quality and performane requirements. For example, one may want to use the EXHAUSTIVE strategy and not apply the Split and Merge operations if quality is the driving fator and enumerating all the onfigurations is not prohibiting. Sequene, Constraints Candidate set of strutures Apply split operator to get disjoint sequenes Apply ost based pruning on eah sequene Solve eah sequene independently using EXHAUSTIVE or GREEDY-SEQ Merge results of disjoint sequenes Reommendation Figure 11. Arhiteture of our solution 7.2 Extending to Sequene of Sets Extending our tehniques when the workload is treated as a sequene of sets of statements is straightforward. In the graph formulation disussed in Setion 3 eah stage represents a set of statements rather than individual statements and eah node represents a (set, onfiguration) pair. The onfigurations seleted optimize the performane of a set. There are known tehniques [5] that we an leverage to find suh onfigurations effiiently. Here the physial design hanges our at the set boundaries instead of possibly at eah statement. 8. EXPERIMENTS We have implemented the algorithms presented in this paper via a prototype that extend the Database Tuning Advisor [1] in Mirosoft SQL Server 25. In our experiments we demonstrate the following: In the presene of updates and/or at low storage bounds, sequene-based approah leads to muh better quality reommendations than a set-based approah. Cost-based pruning tehnique disussed in Setion 4 results in signifiant pruning in number of nodes that need to be generated in the graph. The quality of reommendations from our greedy approah disussed in Setion 6 is lose to that ahieved by an exhaustive solution that enumerates all onfigurations. The performane of greedy sales linearly with the workload size in pratie and quadratially with the number of strutures. Our split and merge approah for exploiting disjoint sequenes (Setion 5) results in lose to optimal quality solutions and is muh more effiient than the alternative that treats the input as a single sequene on real workloads. 8.1 Experimental Setup The experiments were performed on a HP workstation with 2.8 GHz CPU with hyper-threading and 1 GB RAM on a ommerially available database server. All the databases used in the experiment were stored loally on 8 GB hard disk. We used the available what-if APIs on the server to simulate the reate and drop osts of various strutures. This was further used to simulate the ost of transitions between various onfigurations. We use the following workloads in our experiments to demonstrate the effetiveness of our tehniques. TPCH-1-n onsists of the first n queries from TPC-H 22 query benhmark [14]. TPCH-1-n-M-m is same as TPCH-1-n exept the queries of TPCH-1-n are repeated m times. TPCH-1-n-U-k-MID onsists of TPCH-1-n and updates to LINEITEM table that inreases the #rows by k%. Updates are in the middle of the workload. TPCH-1-n-U-k-END same as TPCH-1-n-U-k-MID exept updates are present at the end. Two real workloads and databases, WKLD1 and WKLD2 on databases DB1 (~25 MB with about 1 tables) and DB2 (~5 MB with about 25 tables) respetively. WKLD1 has a mix of 1 insert/update/delete/selet statements. WKLD2 ontains omplex stored proedures as statements. 8.2 Experimental Results Effetiveness of sequene-based approah Here we ompared the quality of sequene and set-based tuning approahes. We used TPCH1G database and workloads TPCH-1-22, TPCH-1-22-U-1-MID and TPCH-1-22-U-1-END at two storage bounds (i) a low storage bound of 1.2 GB whih allows for maximum 2% of data size for redundant strutures and (ii) a high storage bound of 3 GB whih allows for an extra 2 GB spae for redundant strutures. Table 2 below ompares the quality of the two approahes. The % improvement is relative to the optimal output of set-based approah and thus getting any further