Logical Optimization of ETL Workflows

Transcription

1 Logical Optimization of ETL Workflows Alkis Simitsis Nat. Tech. University of Athens Panos Vassiliadis University of Ioannina Timos Sellis Nat. Tech. University of Athens Outline Introduction Modeling ETL Optimization as a State-Space Problem Problem formulation State Generation Transition Applicability State Space Search & Experimental Evaluation Conclusions & Future Work HDMS'05 - A. Simitsis - 25/08/

2 Outline Introduction Modeling ETL Optimization as a State-Space Problem Problem formulation State Generation Transition Applicability State Space Search & Experimental Evaluation Conclusions & Future Work HDMS'05 - A. Simitsis - 25/08/ Extraction Transformation Loading (ETL) Workflows HDMS'05 - A. Simitsis - 25/08/

3 ETL workflows are not big queries S1 γ V1 U S2 γ V2 Sources TIME DW HDMS'05 - A. Simitsis - 25/08/ ETL workflows are not big queries DS.PS_NEW 1 DS.PS_NEW 1.PKEY, DS.PS_OLD 1.PKEY SUPPKEY=1 DS.PS 1.PKEY, LOOKUP_PS.SKEY, SUPPKEY COST DATE DS.PS_OLD 1 DIFF 1 DS.PS 1 Add_SPK 1 SK 1 rejected $2 rejected A2EDate rejected U Log Log Log DS.PS_NEW 2 DS.PS_NEW 2.PKEY, DS.PS_OLD 2.PKEY SUPPKEY=2 DS.PS 2.PKEY, LOOKUP_PS.SKEY, SUPPKEY COST DATE=SYSDATE QTY>0 DIFF 2 DS.PS 2 Add_SPK 2 SK 2 NotNULL AddDate CheckQTY DS.PS_OLD 2 Log rejected Log rejected DSA PKEY, DAY MIN(COST) S 1 _PARTSUPP FTP 1 DW.PARTSUPP Aggregate 1 V1 DW.PARTSUPP.DATE, DAY PKEY, MONTH AVG(COST) S 2 _PARTSUPP FTP 2 TIME Aggregate 2 V2 Sources DW HDMS'05 - A. Simitsis - 25/08/

4 Problems The well known query optimization techniques are not sufficient, mostly due to the existence of data manipulation functions when is it valid to push an ETL activity in front of a function? black-box activities unknown semantics difficult/impossible/meaningless to express in relational algebra we cannot interfere with their interior (e.g., their source code) naming conflicts e.g., an attribute named COST in source 1 contains values in Dollars, while in source 2 contains values in Euros HDMS'05 - A. Simitsis - 25/08/ Related Work ETL and optimization Data Cleaning (Rahm & Do 2000) Duplicate detection (Monge 2000) Cleaning of data from the web (AJAX 2000) CPU optimization for ETL (Potter s Wheel 2001) Extraction, Recovery and Lineage Tracing Detection of differentials (Labio et al. 1997) Recovery (Labio et al. 2000) Lineage tracing (Cui 2001) Streams and optimization AURORA 2003 HDMS'05 - A. Simitsis - 25/08/

5 Equivalent workflows Can we push selection early enough? Can we aggregate before $2 takes place? What about naming conflicts? HDMS'05 - A. Simitsis - 25/08/ Aims How can we minimize the execution time of an ETL workflow? How can we deal with naming problems? When is it valid to change the position of an activity in an ETL workflow? How can we produce equivalent ETL workflows? In summary, the two problems that have risen are to determine which modifications of the workflow are legal which configuration is the best in terms of performance gains HDMS'05 - A. Simitsis - 25/08/

6 Problem Formulation and Contribution We set up the theoretical framework for the problem of the optimization of an ETL workflow by modeling the problem as a state-space search problem each state represents a particular design of the workflow as a graph we define transitions from one state to another we construct the state-space we search for a state that minimizes the execution time of the workflow HDMS'05 - A. Simitsis - 25/08/ Outline Introduction Modeling ETL Optimization as a State-Space Problem Problem formulation State Generation Transition Applicability State Space Search & Experimental Evaluation Conclusions & Future Work HDMS'05 - A. Simitsis - 25/08/

7 States Each activity is characterized by A unique ID Input schemata Output schemata Semantics, expressed in an Input 2 extended relational algebra with black-box functions Each ETL workflow is a state, i.e., a DAG: Nodes: activities and relations Edges: provider relations Input 1 Activity Output HDMS'05 - A. Simitsis - 25/08/ State Generation through Transitions State Generation: a transition is a modification to a state (i.e., a variant of the workflow) that generates a semantically equivalent workflow Three categories for five transitions Swap Distribute and Factorize Merge and Split HDMS'05 - A. Simitsis - 25/08/

8 Transitions Swap two activities Locally swap two unary activities HDMS'05 - A. Simitsis - 25/08/ Transitions Distribute and Factorize Factorize: replacement of two homologous activities placed in two converging data flows by a new one Distribute: distribution of an activity into two converging data flows a,a 1,a 2 : homologous activities HDMS'05 - A. Simitsis - 25/08/

9 Transitions Distribute and Factorize An example a simple cost model: #rows cost: c SK =nlog 2 n and c σ =n application: input: 8 records in each flow selectivity: 50% for σ, 100% for the rest C 1 =2nlog 2 n+n=56 C 2 =2(n+(n/2)log 2 (n/2))=32 C 3 =2n+(n/2)log 2 (n/2)=24 HDMS'05 - A. Simitsis - 25/08/ Transitions Merge and Split Merge / Split unary activities HDMS'05 - A. Simitsis - 25/08/

10 Transition Applicability We resolve naming problems by reducing all attribute names (for all activity and relation schemata) to a common terminology We annotate each activity with three extra schemata: Functionality schema: the set of input attributes that participate in the computation Generated / Projected-out schemata Easy to obtain these through a set of templates (Vassiliadis et CAiSE 03, Inf. Systems) E.g., the Surrogate Key Transformation requires an input production key, a generated surrogate key and a parameter specifying the source of the data HDMS'05 - A. Simitsis - 25/08/ Transition Applicability When is it valid to swap two activities? the functionality schema should be a subset of the input schema HDMS'05 - A. Simitsis - 25/08/

11 Transition Applicability When is it valid to swap two activities? the functionality schema should be a subset of the input schema input schemata should be subset of provider schemata HDMS'05 - A. Simitsis - 25/08/ Transition Correctness To prove correctness we employ cond: post-condition for activities true, when the activity is successfully executed involves activity s functionality schema e.g., $2 (#vrbl1) Cond: post-condition for ETL workflows $2 (COST) $2 (COST) conjunction of the post-conditions of all the workflow activities, arranged in the order of their execution ETL workflow equivalence: the schema of the data propagated to each target recordset, at the end of each flow, is identical the flows have the same post-conditions Theorem: All transitions produce equivalent workflows HDMS'05 - A. Simitsis - 25/08/

12 Outline Introduction Modeling ETL Optimization as a State-Space Problem Problem formulation State Generation Transition Applicability State Space Search & Experimental Evaluation Conclusions & Future Work HDMS'05 - A. Simitsis - 25/08/ Algorithms Simple to produce an Exhaustive search (ES algorithm) We generate all the possible states that can be generated by applying all the applicable transitions to every state Heuristic search (HS algorithm) Pre-Processing: Merge before any other transition Phase 1: Swap only in linear paths Phase 2: Factorize only homologous activities placed in two converging paths Phase 3: Distribute only if transformation is applicable Phase 4: Swap again only in the linear paths of the new states produced in phases 2 and 3 Greedy variant of the Heuristic search (HS-G algorithm) Swap only if we gain in cost HDMS'05 - A. Simitsis - 25/08/

13 Experiments Setup Three categories for 40 different ETL workflows Small: 20 activities (avg) Medium: 40 activities (avg) Large: 70 activities (avg) Quality of solution HS: near optimal HS-G: good for small/medium and over 60% for large workflows Improvement of solution HS: over 70% HS-G: over 60% for small and medium, and over 47% for large workflows Quality Ratio Avg Improvement (%) Quality of Solution (% avg) 40 small medium large Type of Scenario 80,00% 70,00% 60,00% 50,00% Improvement of Solution 40,00% small medium large Type of Scenario HS HS-G HS HS-G HDMS'05 - A. Simitsis - 25/08/ Experiments Volume of state space ES: exponentially increases HS: grows for large scenarios HS-G: keeps state space small Evolution of time ES: in medium/large scenarios did not terminate within 40h HS: avg worst case is ~35min for large scenarios, while the gain in the execution time outreaches 70% 1,E+06 HS-G: for large 1,E+05 scenarios is quicker 1,E+04 1,E+03 than HS, but the gain 1,E+02 is only 47% (avg) 1,E+01 Time (logarithmic scale) 1,E+00 Evolution of Time small medium large Type of Scenario Number of Visited States Volume of State Space small medium large Type of Scenario Evolution of Time ES HS HS-G Time (sec) small medium large Type of Scenario ES HS HS-G HS HS-G HDMS'05 - A. Simitsis - 25/08/

14 Outline Introduction Modeling ETL Optimization as a State-Space Problem Problem formulation State Generation Transition Applicability State Space Search & Experimental Evaluation Conclusions & Future Work HDMS'05 - A. Simitsis - 25/08/ Conclusions & Future Work We set up the problem of optimization of ETL workflows as a state-space search problem, by modeling an ETL workflow as a state We define transitions in the search space, along with their applicability and prove the correctness of applicability rules We provide three search algorithms and explore their performance Future work can be pursued in different directions: Optimization at the physical level ETL Optimization for non-traditional data (XML, biomedical, ) Parallel ETL processing HDMS'05 - A. Simitsis - 25/08/

15 Thank you HDMS'05 - A. Simitsis - 25/08/