On the Logical Modeling of ETL Processes

Transcription

1 On the Logical Modeling of ETL Processes Panos Vassiliadis, Alkis Simitsis, Spiros Skiadopoulos National Technical University of Athens

2 What are Extract-Transform-Load (ETL) activities? Extract Transform & Clean Load Sources DSA DW Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

3 Preliminaries Data types (name, domain) and Constants Attributes Schemata (finite lists of attributes) RecordSets (name, schema, extension) standing for tables and record files Function types (name, a finite list of parameter data types, and a single return data type). A function is an instance of a function type Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

4 Elementary Activities Name Input Schemata Output Schema Rejections Schema Parameter List Output/Rejection Operational Semantics: an SQL statement describing the content passed to the output of the operation, with respect to its input. Data Provider/Consumer Semantics: is the output appended to the target, or overwritten? Are the input rows simply read, or removed from the sources as well? Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

5 Naïve Example of Elementary Activity Parameters Input 1 Elementary Activity + Output Input 2! Rejected Rows Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

6 Complex Example POPULATED_FIELD NotNull_A1 IN IN.A 1 OUT IN A 1 A 1 A 1 A 1 A 2 A 2 A 2 A 2 R A 3 A 3 A 3 A 3 A 4 A 4 A 4 A 4 Legend: OUT OPERATIONAL SEMANTICS R NotNull_A1 SK_A2 SELECT IN.A1 AS OUT.A1, IN.A2 AS OUT.A2,IN.A3 AS OUT.A3, IN.A4 AS OUT.A4 FROM NOTNULL_A1.IN WHERE IN.A1 NOT NULL Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

7 A Typical ETL Scenario DS.PS_NEW 1.PKEY, DS.PS_OLD 1.PKEY SUPPKEY=1 DS.PS 1.PKEY, LOOKUP_PS.SKEY, SUPPKEY COST DATE DS.PS_NEW 1 DIFF 1 DS.PS 1 Add_SPK 1 SK 1 rejected $2 rejected A2EDate rejected U DS.PS_OLD 1 Log Log Log DS.PS_NEW 2.PKEY, DS.PS_OLD 2.PKEY SUPPKEY=2 DS.PS 2.PKEY, LOOKUP_PS.SKEY, SUPPKEY COST DATE=SYSDATE QTY>0 DS.PS_NEW 2 DIFF 2 DS.PS 2 Add_SPK 2 SK 2 NotNULL AddDate CheckQTY DS.PS_OLD 2 Log rejected Log rejected PKEY, DAY MIN(COST) S 1 _PARTSUPP FTP 1 DW.PARTSUPP Aggregate 1 V 1 DW.PARTSUPP.DATE, DAY PKEY, MONTH AVG(COST) S 2 _PARTSUPP FTP 2 TIME Aggregate 2 V 2 Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

8 Template Activities Metamodel layer Elementary Activity ISA RecordSet Template layer Not Null Selection Aggregate IN Custom Layer myscenario mynot Null myselection Supplier LineItem PartSupp Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

9 Template Activities Filters SELECTION (φ(a n+1,,a n+k )) UNIQUE VALUE (R.A) NOT NULL (R.A) DOMAIN MISMATCH (R.A,x low,x high ) PRIMARY KEY VIOLATION (R.A 1,,R.A K ) FOREIGN KEY VIOLATION ([R.A 1,,R.A K ],[S.A 1,,S.A K ]) Unary Transformations PUSH AGGREGATION ([A 1,,A k ],[γ 1 (A 1 ),,γ m (A m )]) PROJECTION([A 1,,A k ]) FUNCTION APPLICATION(f 1 (A 1 ),,f k (A k )) SURROGATE KEY ASSIGNMENT (R.PRODKEY,S.SKEY,x) TUPLE NORMALIZATION (R.A n+1,,r.a n+k,a CODE,A VALUE,h) TUPLE DENORMALIZATION (A CODE,A VALUE,R.A n+1,,r.a n+k,h ) Binary Transformations UNION(R,S) JOIN(R,S,[(A 1,B 1 ),,(A k,b k )]) DIFF(R,S,[(A 1,B 1 ),,(A k,b k )]) Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

10 Example of Template Activity Activity type Input schema Parameters Output schema Rejection schema Equivalent SQL statement for the output of the activity DOMAIN MISMATCH (R.A,x low, x high ) [A 1,,A n ] [$FIELD,R.A] [$LOW,x low ] [$HIGH,x high ] [B 1,,B n ] [C 1,,C n ] SELECT A 1 AS B 1,, A n AS B n FROM R WHERE $FIELD IN [$LOW,$HIGH] Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

11 The ARKTOS II project Administration of DW Metrics Logical Model for DW, Sources & Activities Logical Design Tuning Full Activity Description Physical Model for DW, Sources & Activities Conceptual Model for DW, Sources & Activities Reverse Engineering of Sources & Requirements Collection Software Construction Software & SW Metrics Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

12 On-going & Future Work Conceptual model for the initial phases of ETL projects Logical model for activities Graph modeling for ETL scenarios [DMDW02] Optimization of ETL scenarios On-going development of a set of loosely coupled tools Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

13 Conceptual Model S2. Li nei t ems S3. Li nei t ems {Dur at i on<4h} U DW. Li n e I t e ms Or d e r Ke y Or d e r Ke y Par t Key SK Par t Key Qt yfor Li ne1 Li ne Numbe r Qt yfor Li ne2 N Qu a l i t y Qt yfor Li ne3 Pr i ce Amount Key Amount N Di s c o u n t Ta x St at usforli ne1 N St at us St at usforli ne2 N Co mmi t Da t e St at usforli ne3 Shipdate Ameri can t o Eur ope a n Co mmi t Da t e Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,

14 Graph Modeling [DMDW 02] IN AddSPK1 OUT IN SK1 OUT PAR PAR PKEY PKEY PKEY PKEY PKEY QTY QTY QTY QTY QTY COST COST COST COST COST DATE DATE DATE DATE DATE Add_ const1 SUPPKEY SUPPKEY SUPPKEY DS.PS1 SKEY in out PKEY SOURCE 1 PKEY LU_PKEY LOOKUP_PS SOURCE SK LU_SOURCE LU_SKEY Vassiliadis,Simitsis,Skiadopoulos. On the Logical Modeling of ETL Processes. CAiSE'02 Toronto,