E(xtract) T(ransform) L(oad)

Transcription

1 Gunther Heinrich, Tobias Steimer E(xtract) T(ransform) L(oad) OLAP

2 Agenda 1 Introduction 2 Extract 3 Transform 4 Load 5 SSIS - Tutorial 2

3 1 Introduction 1.1 What is ETL? 1.2 Alternative Approach - ELT 1.3 Motivation 1.4 Goals

4 1.1 What is ETL? ETL stands for Extract, Transform, Load Process to join multiple datasources in different formats into one target database (doesn t have to be a Data-Warehouse) Extracting data from different sources (DB2, Oracle ) Transforming into final targetformat (currencies, dates ) Loading it into the end target, e.g. the Data Warehouse 4

5 1.1 What is ETL? (cont.) 2 different approaches: With staging area Without staging area With staging area (big amount of data, slower) Pull data from source system Put in staging area (database or file = temporal data store) Transform and load it into the warehouse Without staging area (small amount of data, faster) Transformation is done in memory of ETL server 5

6 1.1 Example ETL with and without staging area 6

7 1.2 Alternative approach - ELT ELT (Extract, Load, Transform) Raw data is loaded in the Data Warehouse Transformations are performed inside the Data Warehouse Advantage: No ETL server necessary 7

8 1.2 Example - ELT 8

9 1.3 Motivation Why ETL? DB 2 DB 2 Oracle Department A Oracle Department B MS SQL Server MS SQL Server Different content & structure: Currencies Dates Measurements Datatypes Datavalues Redundancy 9

10 1.4 Goals of ETL Consolidate heterogenous & distributed datasoures in one consistent Data Warehouse Eliminiation of duplicates Adjustments of datatypes Etc To decrease the amount of datas to be used for BI irrelevant data will be ignored To enable a high quality data analysis To extract valuable data for Business Intelligence or other analytical activities 10

11 1 Overview DB 2 Extract Oracle Transform Load Data Warehouse MS SQL Server 11

12 2 Extract 2.1 Periodic Extraction 2.2 Query-based Extraction 2.3 Event-based Extraction 2.4 Immediate Extraction

13 2 Extract Extraction of data form different relevant data sources Filesystems, Relational databases Load selected data into intermediate database, called stage 4 different approaches to extract data: Periodically Query-based Event-based Immediately 13

14 2.1 Extraction - Periodically Extraction is being done periodically E.g. Daily, a few times a day Frequency depends on: dynamic of source data business requirements demand of freshness of data e.g. stocks, products 14

15 2.2 Extraction Query-based Extraction is being executed by explicit query E.g. New insert of product in database Execution of Extraction Strictly speaking, both types periodic extraction and querybased extraction can be assumed to be a form of event-based query 15

16 2.3 Extraction Event-based Extraction is being executed by A time event E.g. each time at 12pm A Database event E.g. a certain number of entries is reached An external event E.g. Stock exchange index reaches a certain amount 16

17 2.4 Extraction Immediately Used when data in target system has to be up to date Data Warehouse System uses same philosophy as the source system Changes will be saved / updated immediately 17

18 2 Overview DB 2 Extract Oracle Transform Load Data Warehouse MS SQL Server 18

19 3 Transform 3.1 Data Migration Examples 3.2 Cleansing of Data

20 3 Transform Data of heterogeneous data sources transformed to one consistent internal format to suit the format and criteria of the target system Data, schemas and data quality gets adjusted to user requirements Two types of transformations Data Migration / Integration Cleansing of Data 20

21 3.1 Transform Data Migration / Integration Data itself is being transformed between formats to be consistent throughout the target database Types of Data Integration Data type adjustment Coding conversion String harmonization Date harmonizing Measurement conversion Attribute separation / combination 21

22 3.1 Transform Datatype adjustment Examples: Character Date String Number

23 3.1 Transform Coding conversion Examples: Plain Text Purchase Price Coding 1 Coding A PuPr Coding B PP 23

24 3.1 Transform String harmonization Examples: Client Customer VCR Video Cassette Recorder 24

25 3.1 Transform String harmonization cont. Examples: Customer CUSTOMER John Doe Doe, John 25

26 3.1 Transform Date harmonization Examples: DD.MM.YYYY DD.MM.YY MM-DD-YYYY DD.MM.YYYY 26

27 3.1 Transform Measurement conversion Examples: 10 inches 25,4 cm 1 km 1000 m 27

28 3.1 Transform Attribute separation / combination Examples: Day = 18, Month = 10, Year = Choco Max, 100g Name = Choco Max Weight =

29 3.2 Transform Cleansing of Data incorrect, inconsistent and incomplete data is being corrected Data quality rules define what dirty (e.g. inconsistent) data is Processes Domain specific (address loading and corrections) Domain independent (statistical approach to compare data) Main problems which has to be solved by cleansing-process Missing Referential integrity Incorrect and illegal values Missing values and Nulls (empty strings ) Redundancy (missing normalization) 29

30 3.2.1 Transform Referential Integrity and Illegal Values Referential Integrity E.g. foreign Key leads to missing data set correction by using other data sources marking data set as erroneous or deletion of data set Illegal Values E.g. wrong address or wrong value correction by using other data sources (internal and external) if possible calculation of correct value 30

31 3.2.2 Transform Missing Values and Redundancy Missing Values Not necessarily wrong (e.g. best before date for tin opener ) consistent representation for missing values If empty: correction to NULL-Entry If certain type of standard value used (e.g. -1,, etc.): correction to NULL-Entry Knowledge about used values essential to spot empty entries Redundancy E.g. data of one dataset repeated in another (missing normalization) 31

32 3 Overview DB 2 Extract Oracle Transform Load Data Warehouse MS SQL Server 32

33 4 Load

34 4 Load Transformed data is loaded into Data Warehouse During the load-process the data quality rules checks the quality of data ( bad data will be corrected or rejected) Two types of loading Initial Loading All Data submitted Subsequent Loading Changed Data submitted Tools Used: Bulk Loader optimized for high data volume 34

35 4 Load Time of Action In night times with low workloads Reason: loading operation demand high processor loads Systems get slow or unusable Problem: Globalization of companies reduces time slots for loading Possible Solutions: Parallel Loading Partition Loading Incremental Updates (Download only changed rows) 35

36 5 SSIS - Tutorial

37 5 SSIS Demonstration of 3 lessons: Lesson 1: Creating the Project and Basic Package Lesson 2: Adding Looping Lesson 3: Adding Package Configurations 37

38 5.1 The Situation - Source Flatfile: Average Rate Currency Date EndOfDayRate 38

39 5.2 The Situation Target (Data-Warehouse) Star-Schema FactCurrencyRate DimCurrency DimTime 39

40 5.3 Process of ELT-Example Extract Load Data of flatfile Lookup / Transform Look up if CurrencyAlternateKey is in dimension-table (e.g. USD) Load Write primary key of corresponding row in new column in fact-table 40

41 5.4 Usage Queries like: Show me the daily developement of the USD Take all rows where currency = USD SELECT * FROM Fact-Table WHERE currencykey = 106 Show me the rates of all currencies on Sunday, July 24th 2008 Take all rows where date = SELECT * FROM Fact-Table WHERE timekey = 2 Time 41

42 Demonstration

43 Lesson 1: Creating the Project and Basic Package Creation of a simple ETL package extracts data from a single flat file transforms the data using lookup transformations loads the result into a fact table destination. 43

44 Lesson 2: Adding Looping Expansion of Lesson 1 By adding looping features to extract multiple flat files into a single data flow process 44

45 Lesson 3: Adding Package Configurations Expansion of Lesson 2 By adding new package configuration options Settings are outsourced to a XML-File Configuration file (XML) 45

46 Thank you for your attention