DATA WAREHOUSE CASE STUDY ANDREAS BUCKENHOFER, DAIMLER TSS

Transcription

1 A company of Daimler AG DATA WAREHOUSE CASE STUDY ANDREAS BUCKENHOFER, DAIMLER TSS

2 ABOUT ME Andreas Buckenhofer Senior DB Professional Since 2009 at Daimler TSS Department: Big Data Business Unit: Analytics

3 NOT JUST AVERAGE: OUTSTANDING. As a 100% Daimler subsidiary, we give 100 percent, always and never less. We love IT and pull out all the stops to aid Daimler's development with our expertise on its journey into the future. Our objective: We make Daimler the most innovative and digital mobility company. Daimler TSS

4 INTERNAL IT PARTNER FOR DAIMLER + Holistic solutions according to the Daimler guidelines + IT strategy + Security + Architecture + Developing and securing know-how + TSS is a partner who can be trusted with sensitive data As subsidiary: maximum added value for Daimler + Market closeness + Independence + Flexibility (short decision making process, ability to react quickly) Daimler TSS 4

5 LOCATIONS Daimler TSS Germany 7 locations 1000 employees* Ulm (Headquarters) Stuttgart Berlin Karlsruhe * as of August 2017 Daimler TSS India Hub Bangalore 22 employees Daimler TSS China Hub Beijing 10 employees Daimler TSS Malaysia Hub Kuala Lumpur 42 employees Daimler TSS Data Warehouse / DHBW 5

6 CASE STUDY Build a DWH using Oracle APEX Relational Database Management System (Oracle) Frontend / GUI Apex workshop: Flinkster open data: Deutsche Bahn AG / DB DB Rent GmbH (Creative Commons Attribution 4.0 International CC BY 4.0) Prerequisite: SQL knowledge Daimler TSS Data Warehouse / DHBW 6

7 CASE STUDY - SOURCES ON GITHUB GitHub Contains source data (folder: sourcedata) Subset of Flinkster data + additional data Contains solutions (folder: scripts) Reference to Scripts / files stored in gitub: C2CS.sql Daimler TSS Data Warehouse / DHBW 7

8 CASE STUDY - GITHUB C2CS.sql Daimler TSS Data Warehouse / DHBW 8

9 INTEREST IN RDBMS COURSE AND / OR CERTIFICATION? OR JUST REFRESH SQL KNOWLEDGE? Oracle academy database courses for SQL, modeling, and PL/SQL Self-learning courses + confirmation of certification Online exam + certificate Preparation course for professional industry certification (+ discount possible Job postings often refer to certificates If interested, contact your lecturer Daimler TSS GmbH Data Warehouse / DHBW 9

10 JUNIOR CERTIFICATION Daimler TSS GmbH Data Warehouse / DHBW 10

11 DATABASE FOUNDATIONS Daimler TSS Data Warehouse / DHBW 11

12 PROFESSIONAL CERTIFICATION Daimler TSS GmbH Data Warehouse / DHBW 12

13 DATABASE DESIGN & PROGRAMMING WITH SQL Daimler TSS Data Warehouse / DHBW 13

14 PROGRAMMING WITH PL/SQL Daimler TSS Data Warehouse / DHBW 14

15 INTEREST IN DATABASES? #NEXTGEN = DOAG NEXT GENERATION COMMUNITY DOAG conference Javaland conference If interested, contact your lecturer Daimler TSS GmbH Data Warehouse / DHBW 15

16 ACCOUNT & IMPORT SOURCE DATA Setup Oracle APEX (apex.oracle.com) Create tables in staging layer Import source data into staging layer Daimler TSS Data Warehouse / DHBW 16

17 CREATE DATA VAULT IN CORE WAREHOUSE LAYER Create Data Vault model Create tables in core warehouse layer Transfer data into Data Vault tables Daimler TSS Data Warehouse / DHBW 17

18 CREATE STAR SCHEMA IN MART LAYER Create Star schema Create tables in mart layer Transfer data into Star schema tables Daimler TSS Data Warehouse / DHBW 18

19 CREATE REPORTS AND ANALYZE DATA Create SQL reports Create graphical reports Analyze data Daimler TSS Data Warehouse / DHBW 19

20 SETUP ORACLE APEX Daimler TSS Data Warehouse / DHBW 20

21 REGISTER AT APEX.ORACLE.COM Oracle Application Express (APEX) Enter select Sign In Select request a workspace Daimler TSS Data Warehouse / DHBW 21

22 REQUEST A WORKSPACE A workspace is a virtual private database allowing multiple users to work within the same Oracle Application Express installation while keeping their objects, data and applications private Daimler TSS Data Warehouse / DHBW 22

23 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 23

24 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 24

25 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 25

26 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 26

27 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 27

28 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 28

29 REQUEST A WORKSPACE Daimler TSS Data Warehouse / DHBW 29

30 CREATE TABLES IN STAGING LAYER IMPORT SOURCE DATA INTO STAGING LAYER Daimler TSS Data Warehouse / DHBW 30

31 CREATE STAGING LAYER INTRODUCTION There are four source files that must be loaded into tables File: File: File: File: File descriptions: DokumentationDatenOpenData.pdf Additional file from 'faked' CRM system: File: OPENDATA_BOOKING_CARSHARING_STR.csv OPENDATA_CATEGORY_CARSHARING.csv OPENDATA_RENTAL_ZONE_CARSHARING.csv OPENDATA_VEHICLE_CARSHARING.csv CUSTOMER.csv Daimler TSS Data Warehouse / DHBW 31

32 CREATE STAGING LAYER TASK Examine the source files and get acquainted with the data How many columns? Which data types? Create tables in the staging area Use prefix stg_ for table names in staging area No constraints in staging area No indexes in staging area Load csv files into the staging tables Examine the data in the staging tables with SQL commands Daimler TSS Data Warehouse / DHBW 32

33 CREATE TABLES First create tables by using script No constraints in staging area No indexes in staging area STG_CATEGORY_CARSHARING -> STG_BOOKING_CARSHARING -> STG_RENTAL_ZONE_CARSHARING -> STG_VEHICLE_CARSHARING -> STG_CUSTOMER -> HKG1.sql H2KG.sql HKG3.sql HK4G.sql HKG9.sql Daimler TSS Data Warehouse / DHBW 33

34 LOAD DATA INTO TABLES IN STAGING LAYER Select SQL Workshop -> Utilities -> Data Workshop Then select Data Load -> Text Data Daimler TSS Data Warehouse / DHBW 34

35 TARGET AND METHOD Daimler TSS Data Warehouse / DHBW 35

36 TABLE OWNER AND NAME Daimler TSS Data Warehouse / DHBW 36

37 LOAD DATA FILES DETAILS Daimler TSS Data Warehouse / DHBW 37

38 LOAD DATA COLUMN MAPPING Daimler TSS Data Warehouse / DHBW 38

39 HOW MANY ROWS HAVE BEEN LOADED? Table name Number of rows STG_BOOKING_CARSHARING 1510 STG_CATEGORY_CARSHARING 20 STG_RENTAL_ZONE_CARSHARING 628 STG_VEHICLE_CARSHARING 1773 STG_CUSTOMER 8636 SGN1.sql Daimler TSS Data Warehouse / DHBW 39

40 EXAMINE STG_BOOKING_CARSHARING FOR DATA QUALITY ISSUES: WHICH FKS ARE MISSING? There are bookings in STG_BOOKING_CARSHARING that do not reference data in the table STG_RENTAL_ZONE_CARSHARING. The columns have no Foreign Key: START_RENTAL_ZONE_HAL_ID and END_RENTAL_ZONE_HAL_ID SGN2.sql Daimler TSS Data Warehouse / DHBW 40

41 WHICH 3 CUSTOMERS DROVE THE LONGEST DISTANCES (KM)? Last name First name Distance (km) Mills Lucinda Bailey Beulah 8587 Palmer Jose 7786 S5GN.sql Daimler TSS Data Warehouse / DHBW 41

42 CREATE TABLES IN CORE WAREHOUSE LAYER Daimler TSS Data Warehouse / DHBW 42

43 CREATE CORE WAREHOUSE LAYER TASK Create a Data Vault data model in the Core Warehouse Layer Which Hub tables, Link tables and Sat tables do you identify? Focus on the tables (+ business keys + primary keys + foreign keys) and not on individual columns first Design the tables on paper before creating the tables in apex.oracle.com Build teams with 4-6 members to create the model on paper Often done by using Post-its that can easily be moved around Data modeling tools or DWH generators are used afterwards to capture the model and generate database scripts Description of Flinkster data on github: DokumentationDatenOpenData.pdf Daimler TSS Data Warehouse / DHBW 43

44 CREATE CORE WAREHOUSE LAYER SCRIPTS There are two scripts at github: Drop tables in Core Warehouse Layer Create table in Core Warehouse Layer C7AN.sql C8AN.sql Daimler TSS Data Warehouse / DHBW 44

45 CREATE CORE WAREHOUSE LAYER REMARKS ABOUT THE SOLUTION Alternative solution: H_BOOKING could be integrated into the link Data transformations and data cleansing is postponed to another layer (eg data types are still varchar for date). In this case "Data Vault 2.0 architecture" is followed S_BOOKING_FLK contains columns START_RENTAL_ZONE, END_RENTAL_ZONE which are redundant in regards to the connections via link table to H_RENTAL_ZONE Columns COMPANY / COMPANY_GROUP should be stored in a reference table (or in a Hub / Sat if the business claims the data as highly business relevant) Daimler TSS Data Warehouse / DHBW 45

46 CREATE CORE WAREHOUSE LAYER REMARKS ABOUT THE SOLUTION 1 link vs 2 links: if booking data are delivered with start zone and later with end zone, then 2 links are required Primary Keys are defined as MD5 result for data type varchar2(32) for simplicity reasons. The better (= faster) alternative is to store the raw result as binary field (and column length 20). Daimler TSS Data Warehouse / DHBW 46

47 SAMPLE DATA - STAGING Data for STG_CUSTOMER from CRM system delivered on Customer_id First_name Last_name Birthday gender 77XT John Bush Male 5TF3 Susan Bird Female Data for STG_CUSTOMER from CRM system delivered on Customer_id Last_name First_name Birthday gender 77XT John Miller-Bush Male 65RK Martin Watt Male Daimler TSS Data Warehouse / DHBW 47

48 SAMPLE DATA CORE WAREHOUSE LAYER HUB TABLE Data for H_CUSTOMER H_CUSTOMER_KEY LOADDATE RECORD_SOURCE CUSTOMER_ID CRM 77XT CRM 5TF CRM 65RK Daimler TSS Data Warehouse / DHBW 48

49 SAMPLE DATA CORE WAREHOUSE LAYER HUB TABLE Data for H_CUSTOMER H_CUSTOMER_KE Y LOADDATE RECORD_SOURC E Last_name First_name Birthday Gender CRM John Bush Male CRM Susan Bird Female CRM John Miller-Bush Male CRM Martin Watt Male Daimler TSS Data Warehouse / DHBW 49

50 TABLES Daimler TSS Data Warehouse / DHBW 50

51 CREATE TABLES IN DATA MART LAYER Daimler TSS Data Warehouse / DHBW 51

52 CREATE DATA MART LAYER TASK Create a Star Schema in the Data Mart Layer I want to analyze number of bookings and distances by vehicle, rental zone, customer, category, traverse usage, booking date, and (bonus) time from + time to Design the tables on paper before creating the tables in apex.oracle.com Build teams with 4-6 members to create the model on paper Daimler TSS Data Warehouse / DHBW 52

53 CREATE DATA MART LAYER SCRIPTS There are two scripts at github: Drop tables in Data Mart Layer Create table in Data Mart Layer C3TU.sql C4TU.sql Daimler TSS Data Warehouse / DHBW 53

54 TABLES Daimler TSS Data Warehouse / DHBW 54

55 ETL LOAD DATA Daimler TSS Data Warehouse / DHBW 55

56 CORRECTION Number of bookings must reduced because of space limitations Truncate table stg_booking; Load data again into table stg_booking. Source File (contains data for Stuttgart only): OPENDATA_BOOKING_CARSHARING_STR.csv Repeat steps in slides for OPENDATA_BOOKING_CARSHARING_STR.csv and load into existing table stg_booking Daimler TSS Data Warehouse / DHBW 56

57 LOAD DATA FROM STAGING INTO CORE DWH LAYER TASK Create insert statements to load Hub tables Implement the loadings pattern for Hub, Link, and Sat tables defined during the lecture If you execute the insert again, the statement must be fault tolerant Limit data amount: CITY_RENTAL_ZONE = 'Stuttgart' Create insert statements to load Sat, Link, and reference tables Daimler TSS Data Warehouse / DHBW 57

58 LOAD DATA FROM STAGING INTO CORE DWH LAYER SCRIPTS There are several scripts at github: Truncate/delete data in Core Layer Load reference table Load Hub tables Load Link table Load Sat tables K1UL.sql K3UL.sql K4UL.sql K5UL.sql K6UL.sql Daimler TSS Data Warehouse / DHBW 58

59 LOAD DATA FROM STAGING INTO CORE DWH LAYER ARE THE SCRIPTS PRODUCTION READY? There is no logging implemented Did the statement succeed? How long did it take? How many rows where inserted? If a DML statement fails, the whole transaction is rolled back Use DML error logging (e.g. Oracle) Use sub-setting: check in the where-clause for data correctness Use loops with bulk fetch and check for data quality within the loop Daimler TSS Data Warehouse / DHBW 59

60 LOAD DATA FROM CORE DWH LAYER INTO MART LAYER SCRIPTS There are several scripts at github: Load Dimensions Load Fact table KI1X.sql KI2X.sql Daimler TSS Data Warehouse / DHBW 60

61 LOAD DATA FROM CORE DWH LAYER INTO MART LAYER ARE THE SCRIPTS PRODUCTION READY? There is no logging implemented Did the statement succeed? How long did it take? How many rows where inserted? If a DML statement fails, the whole transaction is rolled back Use DML error logging (e.g. Oracle) Use sub-setting: check in the where-clause for data correctness Use loops with bulk fetch and check for data quality within the loop No delta processing; initial/full load only Daimler TSS Data Warehouse / DHBW 61

62 SUMMARY Daimler TSS Data Warehouse / DHBW 62

63 COMPLETE SCRIPTS TABLE CREATION Drop staging tables (caution!!! Be 100% sure to execute) Create staging tables HKG0.sql HKG1.sql H2KG.sql HKG3.sql HK4G.sql HKG9.sql Drop tables in Core Warehouse Layer Create tables in Core Warehouse Layer C7AN.sql C8AN.sql Drop tables in Mart Layer Create tables in Mart Layer C3TU.sql C4TU.sql Daimler TSS Data Warehouse / DHBW 63

64 HOW MANY TABLES DO YOU HAVE? Execute script: K5XL.sql Daimler TSS Data Warehouse / DHBW 64

65 COMPLETE SCRIPTS TABLE LOAD Truncate stg tables (caution!!! Be 100% sure to execute) Load data from csv files manually with Data Workshop utility SGN0.sql Truncate tables in Core Warehouse Layer Load tables in Core Warehouse Layer K1UL.sql K3UL.sql K4UL.sql K5UL.sql K6UL.sq Load Dimensions Load Facts KI1X.sql KI2X.sql Daimler TSS Data Warehouse / DHBW 65

66 HOW MANY ROWS DO YOU HAVE? Execute script: K6XL.sql Daimler TSS Data Warehouse / DHBW 66

67 DATA VAULT 2.0 ARCHITECTURE (DAN LINSTEDT) MAPPING OF CREATE AND INSERT SCRIPTS Internal data sources OLTP OLTP External data sources OPENDATA_RENTAL_ZONE_CARSHARING.csv OPENDATA_VEHICLE_CARSHARING.csv OPENDATA_CATEGORY_CARSHARING.csv OPENDATA_BOOKING_CARSHARING_STR.csv CUSTOMER.csv HKG1.sql H2KG.sql Staging Layer (Input Layer) HKG3.sql HK4G.sql HKG9.sql Data Warehouse Backend C8AN.sql Raw Data Vault K3UL.sql K4UL.sql K5UL.sql K6UL.sql Metadata Management Business Data Vault C4TU.sql Mart Layer (Output Layer) (Reporting Layer) KI1X.sql KI2X.sql Frontend Security DWH Manager incl. Monitor Daimler TSS Data Warehouse / DHBW 67

68 AND NOW COMES THE FRONTEND / REPORTING Which 3 vehicles drove the longest distances? Which vehicle model is most popular by number of bookings? In which 5 rental zones did the most rentals begin? Which customer drove the longest distance? Distance for 1 rental Distance summed up for all rentals by customer How many rentals can be forecasted for tomorrow? In which rental zones should vehicles be available tomorrow as highest demand is expected in these zones? Daimler TSS Data Warehouse / DHBW 68

69 THANK YOU Daimler TSS GmbH Wilhelm-Runge-Straße 11, Ulm / Telefon / Fax tss@daimler.com / Internet: Domicile and Court of Registry: Ulm / HRB-Nr.: 3844 / Management: Christoph Röger (CEO), Steffen Bäuerle Daimler TSS Data Warehouse / DHBW 69