DATA WAREHOUSE PART XXI: DB SPECIFICS ANDREAS BUCKENHOFER, DAIMLER TSS

Transcription

1 A company of Daimler AG DATA WAREHOUSE PART XXI: DB SPECIFICS ANDREAS BUCKENHOFER, DAIMLER TSS

2 ABOUT ME Andreas Buckenhofer Senior DB Professional Since 2009 at Daimler TSS Department: Big Data Business Unit: Analytics

3 ANDREAS BUCKENHOFER, DAIMLER TSS GMBH Forming good abstractions and avoiding complexity is an essential part of a successful data architecture Data has always been my main focus during my long-time occupation in the area of data integration. I work for Daimler TSS as Database Professional and Data Architect with over 20 years of experience in Data Warehouse projects. I am working with Hadoop and NoSQL since I keep my knowledge up-to-date - and I learn new things, experiment, and program every day. I share my knowledge in internal presentations or as a speaker at international conferences. I'm regularly giving a full lecture on Data Warehousing and a seminar on modern data architectures at Baden-Wuerttemberg Cooperative State University DHBW. I also gained international experience through a two-year project in Greater London and several business trips to Asia. I m responsible for In-Memory DB Computing at the independent German Oracle User Group (DOAG) and was honored by Oracle as ACE Associate. I hold current certifications such as "Certified Data Vault 2.0 Practitioner (CDVP2)", "Big Data Architect, Oracle Database 12c Administrator Certified Professional, IBM InfoSphere Change Data Capture Technical Professional, etc. Contact/Connect Daimler TSS Data Warehouse / DHBW 3 DOAG DHBW xing

4 NOT JUST AVERAGE: OUTSTANDING. As a 100% Daimler subsidiary, we give 100 percent, always and never less. We love IT and pull out all the stops to aid Daimler's development with our expertise on its journey into the future. Our objective: We make Daimler the most innovative and digital mobility company. Daimler TSS

5 INTERNAL IT PARTNER FOR DAIMLER + Holistic solutions according to the Daimler guidelines + IT strategy + Security + Architecture + Developing and securing know-how + TSS is a partner who can be trusted with sensitive data As subsidiary: maximum added value for Daimler + Market closeness + Independence + Flexibility (short decision making process, ability to react quickly) Daimler TSS 5

6 LOCATIONS Daimler TSS Germany 7 locations 1000 employees* Ulm (Headquarters) Stuttgart Berlin Karlsruhe * as of August 2017 Daimler TSS India Hub Bangalore 22 employees Daimler TSS China Hub Beijing 10 employees Daimler TSS Malaysia Hub Kuala Lumpur 42 employees Daimler TSS Data Warehouse / DHBW 6

7 WHAT YOU WILL LEARN TODAY After the end of this lecture you will be able to Understand DB techniques that are specific for DWH Analytic/windowing functions Bitemporal data Indexing, Partitioning, Parallelism, Compression Daimler TSS Data Warehouse / DHBW 7

8 SQL STANDARD IS EVOLVING Source: Daimler TSS Data Warehouse / DHBW 8

9 EXERCISE: COMPUTE MOST RECENT ROWS Customer_key Name Status Valid_from 1 Brown Single 01-MAY Bush Married 05-JAN Miller Married 15-DEC Stein 15-DEC Stein Single 18-DEC-2015 Write an SQL statement that computes the most recent data for each customer. Script to create the table including data: SIN1.sql Daimler TSS Data Warehouse / DHBW 9

10 EXERCISE: COMPUTE MOST RECENT ROWS Daimler TSS Data Warehouse / DHBW 10

11 EXERCISE: COMPUTE MOST RECENT ROWS create table s_customer ( customer_key integer NOT NULL, cust_name varchar2(100) NOT NULL, status varchar2(10), valid_from date NOT NULL, CONSTRAINT s_customer_pk PRIMARY KEY (customer_key, valid_from) ); insert into s_customer (customer_key, cust_name, status, valid_from) values (1, 'Brown', 'Single', to_date(' ', 'DD.MM.YYYY')); insert into s_customer (customer_key, cust_name, status, valid_from) values (2, 'Bush', 'Married', to_date(' ', 'DD.MM.YYYY')); insert into s_customer (customer_key, cust_name, status, valid_from) values (1, 'Miller', 'Married', to_date(' ', 'DD.MM.YYYY')); insert into s_customer (customer_key, cust_name, status, valid_from) values (3, 'Stein', NULL, to_date(' ', 'DD.MM.YYYY')); insert into s_customer (customer_key, cust_name, status, valid_from) values (3, 'Stein', 'Single', to_date(' ', 'DD.MM.YYYY')); commit; Daimler TSS Data Warehouse / DHBW 11

12 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 1: MAX-FUNCTION SELECT s.* FROM S_CUSTOMER s JOIN (SELECT i.customer_key, max(i.valid_from) as max_valid_from FROM S_CUSTOMER i GROUP BY i.customer_key) b ON s.customer_key = b.customer_key AND s.valid_from = b.max_valid_from; S2IN.sql Daimler TSS Data Warehouse / DHBW 12

13 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 2: EXISTS SELECT s.* FROM S_CUSTOMER s WHERE NOT EXISTS (SELECT 1 FROM S_CUSTOMER i WHERE s.customer_key = i.customer_key AND s.valid_from < i.valid_from); S2IN.sql Daimler TSS Data Warehouse / DHBW 13

14 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 3: MAX IN CORRELATED SUB-SELECT SELECT s.* FROM S_CUSTOMER s WHERE s.valid_from = (SELECT MAX(i.valid_from) FROM S_CUSTOMER i WHERE s.customer_key = i.customer_key); S2IN.sql Daimler TSS Data Warehouse / DHBW 14

15 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 4: COALESCE WITH SUB-SELECT SELECT * FROM (SELECT coalesce ((SELECT min (i.valid_from) 'DD.MM.YYYY')) as end_ts, s.* FROM S_CUSTOMER s) FROM S_CUSTOMER i WHERE s.customer_key = i.customer_key AND s.valid_from < i.valid_from ), to_date (' ', WHERE end_ts = to_date (' ', 'DD.MM.YYYY'); S2IN.sql Daimler TSS Data Warehouse / DHBW 15

16 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 5: MAX-FUNCTION AND WITH-CLAUSE WITH max_cust as ( SELECT i.customer_key, max(i.valid_from) as max_valid_from FROM S_CUSTOMER i GROUP BY i.customer_key) SELECT s.* FROM JOIN S_CUSTOMER s max_cust b ON s.customer_key = b.customer_key AND s.valid_from = b.max_valid_from; S2IN.sql Daimler TSS Data Warehouse / DHBW 16

17 EXERCISE: COMPUTE MOST RECENT ROWS SQL ANALYTIC / WINDOWING FUNCTIONS partition data compute functions over these partitions Rank [sequential order], first [first row], last [last row], lag [previous row], lead [next row] return result Daimler TSS Data Warehouse / DHBW 17

18 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 6: ANALYTIC / WINDOWING FUNCTION LEAD WITH lead_cust as ( SELECT lead (s.valid_from, 1) OVER (PARTITION BY, s.* FROM s_customer s) SELECT * FROM lead_cust b WHERE b.end_ts IS NULL; s.customer_key ORDER BY s.valid_from ASC) as end_ts S3IN.sql Daimler TSS Data Warehouse / DHBW 18

19 EXERCISE: COMPUTE MOST RECENT ROWS SOLUTION 7: ANALYTIC FUNCTION ROW_NUMBER WITH lead_cust as ( SELECT row_number() OVER(PARTITION BY s.customer_key, s.* FROM s_customer s) SELECT * FROM lead_cust b WHERE b.rn = 1; ORDER BY s.valid_from DESC) as rn S3IN.sql Daimler TSS Data Warehouse / DHBW 19

20 MAX OR ANALYTIC / WINDOWING FUNCTIONS WHICH ALTERNATIVE WOULD YOU RECOMMEND? Check execution plans, execution time including service + response time, resource usage for final decision Solutions with Analytic / Windowing do not need self-join and show better statistics compared to the other shown solutions Analytic / Windowing functions are very powerful Remark: Usage of with-clause in SQL statements is preferable compared to sub-selects as it improves readability, understandability, maintainability Daimler TSS Data Warehouse / DHBW 20

21 TEMPORAL DATA STORAGE (BITEMPORAL DATA) Daimler TSS Data Warehouse / DHBW 21

22 TEMPORAL DATA STORAGE (BITEMPORAL DATA) Price: 15EUR Price: 16EUR Valid Time (20.09.) Time New Price of 16EUR is entered into the DB Transaction Time (10.09.) Daimler TSS Data Warehouse / DHBW 22

23 TEMPORAL DATA STORAGE (BITEMPORAL DATA) DEFINITION Business validity: Valid time Time period when a fact is true in the real world The end user determines start and end date/time (or just a date/time for events) Technical validity: Transaction time Time period when a fact stored in the database is known ETL process determines start and end date/time Bitemporal data Combines both Valid and Transaction Time Daimler TSS Data Warehouse / DHBW 23

24 TEMPORAL DATA STORAGE (BITEMPORAL DATA) SQL STANDARD SQL standard SQL:2011 But different implementations by RDBMSes like Oracle, DB2, SQL Server and others Different syntax! Different coverage of standard! Very useful for slowly changing dimensions type 2, but also for other purposes Daimler TSS Data Warehouse / DHBW 24

25 DB2 VALID TIME EXAMPLE CREATE TABLE customer_address ( customerid INTEGER NOT NULL, name VARCHAR(100), city VARCHAR(100), valid_start DATE NOT NULL, valid_end DATE NOT NULL, PERIOD BUSINESS_TIME(valid_start, valid_end), PRIMARY KEY(customerID, BUSINESS_TIME WITHOUT OVERLAPS) ); Daimler TSS Data Warehouse / DHBW 25

26 DB2 VALID TIME EXAMPLE INSERT INTO customer_address VALUES (1, 'Miller', 'Seattle', ' ', ' '); UPDATE customer_address FOR PORTION OF BUSINESS_TIME FROM ' ' TO ' ' SET city = 'San Diego' WHERE customerid = 1; customerid Name City Valid_start Valid_end 1 Miller Seattle Miller San Diego Daimler TSS Data Warehouse / DHBW 26

27 DB2 VALID TIME EXAMPLE SELECT * FROM customer_address FOR BUSINESS_TIME AS OF ' '; Daimler TSS Data Warehouse / DHBW 27

28 DB2 TRANSACTION TIME EXAMPLE CREATE TABLE customer_info( customerid INTEGER NOT NULL, comment VARCHAR(1000) NOT NULL, sys_start TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW BEGIN, sys_end TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW END, PERIOD SYSTEM_TIME (sys_start, sys_end) ); Daimler TSS Data Warehouse / DHBW 28

29 DB2 TRANSACTION TIME EXAMPLE Transaction on : INSERT INTO customer_info VALUES( 1, 'comment 1'); Transaction on UPDATE customer_address SET comment = 'comment 2' WHERE customerid = 1; CustomerId comment Sys_start Sys_end 1 Comment Daimler TSS Data Warehouse / DHBW 29

30 DB2 TRANSACTION TIME EXAMPLE SELECT * FROM customer_info FOR SYSTEM_TIME AS OF ' '; Data comes from a history table: CustomerId comment Sys_start Sys_end 1 Comment Valid Time and Transaction Time can be combined = Bitemporal table Daimler TSS Data Warehouse / DHBW 30

31 INDEXING - WHY Very important performance improvement technique Good for many reads with high selectivity, write penalty B-trees most common root branch branch leaf leaf leaf Table Daimler TSS Data Warehouse / DHBW 31

32 INDEXING A STAR SCHEMA WHICH COLUMNS ARE CANDIDATES FOR AN INDEX? DBs index Primary Keys by default Dimension table columns that are regularly used in where clauses are candidates Maybe foreign Key columns in Fact table (see also later Star Transformation) Daimler TSS Data Warehouse / DHBW 32

33 STAR TRANSFORMATION Fact table has normally much more rows compared to dimension tables Common join techniques would need to join first dimension table with the fact table Alternative technique: evaluate all dimensions (cartesian join) Then join into fact table in last step Oracle uses Bitmap indexes on foreign key columns in fact tables to achieve Star Join; not supported by many DBs Daimler TSS Data Warehouse / DHBW 33

34 PARTITIONING Col1 Col2 Col3 col4 1 A AA AAA 2 B BB BBB 3 C CC CCC Vertical partitioning Horizontal partitioning Col1 Col2 1 A 2 B 3 C Col3 AA BB CC col4 AAA BBB CCC Col1 Col2 Col3 col4 3 C CC CCC Col1 Col2 Col3 col4 1 A AA AAA 2 B BB BBB Daimler TSS Data Warehouse / DHBW 34

35 HORIZONTAL PARTITIONING Very powerful feature in a DWH to reduce workload Split table into logical smaller tables Avoidance of full table scans How could a table be split? Introduction to (Oracle) partitioning: Daimler TSS Data Warehouse / DHBW 35

36 HORIZONTAL PARTITIONING SPLITTING OPTIONS By range Most common Use date field like order data to partition table into months, days, etc By list Use field that has limited number of different values, e.g. split customer data by country if end users most likely select customers from within a country By hash Use a filed that most likely splits the data in evenly distributed chunks Daimler TSS Data Warehouse / DHBW 36

37 PARALLELISM Statements are normally executed on one CPU Parallelism allows the DB to distribute the execution to several CPUs Powerful combination with partitioning Parallelism is limited by the number of CPUs: if parallelism is too high, performance will degrade Intra-query parallelism and inter-query parallelism Daimler TSS Data Warehouse / DHBW 37

38 COMPRESSION Data compression + Index compression Store more data in a block/page = read more data during I/O If CPU resources are available, often a very powerful feature to improve performance Additionally reduce storage Additionally reduce backup time Daimler TSS Data Warehouse / DHBW 38

39 ALREADY COVERED IN A PREVIOUS LECTURE Relational columnar In-Memory DB Materialized Views / Query Tables Daimler TSS Data Warehouse / DHBW 39

40 EXERCISE - RECAPTURE ETL AND DB SPECIFICS Recapture ETL and DB specific topics Which topics do you remember, or do you find important? Write down 1-2 topics on stick-it cards. Daimler TSS Data Warehouse / DHBW 40

41 THANK YOU Daimler TSS GmbH Wilhelm-Runge-Straße 11, Ulm / Telefon / Fax tss@daimler.com / Internet: Domicile and Court of Registry: Ulm / HRB-Nr.: 3844 / Management: Christoph Röger (CEO), Steffen Bäuerle Daimler TSS Data Warehouse / DHBW 41