DATA WAREHOUSE PART LX: PROJECT MANAGEMENT ANDREAS BUCKENHOFER, DAIMLER TSS

Transcription

1 A company of Daimler AG DATA WAREHOUSE PART LX: PROJECT MANAGEMENT ANDREAS BUCKENHOFER, DAIMLER TSS

2 ABOUT ME Andreas Buckenhofer Senior DB Professional Since 2009 at Daimler TSS Department: Big Data Business Unit: Analytics

3 ANDREAS BUCKENHOFER, DAIMLER TSS GMBH Forming good abstractions and avoiding complexity is an essential part of a successful data architecture Data has always been my main focus during my long-time occupation in the area of data integration. I work for Daimler TSS as Database Professional and Data Architect with over 20 years of experience in Data Warehouse projects. I am working with Hadoop and NoSQL since I keep my knowledge up-to-date - and I learn new things, experiment, and program every day. I share my knowledge in internal presentations or as a speaker at international conferences. I'm regularly giving a full lecture on Data Warehousing and a seminar on modern data architectures at Baden-Wuerttemberg Cooperative State University DHBW. I also gained international experience through a two-year project in Greater London and several business trips to Asia. I m responsible for In-Memory DB Computing at the independent German Oracle User Group (DOAG) and was honored by Oracle as ACE Associate. I hold current certifications such as "Certified Data Vault 2.0 Practitioner (CDVP2)", "Big Data Architect, Oracle Database 12c Administrator Certified Professional, IBM InfoSphere Change Data Capture Technical Professional, etc. Contact/Connect Daimler TSS Data Warehouse / DHBW 3 DOAG DHBW xing

4 NOT JUST AVERAGE: OUTSTANDING. As a 100% Daimler subsidiary, we give 100 percent, always and never less. We love IT and pull out all the stops to aid Daimler's development with our expertise on its journey into the future. Our objective: We make Daimler the most innovative and digital mobility company. Daimler TSS Data Warehouse / DHBW 4

5 INTERNAL IT PARTNER FOR DAIMLER + Holistic solutions according to the Daimler guidelines + IT strategy + Security + Architecture + Developing and securing know-how + TSS is a partner who can be trusted with sensitive data As subsidiary: maximum added value for Daimler + Market closeness + Independence + Flexibility (short decision making process, ability to react quickly) Daimler TSS Data Warehouse / DHBW 5

6 LOCATIONS Daimler TSS Germany 7 locations 1000 employees* Ulm (Headquarters) Stuttgart Berlin Karlsruhe * as of August 2017 Daimler TSS India Hub Bangalore 22 employees Daimler TSS China Hub Beijing 10 employees Daimler TSS Malaysia Hub Kuala Lumpur 42 employees Daimler TSS Data Warehouse / DHBW 6

7 WHAT YOU WILL LEARN TODAY After the end of this lecture you will be able to Understand lifecycle of DWH projects Daimler TSS Data Warehouse / DHBW 7

8 LOGICAL STANDARD DATA WAREHOUSE ARCHITECTURE Internal data sources Data Warehouse Backend Frontend OLTP OLTP Staging Layer (Input Layer) Integration Layer (Cleansing Layer) Top Down Core (Inmon) Warehouse Layer (Storage Layer) Aggregation Layer Mart Layer (Output Layer) (Reporting Layer) External data sources Metadata BottomManagement Up (Kimball) Security DWH Manager incl. Monitor Daimler TSS Data Warehouse / DHBW 8

9 TOP-DOWN VS BOTTOM-UP APPROACH Top-Down (Inmon) Comprehensive approach regarding available data Design Core Warehouse Layer = integrated data model first considering all requirements Design data marts afterwards Bottom-Up (Kimball) Approach focusing on fast delivery of first results Design one data mart first Next Marts are modeled afterwards usually using Kimball architecture conformed dimensions to integrate different data marts / fact tables Daimler TSS Data Warehouse / DHBW 9

10 TOP-DOWN VS BOTTOM-UP APPROACH ADVANTAGES AND DISADVANTAGES Top-Down (Inmon) Core Warehouse Layer is designed optimal Data from Core Warehouse Layer is reused in many Marts Time-consuming approach with high preparatory effort High risk with changing requirements Bottom-Up (Kimball) Early involvement of end users Fast results Focus on single Marts leads to risk that overall view is lost, esp. properly designed Core Warehouse Layer Data often not reused but inconsistently copied across Marts Daimler TSS Data Warehouse / DHBW 10

11 THINK BIG, START LOCAL Both approaches have their down-sides Top-Down takes enormous initial effort to build data model for Core Warehouse Layer Bottom-Up is risky as central / integrated focus is lost Think big, start small Think Big: Design conceptual data model for Core Warehouse Layer covering whole enterprise Start small: Implement physical data model for Core and Mart Layer in iterations by each business department Daimler TSS Data Warehouse / DHBW 11

12 WHAT S DIFFERENT IN DWH PROJECTS? DWH is not a product DWH databases are more complex with different layers and data models Data first, code is secondary Data quality is a major concern Data integration is a challenging objective Business need difficult to justify quantitatively Daimler TSS Data Warehouse / DHBW 12

13 WHY DO DWH PROJECTS FAIL? Daimler TSS Data Warehouse / DHBW 13

14 AGILITY IN THE DWH: CASE Source: Daimler TSS Data Warehouse / DHBW 14

15 EXERCISE Define 3-5 criteria for the evaluation of an ETL tool How does a relational DBMS (like Oracle, DB2, MS SQL Server) meet these requirements? Daimler TSS Data Warehouse / DHBW 15

16 EXERCISE - DEFINE 5 CRITERIA FOR THE EVALUATION OF AN ETL TOOL Supplier profile Support HW/SW requirements License / maintenance Costs Usability Reliability Performance and scalability Multi-tenant Interfaces Scheduling Daimler TSS Data Warehouse / DHBW 16

17 EXERCISE - HOW DOES A RELATIONAL DBMS MEET THESE REQUIREMENTS? RDBMS provide many of the functionalities but additional programming required RDBMS are often used for ETL/ELT by programming with SQL, PL/SQL, SQLT, etc ETL Tool Informatica, Talend, Oracle ODI, etc. Separate license Workflow, error handling, and restart/recovery functionality included Impact analysis and where-used (lineage) functionality available Faster development, easier maintenance Additional (Tool-) Know How required Manual ETL SQL, PL/SQL, SQLT, etc. No additional license Workflow, error handling, and restart/recovery functionality must be implemented manually Impact analysis and where-used (lineage) functionality difficult Slower development, more difficult maintenance Know How often available Daimler TSS Data Warehouse / DHBW 17

18 THANK YOU Daimler TSS GmbH Wilhelm-Runge-Straße 11, Ulm / Telefon / Fax tss@daimler.com / Internet: Domicile and Court of Registry: Ulm / HRB-Nr.: 3844 / Management: Christoph Röger (CEO), Steffen Bäuerle Daimler TSS Data Warehouse / DHBW 18

19 PROJECT PHASES SMALL ITERATIONS INSTEAD OF LONG PHASES Feasibility study Analysis Design Implementation Test Operations and maintenance Daimler TSS Data Warehouse / DHBW 19

20 DATA OPS Gartner: a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and consumers across an organization. The goal of DataOps is to create predictable delivery and change management of data, data models and related artifacts. DataOps uses technology to automate data delivery with the appropriate levels of security, quality and metadata to improve the use and value of data in a dynamic environment. Source: Daimler TSS Data Warehouse / DHBW 20

21 HYPE CYCLE FOR DATA MANAGEMENT Daimler TSS Data Warehouse / DHBW 21

22 DATAOPS IS ABOUT ORGANIZATIONAL CHANGE DataOps is a new way of working and collaborating (same with DevOps) DataOps collaboration typically occurs between technical and nontechnical staff compared to DevOps Language barrier between these two parties (e.g. skills mismatch) Therefore required is a core enabler like data literacy Data literacy is the ability to understand data, to build knowledge from data, and to communicate information/meaning to others DataOps can 't be achieved by buying tools Source: Daimler TSS Data Warehouse / DHBW 22

23 BICC: BI CENTER OF EXCELLENCE Organizational team that coordinate and standardize DWH activities within an (end user) organization Define standards and create BI portfolio (e.g. which tools/products to use) Create DWH architecture and govern BI activities Establish processes for business and IT interaction Monitor DWH/BI market for new trends Determine skills and experience of Business users Daimler TSS Data Warehouse / DHBW 23

24 4-QUADRANT MODEL (RONALD DAMHOF) Source: Daimler TSS Data Warehouse / DHBW 24