Real World Business Intelligence and Data Warehousing. Dr. Thomas Zurek VP In-Memory Platform Business Warehouse January 2012

Transcription

1 Real World Business Intelligence and Data Warehousing Dr. Thomas Zurek VP In-Memory Platform Business Warehouse January 2012

2 Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or noninfringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent. 2

3 Agenda 1. Business Intelligence and Data Warehouses definition examples 2. What are the Challenges? 3. SQL and OLAP 4. What SAP does 5. Take Aways 3

4 Agenda 1. Business Intelligence and Data Warehouses definition examples 2. What are the Challenges? 3. SQL and OLAP 4. What SAP does 5. Take Aways 4

5 Examples of Business Intelligence Scenarios fraud detection retail company point-of-sales data & given discounts huge amounts of data a prototypical BI question screencam production analysis solar power production long tail analysis e-commerce companies like Amazon, Ebay, itunes, Netflix, translate sales of popular products into (additional) sales in the long tail BI integrated into operational processes 5

6 Long Tail Analysis (1) An Example from Amazon 6

7 Long Tail Analysis (2) Source: Chris Anderson, The Long Tail, Wired, October 2004, 7

8 Long Tail Analysis (3) Source: Chris Anderson, The Long Tail, Wired, October 2004, 8

9 Business Intelligence and Data Warehouses Business Intelligence An environment in which business users conduct analyses that yield overall understanding of where the business has been, where it is now, and where it will be in the near future (i.e. planning). Data Warehouse An implementation of an informational database used to collect, integrate and provide sharable data sourced from multiple operational databases for analyses. Provide data that is reliable, consistent, understandable. It typically serves as the foundation for a business intelligence system. 9

10 IT Governance Project Governance A Typical Data Warehouse Architecture End-user access / Presentation Reporting / Analyses / Planning Main Service : Make data available for reporting & planning tools Transform : Application specific/(dis-)aggregate/lookup Content : Application specific History : Application specific Store : IC,DSO, Info Set, Virtual Provider, Multi Provider. BI Layer ODS Data Propagation Main Service : Spot for apps/delta to app/app recovery Transform : Enriched General Business logic Content : Data source Business domain specific History : Determined by rebuild requirements of apps Store : DSO(can be logical partitioned) Business transform Harmonization Main Service : Integrated, harmonized Transform : Harmonize quality assure (in flow lookup) Content : Defined fields History : Short or not at all Long term Store : Info source IO/DSO/Z-table Data Warehouse Corp. Memory Data Acquisition Main Service : Decouple, Fast load and distribute Transform : 1:1 Content : 1 data source, All fields History : 4 weeks Store : PSA, DSO-WO. Provide data Source 1 Source 2 Source 3 Source 4 Source 5 10

11 Agenda 1. Business Intelligence and Data Warehouses definition examples 2. What are the Challenges? 3. SQL and OLAP 4. What SAP does 5. Take Aways 11

12 IT Governance Project Governance A Typical Data Warehouse Architecture End-user access / Presentation Reporting / Analyses / Planning Main Service : Make data available for reporting & planning tools Transform : Application specific/(dis-)aggregate/lookup Content : Application specific History : Application specific Store : IC,DSO, Info Set, Virtual Provider, Multi Provider. BI Layer ODS Data Propagation Main Service : Spot for apps/delta to app/app recovery Transform : Enriched General Business logic Content : Data source Business domain specific History : Determined by rebuild requirements of apps Store : DSO(can be logical partitioned) Business transform Harmonization Main Service : Integrated, harmonized Transform : Harmonize quality assure (in flow lookup) Content : Defined fields History : Short or not at all Long term Store : Info source IO/DSO/Z-table Data Warehouse Corp. Memory Data Acquisition Main Service : Decouple, Fast load and distribute Transform : 1:1 Content : 1 data source, All fields History : 4 weeks Store : PSA, DSO-WO. Provide data Source 1 Source 2 Source 3 Source 4 Source 5 12

13 Main Challenges in the Data Warehousing Layer physical connectivity to source systems many protocols many formats, code pages, unicode / non-unicode network quality source system dependency (down times, peak times, ) transformation, cleansing, scrubbing Jun 1, 2011 = = 06/01/11 = VW Touareg = VW TOUAREG = [product] = currency and unit conversions: e.g. box kg resolve ID clashes: e.g. same product no. used in different subsiduaries enrich data: add attributes from source A to data from source B consistency, integrity, compliance create one version of the truth track data flows; know where the data originated ("data provenance") keep log and other change information for audits 13

14 Main Challenges in the BI Layer calculations aggregation of facts: SUM, MIN, MAX, AVG, COUNT, COUNT DISTINCT, formulas: e.g. revenue per employee, profitability, multi-dimensionality: e.g. time region product sales org hierarchies: versioning, logic, various types of hierarchies currency and unit conversions exceptions: e.g. "good": revenue > 1 mio, "bad": revenue < security performance use efficient data structures caching precalculation planning actuals (read-only) vs plan data planning session / transaction 14

15 Main Challenges in the BI Frontend Layer The frontend layer exposes the rich functionality of the platform. many user groups casual user advanced user expert user: familiar w/ domain, data model, technology many contexts operational: any employee supervising operations, processes tactical: managers strategical: higher management, board many technologies web: browser, portals, Office (esp. Excel) specific tools dissemination via , collaboration spaces, 15

16 Agenda 1. Business Intelligence and Data Warehouses definition examples 2. What are the Challenges? 3. SQL and OLAP 4. What SAP does 5. Take Aways 16

17 SQL and OLAP: Example of a Simple Query (Standard) key figure aggregated by SUM COUNT DISTINCT key figure Calculated key figure, normalizing to the subtotal Country Material Quantity No. of Customers DE US Share per Country Pencil % (10/15) Paper % (5/15) Subtotal % Pencil % (7/18) Glue % (11/18) Subtotal % Grand Total % 17

18 SQL and OLAP: Data to Calculate the Query Result SELECT Country, Material, Customer, SUM(Quantity), 1 FROM Country Material Customer Quantity No. of Customers DE Pencil Aral 2 1 BP 3 1 Esso 1 1 Shell 2 1 Texaco 2 1 BP 1 1 This is what can be retrieved by SQL. This is the starting point for further calculations. Paper Esso rows Jet 3 1 imagine a retailer Agip 1 1 o 10000s of materials Pencil Chevron 3 1 o 10000s of customers US Glue Texaco 3 1 Agip 3 1 Elf 3 1 Exxon 1 1 Repsol 2 1 imagine a utilities or mobile phone company o millions of customers combinatorics let this result explode Shell

19 SQL and OLAP: Layer Definition for Example Query LQ: Coun, Mat,Cust, SUM(Quan), 1 L1: Coun, SUM(Quan) L5: Coun, Cust, 1 L6: Cust, 1 L2: LQ.Coun, LQ.Mat, SUM(LQ.Quan)/SUM(L1.Quan) from LQ join L1 L3: LQ.Coun, SUM(LQ.Quan)/SUM(L1.Quan) from LQ join L1 L4: SUM(LQ.Quan)/SUM(L1.Quan), from LQ join L1 19

20 SQL and OLAP: Assemble Query Result Country Material Quantity No. of Customers Share per Country LQ: Coun, Mat, SUM(Quan) LQ: Coun, Mat, SUM(1) L2 Subtotal L1 L5: Coun, SUM(1) L3 Grand Total L1: SUM(Quan) L6: SUM(1) L4 20

21 Agenda 1. Business Intelligence and Data Warehouses definition examples 2. What are the Challenges? 3. SQL and OLAP 4. What SAP does 5. Take Aways 21

22 IT Governance Project Governance What SAP Offers in this Context SAP Business Objects portfolio End-user access / Presentation o frontend tools o data quality and extraction Reporting / Analyses / Planning Main Service : Make data available for reporting & planning tools Transform : Application specific/(dis-)aggregate/lookup Content : Application specific History : Application specific Store o analytic : IC,DSO, Info applications Set, Virtual Provider, (EPM) Multi Provider. o modeling tools SAP Sybase portfolio Data Propagation o databases (ASE, IQ, ) Main Service : Spot for apps/delta to app/app recovery Transform : Enriched General Business logic Content : Data source Business domain specific History o modeling : Determined tools by rebuild requirements of apps Store : DSO(can be logical partitioned) SAP Business Warehouse Harmonization Data Acquisition Main Service : Decouple, Fast load and distribute Transform SAP : HANA 1:1 Content : 1 data source, All fields History : 4 weeks Store : PSA, DSO-WO. Business transform o DW application on top of DB Main Service : Integrated, harmonized Transform : Harmonize quality assure (in flow lookup) Content : Defined fields History : Short or not at all Long term o best practice approach Store : Info source IO/DSO/Z-table o built-in SAP semantics o in-memory DB appliance Provide data BI Layer Data Warehouse Corp. Memory ODS Source 1 Source 2 Source 3 Source 4 Source 5 22

23 SAP HANA + SAP Business Warehouse (BW) In general: DW = DB + X e.g. with X = BW Now: DB HANA Thus: DW = HANA + Y with Y = BW optimized for HANA 23

24 SAP Business Warehouse: the X or Y in more detail Data Warehouse o modeling of data flows transformations data containers o data movement and transformation processes design tools for such processes scheduling monitoring archiving o connectivity and extraction native connectivity to SAP systems and extractors first-class integration of Data Services (ETL) BI Layer o analytic modeling shared dimensions hierarchies measures + KPIs currency and unit handling time dependency / versioning formulas o dimensional data containers (cubes) o planning infrastructure modeling planning session concept planning functions o security 24

25 SAP HANA: Key Impacts on Modern DBMS Advances in Technology column-store in-memory multi-core processors data compression infiniband hard- and software bundling NoSQL (i.e. no-acid) Application-Awareness DB tailored towards the applications providing generic operations frequently used by those applications not in standard SQL (or else) examples currency conversion unit of measure conversion hierarchy logic delta management BW's DSO calculation engine planning engine 25

26 SAP HANA: In-Memory Computing Programming Against a New Scarce Resource Type of Memory L1 CPU Cache L2 CPU Cache L3 CPU Cache Main Memory Size Latency (~) 64K 256K 8M GBs up to TBs 1 ns 5 ns 20 ns 100ns Disk TBs > ns need cache-conscious data-structures and algorithms! 26

27 SAP HANA SAP Business Objects tools SQL SAP Business Suite BICS SAP HANA Other query tools / apps SQL MDX SAP In-Memory Computing Studio SAP In-Memory Database Calculation and Planning Engine Real-Time Data Replication SAP NetWeaver Business Warehouse Row & Column Storage SAP Business Objects Data Services Other data sources SAP HANA in-memory software + hardware (HP, IBM, Fujitsu, Cisco, Dell, Hitachi) data modeling and data management data acquisition Current Scenarios stand-alone data marts operational data marts analytic data marts accelerator for ERP scenarios e.g. controlling & profitability analysis (CO-PA) transparent, i.e. consumption stays with ERP DB for Business Warehouse (BW) BW optimized for HANA HANA optimizations for BW 27

28 Agenda 1. Business Intelligence and Data Warehouses definition examples 2. What are the Challenges? 3. SQL and OLAP 4. What SAP does 5. Take Aways 28

29 Take Aways 1. What are Business Intelligence and Data Warehousing? 2. What are some of the challenges? 3. SAP's efforts and products in that space. we are looking for developers contact me: 29

30 2012 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iseries, pseries, xseries, zseries, eserver, z/vm, z/os, i5/os, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, ianywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase, Inc. Sybase is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. 30