Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics. Erich Schneider, Daniel Rutschmann June 2014

Transcription

1 Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics Erich Schneider, Daniel Rutschmann June 2014

2 Disclaimer This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent SAP AG or an SAP affiliate company. All rights reserved. 2

3 Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics Key Trends Impacting Data Warehousing 2014 SAP AG or an SAP affiliate company. All rights reserved. 3

4 Machine Data Run Connected: B2C and IoT 2014 SAP AG or an SAP affiliate company. All rights reserved. 4

5 Un-Structured Data Run Connected: B2C 2014 SAP AG or an SAP affiliate company. All rights reserved. 5

6 More Data in More Areas e.g. Healthcare - Genomics, DNA analysis 2014 SAP AG or an SAP affiliate company. All rights reserved. 6

7 Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics Big Data Introduces More Complexity to Traditional System Architectures 2014 SAP AG or an SAP affiliate company. All rights reserved. 7

8 Challenges and Inefficiencies Analysts: High Latency Talent Shortage Model Proliferation Usability Shortcomings Lack of Visualization Planning 1 Order Processing Operational Reporting RT Risk & Fraud Trend Analysis Sentiment Analytics Predictive Analytics Pattern Recognition Spatial Processing Predict Monitor 0 Communicate Cache Cache Cache Cache Cache Cache Analyze Explore 0 Report Data Stores Integrate/Load 1 0 Complex Slow Costly Staging Clean-Data Quality 1 0 Collect Transact Operational Datastore Data Warehouse Sensors Mobile Archives Social & Text Geo-Spatial Business & IT: Fragmented Point Solutions Lack of Decision Support Segregated Organization Structure Lack of Data Governance 2014 SAP AG or an SAP affiliate company. All rights reserved. 8

9 Traditional Data and Information Architecture (example) RMS-based Legend Traditional RMS Big Data File System ERP Business Suite Custom OLTP Non-SAP ERP Machine Data Social Data ETL EIM ETL Events Operational Data Stores OLAP DW OLAP EDW Data Mart #1 Data Mart #2 Data Mart #3 Data Mart #4 Data Mart #N Planning Systems GRC Systems SAP BI Systems 3 rd party BI Systems Custom BI Systems Predictive OLAP Systems Sentiment OLAP Systems Transactional Systems Analytical Systems Data Access systems 2014 SAP AG or an SAP affiliate company. All rights reserved. 9 3 rd party ETL

10 Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics Simplification with SAP In-Memory Computing 2014 SAP AG or an SAP affiliate company. All rights reserved. 10

11 Re-think IT landscape simplicity with SAP HANA in-memory Eliminate redundant data copies and simplify applications Separated Transactions + Analysis + Acceleration processes An innovative data management and application approach for transactions, analytics and custom development using an in-memory platform One in-memory atomic copy of data for Transactions + Analysis ETL ETL Cache Transact Analyze Accelerate SAP HANA (DRAM)! Redundant data in and across applications! Inherent data latency VS! Eliminate unnecessary complexity and latency! Accelerate through simplification 2014 SAP AG or an SAP affiliate company. All rights reserved. 11

12 Simpler landscape Integration of data types, data operations and applications processing in on platform Any Apps SQL MDX R JSON Open Connectivity SAP Business Suite & SAP BW SAP HANA Platform SQL, SQLScript, JavaScript Spatial Business Function Library Search Predictive Analysis Library Text Mining Database Services Stored Procedure & Data Models Planning Engine Application & UI Services Rules Engine SAP HANA One System Integration Services 2014 SAP AG or an SAP affiliate company. All rights reserved. 12

13 Unification via SAP HANA Live in SAP Business Suite on HANA Operational Reporting and Foundation for new class of applications SAP Business Suite Applications SAP HANA PLATFORM Zero latency! Operational Reporting HANA Views Database Layer Physical Tables Atomic data set for detailed drill-down information Purchasing Manager Operational data available instantaneously Pre-defined models across entire suite 2014 SAP AG or or an an SAP SAP affiliate company. All rights All rights reserved. reserved. 13

14 Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics So Why an EDW at all? 2014 SAP AG or an SAP affiliate company. All rights reserved. 14

15 Simplification with SAP HANA And what it means for Data Warehousing BI Data Warehouse ERP Some Simple Querying CRM Some Simple Querying Some Simple Querying Historical Reporting Planning Consolidation Integration / Harmonization Operational Reporting ERP Operational Reporting CRM Operational Reporting HANA HANA HANA Legacy 2014 SAP AG or an SAP affiliate company. All rights reserved. 15

16 Traditional EDW s can be Streamlined by Focusing on Original Intent Eliminate the misappropriation Provide a single source of truth! Data harmonization and integration capabilities for heterogeneous data! Audit proof Sealed, Signed and Delivered data persistence (instead of Excel spreadsheets) Regulatory Legal Enterprise-compliant Enable + Design + Maintain + Govern consistent meta data, master data and KPI s from! Diverse Information sources! Multiple Technologies! SAP Data! non-sap Data 2014 SAP AG or an SAP affiliate company. All rights reserved. 16

17 EDWs provide Data Management and Transformation Capabilities In addition to on-the-fly Analytics Centralized processes to move/manage data flows and transformations to harmonize! Enterprise-wide master data like hierarchies, time-dependent data etc.! Calculated and restricted KPI s Data Snapshots in general, e.g.:! Inventory by time period! Data from batch interfaces! Consistent Real-Time data (for example Headcount KPI s, reports for Board meetings etc.)! Any versions of data based on simulations or manipulations which need to be shared across the enterprise, as results of on-the-fly calculations 2014 SAP AG or an SAP affiliate company. All rights reserved. 17

18 Enterprise Data Warehouse Capabilities will be Required in Support of Real-time Enterprises Provide Information Lifecycle Management! Data from Legacy systems as from Mergers & Acquisition! Non-real-time required data from real-time systems as ERP, CRM etc.! Corporate Memory data required to adjust historical information to new business rules Data not ready to be archived yet! Streamed data like un-structured social media data, machine data storage Not required for real-time business processes Long-term trending analysis Optimize overall TCO! Manage multi temperature storage media! Minimize hot data back-up Accelerate time to restore mission-critical data from hot data back-up in real-time systems 2014 SAP AG or an SAP affiliate company. All rights reserved. 18

19 Data Warehousing in the Age of In-Memory Computing and Real-Time Analytics The Logical Data Warehouse is the New EDW 2014 SAP AG or an SAP affiliate company. All rights reserved. 19

20 Logical Data Warehousing (LDW) for Big Data and Business Data Traditional EDWs have outlived their purpose Business in Real-Time: Volume Variety Velocity Value Structured Data Un-Structured Traditional Data Warehouse on RMS Real-Time Connected SAP HANA Platform In-Memory Batch Real-Time 2014 SAP AG or an SAP affiliate company. All rights reserved. 20

21 Journey to an SAP HANA based LDW Logical EDW for SAP and non-sap platforms powered by SAP IMDF* Microsoft Smart Data Access (SDA) Virtual Data consumption of SAP and non-sap data across different data bases and storage media SAP UI HTML5 Mobile SAP BI 4 Fiori IBM 2 IBM Netezza Custom Apps ERP SAP Business Suite SRM SCM CRM PLM SAP EIM SAP BW HANA Native Apps SAP HANA Hot Oracle SDA SAP HANA Platform Warm Teradata SAP IQ SAP ASE SAP SA SAP ESP Cold * In-Memory Data Fabric 2014 SAP AG or an SAP affiliate company. All rights reserved. 21

22 Information Architecture with the SAP HANA Platform smarter 2014 SAP AG or an SAP affiliate company. All rights reserved. 22

23 SAP HANA and SAP ESP Streaming Data as IoT Enabler Analyze after 24h Limited value in isolated events Traditional Data Warehouse Event window e.g. 30 min vs. Continuous Sensor readings - single server 1 Mio/sec constant stream 2 Mio/s (peak) multi server: 5 Mio+/sec Analyze in Real-Time ESP HANA Studio Long-Term Trending History 2014 SAP AG or an SAP affiliate company. All rights reserved. 23

24 Information Architecture for Internet of Things - IoT Streaming Data using the SAP HANA Platform Real-Time Access and Action Streaming Real-time Replication Data Federation Transformation Loading Many Sources SDA Data Services SLT / SRS SAP ESP Engine Real-Time Analysis without latency and redundancy SAP Business Warehouse SAP HANA Studio Data Provisioning Workbench SAP HANA Warm / Cold Data Management (NLS / Extended Storage / SDA) SAP IQ SAP PowerDesigner SAP Logical Data Warehouse Data Exploration and Visualization BI Tool 2014 SAP AG or an SAP affiliate company. All rights reserved. 24

25 Information Architecture with the SAP HANA Platform Velocity aspect of Big Data faster 2014 SAP AG or an SAP affiliate company. All rights reserved. 25

26 SAP Logical Data Warehouse on HANA Load more data in less time Faster Data Loads! Faster Activation on database level Less data layers to be propagated Elimination of data aggregation layers! Less Redundancies Real-Time replication for immediate consumption in Mixed Scenarios! Petabyte-Scale Data Management Elimination of Data Loads because of SDA Smart Data Access 2014 SAP AG or an SAP affiliate company. All rights reserved. 26

27 SAP Logical Data Warehouse on HANA Consume more data in almost real-time Faster BI! Faster Reporting! Faster Analytics! Faster Data Exploration! Faster Planning! Faster Financial Consolidation Across Real-time data flow via streaming engine In-Memory Hot Storage Warm storage Cold storage Hadoop 2014 SAP AG or an SAP affiliate company. All rights reserved. 27

28 Information Architecture with the SAP HANA Platform simpler 2014 SAP AG or an SAP affiliate company. All rights reserved. 28

29 SAP Logical Data Warehouse on HANA Lower TCO with more Agility Simpler Data Model with BW and HANA! No Aggregates! No Indices! No separate layers for performance On-the-fly Transformation! Master Data! Data Model Less IT Involvement On-the-fly BI! Data exploration of SQL models and BW data models! On-the-fly joins using CompositeProviders and SQL data models for (near-)real-time reporting 2014 SAP AG or an SAP affiliate company. All rights reserved. 29

30 Data Warehousing in the Age of In-Memory Computing Bridging the separation between SQL and BW data modeling SAP BW on HANA = BW + HANA Studio! BW Enterprise-grade Governance! SQL Data Mart Agility Mixed Scenarios! Agile SQL data modeling complementary to BW! Virtual data model across SQL and BW data model Enables BW and SQL skills! Single environment! Extract once Use multiple times! Shared master data for both types of data models 2014 SAP AG or an SAP affiliate company. All rights reserved. 30

31 EDW Landscape Consolidation Logical Data Warehouse with BW and HANA Platform Less Data redundancy Less Data persistency Fewer Data transfers Less Data latency Less Data reconciliation Less Data correction Less Data confusion HTML5 Mobile Fiori SAP LDW SAP HANA Platform Fewer Data back-ups Less IT involvement SAP HANA... Runs smarter Runs faster Runs simpler 2014 SAP AG or an SAP affiliate company. All rights reserved. 31

32 SAP Data Warehousing Applications* SAP 3 rd -party SAS, Cognos,. SQL MDX R JSON SAP HANA Studio Open Connectivity SAP Information Steward SAP DataServices SAP HANA Platform SAP PowerDesigner SAP Business Warehouse SAP Business Intelligence SAP Event Stream Processor SAP LT Replication Server SAP InfiniteInsight - KXEN SQL, SQLScript, JavaScript Spatial Business Function Library Search Predictive Analysis Library Text Mining Database Services Stored Procedure & Data Models Planning Engine Application & UI Services Rules Engine SAP HANA One System Integration Services * Some restrictions depending on release level might apply, please refer to official SAP roadmap for details 2014 SAP AG or an SAP affiliate company. All rights reserved. 32

33 THE BW and HANA EDW STRATEGY All customers adopting SAP BW on HANA are on the right track SAP takes care that all customers will be guided to the HANA future Every new BW release will make progress on the HANA roadmap 2014 SAP AG or an SAP affiliate company. All rights reserved. 33

34 Options TODAY All Options Converge Migrate to BW on HANA and be happy SAP data dominates, external data augmented into BW Migrate to BW on HANA and leverage HANA natively SAP data and external data equally important, e.g. FI/CO data and POS data Build a Data Warehouse natively with HANA Non-SAP data dominates, e.g. Health Care, Research, TelCo, Sports, 2014 SAP AG or an SAP affiliate company. All rights reserved. 34

35 Thank You! Daniel Rutschmann Global HANA Center of Excellence RutschmannD Erich Schneider SAP HANA Solution Management ErichSap Erich Schneider 2014 SAP AG or an SAP affiliate company. All rights reserved.

36 THANK&YOU&FOR&PARTICIPATING!! Please!provide!feedback!on!this!session!by!comple6ng! a!short!survey!via!the!event!mobile!applica6on.!! SESSION&CODE:&0411& & For&ongoing&educaAon&on&this&area&of&focus,& visit& SAP AG or an SAP affiliate company. All rights reserved.

37 Useful Links SAP Community Network (SCN) SAP HANA website SAP BW on HANA FAQ SAP Suite on HANA FAQ SAP AG or an SAP affiliate company. All rights reserved. 37