Modernize Data Warehousing

Transcription

1 Modernize Data Warehousing with Hadoop, Data Virtualization, and In-Memory Techniques Philip Russom TDWI Research Director for Data Management July 24, 2014

2 Sponsor

3 Speakers Philip Russom TDWI Research Director, Data Management Tom Traubitz Senior Director, Product Strategy SAP 3

4 Introduction Data Warehouse Modernization Definition Business drivers Technology drivers Key Emerging Technologies Data virtualization for agility and data integration at run time In-memory data management & analytic processing for speed Hadoop for massively scalable storage and analytic processing Data Fabric Combination of in-memory functions, virtualized data, and massive big data in Hadoop Recommendations PLEASE #Hadoop, #BigData, #Analytics, #DataFabric, #LogicalDW, #RealTime

5 DW Modernization takes many forms Additions to existing data warehouse environment (or ecosystem) New data subjects, sources, tables, dimensions, etc. More server instances, nodes, bigger storage More standalone data platforms and tools Complement DW without replacing it Tools for analytics, real time, new data types, new interfaces New appliances, columnar databases, Hadoop, NoSQL, etc. Architectural Adjustments Logical DW design across multiple platforms Extending data integration (DI) Upgrades Newer versions of current database or integration software Bigger and faster hardware Rip and Replace Decommission current DW platform or DI tools; migrate to others

6 Reasons for DW Modernization Does your primary enterprise data warehouse have an architectural design? Yes 79% No 18% Don t know 3% Is the architecture of your data warehouse environment evolving? Yes moderately 54% Yes dramatically 22% No except with DW updates 22% Don t know 2% What technical issues or practices are driving change in your DW architecture? Advanced analytics 57% Increasing data volumes 56% Real-time operations 41% Business performance mgt 38% OLAP 30% Non-relational data 25% Virtualization of data 23% Cloud adoption 21% Streaming data 15% What business issues or practices are driving change in your DW architecture? Competitiveness 45% Fast-paced business processes 43% Compliance 29% Funding 29% Sponsorship 26% Reorganizations 25% Centralizing business control 30% Departmental power struggles 19% Mergers and acquisitions 18% Source: TDWI survey run in late Based on 538 respondents.

7 Modern, Data-Driven Business Practices Run the business by the numbers Requires fresh data, from best sources, delivered fast, to key people Complete information Complete customer views, enterprise-scope data, social media, big data Predict the future via analytics This is the next step for many orgs Goes beyond real-time operations Operational Excellence Efficiency & accuracy result from practices that are data-driven, real time, lean, agile, based on all data Keep pace with accelerating business processes See that a customer is about to churn; immediately reach out to retain him Monitor operations for greater speed, yield, quality Compete by being faster that competitors

8 Why are data-driven applications challenging? Because it takes time To develop a new data solution Plan Learn about business goals, find their right business people to work with, collect business requirements for data, translate these into a technical specification, get approval to access preferred sources, presenting a prototype, getting feedback from business users Develop Coordination with technical staff, profiling source data, improving meta and master data, developing data transforms, modeling data for targets, assuring trustworthy data, mapping from sources to targets, designing a test, actual testing, peer review, code approval, roll out To execute a data solution at run time Get and prepare data The solution boots up, attaches to source systems, extracts data, merges and transforms data, persists (or pipes) the data to a target system Present data Refresh a report, analysis, or app; alert a user or system

9 Data Warehouse Modernization Demands A Reformation of Data Mgt Practices Less a priori data preparation. Do more ETL, staging & aggregation on the fly, as data s needed. Less up front modeling and transformation. Remodel & transform data on the fly, close to real time, as form of late binding. Less bulk data cleansing. Instead of scrubbing whole databases regularly, cleanse data as it is created, accessed or updated. Less data movement. Instead, more direct access to operational data (& Hadoop) in real time or close to it. Less redundancy of data stores. Instead of generating yet another mart or data store, access sources, as needed; join and aggregate on the fly. Free up compute cycles for: More data, fewer samples. For more complete views, better insights.

10 Data Mgt is like two decks of cards Today s traditional techniques, tools, teams, data types New & upcoming techniques, tools, teams, data types The two will be shuffled and folded together into one card deck. Note that the new does not replace the old. They will coexist.

11 ENABLING TECHNOLOGIES for DW MODERNIZATION Data Virtualization Purely semantic views of data structures No physical data, until view is materialized Benefits of data virtualization Doesn t prep & persist a lot of data on the off chance a user or app might need it Collects fresh data, as needed, instead of hoarding stale data Various processing available Some views are read only Others can write data and perform data processing functions (or call them) E.g., views that represent joins or aggregates Virtualization intersects with real time Most views (but not all of them) execute in real time (or close) when materializing data

12 ENABLING TECHNOLOGIES for DW MODERNIZATION In-Memory Data Functions Data mgt & processing in server memory Rarely a DBMS in memory Usually a data subset in memory Benefits of in-memory data Eliminates disk IO, which is traditional bottleneck for data mgt Provides high performance for many datadriven applications, including data virtualization Various processing available Simple table cached in memory E.g., table of metrics/kpis for dashboards Multidimensional data E.g., cube of sales data for intraday analysis Analytic models and scores E.g., rescored intraday to spot/report likely churn

13 ENABLING DW MODERNIZATION Why Hadoop Now? Organizations want more business value from big data The primary path to value from big data is through analytics Analytics, in turn, demands new best practices in data warehousing (DW) and data integration (DI) Hadoop is built to manage multi-terabyte big data, process algorithmic analytics, and execute ETL logic at massive scale Hadoop configurations scale & perform at very low cost Compared to similar configurations with relational tech Hadoop complements DW, DI, Analytics; doesn t replace them Hadoop expands the biz value of these traditional platforms Hadoop does more than data warehousing Also enables enterprise storage, data archiving, content mgt

14 Good Reasons for Integrating Hadoop with Relational DWs, etc. A Relational DBMS is good at: Metadata management, anything relational Complex query optimization Query federation Table joins, views, keys, etc. Security, including roles, directories Administration via mature tools HDFS & other Hadoop tools are good at: Massive scalability File-based data (logs), sensor data, machine data Algorithmic analytics: data mining, statistics, AI, NLP Lower cost than most DW platforms & analytic DBMSs Multi-structured data, no-schema data, NoSQL processing Some ETL functions; late binding; custom code for analytics Use Hadoop Like a very scalable operational data store or data staging area To modernize your existing DW environment & prepare for big data

15 COMBINING TECHNOLOGIES for DW MODERNIZATION In-Memory Data Fabric Data Fabric is a unified view (or collection of views) of data in multiple systems across an enterprise Plus a simplified (yet diverse & performing) collection of interfaces into such sources and targets The point of a data fabric is to provide: A fairly comprehensive big picture of enterprise data A single layer through which data can be accessed, thereby reducing data redundancy, movement, processing A simplified view & mechanism that enables more user types In-Memory Data Fabric (IMDF) is combination of things: The data fabric, in-memory data functions, data virtualization, possibly Hadoop, integrated w/usual applications, databases, & data mgt tools Benefits of IMDF A high-performance form of a data fabric, due to in-memory data functions, parallel processing, direct interfaces, optimization, massive Hadoop data, etc. Real-time speed for time-sensitive biz practices, lean data mgt, scalability, embedding analytics in applications, operationalization, etc.

16 Data Fabric Example 1 3 rd Party Data Public Cloud Mart Mart DATA FABRIC Dashboards DW Reports ODS ODS OLAP Cubes Doc Mgt Data Archive Workflow Graph Hadoop NoSQL ERP SFA CRM Call Ctr Finance Supply Chain Billing Ship g

17 Data Fabric Example 2 3 rd Party Data Public Cloud Mart Mart DW FABRIC Dashboards DW Reports ODS ODS OLAP Cubes App FABRIC Doc Mgt Data Archive Workflow Graph Hadoop NoSQL ERP SFA CRM Call Ctr Finance Supply Chain Billing Ship g

18 Recommendations Revaluate your data warehouse and enterprise data management infrastructure There s always room for improvement Prepare to embrace big data, analytics, real time Prioritize modernization by putting biz goals first Biz wants to manage big data and leverage it Biz wants to compete on analytics Biz needs real-time tech to operate faster Biz needs data-driven solutions sooner, with better alignment Technology goals are also important Assure capacity for growth consider Hadoop for big data Put in-memory functions & data virtualization together in a data fabric Use in-memory functions for speed and as a point of integration Use data virtualization agile development & integration at run time For greatest success with a data fabric, infuse it with ample data virtualization and in-memory caching and processing. The result is an In-Memory Data Fabric (IMDF)

19 Reinventing the Data Warehouse with Big Data

20 How will you turn new signals into business value? :-) Brand Sentiment Predictive Maintenance Network Optimization Insider Threats Smart Equipment Asset Tracking Personalized Care Product Risk Mitigation, Smart Vending Recommendation Real-time Propensity to Churn Real-time Demand/ Supply Forecast 360 O Customer View Fraud Detection Smart Cities

21 CANCER PATIENTS RECEIVE TREATMENT OPTIMIZED TO THEIR DNA SAP MAKES BIG DATA REAL

22 Simplified, Accelerated, Predictive Transactions Traditional: OLTP and OLAP Separate ETL Transactions Streams OLTP + OLAP in SAP HANA Current Data 48/hr Old Data SAP HANA Multiple Data Sources EDW Smart Data Access Streams Staging DB Multiple Data Sources available with Live Access 10:00 AM Lots of separate ETL processes! 10:00 AM 48 Hours 10:00 AM 10:00 AM Immediate

23 Reinventing the Data Warehouse with an In-Memory Data Fabric Results at the Speed of Memory In-memory Platform SQL or SAP River Business Applications Data Fabric Layer Orchestrator Tightly integrated orchestration for management, monitoring, and control Stream SDA Column Storage SDA MapReduce/ Hive ETL & Rep for RT sync Other Sources Real-time Events/ Machinegenerated Data Petabytes of Structured Data Op RDBMS Load Source Databases

24 World s Largest Data Warehouse NEW Guinness World Record Largest Data Warehouse Audited Record: 12.1 Petabytes Tested Configuration 22x HP ProLiant DL580 G7 4x Intel Xeon 2.40GHz 1TB RAM 20x NetApp Storage Arrays E5460s 60/120 x 3TB 7.2Krpm HDD 4 x Fibre Chanel connections SAP IQ 16 (20 nodes) SAP HANA (5 nodes) BMMsoft Federated EDMT 9 with UCM Red Hat Enterprise Linux 6.4 X

25 Streams Real-time Applications on Business + Context Data Across Data Domains Real-time Applications, Interactive Analysis SQL.NET NodeJS Javascript MDX Other SQL Java Scala Python Other In-Memory Processing SAP HANA Geospatial Predictive Planning/ Rules Text/NLP SAP HANA smart data access SHARK MLlib Spark Spark Streamnig GraphX In-Memory Persistence Columnar Data Tachyon Data Access Distributed File Persistence HDFS / Any Hadoop Data Source ERP SCM CRM Text Geospatial Sensor Social Media Logs

26 Real-time insights mean real-time results 100% accuracy in early signal detection 216x faster DNA results from 2 days to 20 minutes $1.1M increased revenue with 1% increased retention rate 500k Euro working capital reduction with 1 week 50,000 daily sports betting games analyzed in real-time 3.2M reclaimed by identifying fraudulent insurance charges

27 Isn t It Time To Reinvent Your EDW Strategy? SIMPLIFY ACCELERATE PREDICT SAP In-Memory Data Fabric A Complete EDW Architecture for All Your Knowledge Workers

28 Thank you Contact information: Tom Traubitz

29 Questions? 29

30 Contact Information If you have further questions or comments: Philip Russom, TDWI Tom Traubitz, SAP 30