Orchestration of Data Lakes BigData Analytics and Integration. Sarma Sishta Brice Lambelet

Size: px

Start display at page:

Download "Orchestration of Data Lakes BigData Analytics and Integration. Sarma Sishta Brice Lambelet"

Gwen Simon
6 years ago
Views:

1 Orchestration of Data Lakes BigData Analytics and Integration Sarma Sishta Brice Lambelet

2 Introduction

3 The Five Megatrends Driving Our Digitized World And Their Implications for Distributed Big Data Management Hyper Connectivity Super Computing Cloud Computing Smart World Cyber- Security Everybody has access Super computers power everywhere The cloud is where we compute Your fridge knows what you want for dinner High-powered security is now the norm 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 3

Common Types of Big Data New Types of Data: 2016 SAP SE

How Big is the Need for Digital Enterprise?

Transforming Businesses of Any Kind into Digital Enterprise Employees Workforce Engageme nt Digital Core Customer Omni-Channel Customer Experience N E W Assets & Internet of Things 01001010111010010

6 Transforming Businesses of Any Kind into Digital Enterprise Employees Workforce Engageme nt Digital Core Customer Omni-Channel Customer Experience N E W Assets & Internet of Things Partners Supplier Collaboration T R A D I T I O N A L IT Budget 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 6

7 The Challenges with Distributed Big Data in Our Digital Economy With all this new data connecting us we should be sailing smoothly. Unfortunately we re drowning in our own data. Inefficient Data Processing Real-time drill-down interaction is impossible when data is distributed across thousands of nodes and processed in batches Lack Business Alignment Need to align business decisions to changing external market conditions by processing data in business systems with Hadoop Data Lakes together. Costly Managing of Big Data Extensive amounts of data start clogging business systems with data that can be more efficiently archived to less expensive systems 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 7

8 Businesses turn to Hadoop & other Big Data platforms for Low Cost & Large Scale Schema Flexibility All Data N E W T R A D I T I O N A L IT Budget 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 8

10 BigData Architecture

11 Introduction Technical Architecture for Big Data In the last years more and more business scenarios are collecting and/or generating large amount of data (e.g. web applications, utility) and therefore scalable and reliable storage is needed. The data volume could be up to several Terabyte per day and several hundreds of Petabyte in total. The Hadoop framework offers a scalable and reliable storage framework. It is open source and has a large community which support the various projects. SAP is embracing Hadoop as part of the HANA platform. Data Volume: Data Lakes Data Variety: Data Velocity: 1 TB per day Data generated by 1 productive well 11,475 Production wells in Canada Social Media Sensor Data GPS Meter Data Real-time Event Streaming IT Devices *Example from Oil&Gas Industry Clickstream 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 11

12 Introduction Hadoop is based on work done by Google in the early 2000s Specifically, on papers describing the Google File System (GFS) published in 2003, and MapReduce published in Hadoop is Open Source (using the Apache license) Hundreds of contributors writing features, fixing bugs There are many related projects, applications and tools which are enhancing the Hadoop eco system. Data distribution and parallel processing framework The Hadoop frame work is based on a Shared nothing architecture. Computation happens where the data is stored, wherever possible. The goal is that nodes talk to each other as little as possible. Reliability and Fault Tolerance Data is replicated multiple times on the system for increased availability and reliability. If a node fails, the master will detect that failure and re-assign the work to a different node on the system SAP SE or an SAP affiliate company. All rights reserved. Customer 12

Introduction Find Balance between Performance and Storage Costs HANA Data Management Platform Data Replication Streaming Data Documents, Logs, SAP HANA Processing Engines (e.g. Spatial, R, Graph) Hadoop Processing Engines (e.

13 Introduction Find Balance between Performance and Storage Costs HANA Data Management Platform Data Replication Streaming Data Documents, Logs, SAP HANA Processing Engines (e.g. Spatial, R, Graph) Hadoop Processing Engines (e.g. Spark, Hive, ) HDFS HANA native BigData Smart Data Streaming NoSQL Graph Time Series Dynamic Tiering HANA & Hadoop SDA MapReduce Admin & Monitoring Hadoop Extension SAP HANA Vora 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 13

HDFS Store historic data in Hadoop, Archive Integr

14 Introduction Different Scenarios for HANA Data Management Platform: Part of SAP ILM Data Science S/4HANA, BWoH Dynamic Tiering Data Aging SAP HANA Processing Engines (e.g. Spatial, R, Graph) Hadoop HDFS Streaming Data Documents, Logs, Hive Hadoop Processing Engines HDFS Store historic data in Hadoop, Archive Integration with Information Lifecycle Management (ILM) Store volume data (e.g. streaming data, logs) in Hadoop ETL (Extract, Transform, Load) in Hadoop 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 14

15 Append Batch Processing Introduction Excursus: Lambda Architecture Realtime Stream Processing increment Realtime View New Data Speed Layer Query and Merge Master Data Set overwrite Precomputed Batch Views Batch Layer Serving Layer 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 15

16 Append Batch Processing Introduction Excursus: Lambda Architecture with SAP SAP HANA SAP Realtime HANA Smart Stream Data Processing Streaming increment SAP HANA Realtime In-Memory View Store New Data Speed Layer SAP HANA Query and Processing Merge Engines Master Data Set overwrite Precomputed HANA SAP Batch Vora Views Batch Layer Hadoop Serving Layer 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 16

17 HANA Data Platform Components Overview (1) Databases and Extensions Component Explanation SAP HANA In-Memory database and central component of the data platform SAP IQ A fast disk-based column-store database scaling up to petabytes of data SAP HANA Dynamic Tiering Accessing other data tiers within on SQL utilizing an Extended Store concept SAP HANA Vora In-Memory relational-mapper for Hadoop and Spark HANA Interfacing and Integration tools Component Explanation Smart Data Access (SDA) Accessing other databases and relational mappers from HANA Smart Data Integration (SDI) HANA integrated ETL/ replication solution (created from data services and SRS) Data Lifecycle Manager Automatic data movement to other tiers 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 17

18 HANA Data Platform Components Overview (2) Streaming Component Explanation Smart Data Streaming (SDS) Integrated HANA streaming solution Event Streaming Processor Standalone streaming solution Event Streaming Processor Lite Solution for decentralized streaming based on SQL Anywhere Standalone Replication Components Component Explanation Data Services Standalone replication solution for ETL SAP Landscape Transformation Realtime replication solution based on SAP NetWeaver ABAP SAP Replication Server Realtime replicaton solution based on log level database access Remote Data Sync Remote databses which are synced with an backend synchronous or asynchronously 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 18

19 SAP Vora & BigData Analysis

20 What s Stopping Us? The Digital Divide between Enterprise and Big Data ENTERPRISE BIG DATA Too Complex Too Slow Unable to Work Together SAP SE or an SAP affiliate company. All rights reserved. Customer Internal 20

23 In-Memory Data Fabric for Enterprise + Distributed Compute ALL IN-MEMORY CONSUME COMPUTE STORE HANA OLTP + OLAP Scale Up + Scale Out SAP HANA + Tiering Appliance TDI Federated Queries & Programming Model SAP Vora Vora Vora Vora Vora Vora Vora Vora Vora Vora Vora Vora Vora Distributed compute - Massive Scale Out Distributed File System Network Storage Cloud Persistence Any Hardware SAP Vora Benefits Add functionality for enterprise applications Hierarchies OLAP modeling Boost SQL performance Federate access across SAP HANA and Hadoop Integrate tooling 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 23

24 Hadoop Background

Hadoop Background Cluster Overview 2016 SAP SE or an

26 Required Components Software Requirements for installing Vora 1.4 Hadoop Distributions: HDP 2.5+ CDH 5.8+ MapR 5.1+ Spark 1.6 Java JDK 8.0 Supported OS Redhat Linux, Suse Linux, CentOS SAP GCC compatibility package ver 4.8 SAP HANA Spark Controller 2.0 (Optional) Zeppelin (Optional) Refer to the Vora Admin Guide at help.sap.com/hana_vora and SAP Note Prerequisites for Installing SAP Vora for the latest information SAP SE or an SAP affiliate company. All rights reserved. Customer 26

27 Vora Use Cases

Customer Use Case Scenarios 2016 SAP SE or an SAP

30 HANA Connectivity

SDA and HANA Connectivity SAP HANA and Hadoop Integration 2016

32 SAP HANA integration - voraodbc SAP HANA delivers voraodbc adapter from SPS12 onwards Introduced support for Vora in-memory Relational tables Kerberos support when connecting to SAP HANA Vora views can be consumed thru HANA using voraodbc Once HANA Wire is enabled you can use HANA Studio or Web IDE to create Virtual Tables on Vora and join that with SAP HANA 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 32

SDA and HANA Connectivity Bi-directional integration of SAP HANA and Vora

SDA and HANA Connectivity Benefits of SAP Vora connectivy vs past SDA

35 Data Lifecycle Manager (DLM) for Hadoop as a tier Define a data aging strategy with DLM Leverage SAP HANA Dynamic Tiering (Warm-Store), Hadoop or SAP Sybase IQ in SAP HANA native use cases with a tool based approach to model aging rules on tables to displace aged data to HANA extended tables to optimize the memory footprint of data in SAP HANA. HOT-STORE (Column Table) WARM-STORE (Extended Table) DATA MOVEMENT * Data Lifecycle Manager SAP HANA 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 35

36 Vora Key Features

Key Features SAP Vora SQL Engine 2016 SAP SE or an

Key Features Distributed in-memory computing architecture 2016

Key Features Hierarchy Processing A Hierarchy

time the data is queried 2016 SAP SE or an SAP

Key Features SAP Vora Latest Innovations 2016 SAP SE or

42 Key Features Key features as of 1.4 Installed natively as a query engine on Hadoop/Spark nodes through Hadoop admin tools (Ambari, Cloudera Manager, MapR CS) and Vora Manager Support for most recent version of Apache Spark (v ) Leverage Spark core and Spark SQL APIs to deliver distributed queries Extend Spark to build enterprise analytics features like hierarchies Build SAP Vora tables on HDFS, MapR FS, Parquet, S3, and local file systems Bidirectional data virtualization between SAP HANA and Hadoop/Spark using SAP HANA Spark controller through SDA and Apache Spark Data Source API framework (data scientist workflows) SAP Vora Modeler to build OLAP cubes on Vora tables with facts, dimensions, and annotations Introduce new services like Dlog, Discovery (Consul) Improved performance through partitioned tables and co-located joins Extend Notebook (IDE) support to Jupyter and Zeppelin New processing engines like Time Series, Graph, DocStore, Diskstore 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 42

43 Customer Use Case Examples

Customer #1 Data Tiering Use Case HANA Index Server

Stores SDA (Virtual Table) HANA Spark Controller DLM

Dynamic Tiering XS Engine HDFS Data Lifecycle

Writes Data DLM Writes Data to ORC File 2016 SAP SE

44 Customer #1 Data Tiering Use Case HANA Index Server Hadoop Spark Processing Engines Spark SQL In-Memory Stores SDA (Virtual Table) HANA Spark Controller DLM Reads Data from HANA Vora Upload Table into Vora Dynamic Tiering XS Engine HDFS Data Lifecycle Manager Extended Storage (DLM) Files Files Files DLM Writes Data DLM Writes Data to ORC File 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 44

45 Response Time [s] Customer #1 Test Results Hadoop HANA / DT / Spark/ Vora DLM Customer Bill Store and Retrieval Store 40ms response time to read PDF Run aggregation query (~4 billion records): Move data from HANA to DT records per minute Load data via Sqoop into Hadoop 4min 24s to load 2.5 million records (single thread) Query Response Time [s] 360 Move data from HANA to Hadoop via VORA into HDFS 22 million records per minute 1min 10s (10 threads) Total of 6.2GB ORC files stored in HDFS against original size of 172GB. Compression Rate: 9 (3 copies in HDFS) HANA DynamicTiering Hadoop - Spark Hadoop - Vora 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 45

Customer #2 Marketshare Comparison (Lumira) 2016 SAP SE or

48 Customer #2 Performance Benchmarks Query Vora (s) Hive/MR (s) Impala (s) SELECT SUM(DOLLAR), MASTER_SUBCATEGORY FROM SKYNET_MAPPED_COMPILED GROUP BY MASTER_SUBCATEGORY SELECT distinct(master_subcategory) FROM SKYNET_COMPILED_MAPPED SELECT count(*) from SKYNET_COMPILED_MAPPED select sum(dollar) from SKYNET_COMPILED_MAPPED where UPC = xyz' select * from SKYNET_COMPILED_MAPPED where UPC = xyz' SAP SE or an SAP affiliate company. All rights reserved. Customer 48

49 Vora Tools Overview Appendix

HANA Vora Hadoop Management Tools 2016 SAP SE or an

HANA Vora Working with Tables and Views 2016 SAP SE or

54 Vora Modeling Tool Vora Tools use the Thriftserver to provide access to the Modeler under Perspectives: Data Browser SQL Editor Modeler 2016 SAP SE or an SAP affiliate company. All rights reserved. Customer 54

Statistical Modeling SAP HANA OLAP Modeler SAP Vora & Spark

59 HANA Developer Application Development Business Analyst Data Discovery Data Engineer OLAP Modeling Data Scientist Statistical Modeling SAP HANA OLAP Modeler SAP Vora & Spark Hadoop 2015 SAP SE or an SAP affiliate company. All rights reserved. Internal 59

Analyze Big Data Faster and Store It Cheaper

Analyze Big Data Faster and Store It Cheaper Dr. Steve Pratt, CenterPoint Russell Hull, SAP Public About CenterPoint Energy, Inc. Publicly traded on New York Stock Exchange Headquartered in Houston, Texas