Analyze Big Data Faster and Store it Cheaper. Dominick Huang CenterPoint Energy Henry Le - Utegra8on Russell Hull - SAP

Size: px

Start display at page:

Download "Analyze Big Data Faster and Store it Cheaper. Dominick Huang CenterPoint Energy Henry Le - Utegra8on Russell Hull - SAP"

Charity Fox
6 years ago
Views:

1 Analyze Big Data Faster and Store it Cheaper Dominick Huang CenterPoint Energy Henry Le - Utegra8on Russell Hull - SAP

Over 5000 square miles of electric transmission and distribu8on service area Assets

in business for over 130 years Ø Ø Ø Domes8c Energy Delivery Operate, Serve, and Grow

2 ABOUT CENTERPOINT ENERGY, INC. Ø Ø Ø Ø Ø Ø Publicly traded on New York Stock Exchange Headquartered in Houston, Texas Over 5000 square miles of electric transmission and distribu8on service area Assets total more than $22 billion Over 8,700 plus employees CNP & its predecessor companies in business for over 130 years Ø Ø Ø Domes8c Energy Delivery Operate, Serve, and Grow Smart Grid Enabled Ø Ø Ø Ø Ø Twenty-Eight State Geography Over Five Million Metered Customers 2.3 million Smart Meters 4000 Miles of Transmission 47,000 Miles of Distribu8on Ø Electric Transmission & Distribu8on Ø Natural Gas Distribu8on Ø Compe88ve Natural Gas Sales and Services CenterPoint Energy Proprietary and Confiden8al

3 AGENDA Ø Key Drivers and Strategy of HANA Initiative Ø Use Case Smart Meter Big Data Analytics Ø Technology Overview Ø POC Results Ø Value and Comparison

4 KEY DRIVERS FOR HANA INITIATIVES Ø SAP HANA as CNP strategic platform for critical transactional applications and Analytics Ø Cost effective solution to manage and contain data storage growth Ø Analytics platform simplification and consolidation to HANA Ø Key technology enabler for future business solutions Ø Maximize CNP investment on HANA license (40TB) Ø Enable business resiliency implementation for CRM/ECC/BPC Ø Leverage HANA in-memory capability for real time analytics

5 STRATEGY 3 YEAR HANA ROADMAP Ø Technical Migration and Consolidation Ø Migrate critical business applications (SAP and Mainframe) Ø Consolidate Analytics solutions (BW, ISAS, ema, etc.) onto HANA Ø HANA Platform Optimization Ø Enhance performance of core business process and mass business functions Ø Enable real-time reporting from the HANA (in-memory) database Ø HANA Platform Innovation Ø Innovative solutions to align with long-term business strategy and roadmap Ø SIMPLE Finance, Predictive Asset Health Analytics, Situational Awareness, Internet of Things, Predictive Analytics for customer services, etc.

6 USE CASE SMART METER BIG DATA ANALYTICS

BUSINESS CHALLENGE 1+ PB of SmartMeter Data 2.3MM SmartMeters taking readings every 15 minutes crea8ng 225MM Readings per day, or over 800 Billion Readings in a Year.

7 BUSINESS CHALLENGE 1+ PB of SmartMeter Data 2.3MM SmartMeters taking readings every 15 minutes crea8ng 225MM Readings per day, or over 800 Billion Readings in a Year. Regulatory requirements require historical readings to be available for 10 years. Uncompressed Data Growth of 8TB per month and over 1PB in a 10 year period. Current DW technology is approaching End of Life Massive amounts of data stored in proprietary vendor solu8on, was hard to manage and has a significantly high total cost of ownership. Need a cost effecuve soluuon for today's analy8cs, regulatory requirements and prepara8on for future use cases. CenterPoint Energy Proprietary and Confiden8al

DATA TIER SOLUTION DATA VOLUME MANAGEMENT: MULTI TEMPERATURE DATA APPROACH hot

available warm cold Non-Ac8ve Data Concept Infrequent access On disk, no need to

for read-only data Sporadic access Not stored in HANA DB; stored in Near-line

8 DATA TIER SOLUTION DATA VOLUME MANAGEMENT: MULTI TEMPERATURE DATA APPROACH hot Data is read and/or wri`en frequently In memory No restric8ons, all features available warm cold Non-Ac8ve Data Concept Infrequent access On disk, no need to keep in memory all the 8me No restric8ons, all features available NLS Management for read-only data Sporadic access Not stored in HANA DB; stored in Near-line Storage Restricted to NLS capabili8es Providing lower TCO by op8mized data volume management

9 BUSINESS CASE CAPEX & OPEX SAVINGS Projected Data Capacity (TB) Projected Total Spend (CumulaUve & EsUmated) Millions $25 $20 $15 $10 $5 Capex Saving O&M Saving $ HANA O&M HANA Capital NZ O&M NZ Capital Projected Growth Business as usual Move to HANA/Hadoop Projected Savings 75% Capex and Opex saving Smart Meter Data grows more than 100TB/year, 1PB+ in 10 years CenterPoint Energy Proprietary and Confiden8al Informa8on 9

10 SOLUTION BENEFITS Ø Cost effec8ve HOT+WARM+COLD data management strategy leveraging HANA data compression and data 8ering technology Ø Simplified Big Data ownership by combining SAP HANA, Dynamic Tiering and Hadoop into a single landscape. Ø Single Database Experience. Query Execu8on u8lizes SDA and automa8cally accesses data stored in HANA, Dynamic Tiering and Hadoop/Vora depending on loca8on of data. Ø Data Movement automated between storage 8ers using the Database Lifecycle Manager (DLM). Ø Founda8on for advanced predic8ve analy8cs and future business capabili8es Ø Instant Real 8me Analy8cs via HANA Ø 75% savings in storage cost compared to current solu8on. Ø Data 8ering technology (Dynamic Tiering, Hadoop) to manage data size and growth. Ø Seamless integra8on with Hadoop integra8on allows for data scien8st to use HANA toolset to access and manage Hadoop data Ø Ability to charge business based on the data being stored and performance requirements

11 TECHNOLOGY REVIEW

12 SAP Big Data Plaform CenterPoint Energy Proprietary and Confiden8al

.) Tier 2 (Hadoop) Batch Layer Netezza zos DLM 36TB HANA EDW 50TB Dynamic Tiering Extended Storage Hadoop (Vora) 750TB 13 months of data

13 NEW SMART METER ANALYTICS ARCHITECTURE Current Architecture Planned Architecture ApplicaUon Business Objects / SAS / Custom Applica8on Storage Tiers (Costs and Performance) Aggrega8on Aging Tier 0 (Memory) Speed Layer Tier 1 (SAN,..) Tier 2 (Hadoop) Batch Layer Netezza zos DLM 36TB HANA EDW 50TB Dynamic Tiering Extended Storage Hadoop (Vora) 750TB 13 months of data are stored in HANA for fast analy8cs 26 months of data are stored in DT (Sybase IQ) 10 years of meter data is stored in Hadoop. The plan is to use SAP HANA Vora to access the data

14 DYNAMIC TIERING Ø SAP Dynamic Tiering is a warm store tradi8onal disk based database system fully integrated into HANA. Ø Based upon Sybase IQ: Column Store & Disk based Ø Reduced TCO by lowering HANA memory footprint Ø All HANA func8ons are available. Read/Write/Update Ø Single Database experience: All DB access requests are managed through the HANA planorm. Ø Centralized opera8on control: All administra8on tasks are handled through the HANA interface.

15 SAP HANA DYNAMIC TIERING DISK-BACKED COLUMN STORE EXTENSION TO HANA FOR WARM DATA MANAGEMENT

16 WHAT IS APACHE HADOOP?

17 HADOOP TECHNICAL ARCHITECTURE HADOOP CLUSTER

Programming SAP HANA Vora is an in-memory query engine which leverages and extends the Apache Spark execu8on

18 SAP VORA - HANA/HADOOP INTEGRATION WHAT S INSIDE AND WHAT DOES IT DO? Drill Downs on HDFS Mashup API Enhancements Compiled Queries HANA-Spark Adapter Unified Landscape Open Programming SAP HANA Vora is an in-memory query engine which leverages and extends the Apache Spark execu8on framework to provide enriched interac8ve analy8cs on Hadoop. Make Precision Decisions Democra8ze Data Access Simplify Big Data Ownership Any Hadoop Clusters

19 SAP DATA LIFECYCLE MANAGER (DLM)

SAP DATA TIERING ARCHITECTURE HANA Index

Data from HANA Vora Upload Table into Vora

Lifecycle Manager Extended Storage (DLM)

20 SAP DATA TIERING ARCHITECTURE HANA Index Server Hadoop Spark Processing Engines Spark SQL In-Memory Stores SDA (Virtual Table) HANA Spark Controller DLM Reads Data from HANA Vora Upload Table into Vora Dynamic Tiering XS Engine HDFS Data Lifecycle Manager Extended Storage (DLM) Files Files Files DLM Writes Data DLM Writes Data to ORC File

21 POC REVIEW

22 POC OBJECTIVES Ø Research and test SAP HANA Data Tiering technology, i.e. DLM (Data Life Cycle Management), Dynamic Tiering, Vora Hadoop Integra8on Ø Evaluate Hadoop technology, understand Hadoop ecosystem and TCO Ø Test SAP VORA - HANA and Hadoop integra8on technology Ø Develop and validate solu8on op8ons for several cri8cal 2016 projects: Smart Meter Analy8cs, customer document repository for Mainframe Migra8on Ø Build CNP in-house exper8se in Hadoop and SAP HANA/Hadoop integra8on technology Ø Iden8fy use case and innova8on opportuni8es at CNP

23 POC ENVIRONMENT AND TEST CASES Ø POC Team Ø Ø Environment Test Cases CenterPointEnergy/Utegra8on (Lead and Architects); SAP (CoE, PE, Global ITP); HP (Hardware); IBM(IBM Hadoop and Cloud) Ø Hardware HP Lab: Hadoop 12 nodes cluster, CS500 HANA, HANA Dynamic Tiering Node IBM BigInsights Cloud Ø Sojware SAP HANA SPS 10, DLM, Dynamic Tiering, VORA Hortonworks HDP Hadoop, RedHat Linux IBM Apache Hadoop with BigSQL Data Load - Extract 800GB, 7 Billion Smart Meter records from Netezza and ISAS, load data into HANA (Meter data scrambled to protect data security) DLM Use DLM tool to move data from HANA to Dynamic Tiering Extended Storage and Hadoop Run queries across all data 8ers and measure performance Load, query and display 19 million PDFs of Customer Bills (Dummy PDF files used, no real customer data)

24 POC SUCCESS CRITERIA Ø Data Tiering Move data among different 8ers including HANA, DT and Hadoop Ø Run SQL queries within and across data 8ers Ø Performance Measure response 8me for each data 8er Ø Data Compression evaluate compression ra8o of HANA, DT and Hadoop Ø SAP DLM U8lize the tool to move data from Hot to Warm and Cold 8er Ø Customer document storage Store and retrieve PDF documents with one second Ø Comparison of storage costs: HANA, DT (Dynamic Tiering Extended Storage) and Hadoop

25 POC TEST RESULTS Hadoop HANA / DT / Spark/ Vora DLM HDP Customer Bill Store and Retrieval à 40ms response 8me to search and display a document from 19 million PDFs HDP Batch data load via SQOOP into Hadoop à 4 min 24s to load 2.5 million records (single thread);1 min 10s (10 threads) Data load from HANA to HDP Hadoop via VORA à Total of 6.2GB ORC files stored in HDFS against original size of 172GB. à Compression Rate: 9 (3 copies in HDFS) Run aggregagon query across SAP HANA, HDP Hadoop & DT (~4 billion records): Response Time [s] Query Response Time [s] Move data from HANA to DT à 289 million records moved from HANA to DT à 670K records per minute Move data from HANA to Hadoop via VORA into HDFS à 1.57 billion records moved from HANA to Hadoop à 22 million records per minute

26 VALUE AND COMPARISON BETWEEN DATA TIERS

27 COMPARISON BETWEEN DATA TIERS Component Performance Cost Factor Volume Processing HANA $$$$ Up to 10s TBs (no technical limit) ACID compliant SQL, SQLscript, graph, 8me series, spa8al, text, Dynamic Tiering or Sybase IQ $$ 100s of TB integrated in HANA Several PBs with Sybase IQ ACID compliant SQL Hadoop Spark/Vora $ 100s of PB or more ANSI SQL compliant Read-only SQL when used from HANA via SDA 15 Umes less expensive than T1 storage Transforma8ons and Ac8ons Performance can be improved significantly by increasing compute nodes and using SSD with higher cost Hadoop Vora in Memory $$ 100s of TB (depending on available memory in Hadoop cluster) Data loaded in memory to achieve be`er performance Read-only SQL when used from HANA via SDA

28 RECOMMENDED USE CASES SHORT TERM Component HANA Dynamic Tiering Hadoop - Spark Hadoop - Vora Recommended Use Case Managing up to several TBs of high value data Very high processing performance required SAP HANA na8ve processing features (PAL,..) required OLTP with many fine-granular updates needed Managing up to several PBs of data at T2/T3 storage cost High performance for complex queries required Deep SAP HANA integra8on required (single database experience) Updates and deletes required Managing up to 100s PBs of data at T4 storage cost, 15 Umes less expensive than T1 storage Read-only sufficient (bulk load, no fine granular writes) Compara8vely low-cost storage important Loose integra8on of administra8on and life-cycle management acceptable High OLAP query performance on Hadoop Addi8onal query features (hierarchies)

29 THANK YOU Contact informa8on: Dominick Huang Sr. Manager, Enterprise Technology & Architecture CenterPoint Energy Tel Russell Hull Chief Support Architect SAP America Henry Le VP of Analy8cs Utegra8on Inc.

30 FOLLOW US Thank you for your Ume Follow us on

31 APPENDIX

33 CNP HANA LANDSCAPE - ANALYTICS (BW + OW) AnalyUcs (BW + OW) ES(NLS/DT/Hadoop) ES(NLS/DT/ Hadoop) 0.5TB 0.5TB 0.5TB 0.5TB 0.5TB 0.5TB 0.5TB 0.5TB 0.5TB 0.25TB 0.5TB Exis8ng blade New HP Node 2 TB Failover blade ES Extended Storage (NLS/DT/Hadoop) HIP(PRD) 36TB (Memory) HIQ(QA) HID(DEV) 1 Situa8on Awareness, MfM Tes8ng & other Apps 4.5TBs HIS (SBX)

34 HADOOP ARCHITECTURE

35 RECOMMENDED HADOOP USE CASES MID & LONG TERM

36 MAJOR HADOOP DISTRIBUTIONS

37 HADOOP ECO SYSTEM

Analyze Big Data Faster and Store It Cheaper

Analyze Big Data Faster and Store It Cheaper Dr. Steve Pratt, CenterPoint Russell Hull, SAP Public About CenterPoint Energy, Inc. Publicly traded on New York Stock Exchange Headquartered in Houston, Texas