@Pentaho #BigDataWebSeries

Transcription

1 Enterprise Data Warehouse Optimization with Hadoop Big #BigDataWebSeries

2 Your Hosts Today Dave Henry SVP Enterprise Solutions Davy Nys VP EMEA & APAC 2

3 Source/copyright: The Human Face of Big Data 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866)

4 Pentaho Webinar Series Sign-up at: pentaho.com 4

5 Goals for Today To understand: Challenges with the current EDW architecture Trends and shifts in data processing How Hadoop can help How to leverage Hadoop with Pentaho Visual MapReduce 5

6 Complete Analytics and Visual Data Management Data Ingestion, Manipulation & Integration Data Discovery & Visualization Enterprise & Ad Hoc Reporting NoSQL Databases Hadoop Predictive Analytics & Machine Learning Analytic Databases 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866)

7 Traditional Data Warehouse Architecture Structured Data Extract Transform Load Data Mart(s) / Warehouse Metadata Unstructured Data Source data acquisition / Ingestion Cleansing Transformation Dashboard Initial consolidation as required Change Data Capture Data Warehouse Management Report Analyzer 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866)

8 Trends with Data Processing Data Load Volume of existing data sources are steadily increasing Requirement to make data available for longer periods of time (3 years -> 30 years) New sources of data are desired for analysis machine-generated or external/ 3rd-party data ELTL Approach To Data Load Extract data from source systems Load it (in its raw form) into the EDW Transform it via SQL, creating new tables Load the new tables into the official data warehouse 8

9 Challenges with Traditional Approaches EDW can t handle increasing data and workloads, so companies must: Reduce the volume of data Restrict end-user access (# of users or access windows) to accommodate longer batch processing windows Purchase additional capacity (hardware / licenses), which can be as much as $100K / TB Then, companies are faced with the following challenges: The compromise itself The incremental outlay of capital required to expand the EDW or purchase more proprietary ETL tool capacity The inability of the incumbent ETL vendor to work with Hadoop 9

10 Solution Architecture with Hadoop Structured Data Data Mart(s) / Warehouse Dashboard ETL ETL Metadata Report Analyzer Unstructured Data Data Integration Source data acquisition / Ingestion Initial consolidation as required Data Integration Cleansing Transformation Change Data Capture Data Warehouse Management 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866)

11 Core Benefits 1. Improve performance Meet critical data processing SLAs 2. Retain all data for analysis 3. Lower costs of data management, growth 4. Extend existing EDW capacity Increase ROI from current investments Time Flexibility Costs 11

12 Challenges with Hadoop: Scripting and Coding Flexibility Time Costs 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866)

13 Pentaho: Quickest, Most Complete Solution for Big Data Design, develop and deploy 15x faster: Full continuity from data access to decisions complete data integration & business analytics platform for any big data store Faster development, faster runtime visual development, distributed execution Instant and interactive analysis no coding, no ETL required 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866)

14 Solution Architecture & Demo Solution Architecture & Demo 14

15 Data Warehouse Optimization Data Sources Big Data Architecture ERP CRM Data Warehouse (Master & Transactional Data) Analytic Data Mart(s) Analytic Data Mart(s) Analytic Data Mart(s) CDR Logs Logs Other Data Raw Data Parsed Data Master Data Tape Archive Analytic Datasets

16 Pentaho for Hadoop Data Integration + Analytics Data Integration Analytics Ingestion Structured Data Processing VISUAL MAP REDUCE Analysis & Reporting I N G E S T Raw Data Master Data Parsed Data Analytic Datasets A N A L Y Z E Unstructured Data ORCHESTRATE ERP DW CRM Pig, Oozie, Flume, Hive, HBase, Sqoop 16

17 Example Call Record Processing What are the top 10 states for outbound calls on Fridays, Saturdays and Sundays? Data available: Call records: date/timestamp & source phone # Reference data: area code by country, state & time zone (North American Numbering Plan)? Goal: Parse, enrich and filter the data Load the data into Postgres for analysis Challenge Prepare the data without impacting the EDW (no ELT)

18 Hadoop Data Processing Scenario Ingestion Structured Data Unstructured Data I N G E S T Raw Data Master Data 18

19 Visual MapReduce Processing VISUAL MAP REDUCE 1. MapReduce Input calling data 2. Calculate Month, Day, Day of Week Raw Data 3. Extract 3 digit area code Parsed Data Master Data 4. Lookup geo master data in HDFS 5. Filter for weekend and US only calls Analytic Datasets 6. Create Value field for Key-Value Pair Java Programing 7. Create Key field for Key-Value Pair 8. MapReduce Output Key-Value Pair 19

20 Solution Architecture & Demo End of Demo 20

21 Leveraging Hadoop with Pentaho Data Management Platform Visual Map Reduce, Orchestration, Connectivity Fusion of all data sources & processing Control/Manage/Optimize flow of data Hybrid Leverages non-hadoop infrastructure OEM Flexibility, Extensibility, Architected to Embed Pricing One of top reasons customers choose us Community/Open Source Cache Similar to Hadoop 21

22 Overall Benefits You should keep your EDW, but use Hadoop and Pentaho to optimize data processing Business Benefits You can defer upgrades to expensive EDW hardware You can offload batch processing from the EDW and make it more available to end-users (improve performance / comply with SLAs) With better performance you may need smaller cluster sizes Technical Benefits This is a low-risk use case that lets you get familiar with Hadoop while creating business value It s easy to evaluate you don t need to modify your cluster and risk disrupting the configuration 22

23 Solution Architecture & Demo Q & A 23

24 Contact Us or Sign-up at: pentaho.com 24