Top Five Reasons for Data Warehouse Modernization Philip Russom

Transcription

1 Top Five Reasons for Data Warehouse Modernization Philip Russom TDWI Research Director for Data Management May 28, 2014

2 Sponsor

3 Speakers Philip Russom TDWI Research Director, Data Management Steve Sarsfield Product Marketing Manager, HP Vertica 3

4 Agenda PLEASE #TDWI, #EDW, #DataWarehouse, #DataArchitecture, #Analytics, #RealTime Background Why many users DWs need modernization What is it? There are many reasons, but I ll boil it down to five Top Five Reasons Analytics Scale Speed Productivity Cost Control New DW Architectures Resulting from Modernization Recommendations

5 DW Modernization has many meanings Additions to existing data warehouse New data subjects, sources, tables, dimensions, etc. More standalone data platforms and tools Complement DW without replacing it More marts and ODSs New appliances, columnar databases, Hadoop, NoSQL, etc. Architectural Adjustments All the above Better design Upgrades Newer versions of current DBMS software More hardware Rip and Replace Decommission current DW platform and migrate to another

6 Contact Information If you have further questions or comments: Philip Russom, TDWI Randy Lea, Teradata 6

7 Top Five Goals for DW Modernization I ll mostly focus on improvements to: Analytics, Scale, Speed These regularly rank high in TDWI surveys, for example: SOURCE: 2014 TDWI Report: Evolving Data Warehouse Architectures, Figure 4 1. ANALYTICS 2. SCALE 3. SPEED I ll also mention improvements to: Productivity, Cost Control These regularly come up in TDWI interviews with users

8 DW Modernization Goals are Related Analytics needs better productivity The challenge is to gain improvements with the first four goals without incurring more of the fifth: cost. Speed contributes to scale and productivity SPEED Streaming Big Data Event Processing Real-Time Operation Operational BI Near-Time Analytics Dashboard Refresh Fast Queries SOURCE: 2012 TDWI Report: High Performance Data Warehousing, Figure 1. CONCURRENCY Competing Workloads Reporting, Real Time, OLAP, Adv. Analytics, etc. Intra-Day Data Loads Thousands of Users Ad hoc Queries HIGH PERFORMANCE DATA WAREHOUSING (HiPer DW) SCALE Big Data Volumes Detailed Source Data Thousands of Reports Scale Out Into: Clouds, clusters, grids, distributed architectures COMPLEXITY Big Data Variety Unstructured Data Machine/sensor Data Web & Social Media Many Sources/Targets Complex Models & SQL High Availability

9 BEYOND OLAP & REPORTING TO Advanced Analytics Organizations need more analytic insights To compete, serve customers, be profitable, control costs, improve quality, grow, etc. Analytics is becoming a larger portion of BI work Reporting and OLAP are still important Organizations need advanced forms of analytics Technologies: Extreme SQL, data mining, statistics, natural language processing, text mining, AI, graph, etc. Methods: Predictive, clustering, segmentation, risk, fraud detection, etc. Most users designed EDWs for reporting and OLAP Analytics requirements differ from reports and OLAP Users face multiple paths to enabling advanced analytics Retrofit analytics onto report-focused EDW Deploy an analytic data platform that complements the EDW Replace the EDW s platform with one that handles all workloads

10 Scale TO MORE DATA, USERS, REPORTS, ANALYSES Data s Growing Volumes are a Challenge Large Data Warehouses data for both reporting and analytics Big Data volume aside, also diversity of data type, source, latency Scale is also a Challenge to Basic BI Functions, like Reporting Thousands of Concurrent BI Users; Thousands of Reports Eventually, thousands of analytic users Scale to Increasing Complexity More processing for ETL, integration, quality, analytics, real time, etc. Distributed DW architectures have more moving parts Scale despite Growing numbers of Concurrent Workloads Reporting, Real Time, OLAP, Analytics, Data Loads, Ad hoc Queries Users have a number of choices for scaling Scale Up: More hardware for more data; efficient storage Scale Out: Clouds, clusters, grids, racks, distributed architectures Deploy or migrate to data platforms built for analytics with big data: columnar databases, data warehouse appliances, newer brands of databases, Hadoop, NoSQL, etc.

11 EVERTHING NEEDS MORE Speed Speed involves a temporal continuum From high performance to near time and true real time Speed is enabled by a functional continuum From hardware to perky queries to event processing Many options are available for modernizing EDWs and analytics High performance functionality In-memory databases, in-database analytics, columnar databases, DW appliances, solid-state drives, modern CPUs, big memory in servers, Near-time functionality Microbatches, federation, virtualization, replication, services, query optimization, etc. Real-time functionality Complex event processing (CEP), stream processing, operational intelligence, etc.

12 MORE SOLUTIONS IN LESS TIME Productivity Agile and lean development methods Early prototype, built out iteratively Instead of older big bang deliverables Biz folks review/guide each iteration To assure IT-to-biz alignment Requirements gathering (RG) now done online Data exploration, discovery, profiling replace RG Req s captured online, applied directly to solution Fast tools and platforms make analytics productive Speed of thought iterative analysis Fast queries & bulk loads build analytic datasets fast Less time per project means More projects Organization uses solution sooner Greater agility for the business

13 DATA VARIES IN VALUE; MANAGE IT ACCORDINGLY Economics As you modernize a DW environment, rethink its economics Cost continuum of data platforms: High $/Tb Traditional Platforms New Affordable Platforms, built for DW/Analytics Cheap Open Source: Hadoop, NoSQL Choose a platform that fits a given data workload but also fits the value of data High-value data on the core EDW Modeling, cleansing, aggregating, and documenting data (which is required for reports and OLAP) increases its value Analytic datasets in the mid tier This data is lightly prepared or prepped on the fly; temp sandboxes Source & archival data on the back tier This is more of a data lake that preserves data in its original form, so it can be repurposed repeatedly, as analytic projects arise

14 ONE WAY TO MODERNIZE A DW Multi-Platform Data Warehouse Environments Many enterprise data warehouses (EDWs) are evolving into multi-platform data warehouse environments (DWEs). Users continue to add additional standalone data platforms to their warehouse tool and platform portfolio. The new platforms don t replace the core warehouse, because it is still the best platform for the data that goes into standards reports, dashboards, performance management, and OLAP. Instead, the new platforms complement the warehouse, because they are optimized for workloads that manage, process, and analyze new forms of big data, non-structured data, and real-time data.

15 Modern DW System Architectures can be Complex The technology stack for DW, BI, analytics, and data integration has always been a multi-platform environment. What s new? The trend toward a portfolio of many data platforms has accelerated. Why? More platform types to serve more data and workload types. Over The Passage of Time Federated Data Federated Marts Data Federated Marts Data Marts Customer Mart Customer or ODS Mart or ODS Real Time ODS DW from a Merger Columnar DBMS Columnar DBMS Map Reduce Complex, Event Processing Data Warehouse Star or Multi- Snowflake dimensional Scheme Data Models Data Staging Data Areas Staging Data Areas Staging Areas Metrics for Performance Mgt OLAP Cubes OLAP DBMSs Detailed Source Detailed Data Source Detailed Data Source Data Analytic Sand Box Data Federation & Virtualization Hadoop Distributed Hadoop File Distributed Sys File Sys DW Appliance DW Appliances No-SQL Database No-SQL Database Streaming Data Tools

16 Good Reasons for Integrating Hadoop with Relational EDW A Relational DBMS is good at: Metadata management Complex query optimization Query federation Table joins, views, keys, etc. Security, including roles, directories Much more mature development tools HDFS & other Hadoop tools are good at: Massive scalability Lower cost than most DW platforms & analytic DBMSs Multi-structured data & no-schema data Some ETL functions; late binding; custom code for analytics Use HDFS like a very scalable operational data store or data staging area, to modernize your existing DW environment

17 Recommendations Revaluate your data warehouse and related systems There s always room for improvement Change is afoot, in both biz & tech Prioritize modernization by putting biz goals first Biz wants to manage big data and leverage it Biz wants to compete on analytics Biz needs real-time tech to operate faster Biz needs BI/DW solutions sooner, more agile Technology goals are also important, though secondary Greater productivity from tech personnel Assuring capacity for growth Diversifying data platform and tool portfolio to support more types of data, workloads, development methods, etc. Migration to new platforms that are faster, more scalable, tuned for analytics, cost less, etc.

18 Cost Optimized Storage Steve Sarsfield, Product Marketing Manager, HP Vertica Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

19 Feeling the Pain Recognizing that it s time to modernize TIME IS MONEY What would be the business impact of reducing time from days to hours (hours to minutes)? READY FOR BIG DATA What is your plan for managing the need for real time data analysis as your data volumes continue to scale? ANALYTIC INNOVATION Are you getting the business insights from your organization s data when you need it? 19 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

20 Big Data Warehouse Key Features Joins, Complex Data Types SQL-based Predictive Analytics Petabyte Scale Advanced Analytics Advanced Analytics Manage Huge Data Volumes Manage Huge Data Volumes Python and R Support Data Scientists Support Data Scientists Work with Legacy Tools Deliver Fast Analytics Deliver Fast Analytics What-if, A/B testing Work with Legacy Tools SQL-based Visualization ETL 20 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

21 Analytics Capabilities Reinforced Legacy Architectures Advanced Analytics Support Data Scientists Work with Legacy Tools Advanced Analytics Manage Huge Data Volumes Support Data Scientists Deliver Fast Analytics Purpose-built Big Data Analytics Platform Work with Legacy Tools Manage Huge Data Volumes Deliver Fast Advanced Analytics Analytics Support Data Scientists New NoSQL Architectures Work with Legacy Tools Manage Huge Data Volumes Deliver Fast Analytics 21 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

22 Cost-Optimized Storage - ILM Tier-off older data Interactive Data Frequently queried Vertica data cache Hot Batch Data Vertica data cache Cool Serve Convert data to Vertica storage format Value Discovery Archive Data Vertica data cache Explore Any format Cold Dark Data Location Format Store Any format 22 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

23 Core Capabilities Impact How Do We Achieve Huge Performance Increases? 23 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

24 Secret Sauce of HP Vertica Columnar Storage Compression MPP Scale- Out Distributed Query Projections Speeds Query Time by Reading Only Necessary Data Lowers costly I/O to boost overall performance Provides high scalability on clusters with no name node or other single point of failure Any node can initiate the queries and use other nodes for work. No single point of failure Combine high availability with special optimizations for query performance A B D C E A CPU CPU CPU Memory Memory Memory Disk Disk Disk 24 Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

25 To find out more Purpose built for Big Data from the first line of code Download and Try Community Edition supports up to 1 TB on 3 nodes Contact us for more information or 30 day trial Contact Steve.Sarsfield@HP.com Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

26 Questions? 26

27 Contact Information If you have further questions or comments: Philip Russom, TDWI Steve Sarsfield, HP 27