Baltimore Washington DB2 User s Group z/os Quarterly Meeting. Data Warehousing Market Trends and Best Practices

Transcription

1 Baltimore Washington DB2 User s Group z/os Quarterly Meeting Data Warehousing Market Trends and Best Practices Jonathan Sloan Senior Technical Consultant jonsloan@us.ibm.com Wednesday, September 10th, 2014

2 Agenda Traditional Data Warehousing Workload evolution Velocity, Variety and Volume Information consumer changes Workload characteristics (the operational warehouse) Predictive Analytics Latency requirements Real-time BI, is it real? System z strengths and gaps How we are addressing those gaps

3 Traditional Data Warehousing The Vision Bill Inmon: A data warehouse is a subject oriented, integrated, time variant, non-volatile collection of data in support of management's decision making process." Gartner's Definition of a Data Warehouse* A data warehouse is a collection of data in which two or more disparate data sources can be brought together in an integrated, time-variant information management strategy. Its logical design includes the flexibility to introduce additional disparate data without significant modification of any existing entity's design. A data warehouse can be much larger than the volume of data stored in the DBMS, especially in cases of distributed data management. Gartner clients report that 100TB warehouses often hold less than 30 terabytes of actual data (SSED). * Magic Quadrant for Data Warehouse Database Management Systems G Published: 31 January 2013, Gartner, Inc. Analyst(s): Mark A. Beyer, Donald Feinberg, Roxane Edjlali, Merv Adrian

4 Traditional Data Warehousing What Gartner is Saying The State of Data Warehousing in February 2012 Gartner: Roxane Edjlali, Donald Feinberg, Mark A. Beyer, Merv Adrian [G ] The data warehouse concept emerged to support the reporting and analytical needs of organizations by achieving the following goals: Separating the workloads to avoid affecting transactional systems. Building a consistent data repository (corporate truth) combining multiple sources of data. This led to the myth of the enterprise data warehouse (EDW) as being the single, trusted place to access all enterprise data. The EDW remains a design principle, but is rarely, if ever, actually deployed. Supporting the analytical requirements of the business users with various levels of service agreements, latency requirements, richness of calculations and speed of query execution. Gartner estimates that between 70% and 75% of all systems referred to as enterprise data warehouses are actually single-business departments in nature. Anecdotal evidence from inquiries in business intelligence or data warehousing practice show that between 60% to 70% of small and midsize businesses do not have a formal enterprise data warehouse, but continue to use their systems with departmental data marts or even operational systems to support their business intelligence and analytical needs. At its core, a data warehouse has been a negotiated, consistent logical model, deployed essentially to meet specific service-level expectations for the delivery and management of data. Traditionally, data warehouse architectures deploy a DBMS, with consistency and governance built through the data transformation and loading process into the repository. As the DBMS is such a dominant influence in data warehouse design, it actually serves three roles: repository, execution platform and architectural design.

5 Traditional Data Warehousing What Gartner is Saying Data Warehousing Trends for the CIO, January 2011 Gartner: Mark A. Beyer, Donald Feinberg, Roxane Edjlali [G ] There are six workloads that are delivered by the data warehouse platform: bulk/batch load basic reporting basic online analytical processing (OLAP) real-time/continuous load data mining operational business intelligence. Most data warehouses address the first three listed loads (considered lowintensity loads)...an advanced warehouse delivers at least one of the additional workloads (high-intensity loads). Warehouses delivering all six workloads need to be assessed for predictability of mixed workload performance. Unpredictable workload mixtures have different data warehouse platform requirements than predictable usage patterns. Failing to plan for mixed workloads will lead to increased administration costs over time, as volume and additional workloads are added, potentially leading to major sustainability issues.

6 The Logical Data Warehouse What Gartner is Saying* An emerging set of new best practices representing a significant split in data warehousing approaches began to emerge from new end-user demands and the availability of the latest technology approaches in These are a combination of traditional, integrated repositories with data virtualization and distributed parallel processing operations (such as MapReduce clusters), along with active auditing and optimization approaches in a semantic interpretive layer. This new construct, the logical data warehouse (LDW) focuses on the data processing or information management logic, not the physical infrastructure (see "Does the 21st-Century 'BigData' Warehouse Mean the End of the Enterprise Data Warehouse?"). The LDW is exhibiting partial adoption in less than 5% of the data warehousing market. * The State of Data Warehousing in 2012 G Published: 13 February 2012, Gartner Inc. Analyst(s): Roxane Edjlali, Donald Feinberg, Mark A. Beyer, Merv Adrian

7 The current DW/BA Market Not a one-size-fits all market. Although vendors have traditionally sold their products to solve all problems, the reality is that products have a tendency to focus on specific workloads proven on that platform. It is important to understand the workloads and advantages of the products in order to apply them correctly. Relational databases still provide superior price/performance for index-able workloads and will likely continue to do so in the future. Scanning type workloads are, for relational databases, a "brute-force" method of data access. For these types of workloads, specialized database infrastructures have some advantages. It is these workloads that are "up for grabs" when it comes to performance improvements. We are investing in these technologies in order to apply them to our products. (This does not negate the value of the traditional RDB approach). New database technologies like columnar databases and columnar adaptations to relational databases, in-memory databases and hardware acceleration are all focused on improving database performance for those workloads that are not index-able, such as aggregations and predicate evaluation where there are no supporting indexes. DB2 for z/os still provides the best high concurrency short duration workload support out of all platforms due to its' BI/OLTP heritage. Costs drivers have changed. Services in support of data warehouses add significant costs to implementation and management of those warehouses. As an example, Teradata Service revenue (>50%) exceeded product revenue for the last several years (and has done so historically). "The market has begun to accept higher first costs in exchange for lower administrative and management costs over the life of the data warehouse." [Gartner]

8 Traditional Data Warehousing What we see in the field Few have implemented a true data warehouse Data marts dominate due to warehouse complexity, funding, politics, shortest path to success (economic factors) Some data warehouses are incredibly focused (single application/single function) Silos of information continue to frustrate customers, drive costs higher and deliver sub-optimal results The opportunity to consolidate, cut costs and derive value from cross department/cross application information is exceptional large financial services firm with >40 data warehouses foreign bank with >5,000 analytic applications

9 Hybrid Transactional Analytic Platforms What Gartner is Saying* Hybrid transaction/analytical processing will empower application leaders to innovate via greater situation awareness and improved business agility. This will entail an upheaval in the established architectures, technologies and skills driven by use of in-memory computing technologies as enablers. (for example, IBM) Hybrid transaction/analytical processing will empower application leaders to innovate via greater situation awareness and improved business agility. This will entail an upheaval in the established architectures, technologies and skills driven by use of in-memory computing technologies as enablers. (for example, IBM) * Hybrid Transaction/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation G Published: 28 January 2014, Gartner Inc. Analyst(s): Massimo Pezzini, Donald Feinberg, Nigel Rayner, Roxane Edjlali

10 Agenda Traditional Data Warehousing Workload evolution Velocity, Variety and Volume Information consumer changes Workload characteristics (the operational warehouse) Predictive Analytics Latency requirements Real-time BI, is it real? System z strengths and gaps How we are addressing those gaps

11 Workload Evolution - Velocity, Variety and Volume Gartner states Data velocity, variety and volume are all expanding rapidly and challenging the warehouse architecture with "extreme data." Technology advances will play a role in addressing extreme data needs and real-time data warehousing, which need advanced change detection capability. The recent evolution of programming techniques in massively parallel-processing operations such as Hadoop and MapReduce and the rapid and simultaneous advances in core, memory and storage technology, provide the opportunity to create a new design principle. The State of Data Warehousing in 2012, 13 February 2012, page 2

12 Workload Evolution - Velocity, Variety and Volume Data warehousing is expensive. You want to keep valuable information in the warehouse. Think of it as a safe deposit box.

13 Workload Evolution - Velocity, Variety and Volume You don t just throw everything into a repository which has your valuable information. Is this warehousing?

14 Workload evolution Predictive Analytics What we see in the field Still new technology with implementations in several domains Adoption in financial services, insurance, retail and government Fraud Detection (credit card, debit card, insurance) Retail (next best product) Tax refund requests, Medicare & Medicaid fraud

15 Workload evolution The Operational Warehouse What we see in the field Many warehouses have morphed into an Operational Warehouse Must support operational requirements and information requirements Creates Risks Changes Platform Requirements (HA really does matter) Disaster Recovery Requirements (a must) Requires workload management Gartner says At the high end*, data warehousing is now mission-critical (page 3) Mission critical systems are defined as systems that support business processes and the generation of revenue that, if absent for a period of time, determined by the organization and its service-level agreements, must be replaced by manual procedures to prevent loss of revenue or unacceptable increased business costs. Normally, mission-critical systems require high-availability systems and disaster recovery sites. We have included the use of a DBMS as a data warehouse engine in the mission-critical systems category, as we believe that many (if not most) data warehouses in use today fit the definition of mission-critical. (page 6 & 7) * My experience is that this is not only at the high end, but, a significant trend amongst many of the customers with whom I have spoken. Some of these customers have been frustrated by either the cost and complexity of competitive disaster recovery solutions or by the lack of support (and development attention on) disaster recovery for operational warehouses. Operational is the new strategic! Australian bank

16 What is Business Critical Analytics? An analytics application that is tightly integrated with z/os transaction systems and critical to the optimal running of a business Make decisions and deliver business insight based on real time or near real time data Failure of these applications for any length of time can result in lost business Typically support a large concurrent user population with high volume of requests Preventing Fraud Cross-selling, up-selling customers Reducing Customer Churn Operational Reporting These applications require high degree of reliability, availability, scalability and low data latency

17 1. Organizations are using analytics to outperform their competition More organizations are using analytics to create a competitive advantage Leaders attribute the value of analytics solutions to their ability to increase revenues, make better, faster decisions and generate innovative ideas Respondents who believe analytics creates a competitive advantage % 70% increase 58% % 58% Source for 2012: Analytics: The real-world use of big data. IBM Institute for Business Value and Said Business School at the University of Oxford Copyright IBM Corporation 2012 Source: Analytics: A blueprint for value, IBM Global Business Services, through the IBM Institute for Business Value Copyright IBM Corporation 2013

18 2. More users across the organization want access to business critical analytics

19 3. Business critical analytics demand low latency, high qualities of service and performance Infrastructure must be scalable, available and reliable Data governance and security must be effective Analytics must be timely and accurate

20 4. Spreading analytic components across multiple departments can increase data latency, cost, complexity and governance risk Customer Interaction Data In Transactional Data DB2 for z/os IMS VSAM Non IBM Data Movement, Cleansing & Management Replicate, Integrate, Cleanse, Manage Data Warehousing Data Warehouse, Operational Data Store, Data Mart Data Analysis Business Intelligence, Predictive Analytics Business Insight Out zenterprise Off z Enterprise

21 5. Bringing analytic components to where data originates improves data governance while minimizing data latency, cost and complexity Customer Interaction Data In Transactional Data DB2 for z/os IMS VSAM Non IBM Data Movement, Cleansing & Management Replicate, Integrate, Cleanse, Manage Data Warehousing Data Warehouse, Operational Data Store, Data Mart Data Analysis Business Intelligence, Predictive Analytics Business Insight Out zenterprise

22 Changing Workload Characteristics (we have see this before) Telephone primary communication tool Outages expected Staff centrally located No governance Mission critical High Volume Near real-time Corporate regulatory compliance Global Adapting Infrastructure Strategy to Ensure Success Archiving High Availability Reporting Mass storage Instant Messaging Desktop application Server based Enterprise model No governance Video sharing Availability Text based Telephony integration Governance Bandwidth Business Analytics & Data Warehousing Departmentally defined Ad hoc Query Capability based products Desktop application IT not involved More volume, real-time More types of users, and mobile devices Corporate regulatory compliance Environmental concerns Enterprise BA: Standardization / Consolidation Modernization Data Governance Cloud Computing Big Data

23 Workload evolution Latency Requirements What we see in the field Expectations have changed (is this a surprise?) Reduced latency is an opportunity to differentiate, but, is it worth the price (are you losing customers, revenue, profit, opportunity because data is delayed)? Some customers want to access operational systems directly Anecdotal evidence from inquiries in business intelligence or data warehousing practice show that between 60% to 70% of small and midsize businesses do not have a formal enterprise data warehouse, but continue to use their systems with departmental data marts or even operational systems to support their business intelligence and analytical needs. (Gartner p. 2) IBM sees this as well, but, recommend caution What type of access (fixed reports or ad-hoc queries)? What about locking? What about data not committed? Some customers flash copy data and use copied tables

24 Workload evolution Is real-time real? What we see in the field Right time is still the general market approach Vendor capabilities are limited in real time. If you don t own the data real time is more difficult. Must have real time ingest and expression and support both operational characteristics and analytic characteristics Laws of physics (data can only travel so fast, records can only be inserted so fast, indexes can be updated only so quickly) Is it worth the cost? If it is real-time, then it is mission critical. What about outages, DR, etc? Some customers are asking about Real-Time Some are starting to implement real time systems in order to differentiate themselves from their competitors

25 Agenda Traditional Data Warehousing Workload evolution Velocity, Variety and Volume Information consumer changes Workload characteristics (the operational warehouse) Predictive Analytics Latency requirements Real-time BI, is it real? System z strengths and gaps How we are addressing those gaps

26 System z Strengths and Gaps The System z platform offers unmatched cost of ownership, qualities of service, scalability and performance through integrated hardware, operating system and software. Our strengths are Timely, accurate and secure information Superior scalability, availability and performance Reduced cost and complexity Rapid deployment and expansion Our gaps are/were Software that runs on System z Acquisition cost Performance (scanning limited due to processor availabilty) Perception (z is the operational platform. warehousing and analytics do not require the strengths of z)

27 Agenda Traditional Data Warehousing Workload evolution Velocity, Variety and Volume Information consumer changes Workload characteristics (the operational warehouse) Predictive Analytics Latency requirements Real-time BI, is it real? System z strengths and gaps How we are addressing those gaps

28 How we are addressing those gaps Software that runs on System z Linux for System z alternatives for distributed products Porting of analytics acquisitions Cognos SPSS CPLEX (optimization) Acquisition Cost Different licensing alternatives to address warehouse MIPS requirements DB2 Value Unit Edition (One Time Charge version of DB2 for z/os) Solution Edition (specialty bundles of hardware, software and services) Appliance Add-On Specialty Processors Lower prices for processors, memory Focus on Total Cost (Energy, labor, virtualization, other) Performance More and faster processors IBM DB2 Analytics Accelerator (which also addresses cost) An abundance of resources Hardware optimized for scanning Perception The market is changing to the value of System z due to the change in warehouse requirements

29 IBM zenterprise Analytics System 9700 Flexibility in Critical Data Decision Systems Organized for simplicity and functionality System z EC12 z/os Operating System Stack DS8870 Data Warehouse Data Mart ODS ETL/ELT DB2 for z/os Analytics Accelerator Operational Source Systems Common Metadata 29

30 IBM zenterprise Analytics System 9710 Cost Effective Critical Data Decision Systems Organized for simplicity and functionality System zbc12 z/os Operating System Stack ETL/ELT Operational Source Systems DS8870 Data Warehouse Data Mart ODS DB2 for z/os DB2 Analytics Accelerator 30

31 An enterprise information hub for modern analytics Data Mart Data Mart Data Mart Best in OLTP & Transactional Solutions Transaction Processing Systems (OLTP) Data Mart Consolidation Industry recognized leader for mission critical transactional systems Best in Analytics Data Mart Industry recognized leader in Business Analytics and Data Warehousing solutions Business Analytics Best in Consolidation Predictive Analytics zenterprise Recognized leader in workload management with proven security, availability and recoverability DB2 Analytics Accelerator for z/os Powered by Netezza Recognized leader in costeffective high speed deep analytics Unprecedented mixed workload flexibility and virtualization providing the most options for cost effective consolidation Together Bringing transactional & decision support workloads together on a single platform

32 IBM DB2 Analytics Accelerator Do things you could never do before! What is it? A high performance appliance that integrates Netezza technology with zenterprise technology, to deliver dramatically faster business analytics What does it do? Accelerates complex queries, up to 2000x faster Lowers the cost of storing, managing and processing historical data Minimizes latency Reduces zenterprise capacity requirements Improves security and reduces risk Complements existing investments

33 IBM DB2 Analytics Accelerator Do things you could never do before! More insight from your data Transparent to the application and reporting tools querying DB2 Blends the best attributes of symmetric multiprocessing (SMP) with the best attributes of hardware-accelerated massively parallel processing (MPP) to deliver extraordinary performance cost-effectively Inherits DB2 for z/os attributes No need to create or maintain indices Eliminate query tuning Fast deployment and time-to-value 33

34 IBM DB2 Analytics Accelerator Product Components CLIENT zenterprise PureData System for Analytics (Netezza Technology ) Data Studio Foundation DB2 Analytics Accelerator Admin Plug-in OSA-Express 10 GbE Dedicated highly available network connection Users/ Applications Data Warehouse application DB2 for z/os enabled for IBM DB2 Analytics Accelerator IBM DB2 Analytics Accelerator

35 DB2 for z/os Approach: Hybrid Database Management System Uniform and transparent access for transactional and analytical applications Data Manager Applications DBA Tools, z/os Console,... Application Interfaces (standard SQL dialects) Buffer Manager... IRLM Operation Interfaces (e.g. DB2 Commands) Log Manager IBM DB2 Analytics Accelerator Uniform DB2 service, maintenance, database administration... System z Superior availability reliability, security, workload management, OLTP performance... Powered by PDA True appliance, Industry leading ease of performance

36 PureData System for Analytics Disk Enclosures Slice of User Data Swap and Mirror partitions High speed data streaming High compression rate SMP Hosts DB2 Analytics Accelerator Server SQL Compiler, Query Plan, Optimize Administration Snippet Blades TM (S-Blades, SPUs) Processor & streaming DB logic High-performance database engine streaming joins, aggregations, sorts, etc.

37 S-Blade Data Stream Processing FPGA Core CPU Core Stream via Zone Map From Decompress Project Restrict Visibility SQL & Advanced Analytics From Select Where Group by Select State, Age, Gender, count(*) From From MultiBillionRowCustomerTableWhere Where BirthDate BirthDate < 01/01/1960 < And 01/01/1960 State in And ( FL, State GA, in SC, ( FL, NC ) GA, Group SC, NC ) by State, Group Age, by Gender State, Order Age, by Gender State, Order Age, Gender by State, Age, Gender 37

38 Connectivity Options Multiple DB2 systems can connect to a single Accelerator DB2 Accelerator DB2 A single DB2 system can connect to multiple Accelerators Accelerator DB2 Accelerator Multiple DB2 systems can connect to multiple Accelerators DB2 Accelerator Accelerator DB2 Better utilization of Accelerator resources Scalability High availability Full flexibility for DB2 systems: residing in the same LPAR residing in different LPARs residing in different CECs being independent (non-data sharing) belonging to the same data sharing group belonging to different data sharing groups

39 Query Execution Process Flow Application Interface Optimizer Heartbeat CPU SPU FPGA Application Query execution run-time for queries that cannot be or should not be off-loaded to the Accelerator Accelerator DRDA Requestor SMP Host Memory SPU CPU FPGA Memory SPU CPU FPGA Memory SPU CPU FPGA Memory DB2 for z/os Accelerator Queries executed without DB2 Analytics Accelerator Queries executed with DB2 Analytics Accelerator Heartbeat (Accelerator availability and performance indicators)

40 High Performance Storage Saver Storing historical data in accelerator only Query from Application Part #1 DB2 No longer present on DB2 Storage Or Accelerator Part #1 Part #2 Part #3 Part #4 Part #5 Part #6 Part #7 Active Historical Time-partitioned tables where: only recent partitions are used in a transactional context (frequent data changes, short running queries) the entire table is used for analytics (data intensive, complex queries) High Performance Storage Saver s Archive Process: Data is loaded into Accelerator if not already loaded Automatically takes Image Copy of each partition to be archived Automatically remove data from DB2 archived tablespace partitions Archived partitions as read-only

41 Storage options to match data needs Optimized in both price and performance for differing workloads High Performance Storage Saver Single Disk Store Database Resident Partitions Dual Disk Store Only stored on Accelerator storage (Less Cost) Optimized performance for deep analytics, multifaceted, reporting and complex queries Only full table update or full partition update from backup Same high speed query access transparently through DB2 Stored on both DB2 and Accelerator storage Mixed query workload with transactions, single record queries and record updates with deep analytics, multifaceted, reporting and complex queries. Full table, full partition update, Incremental update from DB2 data Same high speed query access transparently through DB2 Cost The right mix of cost and functionality Functionality 41

42 Learn More! Visit the zanalytics Website Enterprise Insight for Proven Competitive Advantage -- Delivering Real-time Business Critical Analytics Join the zanalytics Networking Community

43