IBM DB2 Analytics Accelerator: Real-Life Use Cases

Transcription

1 Patric Becker IBM BoeblingenLaboratory IBM DB2 Analytics Accelerator: Real-Life Use Cases

2 Legal Disclaimer IBM Corporation All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. 2

3 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 3

4 Hybrid transaction/analytical processing The hybrid computing platform on z Systems Transaction Processing Analytics Workload Supports transaction processing and analytics workloads concurrently, efficiently and cost-effectively Delivers industry leading performance for mixed workloads The unique heterogeneous scaleout platform in the industry Superior availability, reliability and security 4

5 Deep DB2 integration within z Systems Applications Application Interfaces (standard SQL dialects) DBA Tools, z/os Console... Operational Interfaces (e.g. DB2 Commands) DB2 for z/os Data Manager Buffer Manager... IRLM Log Manager IBM DB2 Analytics Accelerator Superior availability, reliability, security Workload management Superior performance on analytic queries z/os on z Systems PureData System for Analytics 5

6 DB2 Analytics Accelerator Usage scenarios How organizations leverage the Accelerator today Rapid acceleration of Existing Business Critical Queries If the data is analyzed on the mainframe Performance improvements and cost reduction while retaining z Systems security and reliability Accelerate existing workload Reduce IT sprawl Reduce IT sprawl for analytics If the data is offloaded to a distributed data warehouse or data mart Simplify and consolidate complex infrastructures, low latency, reliability, security and TCO Derive business insight from z/os transaction systems If the data is not being analyzed yet Derive new business insight Include external & historical data Improve access to historical data and lower storage costs If the analysis is based on a lot of historical data One integrated, hybrid platform, optimized to run mixed workload Performance improvements and cost reduction Simplicity and time to value 6

7 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 7

8 Halkbank Case Study Halkbank extends banking services on social and mobile channels securely and cost-effectively We wanted to improve the performance for complex analytic queries both in order to support rapid and accurate decision making, and in order to meet deadlines for regulatory reporting, where we face penalties of tens of thousands of dollars for late reporting. We expect to be able to move 70 percent of our queries to DB2 Analytics Accelerator, accelerating queries, reducing our DB2 licensing costs on the mainframe, and freeing up processing resources for other workloads. -- Cenk Niksarlı, Head of IT Infrastructure at Halkbank 8

9 Swiss Mobiliar Case Study and Video Swiss Mobiliar accelerates 50% of queries by a factor of 100 Case Study: Video: DB2 Analytics Accelerator makes it possible for us to execute 90% of our queries 25x faster, and 50% of them 100x faster Queries that used to take five hours to complete are now processed in just 20 seconds in the optimized mainframe environment and we can run them any time, day or night, with no interruption to our production systems on the mainframe. -- Thomas Baumann, IT Performance Architect at Swiss Mobiliar 9

10 Banca Carige Case Study and Video Providing superior customer service on a secure platform Case Study: Video: DB2 Analytics Accelerator is key in our vision to provide the right level of data analysis to all users in the bank and possibly outside the bank. By using DB2 Analytics Accelerator, we expect to reach... all criteria we had in mind and create greater benefits for our users because our data are not moved Daniel Cericola, IT & Governance at Banca Carige 10

11 Swiss Re Case Study and Video Speeding high-level decision making that boosts the bottom line Case Study: Video: Our users are getting their reports as much as 70 percent faster reports that took 10 hours are now available the same day, so user satisfaction has increased dramatically, says Estermann. Because those reports contain key analytics that guide pricing and decision making across our lines of business, the solution has the potential to sharpen the company s competitive edge moving. -- Reto Estermann, Director, Information Technology at Swiss Re 11

12 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 12

13 High Performance Storage Saver Storing historical data in Accelerator only Query from Application to either Active only or Active and Historical Active Part #1 Part #1 No longer present on DB2 Storage Part #2 Part #3 Part #4 Part #5 Part #6 Part #7 Historical 13 Support for partitioned-by-range tables, for example: Time-based partitions, where only the recent partitions are used in a transactional context (frequent data changes, short running queries), but the entire table is used for analytics, regulatory/audit purposes, etc. (data intensive, complex queries)

14 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 14

15 In-database transformation and multi-step processing DB2 Analytics Accelerator introduces Accelerator-Only tables to store intermediate or final results of data transformation or multi-step processes Accelerator-Only tables enable complex data transformations taking place on DB2 Analytics Accelerator, called In-database transformation Accelerator-Only tables enable intermediate and final results of reporting or other processes to be generated and stored on DB2 Analytics Accelerator, called Indatabase multi-step processing 15 Advantages: Accelerate in-database data transformations and data movement processes Reduced need of data movement processes to other platforms for data transformation purposes Enables multi-step reporting on the Accelerator Saves disk space and CPU cost on z Systems currently used for transformations and reporting steps Allow data preparation steps for data mining and other advanced analytics to execute on the Accelerator Store a set of data in DB2 Analytics Accelerator only, not on DB2 for z/os, without using the High Performance Storage Saver functionality

16 Introducing Accelerator-only table type in DB2 for z/os Creation (DDL) and access remains through DB2 for z/os in all cases Non-accelerator DB2 table Data in DB2 only Table 1 Accelerator-shadow table Data in DB2 and the Accelerator Table 2 Table 2 Accelerator-archived table / partition Empty read-only partition in DB2 Partition data is in Accelerator only Accelerator-only table (AOT) Proxy table in DB2 Data is in Accelerator only Table 3 Table 4 Table 3 Table 4 16

17 Accelerator-only tables Technical basics AOTs are created and dropped using DB2 DDL statements (CREATE; DROP) Accelerator must be started QUERY ACCELERATION behavior may have any value during CREATE/DROP Syntax: CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1>; DROP TABLE MYTABLE; Recommended to create a database in DB2 to be used for the AOTs CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1> IN DATABASE MYDB; Usual authorization necessary to create objects in database SELECT and INSERT/UPDATE/DELETE operations using AOTs can only run on the Accelerator QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL Accelerator-shadow tables, Accelerator-archived tables and other AOTs can be used in the same statement Dynamic and static SQL can be used with AOTs 17

18 Accelerator-only tables Technical basics INSERT FROM SELECT into an AOT Can only run on the accelerator All tables in the SELECT must reside in the accelerator Accelerated shadow DB2 tables Archive tables AOTs QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL Cross-Loader to load data from AOTs into DB2 for z/os Different table names 18

19 Performance INSERT from SELECT Up to 95% better elapsed time and negligible CPU time in DB2 for INSERT FROM SELECT into accelerator-only tables for large amount of data 600 Class 2 CPU Time 500 Class 2 CPU Time (Lower is Better) # Rows in MIll CL2CPU_DGTT CL2CPU_PBG CL2CPU_IDT 19 19

20 Multi-step reporting applications with DB2 for z/os BEFORE Accelerator-only tables: Report processing on DB2, source data might reside on the Accelerator already Reporting Application Multi-Step Report 1 Credit Card Transaction History Credit Card Transaction History 2 n Customer Summary Mart Temporary results 2 1 Customer Summary Mart Temporary results n Temporary results Reports and Dashboards 20

21 Multi-step reporting applications with DB2 for z/os With Accelerator-only tables: Temporary objects and processing on the Accelerator Reporting Application Multi-Step Report 1 Credit Card Transaction History Credit Card Transaction History 2 n Customer Summary Mart 1 Customer Summary Mart Temporary results 2 Temporary results n Temporary results Reports and Dashboards Data for transactional and analytical processing 21

22 In-database transformation Using Accelerator-only tables and ELT logic in the Accelerator Transaction Processing Systems (OLTP) Customer Transactions Customer Data Customer Transactions Customer Data ELT logic Customer Transaction Summary and History AOTs Analytics Customer Summary Mart AOTs Advantages: Simpler to manage Better performance and reduced latency Data for transactional and analytical processing 22

23 Data scientist work area Using Accelerator-only tables for ad-hoc analysis Transaction Processing Systems (OLTP) Customer Transactions Customer Data Customer Transactions Customer Data Data Scientist John Work database John Work Area AOTs Work database Bob Work Area AOTs Data Scientist Bob Data for transactional and analytical processing 23

24 What can I do with AOTs? Deeper insight into operational status through faster reporting Support multistep reporting applications. Third party BI reporting suites, QMF, home grown applications, etc. Accelerated campaign tuning for IBM Campaign (Unica) Improved performance for iterative campaign tuning. Simplifying data-transformation processes Delivering in-database transformation within DB2 Analytics Accelerator Data mart consolidation through flexible data infrastructure Host data marts on z Systems, where the data originates Deeper insight into customers and markets Data scientist work area Simpler data integration with DB2 Analytics Accelerator Loader for z/os (load non-db2 for z/os data) Assimilate more data sources for analytics to shorten development cycles and speed integration efforts In-database analytics to accelerate predictive analytics Improves the quality of models, speeds calculations and delivers real-time, actionable business processes 24

25 (Relatively) New IBM Redbooks publication: SG Analytics on z Systems environment Warehouse concepts Logical data warehouse Transformation patterns Accelerator-only tables Concepts and architecture Use cases enabled by accelerator-only tables and in-database-analytics Multi-step reporting Using QMF to store query results and importing tables Accelerating IBM Campaign processing In-database transformations Accelerator and accelerator-only table usage within DataStage Accelerator-only tables supporting data scientists ad-hoc analysis Integration of more data sources and archiving for analytics In-database analytics 25

26 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 26

27 In-database analytics Enable acceleration of predictive analytics applications In-database analytics enables SPSS/Netezza Analytics (INZA) data mining and in-database modeling to be processed within the Accelerator Accelerates SPSS/Netezza Analytics (INZA) data mining and in-database modeling Allows frequent model refreshes to enable adequate scoring Reduced need of data movement processes to other platforms for predictive analytics purposes Supports the full lifecycle of a real-time analytics solution on a single, integrated system, combining transactional data, historical data, and predictive analytics 27

28 In-database analytics Technical basics Only SPSS Modeler 17 or higher is supported, which needs 18 INZA stored procedures dectree - Builds a Decision Tree model by growing and pruning a tree grow_dectree - Builds a Decision Tree model predict_dectree - Applies a Decision Tree model to generate classification predictions prune_dectree - Prunes a previously built Decision Tree model regtree - Builds a Regression Tree model by growing and pruning a tree grow_regtree - Builds a Regression Tree model prune_regtree - Prunes a previously built Regression Tree model predict_regtree - Applies a Regression Tree model to generate regression predictions for a dataset naivebayes - Builds a Naive Bayes model predict_naivebayes - Applies a Naive Bayes model to generate classification predictions for a dataset kmeans - Builds a Clustering model that clusters the input data into k centers. The centers are calculated as the mean value of the nearest input data records predict_kmeans - Applies a K-means Clustering model to cluster records of a dataset 28

29 In-database analytics Technical basics two_step - Builds a TwoStep Clustering model that first distributes the input data into a hierarchical tree structure according to the distance between the data records, then reduces the tree into k clusters. A second pass over the data associates the input data records to the next cluster predict_twostep - Applies a TwoStep Clustering model to score records of a dataset split_data - Randomly splits the input data into two separated subsets pmml_model - Stores the given analytics model as PMML document to a table list_all_models Lists all models with given characteristics model_exists - Checks if the given model exists. The model can be searched in the current or in another given database. drop_model - Drops the given model. All managed tables of this model are also dropped drop_all_models Drops all models For more information about these stored procedures, see the IBM Netezza In-Database Analytics Reference Guide and the IBM Netezza In-Database Analytics Developer's Guide. These PDF files are delivered in a compressed archive called nz-analyticsdoc-v3.2.0, which is available for download from IBM Fix Central. 29

30 Generic Wrapper for Remote Stored Procedures External Providers can supply Stored Procedures to execute on an Accelerator DB2 only acts as gateway 30

31 In-database analytics Technical basics Example of how an application calls the DB2 for z/os wrapper stored procedure: CALL INZA.KMEANS( MYACCEL', 'model=adult_mdl, intable=tpch30m.customer, outtable=iwatest.adult_out, id=c_custkey, target=c_nationkey, transform=s, distance=euclidean, k=3, maxiter=5',?, ''); Blue = procedure/algorithm to execute Red = Accelerator to run the procedure on Green = Algorithm parameters Orange = Table information More information on wrapper stored procedures in DB2 Analytics Accelerator Knowledge Center: 01.ibm.com/support/knowledgecenter/SS4LQ8_5.1.0/com.ibm.datatools.aqt.doc/sp_msg/SPs/sp_ida a_wrapper.html 31

32 In-database analytics Data preparation (using AOTs) and SPSS modeling in the Accelerator Data for transactional and analytical processing Transaction Processing Systems (OLTP) With embedded scoring Customer Transactions Customer Data Customer Transactions Customer Data Customer Txn Data Prep AOTs Modeling SPSS Modeler Advantages: Allows fast model refreshes Ensures adequate scoring Better performance and reduced latency 32 Scoring outside accelerator with SPSS Modeler Server Scoring Adapter for DB2 for z/os Model Model

33 Installation and setup Described in Knowledge Center 01.ibm.com/support/knowledgecenter/SS4LQ8_5.1.0/com.ibm.datatools.aqt.do c/installmanual/concept/c_idaa_inst_analytics.html Installation package can be downloaded from Fix Central 33

34 In-database analytics (Customer Example) Data preparation (using AOTs) and SPSS modeling in the Accelerator Data Preparation Modelling Scoring source data Step 1 source process process type modeling nugget score output (select) node (select) result data IDAA Query acceleration Step 2 Step 3 In-database transformation and analytics Step 1: Performance Results Significant acceleration of select statement: Up to 10x Step 2: Performance Results: Data Preparation in minutes which was not possible before Significant acceleration of data preparation : Up to 240x End-to-end effect : Up to 23x Step 3: Performance Results Acceleration of Modelling highly depends on data size. 34 Your mileage can and will vary!!!

35 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 35

36 Workload Balancing with multiple accelerators having the same data Capacity weight DB2 Subsystem or DSG Member T1 T2 T3 Automatic workload balancing based on capacity weight Capacity weight T1 T2 T3 T1 T2 T3 Accelerator 1 Accelerator

37 Query routing after failure of an active accelerator Capacity weight DB2 Subsystem or DSG Member T1 T2 T3 Capacity weight T1 T2 T3 T1 T2 T3 Accelerator 1 Accelerator

38 Mixed setup with a shared accelerator DB2 Production Subsystem or DSG Member T1 T2 DB2 Test system T3 Load / Update T1, T2 T1 T2 T1 T2 T % Production 50 % Test

39 Mixed setup with a shared accelerator after a failure DB2 Production Subsystem or DSG Member T1 T2 DB2 Test system T3 Load / Update T1, T2 T1 T2 T1 T2 T % Production 0 % Test

40 Incremental Update with active/standby capture agents DB2 Log Cache Data Sharing Group CF DB2 Dynamic VIPA Last applied LRSN/RBA Last applied LRSN/RBA Accelerator 1 Accelerator

41 Incremental Update after active capture agent has failed DB2 Data Sharing Group CF Log Cache DB2 Dynamic VIPA Last applied LRSN/RBA Last applied LRSN/RBA Accelerator 1 Accelerator

42 High Performance Storage Saver with multiple accelerators after failure DB2 Subsystem or DSG Member Table Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 archived Image Copy Table Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 Accelerator 1 Accelerator 2 Table Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 42

43 High Performance Storage Saver with multiple accelerators DB2 Subsystem or DSG Member Table Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 archived Image Copy Table Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 Accelerator 1 Accelerator 2 Table Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 43

44 Example: GDPS PPRC active/active with multiple site workload Site 1 Maximum 20 km Site 2 P P P GDPS-PPRC S S S DB2 Data Sharing Group CF DB2 44 Accelerator 1 Accelerator 2

45 Example: Workload flow for GDPS PPRC active/active Site 1 Site 2 DB2 Data Sharing Group DB2 Queries Queries 45 Accelerator 1 Accelerator 2

46 Example: GDPS PPRC active/active after failure of one site Maximum 20 km Site 1 Site 2 P P P GDPS-PPRC S S S DB2 Data Sharing Group CF DB2 46 Accelerator 1 Accelerator 2

47 Example: Workload flow for GDPS PPRC active/active after failure of one site Site 1 Site 2 DB2 Data Sharing Group DB2 Queries Queries Accelerator 1 Accelerator 2 47

48 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 48

49 Loading non-db2 Data Sources into Accelerator Product Comparison IBM DB2 Analytics Accelerator Loader The user must: Extract data from source (IMS, VSAM, Oracle, SMF, etc.) Convert extracted data to DB2 external load file format DataStage or other tooling can be used Create a DB2 table that matches format of extracted data Add newly created table to the Accelerator Construct a DB2 Load utility field specification that describes the input data Run Accelerator Loader batch job to load data to accelerator Accelerator Loader V2.1 Automates entire process: User builds a select statement from data source(s) (IMS, VSAM, SMF, Oracle, ) 49 Automatically creates the DB2 table Automatically adds table to Accelerator Automatically extracts specified source data Automatically converts data to necessary DB2 format (in memory) Automatically loads data to Accelerator Automatically enables table for acceleration Single batch job!

50 IBM DB2 Analytics Accelerator Loader for z/os v2.1 z/os based Server Direct Load from Data Sources 50

51 Loader V2.1 for Direct Load from VSAM into Accelerator Minutes Extract * + DB2 Load + Accelerator Load The Old vs The New 200GB VSAM KSDS z13 LPAR: 2 CPs 6 ziips w/smt ACCEL_LOAD_TASKS = Extract * + Loader External Load Loader Direct Load from Source Traditional Approach Loader V1.1 function Loader V2.1 Elapsed Time GCP Time ziip time 51 * Extract - A bare bones COBOL program was written/utilized for the base Extract

52 Agenda Value Proposition and Use Cases Query Acceleration High Performance Storage Saver In-database transformations In-database Analytics High Availability configurations DB2 Analytics Accelerator Loader V2.1 Summary 52

53 DB2 Analytics Accelerator Version 5.1 Adding new dimensions in functionality to expand use cases Rapid acceleration of existing Business Critical Queries Adding application support for temporary objects (QMF, Multi-step Reporting, IBM Campaign, etc.) Insight into now to maximize business opportunities in today s dynamic environment Accelerate existing workload Reduce IT sprawl Reduce IT sprawl for analytics In-database transformation to support Data Stage Balanced Optimization and the consolidation of ETL/ELT processing in DB2 for z/os Business agility through simplified architecture Derive business insight from z/os transaction systems In-database analytics to accelerate predictive analytics applications; SPSS/INZA data mining and in-database modeling can be processed within the Accelerator Real-time, actionable business processes Derive new business insight Individual ad-hoc analysis that provides a Data Scientist Work Area Environment to efficiently, continuously test and improve analytic results to drive better customer understanding Include external & historical data Improve access to historical data and lower storage costs Integrate more data sources for analytics, using DB2 Analytics Accelerator Loader for z/os to assimilate with IMS data or data from other sources Simplified access to information when you need it 53 Deliver Right-Time Analytics to drive better business outcomes

54 54