Optim DB2 Tools - Overview

Transcription

1 Optim DB2 Tools - Overview Leif Pedersen Solutions Architect, Optim Enablement Leif.Pedersen@dk.ibm.com

2 Disclaimer IBM s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. 2

3 Topic Outline Introduction 3 Performance Management Data Studio Optim Query Workload Tuner Optim Configuration Manager Optim Workload Replay Application Interfaces DB2 Connect z/os Connect Summary

4 Enterprises want to drive business innovation through software But 70% of IT budget locked into maintenance and operations rather than enabling innovation Business and IT not aligned on how to prioritize work to maximize business value More than 75% of projects miss their target delivery dates 4

5 DB2 Tools Portfolio to Manage Complete Data Cycle Data Architect DB2 Connect Design Data Studio Access & Connect Develop Query Workload Tuner Workload Replay Monitor & Optimize Test & Deploy Data Studio Configuration Manager Administer purequery 5

6 What is IBM Data Studio Comprehensive data management tool An integrated environment for managing databases and developing database applications Built on the popular Eclipse framework Support for Red Hat Linux, SUSE Linux, Windows Optional extra component Data Studio Web console: health and availability monitoring FREE to download! 6

7 IBM Data Studio Features Increase Productivity for Developers and Administrators Integration Support for Databases Develop, Deploy and Debug database applications and routines Manage database objects: Create, Alter, Drop Develop and Execute SQL scripts Impact analysis Compare database objects Shell share with eclipse based tooling Reduced install footprint Rational Application Developer, Rational Appliation Developer for z, InfoSphere Data Architect, Optim Query Workload Tuner, ec Launch points to web based tooling Optim Performance Manager, Data Studio web console, etc. Configurable/Tunable platform No Copy Analytics DB2 for Linux, UNIX and Windows Support DB2 for z/os support 7

8 What s New in IBM Data Studio Leverage the latest DB2 11 for z/oscapabilities Business analytics enhancements Better application performance with SQL and XML enhancements Data Sharing enhancements SQL Results View Enhanced Customers can choose RUN SQL perspective with only 2 panes SQL, Results DBA can view results as split pane(history and result) or single pane(history only or result only) Query Tuning Enhancements Workload Analytics Acceleration Advisor Better support of tuning statements using GTT Improved support for capturing SQL statements from triggers Tuning as a Service Support for Big SQL for InfoSphere BigInsights Browse/ Navigate/ Create/ Alter Hadoop tables Better Experience for reliable change script generation Streamline UI so DBA s save time when generating change scripts Enhance DDL generation by using Create or Replace and Alter instead of Drop Recreate 8 DB2 Client Command Line Processor Import/ Export data from/to local disk Execute DB2 LUW command line scripts

9 How to get Data Studio A no charge full licensed copy of Data Studio can be downloaded from developerworks Support is via Data Studio forum Use IBM Installation Manager for efficient install Product Bundles DB2 for Linux, UNIX and Windows DB2 Accessories Suite for z/os IBM Informix 9

10 Data Studio - Eclipse basic terminology Perspective A collection of views and editors View Data Project Explorer Editor (SQL PL example) Editor Properties View SQL Results View Data Source Explorer Customizable Open, close or resize views and editors in the perspective and drag and drop them anywhere. 10

11 Data Studio - Stored Procedure Development Create stored procedure. Choose Language and Template. Finish. 11

12 Data Studio - Stored Procedure Development Data Studio generates SQL procedure that returns all tables for default schema. 12

13 Data Studio - Deploy Build SQL Procedure Run (execute) procedure 13

14 Data Studio - Executing SQL script We can write our own SQL statement. Data Studio allows validate SQL syntax for given database, specifies special register values, and more. 14

15 Data Studio - Executing SQL script and generating access diagram You can see result of execution in the SQL Results Window: 15

16 Data Studio Database administration perspective 16

17 Data Studio - Mind your test data Intuitive insert, update, delete interface 17 17

18 What is Optim Configuration Manager - OCM? A tool to lower overall operational $$$s for the Z Enterprise... Quickly build a client/server inventory for problem determination and migration planning Automatically alert deviations from Best Practices configuration settings for compliance (1000s of clients or Z servers) Continually enforce Best Practices configuration settings to avoid outages (1000s of clients or Z servers) Dynamically optimize workloads so Z resources are utilized efficiently (Throttle, Penalty Box, WLM, WLB) Unique to IBM and Patent Pending! 18

19 InfoSphere Optim Configuration Manager (OCM) Understand Your Environment Explore client and database server properties Create a baseline inventory of clients for upgrades Solve Problems Faster Prevent problems by scheduling configuration compare jobs that issue Alerts for differences Keep configurations in synch Debug problems faster by reviewing recent changes made to client or server configurations 19 Control Clients (JCC, CLI,.NET) Isolate/Throttle rogue Apps to limit impact on the SYSPLEX Enforce best practices configuration WLM, WLB, WAS Pool or any driver, data source, special register settings Redirect client connections for high availability or staged roll-out Note: With OCM client install Centrally Manage Entire DB2 z/os Environment Deploy Location Aliases or Application Profiles with DB2 for z/os v10 Define zparms with DB2 for z/os v9 or DB2 for z/os v10 Show changes made via z Admin Tool and z Admin OC Tool

20 Managing Production Changes More Efficiently Change in DB2 Release upgrades (e.g. v9 to v11) and maintenance (e.g. PTFs) Exploitation of new capabilities, such as DB2 z/os v11 features Workload tuning (e.g. using Optim Query Workload Tuner) Configuration Change in hardware or software Hardware upgrades Maintenance SYSPLEX load-balancing Change in workloads Increased throughput New application rollout 20 Impact analyzed in simulated Environment with fabricated tests

21 Understanding What Impact Changes Have Without Workload Replay a production-like application infrastructure is needed for realistic testing and analysis With Workload Replay replay actual production SQL with original concurrency without the need for complex application infrastructure 21

22 InfoSphere Optim Workload Replay Capture production workloads and replay them in testing environments Record Application InfoSphere Optim Workload Replay for DB2 for z/os Play DB2 for z/os source subsystem or data sharing group DB2 for z/os target subsystem or data sharing group Also available for DB2 LUW and other distributed servers! Requirements Minimize unexpected production problems Shorten testing cycles Develop more realistic subsystem testing scenarios Benefits Identify subsystem problems sooner with validation reports and performance tuning Use actual production workloads for testing rather than fabricated scenarios Extend quality testing efforts to include the data layer 22

23 Optim Workload Replay Popular usecases 1. Performance Testing: Safely make changes in Test environment for DB object changes, Indexes, Bufferpool allocations, Tablespace Configuration etc. and replay production workload to assess performance impact 2. Stress Testing: Test for anticipated increased workload by controlling speed of Replay (replay faster by taking out think time between SQL executions) Stress testing with end of Month, end of Quarter or end of Year captured workloads 3. Database version migration: Establish deeper and easily repeatable performance baseline before migrating Easily create a catalog of workloads from different applications and business cycles that can be replayed for testing any time 4. Ongoing database maintenance: Increase confidence when applying maintenance especially if customer is behind and needs to apply a large amount of maintenance Ensure no unintended side of effects of making configuration changes 5. New Application deployment New applications are often tested in isolation Run new application simultaneously with replay of production workload to validate how new application will interact with other loads on system 6. Platform consolidation: Consolidation of hardware to move from couple of less powerful servers to newer, more powerful single server 7. Disaster Recovery validation: Establish consistent and repeatable process to validate Stand-by system is capable to handle ever increasing production workload 23

24 Solution Overview Production DB2 10 for z/os InfoSphere Optim Workload Replay for DB2 for z/os Record Test DB2 11 for z/os Capture Play Prepare Replay Compare and Analyze Capture all components needed for real-life workload simulation Prepare workload replay Replay workload Compare replay with original capture or subsequent replays Validate correct SQL execution behavior Identify performance regressions and/or improvements Establish baseline; introduce changes and analyze impact 24

25 Architecture S-TAP for DB2 for z/os Installed with each data sharing member Intercepts subsystem traffic Workload Replay controller Controls starting and stopping of OWR S-TAP, on as-needed basis Drives Local Replay Workload Replay server Software component Built on Guardium v9 Processes, replays and analyzes workloads Web based user interfaces 25

26 DB2 Performance Solution Pack for z/os 26 Identify Prevent OMEGAMON for DB2 Performance Expert on z/os Full range system and application monitoring Detailed reporting and performance warehouse with integrated real-time monitor and Extended Insight feature DB2 Query Monitor for z/os Dedicated SQL monitoring Current and historical views of query detail throughout DB2 subsystems with ability to set alerts for early warning Optim Query Workload Tuner for DB2 for z/os Eclipse-based SQL performance analysis Enables visualization of query hotspots and includes expert advisors to provide tuning recommendations DB2 SQL Performance Analyzer for z/os z/os-based SQL performance analysis Cost estimate and explain function with what if analysis Diagnose Solve

27 Extended Insight into Transactions Where transactions spent time across the infrastructure Application, client, application server, network, database Where transactions spent time within the database Compiler, lock wait, I/O, sorting, routine processing, How workloads met their service level objectives By application, application server, transaction, user, Diagnose Drill down into problem detail and related content Analyze captured data OMPE Extended Insight Time spent distribution detail. Hover over shows actuals Summary of workloads with alerts showing over time spent distribution and percent of transactions with alerts 27 OMPE Extended Insight

28 DB2 Query Monitor Low overhead SQL monitor; requires no object traces DATA collected and available to view SQL metrics DB2 object access SQL text and host variables DB2 commands Negative SQLCODES Expanded and grouped Information about exceptions Buffer Pool Statistics Delays History Interval-based VSAM Datasets for data storage Intervals of data viewable online User controlled number of intervals retained Interval length 28

29 SQL-level Early Alerts / Automatic Notifications Quickly identify problems before they impact the business Identify Alert of potential problems Visual quick scan of complex environment Monitor all alerts and exceptions across all domains for specific SQL performance problems Filter specified alert messages to quickly see problems according to your criteria Priority colorization enables quick identification of critical, high, medium, low, warning and informational alerts DB2 Query Monitor 29

30 DB2 Query Monitor 3.1 Browser Client Quickly navigate across DB2 Data Sharing Groups and Subsystems Easily drill into performance problems 30 ISPF based line commands for power users

31 InfoSphere Optim Query Workload Tuner Identify query candidates DB2 catalog Dynamic statement cache Development Studio hot spots Query or performance monitors Facilitate analysis Query formatting Query annotation Access path visualization and annotation Get expert tuning advice Improve query design Improve statistics quality Improve database design 31

32 Integration Speeds Resolution Times Solve problems closing the loop on problem determination Solve Receive expert advise for problem resolution Correct the problem DB2 SQL Performance Analyzer DB2 Query Monitor OMPE and Extended Insight InfoSphere Optim Query Workload Tuner 32

33 Maximize Investment in the IBM DB2 Analytics Accelerator Value added with IBM DB2 Tools Powerful IBM z Tools capabilities to maximize the Accelerator s impact Measure and report on accelerated query improvement to validate ROI both batch and real-time. Alert on IDAA events - OMEGAMON for DB2 Administer IDAA with traditional DB2 administrator tooling (ISPF) - DB2 Administration Tool Gain insight into workloads, get recommendations on acceleration candidates Query Workload Tuner Analyze Visualize Identify Manage Understand Monitor Report Validate Optimize Parallel loads of accelerator, Load from alternate sources Image Copies, Oracle, Teradata, - IDAA Loader Detailed SQL level accelerator reporting. Filter out un-accelerated queries to maximize workload tuning DB2 Query Monitor Future: Load IMS data to extend operational analytics 33

34 Monitor, Analyze and Report on IDAA with OMEGAMON XE for DB2 Performance Expert on z/os Quickly understand appliance utilization and what other opportunities may exist to maximize your ROI Immediate feedback on accelerated query performance via online monitoring Detailed performance measurements via online and comprehensive batch reporting of the accelerator and its applications Easily analyze performance trends with the accelerator specific performance metrics and save them to a performance database Chargeback reporting for usage accountability Real time display of a thread with accelerated queries Statistics batch report showing Accelerator performance metrics 34

35 Identify, Compare & Segregate Accelerated Queries DB2 Query Monitor for z/os Enhanced visibility of offloaded queries on DB2 Analytics Accelerator Activity by SQL Text screen in web client Measurable performance comparisons for ROI justification Increased productivity through drill-down capabilities to identify and tune only those queries that are not being accelerated Query Segregation screen in ISPF interface 35

36 Query Selection and Access Plan Analysis with Optim Query Workload Tuner for DB2 for z/os Shorten the process of selecting queries to be accelerated Visualize access paths of accelerated queries Increase productivity by working with accelerated queries through a unified interface The ACCELERATED node in this Access Plan Graph indicates that queries will be routed through the accelerator everything below represents the access plan from the accelerator 36

37 Streamlined Performance Analysis 1 Define or select a workload 2 Execute Advisors 3 Drill Down into advice 4 Validate improvement 37

38 Integrate OQWT with DB2 QM / OMPE for complete analysis Support different integration scenarios Push a SQL or SQL workload from DB2 QM client into OQWT Push a SQL or SQL workload from OM DB2 PE to OQWT Pull a SQL workload from DB2 QM offload repository on OQWT GUI Demo Video OMPE and Extended Insight

39 Identify Workload For Acceleration Identify the top N most expensive queries in Query Monitor Drill down from QM web interface Transfer top N queries to Query Workload Tuner as a workload for tuning Capture quieries into QWT Order by ave CPU time Transfer to QWT 39

40 Customize Query Workloads Filtering Options Accelerate analysis, reduce downtime 40

41 2 Execute Advisors Statistics Get recommendations on the best statistics to capture to influence access path selection Query Get recommendations regarding how to rewrite the query for better efficiency Index Get recommendations on indexes changes that can reduce database scans Analytics Accelerator Get recommendations on optimizing and managing accelerated analytic queries and applications 41

42 3 Drill down to advice for accelerated workload Determine tables to be routed the Accelerator Simplify use Consolidate tables, queries to provide a single recommendation Enable what-if analysis Run immediately or save Estimated CPU savings Filter recommendations 42

43 3 Drill down to advice for accelerated workload Show Tables referenced by statement Rationale Recommendation and rationale 43

44 4 Test Candidate Analytics Acceleration Compare benefits on multiple accelerators Analyze tables ONLY with existing accelerated tables as consideration Compare benefits on each existing accelerator Recommendation table to add or remove from Accelerator New 44 Test before deployment Utilize virtual capabilities built into the DB2 engine run subset of candidate tables

45 Lower overhead DB2 and Accelerator Workload driven what-if analysis for existing real accelerator Accelerator driven what-if analysis for workload balancing Pick and choose table in workload that are already accelerated Recommends table to add / remove from accelerator Displays tables are not referenced in the workload Results Quick deployment of initially offloaded all tables in database to the real IDAA Lower overhead in keeping tables in DB2 and Accelerator New Recommendations to add or remove from accelerator Filter to focus on not referenced tables by the workload 45

46 Loading data into the Accelerator - Tremendous Possibilities 46 How does data get loaded into the Accelerator today? The standard DB2 UNLOAD utility extracts data Places tables in Read-only mode until Accelerator load is done (when using option for transactional consistent data) Accelerator reflects DB2 data at point in time load was initiated Optionally use CDC for near-real-time replication to Accelerator Introducing the IBM DB2 Accelerator Loader Features External Dual Load Group Consistent Load Image Copy Load Built for performance & usability Loader populates data in the Accelerator Load DB2 & non-db2 data Load in parallel to avoid application downtime Load to historical point-in-time I want to maximize the power of DB2 and System z for business analytics. How do I bring Oracle data to the Accelerator for query optimization? I have to prepare a summary report from my business application as of last Thursday. How do I capture the data from a date that is other than current, that can be considered for query acceleration? How do I improve the loading of objects into DB2 and/or Accelerator without impacting my DB2 production data? I have an image copy from an external OLTP DB2 tablespace. How can I use this image copy to load an accelerator table?

47 Query Tuning Service Exposed as individual callable interfaces of query tuning features Access plan comparison Index advice Access plan graph Query format/annotation Workload management Can be integrated into customers' own performance/change management processes 47

48 Query Tuning Service Exposed as individual callable interfaces of query tuning features Included as part of IBM Data Studio Access Plan Graph Query Formatter Included as part of Query Workload Tuner Index Advisor Access Plan Comparison Query Formatter/ Annotation Workload Management Can be integrated into customers' own performance/change management processes 48

49 How Does It Work Run individually either on a web container inside IBM Data Studio or as embedded service Windows or Linux machine Consume in different ways JCL Batch commands Coding in different languages 49

50 Example: Client implements by JCL Invoke the tuning service from JCL to compare the access plans //COMPARE EXEC PGM=BPXBATCH,DYNAMNBR=20,COND=(4,LT) //SYSPRINT DD SYSOUT=* //SYSOUT DD SYSOUT=* //STDPARM DD * curl --insecure -X POST -H "Content-Type: application/json; charset=utf-8" --data "{ dbtype:zos, dbhost:host, dbport:port_number, dbname:subsystem, dbuser:user_id, dbpassword:password, sql:\"select N_NAME FROM NATION WHERE N_NATIONKEY < 10\" }" 50

51 Example: Output from formatsql Output Output { "code":0, "message":"select authorization_id.nation.n_name\n FROM authorization_id.nation --CARDF=25 --QUALIFIED_ROWS=11 --NPAGESF=0\n WHERE authorization_id.nation.n_nationkey < 10 --COLCARDF=25 --MAX_FREQ=(missing) --FF=0.44" 51

52 Query Tuning Service list Single Query Tuning Show formatted SQL statement and annotation Show web-based visual Explain SQL Workload Management Create a new workload from file (text/xml) Explain a workload Delete a workload Workload Access Plan Comparison Compare the EXPLAIN snapshots of workload(s) Workload Index Advisor Show recommended indexes of a workload 52

53 Query Tuning Service: explain a query workload Input Connection Info Workload Name Output Explained query count 53

54 Query Tuning Service: delete a query workload Input Connection Info Workload Name Output Successful or not 54

55 Query Tuning Service: compare access plans of two query workloads Input Connection Info The name of Workload 1(Baseline) The name of Workload 2 (Target) Filter options Access Plan Changed? Cost Increased Ratio Output Comparison Task ID Used to retrieve the comparison status and result 55

56 Query Tuning Service: get Access Path Graph Input Connection Info Query Text Output The link of access plan graph in HTML format 56

57 DB2 Drivers Panoramic View Javascript PhP Python/Jython Ruby/JRuby node.js Scala Java JDBC API Zend framework adapters node-odbc node.js JSON driver SqlAlchemy/ Django Adapter Rails Adapter Lift purequery API JSON API Python interpreter c java Ruby interpreter c java Hibernate JPA DB2 CLI and ODBC driver DB2 JCC JDBC driver DB2 57

58 DB2 Connect APIs - JDBC JDBC Application SQLJ Application Java common layer, SQLJ runtime T4 subsystem builds/parses DRDA requests/replies T2z subsystem calls into native z/os APIs T2u subsystem calls native LUW APIs CCC DRDA Shared memory to local UDB DRDA over TCP/IP DB2, IDS RRS DB2 for z/os 58

59 purequery Runtime, every Java application benefits JDBC acceleration for any JDBC application Convert dynamic SQL to static SQL Replace problem queries without changing the source Hibernate/OpenJPA/iBatis acceleration for persistence layers Improved SQL batch performance Auto-tuning of Hibernate and OpenJPA peristence options SQL-friendly APIs for OO access to relational Object to relational mapping APIs that can be tailored to return XML, JSON, arrays, etc. 59

60 Client Optimization Improve Java data access performance for DB2 without changing a line of code Captures SQL for Java applications Custom-developed, framework-based, or packaged applications Bind the SQL for static execution without changing a line of code New bind tooling included Delivers static SQL execution value to existing DB2 applications Making response time predictable and stable by locking in the SQL access path preexecution, rather than re-computing at access time Limiting user access to tables by granting execute privileges on the query packages rather than access privileges on the table Aiding forecasting accuracy and capacity planning by capturing additional workload information based on package statistics Drive down CPU cycles to increase overall capability Choose between dynamic or static execution at deployment time, rather than development time 60

61 DB2 Java Data Access Frameworks Acceleration Hibernate: hibernateibatispurequery1/index.html?ca=dnw-1133&ca=dth-i ibatis: hibernateibatispurequery2/index.html Spring: 61

62 62

63 Tooling for zos Connect end to end development and testing Config.xml Deploy to bundled zos Connect Data Web Services Test Client Data Web Services of Data Studio or RDz 63 Database operations WSDL ZOS Connect artifacts Deploy to remote zos Connect z/os Connect CICS IMS WAS DB2 ZOS Connect on LPAR

64 End to End Architecture for Mobile, Web, Cloud applications invoking Z services using APIs Mobile Applications Cloud APIs Cloud-based Services Access to systems of records and enterprise data via APIs Enterprise APIs On-Premise API Enablement DataPower GW appliance DMZ API Management discovery discovery/invocation discovery/ invocation z/os Connect DB2 CICS IMS Batch IBM Worklight Server Mobile, Web, Cloud App Enablement Cloud - Integration Services Enterprise API s Web Services or REST based services Enterprise Transaction Processing Remote Application Development: Invokes services and APIs Cloud Integration Enablement: Enabling integration with APIs as well as technical services API Enablement: Consumability by internal and external developers (creation and look up) Entitlement Management (securing, workload enforcement) Usage monitoring & Analytics Service Enablement: Enables invocation of Z services or applications by remote applications using standard protocols (REST) 64

65 Questions? 65

66 Thank You! 66