Where did Visual Explain Go?

Transcription

1 Where did Visual Explain Go? Doug Clifton Executive IT Specialist Information Management

2 Agenda Visual Explain History Stats Advisor DB2 Management Clients Package Optimization Service Center Optimization Expert Data Studio InfoSphere Optim Query Tuner for z/os Optim Query Workload Tuner for z/os Complimentary Products Questions Appendix 2

3 Performance tuning challenges across the organization Application Developer I don t have time to hone my SQL skills. I need to focus on developing core application functionality. It is very challenging to aggregate performance data across our complex data environment. DBA I rely on programming frameworks to create SQL for me. How can I tell if it s optimized? I don t understand why our developers aren t focused on creating better performing SQL. 3 IT Manager Performance problems seem to appear without warning and deep technical skills are hard to find. QA Manager We can t adequately test for peak workload since we don t have enough human or IT resources.

4 History Visual Explain DB2 V OSC DB2 9 (8.1) 2007 OE Data Studio DB2 8, 9, IOQWT DB2 8, 9, OQWT 4

5 DB2 Management Clients Package V8 Includes: Visual Explain introduced with DB2 V5 DB2 Connect DB2 Administration Server for z/os DB2 UDB for z/os Control Center Enablement Tools provided on CD-ROM DB2 UDB for z/os Control Center Enablement is shipped on a tape or cartridge DB2 Management Clients Package Program Directory has installation directions 5

6 Visual Explain Includes: Graphical Depictions of Access Plans: Relationships between database objects (e.g. tables and indexes) and operations (e.g. tablespace scans and sorts) are illustrated in the graphs Generate customized reports on explainable statements View subsystem parameters View data from the plan table, the statement table, and the function table Stat Advisor: Provides a report of what statistics are suggested to maximize your access path Rewritten for V8 6

7 Visual Explain, OSC and OE 1. DB2 9 for z/os enhancements Profile based Query Performance normal/exception monitoring Virtual Index What-if Optimizer support 1 Server based Administrative Scheduler 1 Expanded Explain facility 1 2. Built on Eclipse New Outside In Design process 3. Optimization Service Center (OSC) Visual Explain functional replacement Workload Control Center 4. Optimization Expert (OE) Includes all OSC functions Index, Query and Access Path Advisors 1 Retrofitted to DB2 for z/os, V8 7

8 Now - Eclipse-based Query Tuning offerings Data Studio InfoSphere Optim Query Tuner for z/os Optim Query Workload Tuner for z/os Queries from all sources Reports Query Formatter Access Plan Graph Query Statistics Advisor Query Annotation Visual Plan Hint Query Index Advisor Query Advisor Access Path Advisor Workload Statistics Advisor Workload Index Advisor Workload Query Advisor 8

9 Data Studio Available in two packaging options Stand-alone Package Integrated Development Environment (IDE) 9

10 Data Studio - Stand-alone Package Lighter weight offering Ideal for administrators to get up and running quickly and easily. Sufficient for most users who do standard database administration or database routine development. Does not include SQLJ, Data Web Services, or XML development capability 10

11 Data Studio - Integrated Development Environment (IDE) Includes all admin capabilities and an integrated Eclipse development environment for Java, XML (for DB2 for Linux, UNIX, and Windows), and Web services development. You can install this package within the same Eclipse instance (shell share) with compatible Eclipse-based products such as: InfoSphere Data Architect Optim Development Studio Optim Database Administrator Optim Query Workload Tuner Provides seamless movement between data-centric roles, or to share objects across geographically distributed teams. 11

12 Data Studio - Uses Design data access queries and routines Take advantage of analysis of current data distribution. Use entityrelationship (ER) diagramming support to visualize designs. Create, edit, run and tune SQL queries. Facilitate development of efficient queries using enhanced Access Plan Visualization Ease tuning and debugging of complex queries using a handy query formatter. Build, test, optimize and deploy SQL and Java stored procedures with the interactive routine debugger. Leverage drag and drop query builders and comprehensive SQL and XQuery editors with interactive feedback for DB2 and IDS. Provides DB2 PL/SQL procedure tooling support for DB2 9.7 for Linux, UNIX and Windows in Oracle compatibility mode. 12

13 Data Studio - Uses Deliver data directly into new or existing Service Oriented Architecture (SOA) solutions. Create Web services from vetted database assets by dragging and dropping existing SQL statements or stored procedures. In situations where guaranteed delivery is a must, install Data Studio with Optim Development Studio to support Java Message Service (JMS) and deployment to WebSphere DataPower XML Integration Appliance XI50 appliances. Develop SQLJ applications for DB2 in a Java project. SQLJ is a way to provide static support for Java applications that access DB2 databases. To take advantage of the purequery Java API across heterogeneous databases or the ability to leverage static SQL for existing Java applications, even those created using Hibernate or other Java frameworks, install Data Studio with Optim Development Studio and Optim purequery Runtime. 13

14 Data Studio - Uses Manage objects and privileges. Connect to DB2 or IDS databases and create, alter, and drop objects using editors and wizards. Export and import database connections. Modify privileges of users, groups, and roles on data objects such as tables, triggers, and more. For DB2 Linux, UNIX, and Windows, Data Studio includes additional capabilities to facilitate the transition from DB2 Control Center (deprecated in Version 9.7): Data Studio Health Monitor Data Studio Health Monitor provides health and availability monitoring support for DB2 for Linux, UNIX, and Windows databases. This solution is focused on the health of the database and is ideal for development and test systems. Comprehensive monitoring available with Optim Performance Manager Extended Edition. 14

15 Expert-Enabled Query Tuning Out of the Box Identify query candidates from numerous sources DB2 catalog Dynamic statement cache Development Studio hot spots Query or performance monitors New! GUI Facilitate analysis with new GUI! Query formatting Query annotation Access path visualization and annotation 15

16 Analyze Access Plans Visualize access path See flow of query processing See indexes and operations See optimizer rationale Assess access path stability to reduce risk of performance regression Is the optimizer able to apply the filtering early? Are there indexes that support an efficient path? Do statistics allow distinction between the choices? 16

17 Visualize Queries and Costs to Speed Analysis Easily see tables, sections, join predicates, etc. Examine table statistics and additional information Accelerate analysis, reduce downtime Spot human errors Identify where filtering should occur 17

18 Improve Statistics Quality and Collection Generates RUNSTATS control statements Provides advice on Missing statistics Conflicting statistics Out-of-date statistics New! Repair and Complete Control Statements Indicates conflicting and missing statistics Results Accurate estimated costs Better query performance Less CPU consumption Improved maintenance window throughput Half of access path PMRs could be resolved by statistics advisor before calling IBM support. IBM Support 18 Conflicting statistics explanation

19 Single Query and Workload Offerings Development Environment Production Environment InfoSphere Optim Query Tuner Tune SQL pre-production while costs and impact are low Optim Query Workload Tuner Optimize workload for peak performance 19

20 IBM Optim Query Tuner Offerings Optimize performance and reduce costs Optimize performance Improve customer satisfaction Improve organizational productivity Satisfy service level objectives Fit workloads into maintenance windows Reduce costs Increase overall system capacity Reduce charge backs Simplify analysis for DBAs, developers, designers, and others Improve statistics collection efficiency Reduce time to respond 20

21 Expert-Enabled Query Tuning Out of the Box Identify query candidates from numerous sources DB2 catalog Dynamic statement cache Development Studio hot spots Query or performance monitors New! GUI Facilitate analysis with new GUI! Query formatting Query annotation Access path visualization and annotation Get expert tuning advice Improve query design Improve statistics quality Improve database design 21

22 Streamlined Analysis with workflow assistant Define or select a workload Execute Advisors New! Workflow assistant Drill Down into advice Validate improvement 22

23 Gather High Cost Queries and Workloads DB2 Plan table Statement table Catalog plan or package Statement cache Define or select workload Optim Development Studio QMF and QMF HPO DB2 Query Monitor OMEGAMON XE for DB2 File, text, or exported workload 23 User defined category

24 Visualize Queries and Costs to Speed Analysis Easily see tables, sections, join predicates, etc. Examine table statistics and additional information Accelerate analysis, reduce downtime Spot human errors Identify where filtering should occur 24

25 View Optimizer Transformations Accelerate analysis, reduce downtime Spot transformations which will occur 25

26 Execute Advisors Statistics Get recommendations on the best statistics to capture to influence access path selection Query Get recommendations regarding how to rewrite the query for better efficiency Index Get recommendations on additional indexes that can reduce database scans 26

27 Analyze Access Plans Visualize access path See flow of query processing See indexes and operations See optimizer rationale Assess access path stability to reduce risk of performance regression Is the optimizer able to apply the filtering early? Are there indexes that support an efficient path? Do statistics allow distinction between the choices? 27

28 Analyze Access Plans (cont.) Interrogate the plan table to discover access plan flags 28

29 Improve Statistics Quality and Collection Generates RUNSTATS control statements Provides advice on Missing statistics Conflicting statistics Out-of-date statistics New! Repair and Complete Control Statements Indicates conflicting and missing statistics Results Accurate estimated costs Better query performance Less CPU consumption Improved maintenance window throughput Half of access path PMRs could be resolved by statistics advisor before calling IBM support. IBM Support 29 Conflicting statistics explanation

30 Improve Query Design View analysis summary Guard against errors and oversights: Further constrain query, increase index utilization, and reduce data reads Highlights relevant components of the query Filter recommendations by severity Query Advisor checks for Missing join predicate for referential constraint Predicates that can be rewritten as indexable Stage 2 predicates that can be rewritten as stage 1 predicates Recommendation and rationale 30

31 Indexing Advice to Improve Query Efficiency Improve query efficiency Indexing foreign keys in queries that do not have indexes defined Identifying index filtering and screening Support for index only access Indexing to avoid sorts Estimated performance improvement Simplify use Consolidate indexes and provide a single recommendation Enables what-if analysis Provides DDL to create indexes Run immediately or save Test before deployment Utilize virtual index capabilities built into the DB2 engine 31

32 New! Modify existing indexes Index elimination 32

33 Validate Improvement Capture workloads repeatedly to see tuning effects Review differences tree Visualize changes in access plan 33

34 Workload Tuning vs. Query Tuning Optimization decisions are based on tradeoffs Statistics CPU costs vs. query savings Indexing query speed vs resource and transaction Workload tuning speeds up analysis Analyzes multiple queries at once Workload tuning consolidates and optimizes recommendation for overall workload Statistics recommendations Index recommendations Speed up analysis, optimize design, 34 and balance resource usage I was able to test a small workload in QWT. 213 dynamic statements were run and captured from cache. Before tuning the workload was 2.49 cpu seconds and elapsed seconds. After the advisors the workload was 0.42 cpu seconds and 6.33 elapsed seconds. Renee Walton, UPS

35 Environment Capture Facilitates Collaboration Enable environment reproduction Speed up service process 35

36 Foster cross-role collaboration Challenge Disconnect between application and data group Reports available via HTML Solution- verbose HTML file Overview Recommended actions Input sql statement Access Plan Summary Transformed/optimized query Predicates Table access Join operation Table and index catalog information: table, column and index 36

37 Reduce Risk of Performance Regression After Migration Compare access path cost of SQL statements in a package via HTML Version to version migrations of DB2 Cases where a mass-rebind is needed Optimize beyond the prior level of service. Determine whether the later version of the collection has degraded performance. Determine whether any packages have errors. Identify which packages have SQL statements that have degraded performance. Obtain Access Plan information Run Access Plan Cost comparison report Filter Criteria 37

38 Prevent Problems Before They Impact the Business Identify Problematic Package Tables View analysis summary based on TOTAL_COST Identify Problematic SQL statement SQL statement with the highest % cost Tune a single query or at a workload level 38

39 Query Tuner Architecture InfoSphere Optim Query Tuner Client Query Tools Formatting Annotating Access plan graph Visual plan hints Reporting Environment capture Query Advice Statistics Query Index Access Path Workload Advice Statistics Index Workload control center Windows Restricted use DB2 Connect Catalog tables QT tables Profile tables DB2 stored procedures for use with InfoSphere Optim Query Workload Tuner products DB2 for z/os, V8 or later with DDF enabled 39

40 Complimentary Products Information Management

41 Streamline Monitor and Respond Scenarios DB2 Query Monitor InfoSphere Optim Query Workload Tuner OMEGAMON XE 41

42 Optim Performance Manager 4.1 New! Integration Optim Query Workload Tuner 42

43 Optim Solutions for managing performance Overall Performance Management Includes both databases and applications Java,.Net, and DB2 Call Level Interface (CLI) applications ** Integrating application insight with Query Tuning Guided Problem Solving Approach Identify, diagnose, solve and prevent problems Integration across Optim family of products and Rational Proactively preventing problems Foster developer and DBA collaboration Leverage best practices for new development Give developers new tools that work with their preferred environment Tune queries during development Supporting Product Offerings Optim purequery Runtime 2.2 Optim Query Workload Tuner 2.2 Optim Development Studio

44 Preventing the problem: Proactively closing the loop Prevent Monitor Improve usage of database Application Developers DBA Improve database performance Execute application 44

45 Prevent Identify Information Management InfoSphere Optim Solutions for Managing Performance Product Roadmap Tivoli OMEGAMON XE & Optim Perf Mgr alerts and overview dashboards Identify Database Problem Prevent Problem Prevent problems by managing workloads and using historical analysis available with OPM WLM & Tivoli OMEGAMON XE Optim Perf Mgr Extended Edition, Tivoli ITCAM & Tivoli OMEGAMON XE Identify Response Time Problem Diagnose Solve O/S dashboard with Tivoli Integration Solve by tuning O/S Optim Query Workload Tuner and Optim purequery Runtime Diagnose Problem Solve by tuning SQL Diagnose with OPM dashboards, DB2 SQL Performance Analyzer and DB2 Query Monitor Optim DB Administrator Solve by modifying DB 45

46 Minimize Production Slowdowns, Tune in Development Visualize query hot spots during development Most frequently executed Longest running Get expert advice View formatted queries with statistical annotations Improve query structure Improve statistics quality and collection efficiency Get index recommendations Find hot spots Get expert advice Advisor overview Compare results Recommendations Validate improvement Replace SQL Run application Compare performance results 46

47 Resources Integrated Data Management Community IBM Optim Query Workload Tuner web page SQL Tuning, Not just for hardcore DBAs anymore DB2 9 for z/os Performance Topics SG IBM DB2 9 for z/os: New Tools for Query Optimization SG Learn how to monitor and tune queries and workloads to improve application performance tuner1/index.html 47

48 Questions? Information Management

49 49

50 Version to Version Migration Saved Challenge Performance regression discovered across range of QMF workloads Client had been using a trick to affect optimizer choices that was not available in the new version Expected 6-8 months delay plus additional staff costs Delay in promised value of new database version Solution Optim Query Workload Tuner Handled complexity of queries and workloads 30 table joins nested behind several layers of views Column names changed in views 100s of queries in multiple workloads Saved hours of time formatting, annotating, and analyzing queries Recommended additional statistics and predicates for better performance Reduced migration from months to weeks Order of magnitude reduction in query workload analysis. All workloads as good as or better than prior version. 50

51 Validation Against a Production System Accelerate development Develop, test, and run queries, procedures, and functions. Enhance productivity Analyze impact of changes Enhance problem isolation Be proactive Leverage best practices for new development Test before deployment Adapt ahead of impact Minimize production slowdowns Clone project Perform analysis Change database connection 51

52 Query Tuner Client Eclipse Query Based Tools and Advisors Workload Advisors Query Tools Query Advisors Query Environment Formatter Capture Query Environment Annotation Capture Query Environment Capture Query Advisor Index Advisor Workload Report Workload Environment Capture Query Access Plan Environment Graph Capture Query Visual Plan Environment Hint Capture Query Query Environment Report Capture Statistics Advisor Access Path Analyzer Workload Statistics Advisor Workload Index Advisor Workload Query Advisor DB2 for z/os 52

53 Query Tuner Client Eclipse Query Based Tools and Advisors Query Tools Query Advisors Query Environment Formatter Capture Query Environment Annotation Capture Query Environment Capture Query Advisor Index Advisor Query Access Plan Environment Graph Capture Query Visual Plan Environment Hint Capture Query Query Environment Report Capture Statistics Advisor Access Path Analyzer DB2 for z/os 53

54 Query Tuning Tools for z/os Positioning Optim Query Workload Tuner DB2 SQL Performance Analyzer Better support for modern dev environment and dynamic SQL Better support for classic z/os, DBRMs, batch analysis, and z/os libraries Eclipse-based, GUI and stored procedure-driven ISPF-based, Command and JCL- driven Integrates with OM/PE and QM GUI interfaces, Data Studio, Optim Integrates with OM/PE VTAM and QM 3270 interfaces Provides virtual index capability for single queries and across workloads Creates real indexes for WHAT-IF scenarios for single queries Recommends rewriting queries only when improvements result Offers a best practices Query Advisor with approx. 150 rules Has limited support to create statistics in Query Environment Capture and Workload Environment Capture service support Has full support for cloning statistics Strategic investment spans databases 54 Continued z/os-specific investment

55 Query Monitoring Tools Positioning OMEGAMON XE for DB2 PE on z/os System level monitor Threads Memory Buffer Pools SQL Statement Summary Alerts and Exceptions Real time, near & long term analysis Tivoli family integration z/os CICS IMS User interfaces 3270 Browser Windows Client Batch DB2 Query Monitor SQL Statement Monitor Execution counts & times CPU Time Getpages & I/O times SQL Codes Hostvars Alerts and Exceptions Real time, near & long term analysis DB2 Tools Integration OMEGAMON for DB2 SQL PA VE / OSC / QWT User interfaces 3270 Browser Windows Client Batch Query Workload Tuner Workload Capture Execution counts & times CPU Time Getpages Capture SQL for tuning Eclipse Integration Optim Development Studio InfoSphere Data Architect Rational Application Dev. Rational Software Architect DB2 Tools Integration OMEGAMON for DB2 Query Monitor Windows and Linux Eclipse 55