Building and Managing Efficient data access to DB2. Vijay Bommireddipalli, Solutions Architect, Optim

Size: px

Start display at page:

Download "Building and Managing Efficient data access to DB2. Vijay Bommireddipalli, Solutions Architect, Optim"

Carol Nicholson
6 years ago
Views:

1 Building and Managing Efficient data access to DB2 Vijay Bommireddipalli, Solutions Architect, Optim September 16, 2010

2 Information Management Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE IBM Corporation

3 Information Management Application on-ramps to DB2 Plain JDBC Data Access Object purequery JPA for WebSphere, Apache OpenJPA Hibernate, ibatis, EclipseLink,... smash PHP, Ruby.Net applications JDBC API DAO API JPA API Open Source Persistence API Web API.Net Applications JPA Persistence Engine Open Source Persistence Engine JDBC JCC driver ADO.Net DB IBM Corporation

4 Layered Application Stack Challenges Simplify development, but Challenge problem resolution Impact performance Obscure impact analysis Impede capacity planning Layers obscure linkages Where is the problem? Which application is running? Which application does this SQL belong to? Application Server Database Server Business Logic Connection Pool EJB Query Language Data Access Logic Persistence Layer DB Java Driver Network A A A 4

5 Layered Application Stack Challenges Simplify development, but Challenge problem resolution Impact performance Obscure impact analysis Impede capacity planning Layers obscure linkages Where is the problem? Which application is running? Which application does this SQL belong to? Application Server Database Server Business Logic Application Developer Connection Pool EJB Query Language Systems Administrator Data Access Logic Persistence Layer DB Java Driver Network Administrator Network Database Administrator A A A 5

6 JSP XML mashup JDBC Spring http SQL JSON IBM Information Management Data Access Two Views of the World QoS goals Runstats Partition Stored strategy Procedures Response REORG Time! Developer DBA No insight into data access performance Databases are a different skills domain DBAs have little to no visibility of application SQL before deployment, no opportunity for review and optimization Not a productive development environment Problem isolation takes days due to inability to trace SQL to Java application and source code No single environment that spans Java application and database development Constantly increasing application workload taxes existing systems need to fit more work into existing systems Problem isolation and resolution SQL injection represents an increasing risk to data security Resource utilization across application and database infrastructure 6

7 Database access development options - Client Applications (business logic) Call Individual SQL statements Application Version 1 Uses individual SQL getcustomers() getcustomersbyzip() getaccountdetails() getaccountreport() Application Version 2 Uses SP Calls getcustomers() getcustomersbyzip() getaccountdetails() getaccountreport() Call Stored Procedures (SQL SPs, Java SPs), Can use individual Statements Using Dynamic or static SQL Db access Using Stored Procs Call can be Dynamic or Static SQL Static SQL Packages Stored Procedures Can be SQL bodied or Java Stored Procedures 7

8 Comparing programming models A. Application SQL only Dynamic SQL B. SQL Stored Procedure Dynamic CALL, Static Procedure Application SELECT C1 FROM T1 DB CALL P1 Application DB SELECT C1 FROM T1 C. Application SQL only Static SQL D. SQL Stored Procedure Static CALL, Static Procedure Application SELECT C1 FROM T1 DB SELECT C1 FROM T1 CALL P1 Application DB SELECT C1 FROM T1 CALL P1 8

9 Information Management IBM Data Studio SQL Execution - Dynamic vs. Static Dynamic SQL Check auth for package / plan Static SQL Check auth for package / plan Parse SQL Statement Check Table / View Auth Calculate access path Extract access path from catalog or package cache Store access path in a temporary package Execute SQL statement Execute SQL statements IBM Corporation 9

10 Information Management IBM Data Studio SQL Execution Advantages Dynamic vs. Static Feature Performance Access path reliability Authorization Monitoring, Problem determination Capacity planning, Forecasting Tracking dependent objects Dynamic SQL (purequery, JDBC) Can approach static SQL performance with help from dynamic SQL cache. Cache misses are costly Unpredictable Any prepare can get a new access path as statistics or host variables change Privileges handled at object level. All users or groups must have direct table privileges Security exposure, and administrative burden Database View is of the JDBC or CLI package No easy distinction of where any SQL statement came from. Difficult to summarize performance data at program level. No record of which objects are referenced by a compiled SQL statement Static SQL (purequery, SQLJ) All SQL parsing, catalog access, done at BIND time. Fully optimized during execution. Guaranteed locked in at BIND time All SQL available ahead of time for analysis by EXPLAIN. Privileges are package based. Only administrator needs table access. Users/Groups have execute authority. Prevent non-authorized SQL execution. Package View of applications makes it simple to track back to the SQL statement location in the application Package Level Accounting gives program view of workload to aid accurate forecasting. Object dependencies registered in database catalog IBM Corporation

11 Have You Heard of SQL Injection? 11

12 Increased throughput as compared to JDBC with 90% Package Cache hit ratio 90% Package Cache hit ratio: DB2 LUW Database ITR comparison to JDBC 120% ITR: % difference from JDBC 100% 80% 60% 40% 20% 97% 115% 0% purequery Client Optimization Static purequery API Static (Method Style) In comparison with JDBC, Static APIs doubled the throughput (ITR*) of the DB2 LUW database server. 12

13 Increased throughput as compared to JDBC with 95% Package Cache hit ratio 95% Package Cache hit ratio: DB2 LUW Database ITR comparison with JDBC 100% ITR: % difference from JDBC 80% 60% 40% 20% 60% 66% 0% purequery Client Optimization Static purequery API Static (Method Style) Compared to JDBC, Static APIs increased the throughput (ITR*) of the DB2 LUW database server by 60 and 66% respectively 13

14 Information Management IBM Data Studio Performance Using IRWW SQLJ and JDBC workloads, CL.1 CPU drops by 19.22%, while CL.2 CPU drops by 20.51%. Throughput increases by 46% SQLJ and JDBC - dynamic vs. static SQL Microseconds Dynamic CL1 Static CL1 Dynamic CL2 Static CL2 0 Dynamic CL1 Static CL1 Dynamic CL2 Static CL IBM Corporation

15 Information Management IBM Data Studio Static CALL to native SQL SP vs. dynamic CALL Results show an 11% reduction in CL.2 CPU time, and 9% reduction in CL.1 CPU time. Elapsed times dropped by roughly 5% IBM Corporation

16 Stored Procedures and DB2 SQL DB2 for LUW Version 8.2, 9.1, 9.5, 9.7 DB2 for iseries V5R3, V5R4 SQL External DB2 for z/os Version 8, 9 SQL Native DB2 for z/os Version 9 Java (JDBC or SQLJ or purequery) DB2 for LUW Version 8.2, 9.1, 9.5, 9.7 DB2 for iseries V5R3, V5R4 DB2 for z/os Version 8, 9 PL/SQL DB2 for LUW Version 9.7 Oracle V10, V11 16

17 Stored procedures in DB2 9 for z/os - SQL External SQL procedures: Pre- DB2 9 SQL stored procedures SQL processor converts SQL statements into a C program Build utility DSNTPSMP, customizable Runs in a WLM environment Debug with Unified Debugger Native SQL procedures: SQL procedure logic runs in the DBM1 address space Extensive support for versioning Native SQL procedures support ziip Processors when invoked over DDF Debug with Unified Debugger 17

18 SQL Execution mode in Java Stored Procedures Dynamic: Uses purequery or JDBC APIs Static: Uses purequery or SQLJ Debug both with Unified Debugger 18

19 Comparing programming models B & D Use SPs when several SQL statements work together with minimal flow logic A. Application SQL only Dynamic SQL B. SQL Stored Procedure Dynamic CALL, Static Procedure Application SELECT C1 FROM T1 DB CALL P1 Application DB SELECT C1 FROM T1 C. Application SQL only Static SQL High QoS D. SQL Stored Procedure Static CALL, Static Procedure High QoS Application SELECT C1 FROM T1 DB SELECT C1 FROM T1 CALL P1 Application DB SELECT C1 FROM T1 CALL P1 19 C & D Higher performance, access path reliability, improved authorization, monitoring, problem determination, capacity planning, and dependent object tracking.

20 High cost of literals IBM Information Management Review SQL coming from applications Consolidate SQL by replacing literals with parameter markers Optimize database resources (e.g. dynamic statement cache) usage 20

21 Using JDBC batching Users Network Table 1, operation 1 Table 1, operation 2 Table 1, operation 3 Application Servers Table 1, operation 4 Table 1, operation 5 Table 1, operation 6 DBMS Batching Multiple operations all execute as one batch minimize chattiness on the network pstmt.setstring(2,"accounting"); pstmt.setstring(3,"a00"); pstmt.addbatch(); int [] numupdates=prepstmt.executebatch(); 21

22 SQLJ using Heterogenous batching Users Network Table 1, operation 1 Table 1, operation 2 Table 1, operation 3 Application Servers Table 2, operation 1 Table 2, operation 2 Table 1, operation 4 Table 3, operation 1 DBMS Heterogenous Batching Multiple operations across different tables all execute as one batch 22

23 Java Data Access Spectrum balancing productivity and control Full SQL control Object-relational mapping Managed objects Code all your SQL JDBC / SQLJ Spring templates ibatis pq DAO Hibernate OpenJPA (EJB3) Use SQL templates, inline only Add basic OR mapping and annotated-method style Complex OR mapping and persistence management, but loss of controls Adds container management option 23

24 SQLJ SQLJ #sql [con] {SELECT firstnme INTO :empname : FROM employee WHERE empno = :empno : empno}; Non-Java syntax, but optimal access Tedious and long code JDBC Sring sqlstatement = SELECT firstnme FROM employee WHERE empno =? ;? java.sql.preparedstatement ps = con.preparestatement(sqlstatement); ps.setstring(1, empno); java.sql.resultset reader = ps.executequery(); reader.next(); empname = reader.getstring(1); Reader.close(); SQLJ Pros Concise compared to JDBC Full control of SQL Most optimal access to the database (static) Heterogenous batching (reduce network traffic) Better diagnostics SQLJ Cons Not a popular common programming model Many IDEs do not have SQLJ support Deployment has additional steps Cannot be used with other programming models 24 IBM Data Studio purequery For DBAs and Application Developers (v2.1) - Part 1 24

25 Data Access Objects core J2EE pattern Using J2EE Data Access Objects Methods for each database access Each method has only your parameters SQL can be in separate file separate business logic from data access 25

26 What is Hibernate IBM Information Management Open source Object-Relational Mapping (ORM) library for Java Simple object oriented database access Application Hibernate JNDI JDBC JTA No complex JDBC coding Data access though object query language and SQL Database 26

27 Hibernate Application Developers code as above Hibernate Hibernate generates the following SQL JNDI JDBC JTA Database 27

28 Goals for performance management Prevent problems before they impact the business Get early warning of emergent problems Isolate problems quickly Improve quality of service Use expert advice for faster problem resolution Stabilize application performance Reduce costs Increase capacity of existing systems Free up DBA time for value creation activities Fix problems in development while costs and impact are low Propagate performance expertise to avoid problems 28

29 IBM Information Management Identify, diagnose, solve, and prevent performance problems Alert of potential problems Visual quick scan of complex environment Drill-down into problem detail and related context Analyze captured SQL Network 1. Identify 2. Diagnose Users Application Servers 4. Prevent DBMS & OS Use best practices to avoid problems in development 3. Solve Use proven performance improvement techniques 29

30 More Visibility, Productivity, and Control of Application SQL Capture Capture the SQL during development Monitor Java Persistence Architecture Capture Optimize Review and share SQL Optimize and replace bad SQL Create approved SQL list Review Optimize Restrict Deploy Configure execution properties Optionally bind for static execution Configure Bind Execute 30

31 Simplifying Problem Determination Scenario Application Developer Available for each db access SQL text generated Access path Cost estimates Estimated response time Database Administrator Available for each SQL Application name Java class name Java method name Java object name 31

32 Facilitate Problem Determination Diagnose Which application has issued the problematic SQL? Use database package information to correlate SQL with Java or.net applications 32

33 Unique Package Names Improves PD Hard to identify unique Java applications Diagnose Most dynamic Java applications use packages SYSLNx00 Unique package names link SQL to Java code modules Static Java SQL Dynamic Java SQL 33

34 Statically invoking Stored Procedures from Java Diagnose Native SQL stored procedure invoked Dynamically from Java application Native SQL stored procedure invoked Statically from Java application 34

35 Client Info. fields Diagnose Voluntary but very useful! java.sql.connection.setclientinfo - setclientinfo("applicationname","orderhistoryapp"); conn.setclientinfo("clientuser", Vijay Bommireddipalli"); conn.setclientinfo("clienthostname, xyz"); 35

36 Tip: Adding Java clientprogramname will set Correlation Id. Diagnose 36

37 Solve using any or all of the following - Solve Stabilize performance and lock access plans using Static SQL Improve performance using JDBC batching or even better Heterogeneous batching capability Improve performance by substituting literals with parameter markers 37

38 Preventing the problem: Proactively closing the loop Prevent Monitor Improve usage of database Application Developers DBA Improve database performance Execute application 38

39 Building high performance applications proactively Prevent Take advantage of optimized database access Static SQL Batching (JDBC or heterogeneous) Avoid literals where possible Avoid unnecessary re-prepares Stored Procedures Add diagnostics where possible Eg. Client info fields Side effects of static SQL Templates Share templates of frequently used SQL statements Review process for SQL Explain analysis Tune early, Tune often 39

40 Resources Information Management Optim solutions for accelerating Java database access IBM Optim Development Studio IBM Optim purequery Runtime web page Articles and tutorials covering the Optim portfolio IBM Corporation

41 Information Management IBM Corporation

purequery Deep Dive Part 2: Data Access Development Dan Galvin Galvin Consulting, Inc.

purequery Deep Dive Part 2: Data Access Development Dan Galvin Galvin Consulting, Inc. Agenda The Problem Data Access in Java What is purequery? How Could purequery Help within My Data Access Architecture?