Best Practices for Boosting Java Application Performance and Availability on IBM DB2

Transcription

1 Best Practices for Boosting Java Application Performance and Availability on IBM DB2 Pallavi Priyadarshini Architect, JCC DB2 Connect, IBM

2 Agenda DB2 JDBC driver architecture and ecosystem API details Best Practices (for Type 2 and Type 4) Connection Management SQL Execution DB2 Dynamic Statement Cache Batching and Multi-row operations Miscellaneous performance tips

3 3 Driver Background, Architecture and Integration

4 JDBC 4 Java Database Connectivity API for the Java programming language that defines how a client may access a database Provides methods for querying and updating data in a database JDBC is oriented towards relational databases Portability of applications across databases and platforms The API is defined in the classes and interfaces that make up the Java packages: Java package java.sql JDBC core API Java package javax.sql - extension providing DataSource interface Connection pooling Distributed transactions Rowsets

5 5 Universal driver - JCC (Java Common Client) Abstract JDBC processor common architecture independent of platform, connectivity type or JDBC/SQLJ access Cross-platform Provides java connectivity to all DB2 servers (Linux, Unix, and Windows, z/os, iseries), Informix Different connectivity paths (local and remote) Type 4 distributed Pure Java standalone remote client (DRDA) Type 2 local - Fastest path when executing on same machine as DB2 engine. Avoids costly construction/parsing of DRDA command stream Type 2 zos Native RRSAF-based; Java and native C Type 2 LUW Native CCC based; Java and native C One driver is loaded and type 2 and type 4 connections can be obtained (type 2 requires proper native libraries) Supports JDBC 3 under JRE 1.4 (db2jcc.jar), and JDBC 4 under JRE 1.6 (db2jcc4.jar) Provides behavioral consistency, API consistency and code reuse between connectivity types and platforms (minimize divergence)

6 6 Universal Driver Architecture SQLJ Application SQLJ runtime JDBC Application Abstract machine (am)/requestor Java common layer T4 subsystem builds/parses DRDA requests/replies T2z subsystem calls into native z/os APIs T2u subsystem calls native LUW APIs CCC DRDA over TCP/IP RRS DB2 for z/os local UDB DB2, Informix

7 7 Driver Technologies Full function JDBC driver, much more than a JDBC driver High availability Scalability Performance Security Monitoring Robust Application Functionality New Data Types Workload balancing, Client Re-route (failover) Extensive multi-threading Connection reuse/efficient socket mgmt, DRDA Chaining/Batching,Defer Prepares, Statement Caching Encryption/authentication (SSL, Kerberos, DB2 security) Extended diagnostics, API for system monitoring Internationalization, data conversion, robust result set, Parm metadata, multiple connection options, Distributed Transactions BIGINT, TIMESTAMP, XML/BLOB/CLOB with multiple streaming options

8 8 Integration Ships with BM data servers. They drive most requirements DB2 on z/os. Includes both T4 and T2 zos drivers DB2 on LUW. Includes both T4 and T2 UW drivers DB2 on System I Informix connectivity (Type 4) Integration with IBM products Websphere products are primary consumer and drive a lot of requirements Many other internal groups like Cognos, Tivoli, Sterling Commerce, Content Manager DB2 tooling like RAD Optim tools portfolio like Data Studio/OPM/OCM/pureQuery/OQWT Certification/Testing by non-ibm products SAP, Weblogic, JBOSS High profile customers Used by top banks, retailers, insurance companies

9 9 Connection Management

10 Client configurations Distributed Connectivity 10 Based on DRDA - Distributed Relational Database Architecture supports direct connectivity to DRDA servers over TCPIP eliminates intermediate servers required by other driver types JDBC/SQLJ/ purequery/ Java based Clients Direct Connect (Type 4) DB2 z/os DB2 Connect Client CLI/ Ruby/ Perl.NET Non-Java based Clients DRDA DRDA DB2 Connect Server Direct Connect DRDA DB2 z/os CF DB2 Connect Clients no longer require DB2 Connect Server DB2 Group DB2 Connect Clients are Sysplex enabled with improved availability

11 11 Getting a connection - DriverManager Driver name com.ibm.db2.jcc.db2driver JDBC 1 java.sql.drivermanager API The actual driver type determined during runtime from the connection URL format: Type 2 - jdbc:db2:database Type 4 - jdbc:db2://host:port/database (default port number 446) Class.forName( com.ibm.db2.jcc.db2driver ); Connection con = DriverManager.getConnection( jdbc:db2://localhost:50000/sample, username, password ); Disadvantage - Reduces portability because the application must identify a specific JDBC driver class name and driver URL.

12 12 Getting a connection - DataSource JDBC 2 - javax.sql.datasource API Preferred approach Increases application portability by using a logical name Logical name mapped to DataSource object via JNDI naming service logical name -> driver info, DB name, IP, port, user, password, etc. Application servers can be used to configure and manage data sources Properties of DataSource object modified if any DataSource information changes. Eg. if data source moved to different server, property for the server can be changed. No change to application code necessary. Connection obtained through a DataSource object is identical to a connection obtained through the DriverManager facility. DataSource also has more flexibility in specifying Properties

13 13 Getting a connection (DataSource examples) Non-WAS datasource example DB2SimpleDataSource ds = new DB2SimpleDataSource(); ds.setservername(xxx); ds.setportnumber(xx); ds.setdatabasename(xxx);... Connection con=ds.getconnection(); WAS datasource example - Application requests DataSource object by logical name Context ctx=new InitialContext(); DataSource ds=(datasource)ctx.lookup("jdbc/sampledb"); Connection con=ds.getconnection();

14 14 Properties JDBC API defines a set of properties to identify and describe a DataSource implementation. May be specified in any of three ways JDBC 1 connection properties passed as a java.util.properties object (consisting of key/value pairs) as argument to DB2Driver.connect() or java.sql.drivermanager.getconnection() As part of the database URL itself jdbc:db2://server[:port]/databasename[:propertykey=value;...] jdbc:db2://localhost:50000/testdb:user=foo;password=bar; Using setxxx methods most properties are defined in the abstract com.ibm.db2.jcc.db2basedatasource class Examples of properties - ProgressiveStreaming, deferprepares, logintimeout, keepdynamic, enablesysplexwlb, currentexplainmode, cursorsensitivity, maxtransportobjects, tracefile, tracelevel, currentschema, currentsqlid, dumppool Driver configuration tuning via external properties (not requiring application or DataSource modification) is available

15 15 Global Properties Global driver properties can be provided through Java system properties or a properties file. If both are set, Java system properties will override any property setting from the properties file. File name of the properties file can be provided through the system property "db2.jcc.propertiesfile". If that property is not set, first "DB2JccConfiguration.properties" and then "com/ibm/db2/jcc/db2jccconfiguration.properties" will be the assumed file name. The global property names must be prefixed with one of the following: db2.jcc. (eg. db2.jcc.tracefile) db2.jcc.default. (eg. db2.jcc.default.tracefile) db2.jcc.override. (eg. db2.jcc.override.tracefile) The properties prefixed with db2.jcc. or db2.jcc.default. will be only used if and only if they are not set on connection or datasource level. To override the properties set on connection or datasource level anyway, use another set of global driver properties prefixed with db2.jcc.override.

16 16 Getting a Connection All DB resources hang off Connection object, must be managed carefully Expensive operation - Average connection object is 1-2M in size and contains a great deal of information on connection context Creating and terminating connections is time consuming operation both in the driver and DB2 and can slow down applications Client obtaining a physical database connection requires multiple network requests to Handshake on drda protocol Validate client user credentials Establish code page, packet size setc.

17 17 Getting connections efficiently Get a connection object one at a time, as needed Reuse a connection object for multiple Statement objects when possible Close connections promptly, don't leave connection cleanup to garbage collection Add another close in the finally block to cover the case when exception occurs before connection close statement Apply close to other JDBC objects too Statement PreparedStatement CallableStatement ResultSet

18 18 Clean up example public test () throws SQLException { Connection con = null; ResultSet rs = null; Statement stmt = null; try { con = ds.getconnection(); stmt = con.createstatement(); rs = stmt.executequery(... // If an exception occurred here, then the next 3 closes are not executed. rs.close(); // Explicit close in same method stmt.close(); con.close(); }... finally { // Close again in finally block, in case there was an exception before explicit rs.close() try {if (rs!= null) rs.close(); } catch (Exception e) {} try {if (stmt!= null) stmt.close();} catch (Exception e) {} try {if (con!= null) con.close();} catch (Exception e) {} } }

19 19 Connection pooling A Connection Pool is a set of client established physical database connections that can be reused by the application Part of DataSource support Connection object pool maintained by Application Server like Websphere Allows cost of connection requests to be spread out over multiple requests and can significantly improve performance Saves creating/destroying Connection objects which is relatively expensive Without connection pooling each Connection represents physical connection and close terminates the physical connection With connection pooling Connection is a logical representation Cost of creating each physical database connection is incurred only once Subsequent getconnection from the pool is fast Completely transparent to the application

20 20 Connection pooling Application Server Application Resource Adapter JCA Connection Manager DB Connection Pool JVM DB2 Universal Driver JDBC/SQLJ Logical Connection 1 Logical Connection 2 Logical Connection 3 disconnect at commit/rollback Transport 1 Transport 2 pooled connections to DB2 Data Sharing Group CF

21 21 SQL Execution

22 22 Creating a Statement JDBC connections support creating and executing statements Update statements such as CREATE, INSERT, UPDATE and DELETE Returns update count indicating how many statements updated on the database Query statements such as SELECT. Returns JDBC result set which has metadata describing names of columns/types Stored procedures JDBC represents statements using one of the following classes: Statement the statement is sent to the database server each and every time. PreparedStatement the statement is cached and then the execution path is pre determined on the database server allowing it to be executed multiple times in an efficient manner. CallableStatement used for executing stored procedures on the database.

23 23 Prepared Statement Cache or Statement Pool Client side optimization Pool of PreparedStatement and CallableStatement objects not active in a Connection Application requesting PreparedStatement/CallableStatement may get from pool Application closes object returns object to the pool Reduced overhead of Java object delete and re-create Also saves on description cost The statement cache exists for the life of an open connection. When connection is closed, driver deletes the statement cache and closes all pooled statements. Least recently used algorithm if cache is full No impact to application

24 24 Statement or PreparedStatement PreparedStatement over Statement 2 DB calls are needed for fetch one for describing column, and other for data PreparedStmt makes description calls at construction time, Statement makes them on every execution. PrparedStatement enables Statement Pooling Use Statement when SQL is not executed often Close explicitly before closing connections Reuse these objects when possible One Statement object can be used to execute multiple SQLs instead of creating multiple Statement objects

25 25 Statement or PreparedStatement Statement example Statement stmt = con.createstatement(); stmt.executeupdate("insert INTO EMPLOYEE VALUES( 'John', 123 )"); stmt.executeupdate("insert INTO EMPLOYEE VALUES( 'Mary', 425 )"); PreparedStatement example PreparedStatement ps = con.preparestatement( INSERT INTO EMPLOYEE VALUES(?,?) ); ps.setstring(1, John ); ps.setint(2, 123); ps.executeupdate(); ps.setstring(1, Mary ); ps.setint(2, 425); ps.executeupdate();

26 26 DB2 Dynamic Statement Caching

27 DB2 SQL Static vs. Dynamic Access path is the set of instructions that DB2 uses to locate data under SQL DB2 Optimizer component: Generates alternative access paths Calculates cost for each access path Chooses cheapest access path Select * from T where A='1' and B='2' CREATE INDEX IDX_A on T(A) CREATE INDEX IDX_B on T(B) Alternative access paths Path 1: Use no indexes Path 2: Use IDX_A only Path 3: Use IDX_B only Path 4: Use both indexes Access path selection is expensive Static SQL Prepared at bind time, execution is fast Dynamic SQL Prepared at run time Complete sql stmt string unknown before execution Popular as distributed queries from non-mainframe platforms use dynamic sql (eg. JDBC) 27

28 Dynamic Statement Cache 28 Feature to improve performance of dynamic sql Allows applications (multiple threads) to reuse and share prepared statements. Cache pool shared by different threads, plans and packages ( global cache ). Prepared stmt is saved in an in-memory cache Subsequent prepares of same statement loads from cache PREPARE Flavors Full prepare Statement is not found in the cache and must be prepared from scratch Short prepare Statement is found in the cache and can be reused; no need to regenerate access path Good cache hit rate produces significant performance benefits A Full Prepare can be X more CPU than a Short Prepare!

29 29 Cache match criteria Sql stmt text code SQLs with same string SELECT * from T where empid = 1 SELECT * from T where empid = 1 (2 entries in cache) Prepare attributes (if for a cursor select) Bind options - CURRENTDATA, DYNAMICRULES, ISOLATION, SQLRULES, QUALIFIER Authorization id Special registers - CURRENT DEGREE, PATH, SCHEMA, RULES, PRECISION, REFRESH AGE etc. Use parameter markers (?) instead of literals to improve cache hit rate SELECT * from T where empid = 1 SELECT * from T where empid = 2 (2 entries in cache) VS. SELECT * from T where empid =? (one entry in cache)

30 30 EXPLAIN EXPLAIN facility can be used to provide information on access plans Populates various explain tables with details on access path information of statements Use Dynamic SQL Stmt section of statistics to monitor the Global Cache Hit Ratio % to determine if the cache size needs to be increased.

31 31 Literal replacement Introduced for improved cache hit for statements with literals Automatically replace literal with ampersand (&) & treated as a parameter marker (?) Set Connection or DataSource Property statementconcentrator to STATEMENT_CONCENTRATOR_WITH_LITERALS (2) Eg Select * from T where productid = '1' Select * from T where productid = '2' Results in 1 cache entry Select * from T where productid = &

32 32 Why SQLJ? Static SQL performance for Java applications Consistent performance Performance advantage over JDBC Static SQL authorization model Provides Java with a stronger authorization model Monitoring/Manageability Static SQL packages for accounting/monitoring Static SQL locks in access path, so that access path changes do not occur without a conscious choice Measurements with the IRWW workload comparing JDBC vs SQLJ with the T2 driver Throughput (ETR) Normalized Throughput (ITR) z/os CPU Utilization CL.1 CPU time JDBC T SQLJ T (+2.20%) (+37.13%) (-25.47%) (-32.00%)

33 33 Multi-Row and Batch operations

34 34 Batching INSERTs DB2 Execute one Multi-row INSERT instead of many individual INSERTs INSERT INTO EMPLOYEE (EMPID) VALUES (:HV) FOR :NUM-ROWS ROWS - HV is host variable array - NUM_ROWS are # rows to be inserted JDBC API enablemultirowinsertsupport=true (default) Connection or DataSource property Eg. ps = conn.preparestatement( INSERT INTO EMPLOYEE VALUES (?) ) ps.setint(1) ps.addbatch() ps.setint(2) ps.addbatch() Int[] returncodes = ps.executebatch()

35 35 Batching SELECTs Not supported by JDBC spec executebatch with SELECT returns exception DB2 driver extension executedb2querybatch PreparedStatement ps = conn.preparestatement( SELECT * from T1 where C1 =? ) ps.setint(1,1) ps.addbatch() ps.setint(1,2) ps.addbatch() ((com.ibm.db2.jcc.db2preparedstatement)pstmt).executedb2querybatch(); While (ps.getmoreresults()) { Rs = ps.getresultset() While (rs.next()){ } rs.close } ps.close

36 36 Multi-Row Fetch Returns many rows in one SQL (ROWSET) EnableRowsetSupport = 1 for Type 2 or Type 4 Scrollable cursors for Type 4 Scrollable or forward-only cursor for Type 2 ((com.ibm.db2.jcc.db2basedatasource)ds).setenablerowsetsupport(com.ibm.db2.jcc.db2b asedatasource.yes); java.sql.statement s = con.createstatement(resultset.type_scroll_insensitive, ResultSet.CONCUR_READ_ONLY); s.setfetchsize(1000); //Set hint for fetch size number of rows per block String selectstring = "select id, name from T"; ResultSet rs = s.executequery(selectstring); while (rs.next()) { }

37 Block fetch DRDA block fetch for forward only, read only cursor Returns multiple rows in query block reply message Query Block size can be tuned to 256K Large block size improves performance, requires more storage on client to buffer You can expect significant performance improvement for applications with the following queries Queries that return LOBs Queries that return more than 1 row These measures show the following results: Limited block fetch increases throughput by 169% Limited block fetch reduces class 1 CPU time by 63% 37

38 38 Transaction Autocommit default is TRUE Every SQL followed by implicit commit which means disk I/O Set Autocommit to FALSE Code COMMIT at frequent intervals to control when disk I/O happens Example 1 many disk I/Os con.setautocommit(true) Stmt = con.createstatement() stmt.execute( insert... ) //Server issues an implicit COMMIT stmt.execute( insert... ) //Server issues an implicit COMMIT Example 2 con.setautocommit(false) Stmt = con.createstatement() stmt.execute( insert... ) stmt.execute( insert... ) stmt.commit()

39 Common Java performance problem and tuning Java application Mismatch of Java and DB2 data types Use standard mapping between java/jdbc types and SQL types in getxxx and setxxx (eg. java.sql.timestamp to SQL TIMESTAMP) Retrieval of unused columns (select * ) JDBC JDBC resources not closed (cursor, statements, connections). No usage of Parameter Markers - E.g. select c1, c2 FROM t1 WHERE c3=? Usage of Statement() instead of preparedstatement() objects 39 Environment JVM heap too small Old System levels (JDK, JDBC/SQLJ driver) DB2 Statement Cache not tuned well Use stored procedures Avoid - Held cursors are held across commit by default, can cause unwanted contention/issues DGTT s that are not dropped/released Held LOB locators Only prepare what will be executed - No deferred prepares under certain conditions

40 40 Useful links utshell.pdf d2c-ae74-0ddcc2a409bc/entry/december_17_2012_6_07_am3?lang=en

41 41 Thank You