SQLJ Implementation Experiences on DB2 for z/os

Transcription

1 Session: G11 SQLJ Implementation Experiences on DB2 for z/os Julian Stuhler Triton Consulting Wednesday May 10, pm 5.10pm. Platform: DB2 for z/os SQLJ's performance and manageability are making it increasingly popular as a means of accessing DB2 for z/os data from within a Java environment. The introduction of the DB2 Universal Driver brings with it a number of additional capabilities. This presentation provides an overview of SQLJ and the Universal Driver, and presents some practical guidelines and recommendations for implementation in a DB2 for z/os environment. 1

2 Topics Introduction SQLJ Overview Deployment Issues Performance Issues Summary & Conclusions After a brief introduction, we will review some SQLJ basics, contrasting SQLJ with JDBC and looking at the new DB2 Universal Driver. The presentation will then outline some of the deployment issues that the author has encountered, including roles & responsibilities and program preparation pitfalls. Finally, some common performance issues and workarounds will be discussed before the presentation summary. 2

3 Your Presenter DB2 consultant with Triton Consulting, based in the UK 17 years DB2 experience Database Administration Systems Programming Application Development IBM Gold Consultant IDUG Board of Directors Redbook Java Stored Procedures: Learning By Example (SG ) The presenter can be contacted using the details below: Tel: +44 (0) Fax: +44 (0)

4 Topics Introduction SQLJ Overview Deployment Issues Performance Issues Summary & Conclusions This section of the presentation provides an overview of SQLJ, and IBM s support for it on the z/os platform. 4

5 SQLJ Overview JDBC v SQLJ JDBC standard (part of core Java API) provides the mechanism for Java programs to access relational databases JDBC uses dynamic SQL, which is flexible but also poses some challenges Performance Response time consistency Authorisation Monitoring & Accounting (same package for all) Coding The SQLJ standard is intended to address these challenges, by allowing embedded static SQL access to relational data Not mutually exclusive - JDBC and SQLJ can co-exist in same program JDBC is the original database connectivity API for Java applications. It is based upon a dynamic SQL model, which provides great flexibility but also has some disadvantages when compared to SQLJ an alternative method which is able to use static SQL. Despite facilities such as the DB2 for z/os Dynamic Statement Cache, static SQL programs generally outperform their dynamic counterparts due to the fact that many processes (including syntax checking, authorisation checking and access path selection) are performed in advance of run time. Perhaps even more importantly for many DBAs, static SQL tends to have consistent run times, whereas the response time for a dynamic SQL query can vary widely depending on the current database statistics. Static SQL can also offer a more secure authorisation model: a user running a dynamic SQL query needs direct access to the tables the query is accessing, but the same user running a static SQL query only needs execute authority on the plan or package associated with the query. Finally, the number of lines of code required to implement an SQLJ program is typically much less than the JDBC equivalent this can make SQLJ programs both quicker to write and easier to read and maintain,. It should be noted that SQLJ is not always the best solution: there are times when the ability to dynamically construct an SQL statement at execution time can greatly simplify coding. It is also perfectly possible to mix both JDBC and SQLJ within the same program, allowing the developer to select the best API for each set of circumstances. 5

6 SQLJ Overview JDBC Driver Types Type 1 Maps JDBC calls to another API (usually ODBC) No longer used / supported Type 2 High performance Driver written partly in Java, partly in native methods Needs platform-specific client library for DB access For z/os, this means RRSAF access to local DB2 Type 3 100% Java Uses proprietary middleware to translate calls to DB-specific format (aka DB2 Net Driver ) No longer recommended use T4 instead Type 4 100% Java Access DB directly using open DB protocol (e.g. DRDA) This slide summarises the various types of JDBC driver. Type 1 drivers were the original implementation, and were essentially a stop-gap solution that mapped JDBC calls onto other API standards such as ODBC. They perform poorly, and have been obsolete since the various database vendors produced dedicated JDBC drivers. Type 2 drivers are written partly in Java and partly in native methods. They don t know how to talk to a remote database directly - they depend upon the existence of a separate platform-specific client-library to provide connectivity to the remote DB2 system. In a z/os Java environment, this means that the Java program can only communicate with a local DB2 subsystem on the same LPAR, using RRSAF. Like the Type 1 drivers, Type 3 drivers are now obsolete and have been replaced by Type 4 drivers. They are 100% Java drivers that talk to a dedicated JDBC server daemon on the database server, which then communicates with the database. While this avoids the need for any sort of driver installation on the client (the driver can be downloaded in an applet/web browser) there are maintenance and scalability issues with this approach. Type 4 drivers are also 100% Java, but they contain all of the code necessary to directly access the database remotely. In DB2 terms, Type 4 drivers provide full DRDA Application Requester functionality. Although the new DB2 Universal Driver incorporates both Type 2 and Type 4 functionality, for best performance it is important to use the right one for a given requirement: Type 2 for local database access, and Type 4 for remote access (note that Type 2 is the only option for z/os Java programs running under CICS, IMS or as DB2 stored procedures) 6

7 SQLJ Overview IBM Support IBM is currently the leading supporter of the SQLJ standard Fully supported in DB2 for z/os and DB2 for LUW Version 8 of both products introduced new DB2 Universal Driver for JDBC & SQLJ aka DB2 Universal Driver for Java Common Connectivity or just JCC driver Single driver that is independent of: Connectivity type supports both Type 2 (local) and Type 4 (distributed) connectivity (determined by connection string) Platform supports both LUW and z/os SQLJ is an open Java standard, but unfortunately IBM is the only major database vendor to fully support it (Oracle were initially keen, but seem to have recently stepped back from complete support). Previously, each DB2 codebase had its own T2 and T4 drivers, but Version 8 of both DB2 for z/os and DB2 for LUW introduced a new driver, known as the DB2 Universal Driver for JDBC & SQLJ (or DB2 Universal Driver for Java Common Connectivity - JCC). This single driver can be deployed on both z/os and LUW, and supports both Type 2 and Type 4 connectivity. This greatly enhances application portability, eliminating the annoying inconsistencies present in the previous products. 7

8 SQLJ Overview IBM Support High performance Old versions known as legacy drivers T2 connectivity must be used under CICS, IMS or Java Stored Procedure environment The new driver also introduces a number of valuable performance and functionality enhancements, some of which are covered later in this presentation. In order to differentiate them from the new DB2 Universal Driver, the older platform specific JDBC drivers are now known as the legacy drivers. 8

9 SQLJ Overview DB2 UCC Driver Advantages Single driver across all platforms Common JDBC/SQLJ behaviour No need to tweak code (such as driver name) to get it to run on another platform Common SQLJ utilities Program preparation now identical across platforms Reduce development / maintenance costs Write anywhere, deploy anywhere This slide summarises the advantages of the new universal driver. The value of being able to write a Java application on one platform and deploy it on another unchanged should not be underestimated. Many DB2 customers are using workstation development tools (such as the DB2 Development Center) to rapidly develop / prototype / test Java applications before they are deployed on the z/os platform. With no requirement for code changes and a 100% common program preparation process across z/os and LUW, developers have maximum flexibility in how and where to write and test their applications. This in turn can lead to significantly reduced development/maintenance costs. 9

10 SQLJ Overview T4 Driver for XA Implementation of DB2 Universal Driver for the z/os environment aka IBM z/os Application Connectivity to DB2 for z/os and OS/390 No charge optional feature for DB2 for z/os & OS/390 Version 7 APAR PQ80841, plus some pre-reqs Included with DB2 for z/os Version 8 Refer to APAR PQ85053 for latest version Can also be used to access V6 subsystems, but some features (e.g. XA functionality) disabled Applicable to z/os Java client only Still need to buy DB2 Connect for other scenarios A special implementation of the DB2 Universal Client is worth mentioning: the IBM z/os Application Connectivity to DB2 for z/os and OS/390. This is an optional nocharge feature for DB2 for z/os V7 (implemented via APAR PQ80841) that provides full Type 4 driver functionality for z/os Java clients needing to access a remote DB2 subsystem. The feature is also included with DB2 for z/os V8 (but you should refer to APAR PQ85053 for latest version). It should be noted that this feature allows access from within z/os only access from other environments (such as z/linux) still require the relevant DB2 Connect licence to be purchased. 10

11 SQLJ Overview T2 Driver xseries Browser zseries HTTP Server Websphere AS NEWCUST CALL NEWCUST zseries DB2 for z/os DDF DBM1 SPAS NEWCUST SP Exec SQL Exec SQL JDBC T2 DB2 for z/os DBM1 DDF Data This diagram illustrates the required connectivity when using a Type 2 driver to talk to a remote DB2 for z/os system from within a WebSphere application on a separate LPAR. As you can see, a DB2 for z/os subsystem is required on the WebSphere LPAR in order to provide the required remote connectivity to the DB2 for z/os subsystem containing the data: the Type 2 driver does not contain the necessary DRDA protocols to allow direct access to the remote DB2 system. Many DB2 customers found themselves having to pay for an additional DB2 licence on their WebSphere LPAR merely to provide this connectivity, which proved prohibitively expensive for many. 11

12 SQLJ Overview T4 Driver for XA xseries Browser zseries HTTP Server Websphere AS NEWCUST CALL NEWCUST zseries DB2 for z/os DDF DBM1 SPAS NEWCUST SP Exec SQL Exec SQL JDBC T4 XA Data This diagram shows a similar situation, but this time a Type 4 driver is being used instead (the T4 Driver for XA, also known as the IBM z/os Application Connectivity to DB2 for z/os and OS/390 ). As a Type 4 driver is able to directly connect to a remote database system (i.e. it contains DRDA application requester functionality in DB2 terms) no local DB2 system is required on the WebSphere LPAR. Although the feature still needs to be correctly licensed, this approach can yield significant cost savings where DB2 is not needed on the WebSphere LPAR for any other reason. 12

13 T4 Driver v DB2 Connect Common cause for confusion T4 driver will give you capability to connect to DB2 for z/os from other environments, but you are not necessarily licensed to do so! Distributed client DB2 for LUW: Connect not required Distributed client DB2 for z/os: Connect licence is required z/os client DB2 for z/os: Application Connectivity licence is required In all cases, T4 driver only gives basic JDBC/SQLJ connectivity. Full DB2 Connect gives many other functions, such as: Additional API support (ODBC, ADO, etc) Heterogeneous database access Connection concentrator Failover support 13

14 Topics Introduction SQLJ Overview Deployment Issues Performance Issues Summary & Conclusions This part of the presentation describes some SQLJ deployment experiences using the new universal driver on a z/os platform. 14

15 Installation Set-up is relatively straightforward Edit USS.profile to set environment settings (PATH, LIBPATH, CLASSPATH) Customise JCC driver configuration properties file Ensure IBM-supplied JDBC stored procedures and tables are defined Bind JCC packages and set EXECUTE authority using DB2binder Upgrading from old driver Save old copies of everything Change driver names and URLs Re-prepare programs (or use db2sqljupgrade) Installation of the new driver is relatively straightforward, and is documented in the DB2 Programming Guide and Reference for Java, and not the DB2 Installation guide as you might initially think. The process for upgrading from the legacy drivers is also covered. Because the new JCC driver is a completely new codebase, all SQLJ programs have to be reprepared from scratch or upgraded using the supplier db2sqljupgrade utility. 15

16 Legacy Driver / Universal Driver Compatibility SQL VALUES statement Could be used by legacy driver to invoke stored procedures no longer supported Rewrite to use normal CALL statement DECIMAL SP output parameters Depending on DB2 server, driver may not be able to online check the output parameter length In this case a default of DEC(31,8) is used Truncation will occur if actual parm is bigger Although the new universal driver is mostly compatible with the old one, there are a few areas where you may need to make some application changes as a result of implementing the new driver. It is worthwhile checking the readme file for the universal client to get a full list of these, so that you can assess the application impact of deploying the new drivers. This slide lists two of the incompatibilities that we experienced: The legacy drivers used to allow a call to a stored procedure to be invoked via the SQL VALUES statement. The JCC driver, such calls are interpreted as a stored function call, which is not supported. This is a simple one to fix merely re-code the VALUES statement in the form of a standard stored procedure CALL. The second issue we encountered related to truncation of decimal output parameters. Most DB2 servers do not support dynamically prepared CALL statements, so the driver will not be able to perform online checking of the parameter length. In such cases, it assumes the DECIMAL value will have a precision of 31 and a scale of 8. Where the actual result exceeds this, truncation will occur. 16

17 New SQLJ Program Preparation Overview.java Java Compile.class USS commands javac SQLJ Source SQLJ Translator Can optionally be invoked automatically by sqlj command sqlj.ser Customise Profile db2sqljcustomize.ser Can optionally be invoked automatically by db2sqljcustomize command All commands and files are 100% platform portable SQLJ Bind db2sqljbind DB2 Package This diagram summarises the program preparation process for the DB2 Universal Driver. This is somewhat simpler than the process for the older legacy drivers, as a single db2sqljcustomize command can perform all of the steps necessary to customise the SQLJ profile and bind the associated package(s) to DB2. The preparation process begins by running the Java source code through the SQLJ translator (using the sqlj command). This is analogous to the DB2 pre-compiler its job is to extract the SQLJ code and place it in a serialised profile, so that the modified Java source code can be run though a standard Java compile (using the javac command) to produce a Java class file. The sqlj command can optionally be configured to invoke the javac command automatically. At this stage, the serialised profile produced by the SQLJ translator is database-independent it could just as easily be used to access an Oracle database as a DB2 one. In order to support DB2 static SQL it must be customised for DB2. This is performed by the aptly-named db2sqljcustomize command, which will place a DB2-specific section in the serialised profile. It can also optionally invoke a bind process, to create one or more static SQL packages at the remote DB2 server (this can also be done separately using the db2sqljbind command if required). Due to the common codebase on both z/os and LUW platforms, all commands, syntax and output files are 100% portable across platforms. It is possible to begin the preparation process on one platform, port the necessary files to another and complete the process there. Note that on z/os all sqlj program preparation commands are executed from within the USS (Unix System Services) environment, so anyone responsible for running this process on the zseries platform will need some familiarity with that environment. 17

18 SQLJ Program Preparation Issues Use online checking during customisation -onlinecheck option (default) Checks DB2 catalog for object existence, authorisation, data types, etc Beware only dynamically preparable statements are checked If online checking fails, DESCRIBE will be issued at runtime Set DESCSTAT=YES to avoid -516 runtime errors (tells DB2 to include SQLDA in package for static SQL statements during BIND Especially important for performance see later The sqljcustomize command includes the onlinecheck option (which is on by default). This instructs the JCC driver to use the DB2 catalog of the target database system to validate any SQL calls within the program being customised (this includes checking object names, authorisation, data type matching and others). If online checking is not enabled, the customisation process will be quicker but some problems may not be discovered until the SQLJ package(s) are bound, or worse still until the program is actually run. Check your DSNZPARMs to ensure that the DESCSTAT parm (which controls whether DB2 executes DESCRIBEs on static SQL statements when it performs a bind operation) is set to YES. This is necessary because if online checking fails for a given statement, it is described at runtime. This runtime describe will fail with a if DESCSTAT=NO. This option is especially important for performance reasons, as we will see later in this presentation. 18

19 SQLJ Program Preparation Issues Password for BIND Mainframe userid/password for BIND must be explicitly entered in db2sqljcustomize or db2sqljbind no option to hide or be prompted Outstanding feature request Use workstation and/or USS scripting in the meantime Consider omitting customisation during initial testing If customisation is not found in.ser file, JDBC (dynamic) access will be used Speeds up early development Consider using DB2 Development Center Automates entire program preparation process A particular issue we came across when performing the program preparation process entirely within USS on z/os was with passwords for the BIND. A userid and password can be passed to the db2sqljcustomize or db2sqljbind commands, to be used as the authid for performing the BINDs under. However, unlike other commands such as CONNECT, omitting the password does not prompt for it with the characters not being echoed to the screen the password must be typed in the clear, as part of the command. This is an annoying exposure, and the subject of an outstanding feature request. In the meantime, it can be worked around by wrapping the program preparation commands in USS scripts, or using the DB2 Development Center to prepare and deploy the program. Another issue to consider is whether to bother customising during the early development process, where the program is probably going to be subject to frequent change and performance is less of an issue. If no customisation is found, the driver will fall back on dynamic JDBC access to the data (this approach is illustrated in the Possible Implementation Scenario later in this presentation). Finally, do consider using the DB2 Development Center (the new name for the old DB2 Stored Procedure Builder) to build, prepare, test and deploy your SQLJ programs. It provides a host of useful functions to improve productivity when developing Java and PSM applications and stored procedures. 19

20 SQLJ Roles Think carefully about who should do what in an SQLJ development environment Will the developers bind static SQL during testing, or will DBAs do it? If the developers do it, will they have a valid RACF userid/password, with BIND authority? If the developers don t do it, who sorts out the problems found when the SQL is run statically for the first time? Who (or what) moves the SQLJ packages between environments? One aspect of SQLJ implementation that is commonly overlooked until problems start to occur is roles and responsibilities. The slide shows just some of the questions that will have to be answered. Will LUW developers be given authority to BIND packages in a z/os environment? Such developers may not have z/os userids, and even if they do there may be problems in giving them the necessary BIND access, or having them embed their passwords in their program preparation scripts. One solution to this is to defer the customisation until late in the development process, then have the DBA perform the BIND (see next slide). 20

21 Possible Implementation Scenario Developer / DBA Unit Test (Workstation) Prep (SQLJ) System Test (zseries) Endevor / SCLM Production (zseries) Test Amend Move.class &.ser files Move.class &.ser files OK? BIND COPY /db2sqljbind BIND COPY /db2sqljbind Cust Profile & BIND Test Test OK? N.B. class and ser files can also be placed in a jar file This diagram illustrates a possible implementation scenario. Development and testing is performed in a Unit Test environment on the LUW platform, allowing rapid prototyping. Initially, no profile customisation is performed, which will speed up the preparation process and avoid the developers having to be given BIND access, but will also result in the SQLJ calls being executed dynamically. Once the program is ready to move to the System Test environment on z/os, the serialised profile is customised and bound by the DBA in order to enable the static SQL access. A final test is then performed by the developer before the Java.class files and the customised profile are moved to the z/os environment (these can be packaged up into a JAR file if required). Once the necessary files are available on the z/os LPAR, all that is required is a BIND COPY of the SQLJ package(s) to create copies in the System Test collection no other changes are required. The Java application can then be tested ready for deployment into production. The move to Production is identical to the move to System Test: the.class and.ser files for the application are copied to the relevant environment and a BIND PACKAGE COPY is executed to create the production packages. These steps can be easily automated using a code management tool such as Endevor or SCLM. 21

22 Topics Introduction SQLJ Overview Deployment Issues Performance Issues Summary & Conclusions This section of the presentation covers some of the common performance pitfalls associated with SQLJ. 22

23 SQLJ Performance Issues Beware of missing/failed customisations Check customisation messages carefully If customisation is not found in.ser file, JDBC (dynamic) access will be used No warnings given, unless JCC driver configuration properties file has been changed (db2.jcc.sqljuncustomizedwarningorexception) If in doubt, use DB2 monitor to check Use positioned iterators Named iterators are translated into positioned iterators internally, so save this overhead Scrollable iterators can be a big help Not previously available in legacy driver As previously mentioned, if a serialised profile is not customised, by default DB2 will just fallback to using dynamic JDBC access when the program is executed, with no warnings being produced. In such cases, the only way to determine if static SQL is being executed is to use your favourite DB2 performance monitor to check. If you want to avoid this, consider changing the db2.jcc.sqljuncustomizedwarningorexception property in the DB2JccConfiguration.properties file this can be set to 1 (in which case the JCC will issue a warning when an uncustomised SQLJ application is run) or 2 (in which case an exception will be thrown by the driver). SQLJ uses constructs called iterators to store SQL result sets the SQLJ equivalent of a cursor. Two types of iterators are supported: named and positioned. Positional iterators identify the columns within a result set by their position, whereas named iterators identify the columns by their actual names. Although named iterators sound much more attractive (they make the code easier to read, and protect the program from the effects of someone re-ordering the columns in the result table) they actually have to be translated into positioned iterators by DB2 internally. This is a performance overhead that can be avoided by using positioned iterators directly. Although not an SQLJ issue directly, use of scrollable iterators can be a great performance boost for some applications. This feature was not previously supported in the legacy drivers, but the Universal Driver provides the ability to define iterators that can be navigated in either direction using additional SQLJ methods. As for scrollable cursors in other languages, the scrollable iterators can be defined as static or dynamic, sensitive or insensitive. 23

24 SQLJ Performance Issues Use matching data types Additional work for SQLJ runtime to reformat data See Chapter 4 of SC Java Application Programming Reference for Java for DB2/Java data type mappings Always use online checking Even if data types are matching, unless online checking is enabled a predicate may be made non-indexable Java string variables do not have implicit length DB2 needs length of host variable for proper optimisation, so catalog access is needed As for other host languages, it is good practice to ensure that Java variables are properly matched to the DB2 data types you are accessing, in terms of their type and their size. Although the JCC driver will perform any valid conversions at run-time, this requires additional work and may have a significant performance impact on your application. A listing of the recommended Java types for each DB2 data type can be found in the DB2 Java Application & Programming Reference. Online checking during profile customisation is especially important when predicates using host variables are involved. In order for a predicate to use a matching index scan, the definition in the Java package must match the definition in the DB2 catalog, both in terms of data type and length. Because Java String objects do not have a concept of length, that information can only be obtained from the catalog, as part of the online checking process. If online checking is not performed, DB2 does not know if the length of the column and host variable match, and may therefore make the predicate non-indexable. 24

25 Other Performance Issues Usual DB2 / Java / Websphere good practices remain Check access paths, DB2 housekeeping, etc Retrieve only required data Disable autocommit Enable connection pooling (Websphere or Connect) Tune JVM (especially heap size) There are many other performance considerations outside of the SQLJ specific ones, and a few of them are covered on this slide. Generic DB2 performance good practices still apply, of course. These include using EXPLAIN during the BIND of SQLJ packages to ensure that correct access paths are being selected, implementing good housekeeping to ensure that statistics stay up to date and table organisation is good, and others. Another general rule for DB2 performance is only to select the columns required for a given query. This is even more important in a Java environment, where code page conversions and java object overheads make the cost penalty higher than usual. By default, DriverManager database connections have the autocommit property set on, which will force a DB2 COMMIT operation after each SQL statement. In addition to the restart/recovery complications this introduces, it can impair application performance. Generally, this property should be set to false, with explicit commits being made as required. Connection pooling is a framework for caching physical data source connections, which are equivalent to DB2 threads. When JDBC reuses physical data source connections, the expensive operations that are required for the creation and subsequent closing of java.sql.connection objects are minimised. This can provide up to 20x performance improvements in some scenarios. Finally, it is always worth reviewing the Java Virtual Machine (JVM) heap size for your application. In a Java database workload using either JDBC or SQLJ to access relational data, a lot of Java objects are created and then destroyed. The JVM heap size plays an important role in overall Java application performance. In almost all cases the default JVM heap size (8MB) is not sufficient and can lead to poor performance. Increasing the heap size (300Mb-400Mb is not uncommon) can dramatically improve performance, as it reduces the amount of garbage collection performed by Java. 25

26 Topics Introduction SQLJ Overview Deployment Issues Performance Issues Summary & Conclusions 26

27 Summary New universal driver gives major advantages over old legacy driver Portability Performance New function If you have existing applications, conversion is not trivial DBAs need at least some USS skills SQLJ moves closer to prime-time 27

28 Questions? 28

29 Session G11 SQLJ Implementation Experiences in DB2 for z/os Triton Consulting The Royal 25 Bank Plain Norwich NR2 4SF United Kingdom Tel: +44 (0) Fax: +44 (0)