Continuous Availability and Scalability with DB2 for z/os and WebSphere

Transcription

1 Continuous Availability and Scalability with DB2 for z/os and WebSphere Maryela Weihrauch Distinguished Engineer DB2 for z/os, IBM Pallavi Priyadarshini Architect JCC DB2 Connect, IBM

2 Agenda DB2 z/os concepts for availability Client configurations Enabling Sysplex Workload Balancing in WAS environments Algorithms for Workload Balancing and Automatic Client Reroute Recommended WAS, JCC and DB2 settings 2

3 DB2 Data Sharing - Availability Goal: Continuous availability across any planned or unplanned outage Elimination of single point of failure Remove all causes for planned outages rolling on-line upgrades and maintenance online schema evolution online utilities On a failure: Isolate failure to lowest granularity possible Automate recovery and recover fast Challenge: the distributed application server needs to find the (best) available path to the data. 3

4 DVIPA and Sysplex Distributor the Concept CF z/os 1 z/os 2 DB2A connect to Vx:446 SD Vx 1 2 distribute connection to DB2B 3 DB2B Needs to be configured at network layer and used for initial connections to DB2 group Group DVIPA provides a virtual TCPIP address into the Data Sharing Group Sysplex Distributor routes the connection request to the most available member based on WLM recommendation 4

5 DVIPA and Sysplex Distributor Consideration Benefit: Connections are successful as long as one member is up Connection level workload balancing between members Setup is isolated to z/os environment Drawbacks SD on one lpar may route to a member on a different lpar, which results into slightly higher response time compared to direct member access. Information about availability of data sharing members is only considered at creation of a "new" connection but application server typically maintain long-running connections (eg. WAS connection pool) DVIPA and Sysplex Workload Balancing at the application server need to be enabled to ensure the highest availability characteristics in a distr. environment. 5

6 DB2 Client Panoramic view Javascript PhP Python/Jython Ruby/JRuby node.js Scala Java JDBC API Zend framework adapters node-odbc node.js JSON driver SqlAlchemy/ Django Adapter Rails Adapter Lift purequery API JSON API Python interpreter c java Ruby interpreter c java Hibernate JPA DB2 CLI and ODBC driver DB2 JCC JDBC driver DB2

7 DB2 Client Configuration to Access DB2 z/os Strategy JDBC/SQLJ/ purequery/ ObjectGrid/ Data Web services Java based Clients JCC Type 4 DRDA DB2 z/os DRDA DB2 Connect (Gateway) DRDA CF CLI/ Ruby/ Perl.NET C based Clients DRDA Type 4-like DRDA DB2 z/os DB2 Group Clients access DB2 z/os directly (will not change licensing model) Sysplex Workload Balancing is supported by DB2 Connect gateway, JCC type 4, and Data Server Driver for CLI and.net application 7

8 JCC Sysplex Support in WAS environment Set DataSource IP as DVIPA (Initial connection to Sysplex Distributor) Connects applications to a DB2 data sharing group as though it were a single database server Global JCC property OR DataSource specific setting (enablesysplexwlb) Group DVIPA of data sharing group 8

9 JCC Sysplex Support in WAS environment Custom properties available on Data Source 9

10 JCC Sysplex Support in WAS environment Set DataSource property enablesysplexwlb to TRUE 10

11 JCC Sysplex Support in WAS environment Enables Workload Balancing (WLB) Workload spread amongst different members, based on server lists dynamically provided by Work Load Manager (WLM) Transactions distributed amongst best available members for maximum throughput and scalability Enables Automatic Client Reroute (ACR) On by default when WLB enabled Enables client application to recover from loss of communication so that application continues with minimal interruption When one member of the group fails, connection should be re-established to the next best available member of the group 11

12 Basic concepts underlying EnableSysplexWLB Server lists Available members identified by identified by member IP address and port Member priority (Weight) that indicates capacity of each member to handle work relative to others Weights obtained from z/os Workload Manager Returned whenever a connection is established or reused Connection pool Logical connection from the application to the driver No physical network connection while in pool Transport pool Physical connection from driver to DB2 Global pool of Transport objects per JVM Connection reuses/changes transport at transaction boundary EnableSysplexWLB automatically enables use of transport pool 12

13 WebSphere Application Server - End-to-end Example Always use connection pool closest to application Enable Sysplex Workload Balancing for best availability. Application Server Resource Adapter JVM DB2 Universal Driver JDBC/SQLJ Logical Conn 1 Application JCA Connection Manager DB Connection Pool Logical Conn 2 Logical Conn 3 disconnect at commit/rollback Transport 1 Transport 2 pooled connections to DB2 Data Sharing Group CF 13

14 Without enablesysplexwlb Application outage WAS JCC Driver DB2 Data Sharing Group App1 Connection Pool with size 2 C1 T1 DB2 A App2 C2 T2 DB2 B WAS JCC Driver DB2 Data Sharing Group App1 StaleConnection Exception Connection Pool with size 2 C T1 X DB2 A App2 C2 T2 DB2 B 14

15 With enablesysplexwlb Application continuity WAS JCC Driver DB2 Data Sharing Group SysplexWLB=true App1 Connection Pool with size 2 C network failure SQL failure T1 DDF Shutdown X DB2 A App2 C2 T2 DB2 B WAS JCC Driver DB2 Data Sharing Group SysplexWLB=true App1 Connection Pool with size 2 C1 T1 DB2 A App2 C2 T2 T3 DB2 B 15

16 Changed driver defaults for Sysplex properties V9.7FP6/V10.1FP2 and above recommended for Sysplex support Only set enablesysplexwlb to true Stay with driver defaults as much as possible Better defaults for Global properties db2.jcc.maxtransportobjects Max # of connections to DB2 server across all datasources Default value of 1000 (previous default -1, unlimited transports) # of transports driver can go upto = #logical connections * # ds members db2.jcc.maxtransportobjectidletime Maximum elapsed time in seconds before an idle transport is dropped Default value is 60 sec db2.jcc.maxtransportobjectwaittime Number of seconds the client waits for a transport to become available Default value is 1 sec (previous default = -1, unlimited wait) 16

17 Changed driver defaults for Sysplex properties db2.jcc.maxrefreshinterval Max elapsed time in sec before server list is refreshed Default is 10 sec (previous default 30 sec) Better defaults for Connection and DataSource properties maxretriesforclientreroute Maximum number of connection retries for automatic client reroute Default is 1 (previous default -1 unlimited, previous recommendation was to set to # of ds members) Cycling through all members of a group counts as 1 retry (previous behavior was retry against each member was counted as 1) retryintervalforclientreroute Number of sec between consecutive connection Default 0 sec 17

18 Work Load Balancing Initial connection to DB2 data sharing group using distributed IP address Initial connection returns a member IP list with WLM weights Subsequent connection directly to members Any following connection uses client Sysplex workload balancing algorithm to determine which member to use Driver asks for new server list after every maxrefreshinterval secs Workload i.e. Active transactions should be distributed in the ratio of the server returned member weights Single logical connection from an application can be directed to one member for one transaction and directed to different member for another transaction. At start of each transaction, best member computed 18

19 Work Load Balancing At the end of a transaction, server determines whether logical connection can execute next transaction on a different member. This decision, called reuse permission, is conveyed to client in the commit or rollback reply. DB2 returns indicator as part of COMMIT/ROLLBACK if transport can be reused and returns SET statements to replay connection state at reuse No open WITH HOLD cursor No declared global temporary tables must exist Used declared global temp tables must be explicitly or implicitly dropped No reference to packages bound with KEEPDYNAMIC YES If reuse permission, transport given back to the transport pool at the end of the transaction 19

20 Work load balancing (WLB) algorithm DB2/WLM reports following to the driver M1 : priority=70 IPADDR=/ port=7330 M2 : priority=30 IPADDR=/ port=7330 Member 1 has a weight of 70 i.e. ratio is 70/100 = 0.7 Member 2 has a weight of 30 i.e. ratio is 30/100 = 0.3 Goal is for 70% of the work to be allocated to member 1, and 30% to member 2 Next transaction should be assigned to a member where (Member s currently active transactions) / (Total active transactions for the group) <= (Member priority) / (Total of all member priorities) With reference to WLB No of active transactions on a member = No of transports in use

21 Work load balancing (WLB) algorithm What does the new algorithm do Member 1 has 35 active connections (in a UoW) Member 2 has 10 active connections Current ratios - Member 1 = active connections / total connections = 35 / 45 = 0.78 (which is > 0.7 target ratio) Member 2 = 10 / 45 = 0.22 (which is < 0.3 target ratio) Next transaction would be assigned to member 2. Weight ratios are re-calculated each time a new member list is received

22 Transport Pool Behavior Every time a connection needs a transport, WLB algorithm first provides the best member Logical connection requests the pool for a transport to this 'best member' using the Database Name, Server, Port, SSL, Server Type, Auth Mechanism as a composite key Every logical connection remembers the transport object it last used against a given member. It passes in the key to the last transport (if any) that the connection used for this member. Algo attempts to use this transport object when it needs to go to the member Until maxtransportobjects is hit, the pool prefers to create a new transport if one is not available 22

23 Transport Pool behavior This leads to: maxpossibletransports = # of data sharing members X # of logical connections WAS JCC Driver SysplexWLB=true App1 Connection Pool with size 2 C1 T1 Idle DB2 A App2 C2 T2 T3 T4 Idle Active Active DB2 B If maxpossibletransports > maxtransportobjects, the transports should remain at maxtransportobjects, the excess need will result in logical connections being put on a wait queue. At end of maxtransportobjectwaittime, default is 1 second at the end of which connection receives a SQLCODE -4210, SQLSTATE [jcc][t4][10391][12247][ ] Timeout getting a transport object from pool. ERRORCODE=-4210, SQLSTATE=

24 Transport Pool Statistics For monitoring driver Sysplex behavior (global transport object pool) Configuration property settings db2.jcc.dumppool=dump_sysplex_msg DUMP_POOL_ERROR db2.jcc.dumppoolstatisticsonschedule=60 db2.jcc.dumppoolstatisticsonschedulefile=/home/was/logs/srv1/poolstats Example fields npr - total number of requests made to the pool since the pool was created. nsr number of times pool returned an object (successful requests) lwroc light weight reuse hwroc heavy weight reuse lbt longest time thread waited to get an object tpo transport object count Application API also exists to gather transport pool statistics (DB2PoolMonitor class)

25 ACR Behavior in a Data Sharing Group Seamless On a connection failure, driver silently re-establishes the connection to another member. Calling application remains unaware of this. Non-Seamless On a connection failure, driver reports an exception (SQLCODE ) to the calling application if the connection could be established to an alternate member. When working with a Data Sharing Group, driver always evaluates the failing transaction for a Seamless failover. Unless certain conditions are met, Seamless failover can not be performed. Conditions such as 2 nd or more SQL of a transaction, held cursors, open LOB streams prevent seamless failover. 25

26 Behavior at Connection Errors If first SQL stmt in transaction fails and reuse OK (Seamless reroute) No errors reported back to application SET statements associated with the logical connection are replayed with first SQL on another transport If subsequent SQL fails and reuse OK (Non-seamless reroute) reuse error returned to application (transaction is rolled back) Up to application to retry transaction Reconnection happens only when application resubmits (different behavior than older releases) SET statements are replayed to recover connection state If subsequent SQL and reuse not OK (Non-seamless reroute) connection failed error returned to application. Connection returned to initial (default) state application needs to reestablish connection state and retry transaction If all members in the member list are tried and none seems to be available, the initial data source url (distributed IP address) is retried to make sure that really no member is available. 26

27 WebSphere Connection Pooling Connection object pool maintained by WebSphere Saves creating/destroying Connection objects which is relatively expensive Connection object holds references to other Java objects like preparedstatement objects that would be destroyed with the Connection object Connection Pooling Properties Max Connections max connections from JVM instance Min Connections lazy minimum number of connections in pool Reap Time How often cleanup of pool is scheduled in seconds Unused Timeout How long to let a connection sit in the pool unused Aged Timeout How long to let a connection live before recycling Purge Policy After StaleConnection, does the entire pool get purged or only individual connection 27

28 Recommended WebSphere Connection Pool Properties with SysplexWLB Disable Reap Time, Unused Timeout and Aged Timeout by setting values to zero Actual physical connections are handled by driver while connections handled by WAS are logical connections. 28

29 End-to-end Pooling WAS JCC Driver DB2 for z/os App threads Connection pool Transport pool DDF connections DBATs Idle/ Pooled Inactive Pooled 29

30 Important DB2 zparm Values CONDBAT Max. # of distributed connection into DB2 system Includes inactive and active connections, may be large because of many inactive connections Default is 10,000 MAXDBAT - Max # database access threads (DBATs) that can be active concurrently. In many installations, max. value determined by available storage in DBM1 (check IFCID 225) Default is 200 IDTHTOIN time in sec an active server thread remain idle before it is canceled Inactive connections are not subject to idle thread timeout Strongly recommended to not set to 0, default of 120 secs works well To control connections waiting on DBAT, new parms introduced - MAXCONQW how long in seconds a transport/client connection should wait for a DBAT before re-eroute 30 MAXCONQN - how many transports/client connections should wait for a DBAT before reroute

31 WAS/DB2 - Tuning Considerations Application Server Connection Pool: max connection unused timeout DB2 C-based /JCC client: EnablesysplexWLB MaxTransportsObjects MaxTransportObjectIdleTime MaxTransportObjectWaitTime DB2 Zparms: MAXDBAT CONDBAT MAXCONQN MAXCONQW IDTHTOIN 31

32 Summary Highly available environment for strategic, enterprise applications involves: DB2 data sharing group enabled with DVIPA and Sysplex Distributor Application servers/db2 Client drivers enabled for Sysplex Workload Balancing Applications are well-behaved DB2, Driver and WAS teams have worked together to come up with optimal out of box behavior for most scenarios Some tuning may be needed for HA features of DB2, Drivers and WAS 32

33 References DB2 for z/os and WebSphere Integration for Enterprise Java Applications DB2 9 for z/os Data Sharing: Distributed Load Balancing and Fault Tolerant Configuration DB2 Version 9.1 for z/os Data Sharing: Planning and Administration (SC ) topic=/com.ibm.db29.doc.dshare/db2z_dshare.htm DB2 for z/os best practice education 33

34 Thank you 34