Application Continuity

Transcription

1 Application Continuity Checklist for preparation ORACLE WHITE PAPER SEPTEMBER 2018

2 Contents Application Continuity Checklist 1 Align Application Timeouts 1 Current recommended patches for Configure the TNS or URL for High Availability 3 Enable Fast Application Notification (FAN) 4 Monitoring FAN 4 Application Continuity Coverage 5 Steps for using Application Continuity 5 For JDBC Applications 8 Additional Materials 9 0 APPLICATION CONTINUITY CHECKLIST

3 Application Continuity Checklist The following checklist is useful for preparing your environment for using the Oracle Database 12c feature Application Continuity. Even if Application Continuity is not enabled on your database service, or not used by your applications, the points discussed here provide great value in preparing your systems to support Continuous Availability. Align Application Timeouts If an application level timeout is lower than timeouts provided for the detection and recovery times of the underlying system, then there is insufficient time available for the underlying recovery and replay to complete. Misaligned timers can result in replay by Application Continuity starting before the system has recovered, potentially causing multiple replays to be attempted before success, or worse still, requests completely timing out and an being error returned to your applications or users. Consider that an application uses READ_TIMEOUT or HTTP_REQUEST_TIMEOUT or some custom timeout, then the following simple guidelines apply: READ_TIMEOUT > EXADATA special node eviction (FDDN) (2 seconds in ) READ_TIMEOUT > MISSCOUNT (default 30 sec, modifiable in ) READ_TIMEOUT > Data Guard Observer: FastStartFailoverThreshold (default 30 sec, modifiable) FastStartFailoverThreshold > MISSCOUNT (must be at least twice) READ_TIMEOUT > FAST_START_MTTR_TARGET READ_TIMEOUT > NET level: (RETRY_COUNT+1) * RETRY_DELAY and READ_TIMEOUT < Replay_Initiation_Timeout (modifiable on the service, default 300 seconds) To avoid premature cancelling of requests the application timeout should be larger than the maximum of: (MISSCOUNT (or FDNN) + FAST_START_MTTR_TARGET), (FastStartFailoverThreshold + FAST_START_MTTR_TARGET + TIME TO OPEN). Current recommended patches for JDBC BLR for AC: UNITS FOR TRANSPORT_CONNECT_TIMEOUT CHANGED WITH NO QUALIFIER 1 APPLICATION CONTINUITY CHECKLIST

4 BLR for AFTER TIMEOUT OF AN ADDRESS IN URL, NOT TRY OTHER ADDRESS AND RETRY MLR BLR for AC: SPEC-J 4% INCREASE BETWEEN 12.1 JARS AND 12.2 JARS BLR for AC: NEC OBSERVED THAT ALL IPS IN SCAN ADDRESS ARE NOT USED FOR JAVA NET BLR for KERBEROS: "protocol violation" exception while getting a connection ( ) BLR for AFTER TIMEOUT OF AN ADDRESS IN URL, NOT TRY OTHER ADDRESS AND RETRY UCP ConnectionWaitTimeout should exceed CONNECT_TIMEOUT MLR UCP: java.lang.nullpointerexception at ServiceMember.averageAdvisedLoad UCP: ONE THREAD STUCK IN LOADBALANCER.UNDERLOADEDINSTANCE() GROWING CPU USAGE UCP: connection attempt failed with ORA BLR for BLR BACKPORT OF BUG ON TOP OF NEED TO GUARD AGAINST OUTOFMEMORYERROR FOR LARGE/LONG REQUESTS WLS BLR for AC: AQ JMS USES ORACLE CONCRETE CLASSES - DEPRECATED JAVA CODE 2 APPLICATION CONTINUITY CHECKLIST

5 BLR for AC: - WLS DO DB POOL REBUILD AFTER RECEIVE SERVICE DOWNEVENT FOR STANDBY ONS MLR ONS JAVA CLIENT CLOSES SUBSCRIBERS ON GC OF ONS OBJECT BLOCKING WAIT WHEN SHUTTING DOWN ONS CLIENT RDBMS BLR for BLR for AC:SUPPORT FOR REPLAYING ENQUEUE,AQ ID SEQUENCE NEED TO KEEP AC: AQ JMS USES ORACLE CONCRETE CLASSES - DEPRECATED JAVA CODE Configure the TNS or URL for High Availability The following TNS/URL configuration is recommended for successfully connecting at failover, switchover, fallback and basic startup. Set RETRY_COUNT, RETRY_DELAY, CONNECT_TIMEOUT and TRANSPORT_CONNECT_TIMEOUT parameters in the TNSnames or the URL to allow connection requests to wait for the service and connect successfully. Set CONNECT_TIMEOUT to a high value to prevent login storms. Low values can result in feeding frenzies logging in due to the application or pool cancelling and retrying connection attempts. Do not set (RETRY_COUNT+1)*RETRY_DELAY or CONNECT_TIMEOUT larger than your response time SLA. The application should either connect or receive an error within the response time SLA. These are general recommendations for configuring the connections for high availability. Do not use Easy Connect Naming on the client as EZCONNECT has no high availability capabilities. This is the recommended TNS for ALL Oracle drivers for 18c and 12.2: Alias (or URL) = (DESCRIPTION = (CONNECT_TIMEOUT= 120)(RETRY_COUNT=20) RETRY_DELAY=3) (TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521))) (ADDRESS_LIST = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521))) (CONNECT_DATA=(SERVICE_NAME = gold-cloud))) For JDBC connections in 12.1 the following should be used: 3 APPLICATION CONTINUITY CHECKLIST

6 (DESCRIPTION = (CONNECT_TIMEOUT= 15) (RETRY_COUNT=20)(RETRY_DELAY=3) (ADDRESS_LIST = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521))) (ADDRESS_LIST = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521))) (CONNECT_DATA=(SERVICE_NAME = gold-cloud))) Enable Fast Application Notification (FAN) FAN needs to be enabled. FAN is a mandatory component for interrupting the application to failover. When a node or network fails, the application needs to be interrupted in real time. Failing to enable FAN will lead to applications hanging when HARD physical failures occur. Hang times without FAN range from 8-15 minutes. Starting with Oracle Database 12c, there are three important enhancements for FAN: FAN is default, auto-configured and enabled out of the box with Oracle Real Application Clusters (RAC). FAN reads the URL and configures itself at the client. FAN is always configured at the Grid Infrastructure cluster. Using the URL format shown above is important for auto-configuration of FAN (using a different format will prevent FAN from being auto-configured). All Oracle clients use the Oracle Notification Service (ONS) as the transport for FAN FAN is posted by Global Data Services (GDS) to allow FAN events to span data centres. Monitoring FAN Use the FANWatcher utility to verify event posting and receiving: There is an updated (July 2017) version of FANWatcher available from OTN or via WebLogic Server Blogs. Refer to Fast Application Notification - inuity/learnmore/fastapplicationnotification12c pdf Monitoring FAN Events UCP tracing can be enabled via the following procedure - to show FAN is being processed at UCP/Tomcat: Logging can be configured using a properties file. The location of the properties file must be set as a Java property for the logging configuration file property. For example: java Djava.util.logging.config.file=myLog.properties The logging properties file defines the handler to use for writing logs, the formatter to use for formatting logs, a default log level, as well as log levels for specific packages or classes. For example: handlers = java.util.logging.consolehandler java.util.logging.consolehandler.level = ALL java.util.logging.consolehandler.formatter = java.util.logging.simpleformatter oracle.ucp.level = FINEST oracle.ucp.jdbc.pooldatasource = WARNING Feature specific logging can be enabled by setting a property in the logging configuration file. For all features: clio.feature.all = on 4 APPLICATION CONTINUITY CHECKLIST

7 or for specific features, such as: clio.feature.conn_construction = on clio.feature.high_availability = on clio.feature.load_balancing = on clio.feature.transaction_affinity = on More information can be found in the Oracle documentation: Universal Connection Pool Developer s Guide Chapter 12 Overview of Logging in UCP JDBC Developers Guide Chapter 34 Diagnosability in JDBC Application Continuity Coverage Application Continuity for Oracle Database 12c supports the following clients: Oracle JDBC Replay Driver 12c or later. This is a JDBC driver feature provided with Oracle Database 12c for Application Continuity, referred to as the replay driver onwards. Oracle Universal Connection Pool (UCP) Oracle WebLogic Server 12c, and third-party JDBC application servers using UCP Java connection pools or standalone Java applications using Oracle JDBC - Replay Driver 12c or later with Request Boundaries Oracle Tuxedo OCI Session Pool 12c Release 2 or later SQL*Plus 12c release 2 or later (use -ac command line switch) ODP.NET Unmanaged Provider 12c Release 2 or later If using a third party, Java-based application server, the most effective method to achieve high availability is to replace the data source with UCP. This approach is supported by many application servers including IBM WebSphere and Apache Tomcat, Red Hat JBoss, Spring and Hibernate, and others. Using UCP as the data source allows UCP features such as Fast Connection Failover, Runtime Load Balancing and Application Continuity to be used with full certification. Steps for using Application Continuity Use a supported client (see coverage above) Return Connections to the Connection Pool The application should return the connection to the connection pool on each request. It is best practice that an application checks-out a connection only for the time it needs it. Holding a connection when not in use is not good practice. An application should therefore check-out a connection and then check-in that connection immediately the work is complete. The connections are then available for subsequent use by other threads, or your thread when needed again. Following this practice also embeds request boundaries that Application Continuity uses to identify safe places to resume and end capture. Use FAILOVER_RESTORE Check if the application is presetting values on connections. 5 APPLICATION CONTINUITY CHECKLIST

8 Some applications and mid-tiers configure connection pools such that all connections are, for example, in a preset language or time zone. If session state is set intentionally on connections outside requests, and requests expect this state, replay needs to re-create this state before replaying. Most common states are restored automatically by setting FAILOVER_RESTORE to LEVEL1. If you preset session states in addition to the standard set, then you will need to register a callback, or UCP label, to restore these states. Refer to Application Continuity whitepaper listed in Additional Materials below. Choose one of the following options: FAILOVER_RESTORE=LEVEL1 set on the service Connection Initialization Callback for Java or the (older) TAF Callback for OCI Universal Connection Pool or WebLogic Server Connection Labeling Starting with RDBMS 12.2, FAILOVER_RESTORE=LEVEL1 is the recommended method. Enable Mutable Use in the Application Mutable functions are functions that can return a new value each time they are executed. Support for keeping the original results of mutable functions is provided for SYSDATE, SYSTIMESTAMP, SYS_GUID, and sequence.nextval. If the original values are not kept and different values are returned to the application at replay, replay is rejected. Configure mutable objects using GRANT KEEP for application users, and the KEEP clause for a sequence owner. When KEEP privilege is granted, replay applies the original function result at replay. For example: SQL> GRANT [KEEP DATE TIME KEEP SYSGUID] to USER SQL> GRANT KEEP SEQUENCE mysequence to myuser on sequence.object Side Effects may need to be disabled Side effects are replayed unless the application specifies otherwise. Applications that use external actions should be reviewed to decide if requests with side effects make sense to replay or not. For example, does the application want to send again or to audit again or to transfer a file again? Frequently it is desirable to replay the side effect. However, sometimes it may be better not to. If a request has an external action that should not be replayed, that request can use a connection that does not have Application Continuity enabled, or replay can be disabled for that request using the disablereplay() API for Java or OCIRequestDisableReplay for OCI. All other requests continue to be replayed. Protection Levels Using ORAchk Application Continuity s Coverage Analysis reports in advance of any failovers, the percentage of requests that are fully protected by Application Continuity. For those requests that are not fully protected, what they are and where they occur is reported: Measure Coverage ( inuity/overview/application-continuity-wp-12c pdf) 6 APPLICATION CONTINUITY CHECKLIST

9 For JDBC Applications Enable JDBC Statement Cache For JDBC based applications, always use the JDBC statement cache for performance. When using Application Continuity, it is mandatory to disable a statement cache at the application server level (for example Tomcat, WebLogic, WebSphere etc) if used, as described in the JDBC reference documentation: If a statement cache at the application server level is enabled (for example, the WebLogic or third-party application server statement cache), this must be disabled when the replay is used. Instead, configure the JDBC statement cache, which performs better because it is optimized for JDBC and Oracle and because it supports Application Continuity. Use oracle.jdbc.implicitstatementcachesize=nnn Setting nnn to a positive value enables the Implicit Statement Cache. Tune Garbage Collector For many apps the default Garbage Collector tuning is correct. For an app with extreme performance requirements it may be necessary to perform JVM tuning. Our recommendation for extreme performance is to add the following to the Java command line (note that both attributes should be set to the same value): java -Xms 2000m Xmx 2000m COMMIT Application Continuity supports COMMIT of all styles: top-level, AUTOCOMMIT and COMMIT embedded in PLSQL. If your application is using a top-level commit, this is OCOMMIT or COMMIT() that is standalone, then there is full support for replay including when using SESSION_STATE_CONSISTENCY=STATIC mode ( ) and with Transparent Application Continuity (TAC). If you application is using COMMIT embedded in PLSQL or AUTOCOMMIT, it may not be possible to replay for cases where Application Continuity detects that the call including the COMMIT did not run to completion. Application Continuity will do the right thing. For top-level COMMIT, for JDBC applications, you MUST disable AUTOCOMMIT either in the application or as property of UCP. This is particularly important when UCP is embedded in 3rd party application servers such as Apache Tomcat, IBM WebSphere and RedHat JBoss if the application does not want AUTOCOMMIT. Remove deprecated Java Against your Java applications, please run the concrete class analysis as per the Application Continuity white paper found at: Application Continuity with Oracle Database 12c Release 2 : Using AC Checking for Concrete Classes ( ioncontinuity/overview/application-continuity-wp-12c pdf) You must avoid using JDBC deprecated classes (concrete classes). This is a hard restriction in 12.1 but has been relaxed for most concrete classes in 18c. For information about the deprecation of concrete classes, including actions to take if an application uses them, see My Oracle Support Note ( 7 APPLICATION CONTINUITY CHECKLIST

10 Diagnostics Use the following in the OracleLog.properties file (should use absolute paths to the directory hosting the log files): handlers = java.util.logging.filehandler java.util.logging.filehandler.pattern = <REPLACE THIS PATH FIRST>/jdbc%u.log java.util.logging.filehandler.limit = java.util.logging.filehandler.count = java.util.logging.filehandler.formatter = oracle.ucp.util.logging.ucpformatter.level = WARNING oracle.jdbc.internal.replay.level = FINEST oracle.ucp.jdbc.pooldatasourceimpl.level = FINE oracle.ucp.jdbc.oracle.level = FINEST Tracing at the RDBMS server can be enabled via the init.ora file: alter system set event='10602 trace name context forever, level 28:trace[progint_appcont_rdbms]:10702 trace name context forever, level 16' scope = spfile ; Statistic monitoring beans are available in the ucpdemos.jar sample code at: Refer to the code samples UniversalConnectionPoolManagerMBeanSample.java and UniversalConnectionPoolManagerSample.java for information on MBeans. Additional Materials OTN Home page for Application Continuity html FAN whitepaper ioncontinuity/learnmore/fastapplicationnotification12c pdf Application Continuity whitepaper 8 APPLICATION CONTINUITY CHECKLIST

11 ion-continuity-wp-12c pdf Application Continuity OOW 2017 Client Failover ioncontinuity/learnmore/hidingunplannedoutages pdf ioncontinuity/learnmore/client-failover-brief pdf Oracle Corporation, World Headquarters Worldwide Inquiries 500 Oracle Parkway Phone: Redwood Shores, CA 94065, USA Fax: APPLICATION CONTINUITY CHECKLIST

12 CONNECT WITH US blogs.oracle.com/oracle facebook.com/oracle twitter.com/oracle oracle.com Copyright 2018, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group White Paper Title September 2018 Author: Troy Anthony Contributing Authors: Carol Colrain, Carmen Frank