Oracle Client HA Configuration with Oracle

Transcription

1 Oracle Client HA Configuration with Oracle 18c Robert doag2018

2 Who Am I Senior Principal Consultant and Trainer at Trivadis GmbH in Munich Master of Science in Computer Engineering At Trivadis since 2004 Trivadis Partner since 2012 Focus: Data and service high availability, disaster recovery Architecture design, optimization, automation Troubleshooting Trainer: O-RAC, O-DG 2

3 We help to generate added value from data 3

4 With over 650 specialists and IT experts in your region 16 Trivadis branches and more than 650 employees Experience from more than 1,900 projects per year at over 800 customers 250 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 million Financially self-supporting and sustainably profitable 4

5 Agenda 1. Introduction 2. Database Services 3. Timeouts 4. Transparent Application Failover 5. Fast Application Notification/Fast Connection Failover 6. Application Continuity 7. Conclusions 5

6 Introduction 6

7 Oracle Client HA Main Problems To Address CASE 1 CASE 2 New network session (connect) Database Clients Already established network session Database Clients 2 Connect attempts 4 Client failover 1 Connected 5 Client failover 3 Wait for connect timeout Problem Problem 3 Re-connect attempts 4 Wait for re-connect timeout Problem Problem 1 IP not reachable (server/network/ issue) 2 IP not reachable (server/network/ issue) 7

8 Database Services 8

9 The Foundation Dynamic Database Services All of high availability techniques require a properly configured dynamic database service Do not use for application connectivity: Default database services Static database services The service_names initialization parameter Dynamic database services can be created with: srvctl (Oracle Grid Infrastructure installation) dbms_service.create_service() PL/SQL procedure For a Data Guard system, we need a role-based service Service READY Service READY 9

10 Dynamic Database Services Services created with srvctl are managed by Oracle Grid Infrastructure Services created with PL/SQL require a TRIGGER firing AFTER STARTUP ON DATABASE Service srvctl add service -db RAC_SITE1 -service OLTP.TRIVADIS.COM -preferred "RAC1,RAC2" -available "RAC3" -tafpolicy BASIC -failoverdelay 1 -failoverretry 180 -role PRIMARY -clbgoal LONG -rlbgoal SERVICE_TIME... BEGIN DBMS_SERVICE.CREATE_SERVICE ( service_name => 'OLTP.TRIVADIS.COM', network_name => 'OLTP.TRIVADIS.COM', failover_method => 'BASIC', failover_type => 'SELECT', failover_retries => 180, failover_delay => 3); END; / 10

11 Timeouts 11

12 New Oracle Net Session Connect Timeout Connect timeout kicks in during connection request to an unresponsive server Failover to second address is possible, after the first connect request timed out! 1 LSNR Three-way handshake 2 Oracle Net 3 LSNR 12 Default connect timeout value depends on OS ARP cache entry status: REACHABLE 60 sec. STALE or entry already removed 3 sec.

13 New Oracle Net Session Connect Timeout 1 sqlnet.ora parameters Applies to each IP that a host name resolves to! All Oracle client versions supported TCP.CONNECT_TIMEOUT=3 #default 60 sec. SQLNET.OUTBOUND_CONNECT_TIMEOUT=5 #no default LSNR 2 Oracle Net LSNR For clients 11gR2+ the timeouts value can be specified in the connect description OLTP.trivadis.com = (DESCRIPTION = (FAILOVER=ON) (LOAD_BALANCE=OFF) (CONNECT_TIMEOUT=5)(RETRY_COUNT=3)(RETRY_DELAY=1)(TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP )(HOST = italy )(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP )(HOST = sweden )(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = OLTP.trivadis.com))) 13

14 New Oracle Net Session Connect Timeout 1 JDBC Thin driver: TRANSPORT_CONNECT_TIMEOUT ( ) Default value with no qualifier in ms (BUG) and not seconds! JDBC patch LSNR 2 Oracle Net LSNR 18c issue has been solved pds.seturl("jdbc:oracle:thin:@(description =(FAILOVER=ON)(LOAD_BALANCE=OFF)" + "(CONNECT_TIMEOUT=5)(RETRY_COUNT=3)(RETRY_DELAY=1)(TRANSPORT_CONNECT_TIMEOUT=3)" + JDBC Thin clients can alternatively use the following driver property (ms) Overrides CONNECT_TIMEOUT from address description parameters prop.put(oracle.net.ns.sqlnetdef.tcp_conntimeout_str, ""+3000); 14

15 New Oracle Net Session Connect Timeout De-prioritization of down database nodes is activated by default and can be tuned: SQLNET.DOWN_HOSTS_TIMEOUT (OCI driver, ) oracle.net.down_hosts_timeout (JDBC Thin driver, ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP )(HOST = italy )(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP )(HOST = sweden )(PORT = 1521))) italy sweden Timeout Cache entry expired sweden italy Timeout 15 Works fine, if TRANSPORT_CONNECT_TIMEOUT > 3 sec.

16 Established Oracle Net Session Re-Connect Timeout 2 Re-connect timeout kicks in during data send operation to an unresponsive server/ip Default timeout value on Linux > 15 min. LSNR P1 3 Timeout 1 Oracle Net 4 LSNR Client failover (CONNECT_TIMEOUT=5)(RETRY_COUNT=3)(RETRY_DELAY=1)(TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP )(HOST = italy )(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP )(HOST = sweden )(PORT = 1521))) Don t use the following parameters to reduce the re-connect timeout: sqlnet.ora: SQLNET.RECV_TIMEOUT, SQLNET.SEND_TIMEOUT (sec) JDBC Thin driver connection property: ReadTimeout (ms) 16 Better, use FCF/FAN or tune the tcp_retries2 OS kernel parameter

17 Timeouts Summary At least Oracle client connect timeouts should be set: CONNECT_TIMEOUT=5 and TRANSPORT_CONNECT_TIMEOUT=3 is a good starting point Take your network latencies into considerations Depending on your goal, TRANSPORT_CONNECT_TIMEOUT might require a higher value (de-prioritization) To address re-connect times: Use FCF/FAN, or Tune the tcp_retries2 OS kernel parameter 17

18 Transparent Application Failover 18

19 Transparent Application Failover Overview TAF is a feature of the client OCI driver introduced in Oracle 8 (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP )(HOST = italy )(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP )(HOST = sweden )(PORT = 1521))) Allows for automatic re-connection In many cases allows for resumable queries Useful for session migration between RAC instances during some planned downtimes Failover process can only be initiated, after receiving an error for the established connection (re-connect timeout) 19

20 Transparent Application Failover Configuration TAF properties can be set on the client or server side (recommended, higher priority) Example server side TAF BASIC method configuration srvctl add service -db OLTP_SITE1 -service OLTP_RW -preferred OLTP1,OLTP2 -tafpolicy BASIC -failovertype SELECT -failoverdelay 1 -failoverretry 180 BEGIN DBMS_SERVICE.CREATE_SERVICE ( service_name => 'OLTP_RW.TRIVADIS.COM', network_name => 'OLTP_RW.TRIVADIS.COM', failover_method => 'BASIC', failover_type => 'SELECT', failover_delay => 1 failover_retries => 180); END; / 20

21 Transparent Application Failover Session Migration To transparently migrate sessions between RAC nodes during planned downtimes srvctl stop service -db <db_unique_name> -instance <instance> -service <service> EXEC DBMS_SERVICE.DISCONNECT_SESSION('<service>', DBMS_SERVICE.POST_TRANSACTION) Starting with 18c srvctl add service -db <db_unique_name> -service <service> -stopoption [IMMEDIATE TRANSACTIONAL> -drain_timeout <value_in_sec> srvctl relocate service -db <db_unique_name> -oldinst <instance> -newinst <instance> -service <service> To override default service settings srvctl relocate service -db <db_unique_name> -service <service> -oldinst <instance> -newinst <instance> -stopoption TRANSACTIONAL -drain_timeout 60 -force 21

22 Transparent Application Failover (TAF) TAF is simple to configure and use, but is also restricted in many areas: Uncommitted transactions are rolled back (nevertheless, to assure that a database application is failover-aware application, it needs to issue explicitly a ROLLBACK before proceeding with subsequent work: ORA-25402: transaction must roll back) Stored procedure read is not supported PL/SQL and session state is lost (callback functions might be a solution) Parallel query, database links, SYS user are not supported Is not designed to work after server process failure (ORA-03113: end-of-file on communication channel) To kick in, TAF needs an error/timeout! 22

23 Fast Application Notification Fast Connection Failover 23

24 Fast Application Notification Overview Provides rapid notification about status changes (up/down events) for database services, instances and nodes ONS ONS Generating Node: cldb01.trivadis.com Event payload: VERSION=1.0 event_type=servicemember service=soe_app1.trivadis.com instance=rac2 database=rac_site1 db_domain=trivadis.com host=cldb02 status=down reason=failure timestamp= :56:43 timezone=+02:0 FAN Subscribers Delivers workload information about services (runtime load balancing) ONS is used as the FAN transport for all client types Can be used with different client types: JDBC, OCI, ODP.net Well integrated with Universal Connection Pool 24

25 Requirements For Using FAN Events Oracle Grid Infrastructure is necessary to register with ONS ONS default ports local: 6100, remote: 6200 (firewall) Configured and started automatically for GI cluster installations For GI standalone systems needs to be activated and configured manually ONS ONS srvctl enable ons srvctl modify ons -remoteservers <remote_node> verbose srvctl start ons Correct database service configuration is necessary srvctl add service -clbgoal [SHORT LONG] #LONG is the default -rlbgoal [SERVICE_TIME THROUGHPUT NONE] -notification [TRUE FALSE] #To enable FAN for OCI/ODP.net connections 25

26 Fast Connection Failover Overview Pre-configured client side FAN integration for JDBC clients It works in combination with connection pooling mechanism, so the Universal Connection Pool (UCP) or WebLogic Server Active GridLink is necessary Reacts to up/down FAN events Remove dead connections from connection pool after receiving FAN down event and redistributing them, if applicable, to other available nodes Connection re-distribution after receiving FAN up event Do not configure TAF with FCF for JDBC thick (OCI) clients ONS Connection Pool (60 connections) ONS 3 26

27 Fast Connection Failover Setup Example how to use FCF with Universal Connection Pool (UCP) Configure ONS and database service Include UCP and ONS libraries in your CLASSPATH (not part of the Oracle Instant Client installation) CLASSPATH=.:/usr/lib/oracle/18.0/client64/lib/ojdbc8.jar:/usr/lib/oracle/18.0/client 64/lib/ons.jar:/usr/lib/oracle/18.0/client64/lib/ucp.jar To subscribe to FAN events and use HA UCP features you need to activate FCF first... PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource(); pds.setconnectionfactoryclassname("oracle.jdbc.pool.oracledatasource"); pds.seturl(dburl); Setting connection pool properties... pds.setfastconnectionfailoverenabled(true); //not activated per default! 27

28 Fast Connection Failover Restrictions Fast Connection Failover restrictions In-flight transactions are lost as well as calls in the middle of execution As with TAF, FAN is not designed to hide server process failure (No more data to read from socket) Application exception handling is absolutely necessary! 28

29 Application Continuity 29

30 Application Continuity Overview Addresses temporary recoverable outages of instances, databases and network communications Transaction Guard server side component Transaction state is recorded and retrievable within database in order to ensure idempotent execution on replay (DBMS_APP_CONT.GET_LTXID_OUTCOME) Available starting with Oracle 12c Enterprise Edition Oracle 12c JDBC Replay Driver client side component Replays the failed request so that the client may simply continue As of 12.1 implemented only for JDBC thin client, in 12.2 also OCI and ODP.net Application Continuity requires RAC or RAC One Node or ADG (GG) option 30

31 Application Continuity Interaction With UCP Runtime Example AC/TG interaction with UCP Replay Re-Connect 1 Check-out connection (Request begin) 2 Associate LTXID Send LTXID to the driver 3 Work: INS/DEL/UPD/COM 4 Communication Break 5 6 Request new connection Recoverable Error SQL Exception 7 8 If safe, Replay Check the last LTXID outcome 9 Check-in connection (Request end) UCP INS DEL UPD COM Replay Buffer 31

32 Application Continuity Application Changes Application Continuity with UCP PoolDataSource pds = PoolDataSourceFactory.getPoolDataSource(); pds.setconnectionfactoryclassname("oracle.jdbc.replay.oracledatasourceimpl");... conn = pds.getconnection(); // Implicit database request begin // JDBC calls protected by Application Continuity conn.close(); // Implicit database request end Application Continuity without connection pool OracleDataSourceImpl ods = new OracleDataSourceImpl(); conn = ods.getconnection();... ((ReplayableConnection)conn).beginRequest(); // Explicit database request begin // JDBC calls protected by Application Continuity ((ReplayableConnection)conn).endRequest(); // Explicit database request end 32

33 Application Continuity Requirements Database service attributes need to be correctly specified for AC and TG 18c Transparent Application Continuity srvctl add service -failovertype [AUTO TRANSACTION] # to enable Application Continuity -commit_outcome TRUE # to enable Transaction Guard -retention # the number of seconds the commit outcome is retained -replay_init_time 900 # seconds after which replay will not be initiated -failoverretry 20 -failoverdelay 2 -notification TRUE Grant permission on the Transaction Guard package GRANT EXECUTE ON DBMS_APP_CONT TO <APP_USER> Sequences, SYSDATE/SYSTIMESTAMP, SYS_GUID can be configured to keep values on replay 33

34 Application Continuity Restrictions Application Continuity is not supported for Autonomous transactions XA transactions (12.2+ versions available for JDBC Thin driver) Active Data Gurd (ADG) with read/write DB links GoldenGate, Logical Standby databases or 3rd party replication Error handling still necessary (non-recoverable errors, replay not possible, etc.)! 34

35 Conclusions 35

36 Conclusions To achieve high availability, correct client-side configuration is crucial At least Oracle client connect timeouts should be set Be careful with Oracle re-connect timeouts (undesired side effects) Dynamic database services are key to client high availability TAF/FAN/FCF are very powerful But with some limitations and exception handling is still necessary! Application Continuity helps to transparently replay in-flight transactions Exception handling is still necessary! Small improvements introduced with the 18c version 36

37 DOAG 2018 #opencompany Booth: 3rd Floor next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit 37