Oracle Database Service High Availability with Data Guard?

Transcription

1 Oracle Database Service High Availability with Data Guard? Robert Bialek Senior Principal doag2017

2 Who Am I Senior Principal Consultant and Trainer at Trivadis GmbH in Munich. Master of Science in Computer Engineering. At Trivadis since Trivadis Partner since Focus: Data and Service High Availability, Disaster Recovery. Architecture Design, Optimization, Automation. New Technologies (Trivadis Technology Center). Open Source. Technical Project Leadership. Trainer: O-GRINF, O-RAC, O-DG

3 Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems

4 With over 600 specialists and IT experts in your region. COPENHAGEN HAMBURG 14 Trivadis branches and more than 600 employees. 200 Service Level Agreements. Over 4,000 training participants. DÜSSELDORF Research and development budget: CHF 5.0 million. FRANKFURT Financially self-supporting and sustainably profitable. BASEL FREIBURG STUTTGART BRUGG ZURICH MUNICH VIENNA Experience from more than 1,900 projects per year at over 800 customers. GENEVA BERN LAUSANNE

5 Technology on its own won't help you. You need to know how to use it properly

6 Database Service High Availability Goal Increase database service uptime, by: eliminating any single point of failure to avoid unplanned outages. minimizing the effect of an unplanned outage on the end user (automatic failover). reducing downtimes during planned outages. Storage Consider the whole SW/HW stack. Database Server(s) Application Server(s) Clients Find the best cost/risk ratio. Downtime Best cost/risk ratio Effort Costs Complexity Availability

7 Database Service High Availability Options Cluster primarily used option for service high availability: Real Application Clusters. RAC One Node. Cold Failover Cluster. Data Replication used mostly for data, rather than service high availability: Data Guard (Fast-Start Failover/Global Data Services). GoldenGate (Global Data Services). Other replication technologies. HA SPOF HA HA

8 Agenda 1. Introduction 2. Configuration 3. Special Cases 4. Conclusions

9 Introduction

10 Database Service HA with Data Guard? Introduction Yes, it can also be used for service high availability: Planned downtimes manual switchover. Unplanned downtimes fast-start failover or manual failover. Why might we consider Data Guard for service high availability: Less complex than a cluster installation. Infrastructure requirements not that high (even local storage is sufficient). Not subject to additional license fees (EE license assumed). Data Guard Additionally, many other advantages: data high availability, snapshot standby, potentially rolling upgrade capability,... But, with some restrictions we need to consider... Primary Standby FSFO Configuration

11 Database Service HA with Data Guard? Big Picture Database Clients Master Observer Required Backup Observers (optional, 12.2) Ping Ping Ping Primary RW Service Transparency required (failover/ switchover)... Primary Target Failover Standby Candidate Target Failover Standbys (optional, 12.2)

12 Database Service HA with Data Guard? Monitoring Reconnect interval expired Logoff Observer Failover condition detected ObserverReconnect property set and reached Timeout Connect re-tries DGMGRL Threads W000 B001 P001 S001 Failover SLEEP ~ 3sec. Connect SLEEP ~ 3sec. PING PRIMARY Enter PING State PING TARGET STANDBY

13 Configuration

14 Data Guard Protection Modes with FSFO Prerequisites FSFO: Guaranted zero data loss. FSFO: Data loss possible. FSFO: Guaranted zero data loss. MaxAvailability (10.2+) MaxPerformance (11.1+) MaxProtection (12.2) LogXptMode=SYNC or FASTSYNC (12.1+) FastStartFailoverTarget(*) Flashback Database Mostly used protection mode LogXptMode=ASYNC FastStartFailoverTarget(*) FastStartFailoverLagLimit Flashback Database All Protection Modes LogXptMode=SYNC FastStartFailoverTarget(*) Flashback Database Recommended: at least 2 STDBY DBs (protection mode downgrade!) DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverThreshold = <xy>; DGMGRL> ENABLE FAST_START FAILOVER; Value in seconds

15 Fast-Start Failover Observer (1) Monitoring component, initiates a failover procedure. In 12.2, up to 3 observers (in background) can be started: One master and up to two backup (standby) observers. DGMGRL> START OBSERVER OBS1.TRIVADIS.COM IN BACKGROUND FILE IS '$ADMIN_SID/fsfo_$ORACLE_SID.dat' LOGFILE IS '$ADMIN_SID/fsfo_$ORACLE_SID.log' CONNECT IDENTIFIER IS <Alias1>.TRIVADIS.COM; In older releases: Only one running observer (HA needs to be adressed). Ping Observer PRIMARY Oracle wallet required W000 P001 B001 S001 TARGET Failover Standby nohup dgmgrl -logfile $ADMIN_SID/fsfo_$ORACLE_SID.log <<EOD & CONNECT $CONNECT_DATA START OBSERVER FILE='$ADMIN_SID/fsfo_$ORACLE_SID.dat'; EOD

16 Fast-Start Failover Observer (2) Observer Fast start-failover is initiated by the master observer to the target standby database, if one of the following conditions is detected: observer and the target standby database cannot reach the primary database (default: ObserverOverride= FALSE ). user-configurable condition is met. Ping PRIMARY DBMS_DG.INITIATE_FS_FAILOVER function has been executed. W000 P001 B001 S001 TARGET Failover Standby Additonally, other pre-condidtions enforced by a protection mode need to be fulfilled: MaxProtection/MaxAvailability: target failover standby is in SYNC. MaxPerformance: FastStartFailoverLagLimit not reached for the target failover standby

17 Data Guard: Role-Based Services For a Data Guard system, we need a role-based service, that is running only if database has a specific role: Read-write service on a primary database. Optionally, a service on standby databases for reporting. Optionally, a service on snapshot standby databases. Database Clients? To accomplish this task: Use Oracle Grid Infrastructure role-based services. Create your own AFTER STARTUP ON DATABASE trigger. Service R/W Service R/O [SNAP]

18 Data Guard: Example Role-Based Services Example role-based services with Grid Infrastructure. srvctl add service -db DB_SITE1 service SRV_RW.trivadis.com -role PRIMARY srvctl add service -db DB_SITE1 service SRV_RO.trivadis.com -role PHYSICAL_STANDBY srvctl add service -db DB_SITE1 -service SRV_SP.trivadis.com -role SNAPSHOT_STANDBY Services are started only if database and service role match. SvcAgent::start 680 query_db_role SvcAgent::start 710 not starting service srv_rw Role mismatch - Service role:primary, current DB role:physical_standby Depending on used client HA features (TAF, FAN/FCF, AC) additional service properties need to be specified

19 Client-Side Configuration Main Problems To Address CASE 1 CASE 2 New network session (connect) Database Clients Already established network session Database Clients 2 Connect attempts 4 Client failover 1 Connected 5 Client failover 3 Wait for connect timeout Problem Problem 3 Re-connect attempts 4 Wait for re-connect timeout Problem Problem 1 IP not reachable (server/network/ issue) 2 IP not reachable (server/network/ issue)

20 New Oracle Net Session Connect Timeout (1) sqlnet.ora parameters (OCI, ODP.net) Applies to each IP that a host name resolves to! All Oracle client versions supported. TCP.CONNECT_TIMEOUT=3 #default 60 sec. SQLNET.OUTBOUND_CONNECT_TIMEOUT=5 #no default For clients >=11.2: 1 LSNR 2 Three-way handshake Oracle Net 3 LSNR OLTP.trivadis.com = (DESCRIPTION = (FAILOVER=ON) (LOAD_BALANCE=OFF) Introduced in (CONNECT_TIMEOUT=5)(RETRY_COUNT=3)(RETRY_DELAY=1)(TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP )(HOST = italy )(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP )(HOST = sweden )(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = OLTP_RW.trivadis.com)))

21 New Oracle Net Session Connect Timeout (2) JDBC Thin driver TRANSPORT_CONNECT_TIMEOUT is available beginning with 12.2 version To use RETRY_COUNT with , patch is required (BUG ) =(FAILOVER=ON)(LOAD_BALANCE=OFF)" + "(CONNECT_TIMEOUT=3)(RETRY_COUNT=10)(RETRY_DELAY=1)" + "(ADDRESS_LIST = " + "(ADDRESS = (PROTOCOL = TCP )(HOST = blue.trivadis.com )(PORT = 1521)) " + "(ADDRESS = (PROTOCOL = TCP )(HOST = brown.trivadis.com )(PORT = 1521))) " + "(CONNECT_DATA = (SERVICE_NAME = sales_rw.trivadis.com)))"); JDBC Thin clients can alternatively use the following driver property (ms) Overrides CONNECT_TIMEOUT from address description parameters Properties prop = new Properties(); prop.put(oracle.net.ns.sqlnetdef.tcp_conntimeout_str, ""+3000); ods.setconnectionproperties(prop);

22 Established Oracle Net Session Re-Connect Timeout 2 Break established network connection without waiting for long TCP timeouts (>15 min.) In most cases no VIPs in use! Using the following parameters is not a good idea: SQLNET.RECV_TIMEOUT=30 #no default value, OCI driver SQLNET.SEND_TIMEOUT=30 #no default value, OCI driver LSNR P1 3 Timeout 1 Oracle Net 4 LSNR Client failover prop.put ("oracle.jdbc.readtimeout", "5000"); //5000ms, JDBC Thin driver Better solution: If possible use: Fast Application Notification/Fast Connection Failover. Tuning OS kernel parameter tcp_retries2 might be also an alternative

23 Client HA Features Overview Transparent Application Failover: Can be used with Data Guard the same way (advantages/disadvantages) as with a cluster. Fast Application Notification/Fast Connection Failover: Oracle Grid Infrastructure is required to register with ONS. Comparing to RAC only rapid notification about up/down events, no workload balancing. Application Continuity can be used with Data Guard the same way as with a cluster: But requires RAC or RAC One Node or ADG (GG) option. More about this topic: DOAG 2016 presentation: Oracle Client Failover - Under the Hood

24 Special Cases

25 Special Cases with FSFO: Candidate Targets Standby Starting with 12.2, many candidate fast-start failover target databases can be specified, but switchover or FSFO works only to the current target standby database. DGMGRL> EDIT DATABASE db_site1 SET PROPERTY FastStartFailoverTarget = Threshold: Target: 60 seconds db_site2 Candidate Targets: db_site2,db_site3 Observers: 'db_site2,db_site3'; (*) obs1.trivadis.com obs2.trivadis.com DGMGRL> SWITCHOVER TO db_site3; Error: ORA-16655: specified standby database not the current fast-start failover target standby. Current target depends on many conditions FSFO Switchover db_site1 db_site2 db_site

26 Special Cases: Switchover DelayMins >0 Version X: recovery re-started with NODELAY option. Version X: recovery waits until DelayMins reached! OPEN_MODE Primary CLOSED BY SWITCHOVER Standby MOUNTED Application RW service outage within DelayMins time-frame! Application connect attemps fail with ORA-16456: switchover to standby in progress or completed Version : Switchover is not possible. Error: ORA-16672: switchover not permitted to standby database with non-zero DelayMinsFailed. PRIMARY DelayMins>0 TARGET Failover Standby

27 Special Cases with FSFO: Master Observer Failure Master Observer Backup Observers After about 31 sec. the master observer is changed, i.e.: available backup observer is promoted to the master role. Note: to perform a master change, the primary database needs to be available! Data Guard Broker initiated a master observer switch since the current master observer cannot reach the primary database For maintenance, a master change can be performed manually. PRIMARY Logged on the primary database DGMGRL> SET MASTEROBSERVER TO obs2.trivadis.com; Sent the proposed master observer to the data guard broker configuration. Please run SHOW OBSERVER to see if master observer switch actually happens

28 Special Cases with FSFO: Failover Target Failure Observer After about 10 sec. the target failover standby is changed, i.e.: a candidate target failover standby observer is promoted to the current target role. Permission granted to the primary database for target switch. The primary database returned to SYNC/NOT LAGGING state with the standby database db_site3. db_site1 db_site2 db_site3 Note: to perform the target failover standby change, the primary database and the master observer need to be available! If the master observer fails at the same time: LGWR: FSFO SetState("UNSYNC", 0x2) operation requires an ack Primary database will shutdown within 30 seconds if permission is not granted from Observer or FSFO target standby to proceed

29 Special Cases with FSFO: Master Observer/Primary If the primary and the master observer fail: No failover is initiated to a candidate standby. From a backup observer log file: Master Observer Backup Observers Ready to failover check on standby returned RFS_NON_MSTOB. Command READY_TO_FSFO to thread S024 returned status=0 Fast-Start Failover is not possible because this observer is not the master. db_site1 db_site2 db_site3 If the master observer is started at a later time, it waits until FastStartFailoverThreshold timeout is reached again and fails over to the current target standby

30 Special Cases with FSFO: Private Redo Network If the public network on the primary server fails: Broker configuration property: ObserverOverride=FALSE. No failover (HB over private network still works!). Fast-Start Failover is not possible because primary last contacted the standby within FastStartFailoverThreshold seconds HB Master Observer Public Network In this network configuration consider using: db_site1 db_site2 db_site3 Private Network DGMGRL> EDIT CONFIGURATION SET PROPERTY ObserverOverride='TRUE';

31 Master Observer, Primary/Standby DB: Location? (1) For DR HA service protection, do not place the primary and the master observer in the same data center! Data Center 1 Data Center 2 Master Observer Backup Observer No automatic failover! Primary Target Standby No RW application service available!

32 Master Observer, Primary/Standby DB: Location? (2) For DR HA service protection, do not place the primary and the master observer in the same data center! Data Center 1 Data Center 2 Backup Observer Master observer placement correction monitoring: DGMGRL> SET MASTEROBSERVER TO Master Observer To relocate: Disable & Enable FSFO Automatic failover! Potential placement problem! Candidate Target Standby Primary Target Standby RW application service available!

33 Conclusions

34 Conclusions (1) Can Data Guard be a good solution for database service high availability? Yes, with a fast-start failover configuration. Anyway, it is not a replacement for a cluster but rather an alternative. Careful business requirements analysis is necessary. Advantages: It offers a good service high availability, in addition to excellent data high availability and some other features. Fairly simple solution (setup and operation). Not subject to additional license fees (EE license assumed). Infrastructure requirements not that high as for a cluster. Most client HA features can be used the same way as with a cluster

35 Conclusions (2) Disadvantages: Component placement is critical and requires customized monitoring scripts. Some technical restrictions like network latencies (SYNC), flashback database or force logging might limit Data Guard in this area. Re-connect timeouts without FAN/FCF (no VIPs)

36 DOAG 2017 #opencompany Booth: 3rd Floor next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit