Practical Tips for Oracle Enterprise Manager High Availability and Diagnostics

Transcription

1

2 Practical Tips for Oracle Enterprise Manager High Availability and Diagnostics Angeline Janet Dhanarani Senior Product Manager Oracle Enterprise Manager October 29, 2015 Deepen Chakraborty Enterprise Architect Technology Manufacturing Group Intel Corporation

3 Total Cloud Control Superior Enterprise-Grade Management Integrated Cloud Stack Management Complete Cloud Lifecycle Management Scalable, Secure Optimized, Efficient Agile, Automated 3

4 Next Release Builds on a Solid Foundation NEW: Continuous Monitoring NEW: Infrastructure Management NEW: Improved Hybrid Cloud Management Superior Enterprise-Grade Management Integrated Cloud Stack Management Complete Cloud Lifecycle Management Scalable, Secure Optimized, Efficient Agile, Automated 4

5 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 5

6 Program Agenda 1 2 Best Practices on High Availability and Disaster Recovery Monitoring and Diagnostics 6

7 Best practices on High Availability and Disaster Recovery 7

8 Enterprise Manager Architecture Overview Console BI Publisher Connectors HTTP(S) >EMCLI >EMCLI EMCLI JDBC Firewall Oracle Management Repository(OMR) Oracle Management Service(OMS) Software Library Notifications Oracle Store/MOS Targets Plug-ins Agent 8

9 High Availability Levels in Enterprise Manager Level1 : Very minimal failure protection Low implementation cost Recommended for development/test instance Level3: Offers component level failure protection Offers scalability No site level failure protection >EMCLI Users >EMCLI Active OMS OMR with RAC Users OMR OMS Load Balancer Active OMS OMR (Local Standby) Level2: Offers protection against OMS node failure Does not offer scalability Automatic failure detection and automated failover of OMS node Level4: Disaster Recovery Maximum protection (even site failures) Minimized downtime Higher Cost for Setup Users >EMCLI Recommended for 24* 7 production instance Active OMS Active Site OMR NFS Passive OMS OMR(Local Standby) Global Load Balancer /DNS Passive Site 9

10 High Availability Best Practises for Enterprise Manager High Availability Levels Configure Backups of OMS configurations emctl exportconfig Configure Backups - Emkey, Repository, Software Library Configure RAC for Repository High Availability per site Configure Additional OMSes per site Configure Highly Available Storage Configure Dataguard for data replication Configure Continuous Storage Replication Level1 Level2 Level3 Level4 10

11 Disaster Recovery Solution for Enterprise Manager Storage Replication is the only solution supported for Disaster Recovery Standby Weblogic Domain is no longer supported from EM 13c Not supported by FMW (used in Enterprise Manager 13c) Deprecation notice given in EM Migration from Standby Weblogic domain to Storage replication topology Contact Oracle Support for assistance Two supported methodologies for Storage Replication NAS Storage Replication ACFS Server Replication ( Certification in-progress) 11

12 Enterprise Manager Disaster Recovery Solution Disaster Recovery Solution Configuration with NAS Storage Replication PASSIVE Server Load Balancer of Standby data center DNS Lookup Server Load Balancer of Primary data center ACTIVE EM Repository Physical Standby Storage OMS Share OMS1 Share Swlib Share BIP Share Storage OMS Share OMS1 Share Swlib Share BIP Share Storage Continuous Replication Primary OMS DB Replication with Dataguard from Primary to Standby Additional OMS1 EM Repository 12

13 Enterprise Manager Disaster Recovery Solution Disaster Recovery Solution Configuration with ACFS Replication ACFS Volumes OMS OMS1 Swlib BIP EM Repository PASSIVE Server Load Balancer of Standby data center HANFS Configuration ASM Storage DNS Lookup ACFS Replication Server Load Balancer of Primary data center Primary OMS DB Replication with Dataguard from Primary to Standby Additional OMS1 HANFS Configuration ASM Storage ACTIVE ACFS Volumes OMS OMS1 Swlib BIP EM Repository 13

14 High Availability Solution for Enterprise Manager on Windows Platform Microsoft Windows Server Failover Clustering Cold Failover Cluster (Active/Passive) solution for OMS and Agent on Windows Certified on Windows 2012R2 14

15 Monitoring and Diagnostics 15

16 Enterprise Manager Health Monitoring Early Warning Symptoms Console Users EM Administrators Slow access to console Slow Job executions Slow login Jobs failed /suspended Slow response OMS crashes /restarts Intermittent Connectivity Issue Availability Status pending computation Target Administrators ( DBAs) Delay in notification alerts Alerts not triggered Events not triggered Patching Administrators Patch application very slow Blacking out takes more time 16

17 Enterprise Manager Health Monitoring Diagnostic Workflow Is your system capacity sized appropriately? Yes Identify the sub-system needing attention No/Not Sure Follow the Sizing Recommendation for your configuration Small/Medium/Large Monitor-the-monitor (Self monitoring features) is the starting point of this investigation Yes Has the inflow of load increased in Enterprise Has Metric data or Event data inflow increased? Manager? Yes Analyze what has changed No More targets? Do you want the More metrics? changes? Yes Would you like to optimize performance? Yes Identify the target/metric and turn off the collection. Set appropriate alert threshold. Identify the sub-system and introduce RAC services 17

18 Enterprise Manager Health Monitoring Diagnostic Workflow Is your system capacity sized appropriately? Yes Identify the sub-system needing attention No/Not Sure Follow the Sizing Recommendation for your configuration Small/Medium/Large Monitor-the-monitor (Self monitoring features) is the starting point of this investigation Yes Has the inflow of load increased in Enterprise Has Metric data or Event data inflow increased? Manager? Yes Analyze what has changed No More targets? Do you want the More metrics? changes? Yes Would you like to optimize performance? Yes Identify the target/metric and turn off the collection. Set appropriate alert threshold. Identify the sub-system and introduce RAC services 18

19 Is your system capacity sized appropriately? Refer Sizing Recommendations Ensure Java Heap Size settings are optimal for OMS Ensure Hardware requirements are optimal Memory and Storage Ensure database parameters settings are compliant for your configuration Ensure network latency between OMS and repository is less than 1ms Agent to OMS communication can tolerate huge latencies Heap Size Parameters OMS_HEAP_MIN OMS_HEAP_MAX OMS_PERMGEN_MIN OMS_PERMGEN_MAX Setup -> Manage Cloud Control >Repository 19

20 Enterprise Manager Health Monitoring Diagnostic Workflow Is your system capacity sized appropriately? Yes Identify the sub-system needing attention No / Not sure Follow the Sizing Recommendation for your configuration Small/Medium/Large Monitor-the-monitor (Self monitoring features) is the starting point of this investigation Yes Has the inflow of load increased in Enterprise Has Metric data or Event data inflow increased? Manager? Yes Analyze what has changed No More targets? Do you want the More metrics? changes? Yes Would you like to optimize performance? Yes Identify the target/metric and turn off the collection. Set appropriate alert threshold. Identify the sub-system and introduce RAC services 20

21 Does the Enterprise Manager Repository need attention? Check out Top Wait Events metrics for Enterprise Manager Repository Log File Sync wait event should not appear in the top 5 waits Correlate with AWR report findings to see if the impact is from specific sub-system. Could also indicate increase in load inflow. DB CPU should be topmost wait event Expect to see I/O related wait events but only Write Check the SGA Buffer Cache Size if there read wait events in top 5 For an Ideal repository host :single instance or RAC cluster No more than 40% CPU utilization on average across all nodes in RAC Ideal Values for Log File Sync Wait Time Engineered System ~1 millisec Non-engineered System ~5 millisec or less Onwards 21

22 Does the Job sub-system need attention Monitor EM Jobs Service performance charts Look for Growing backlog in Jobs Steps Scheduled at repository Look for consistently High Job Dispatcher processing time (%) but Low throughput. Indicates processing bottleneck. Consistently growing, ensure job pool threads are sized per Sizing recommendation Correlate with cluster Wait times in AWR report. 22

23 Does the Loader sub-system need attention? Loader throughput indicating processing time Look for consistent drastic increase over a time period Agent backlog Check for consistent increase in Back-off requests / Backlogs Correlate with Loader Statistics Report Check SLB configuration ( Recommended: Round robin algorithm) Increase the loader pool size uniformly across OMS If there is consistent backoffs in range of hundreds per hour Setup -> Manage Cloud Control >Health Overview 23

24 Enterprise Manager Health Monitoring Diagnostic Workflow Is your system capacity sized appropriately? Yes Identify the sub-system needing attention No/ Not Sure Follow the Sizing Recommendation for your configuration Small/Medium/Large Monitor-the-monitor (Self monitoring features) is the starting point of this investigation Yes Has the inflow of load increased in Enterprise Has Metric data or Event data inflow increased? Manager? Yes Analyze what has changed No More targets? Do you want the More metrics? changes? Yes Would you like to optimize performance? Yes Identify the target/metric and turn off the collection. Set appropriate alert threshold. Identify the sub-system and introduce RAC services 24

25 Has the inflow of load increased on Enterprise Manager? Crucial for well-tuned Enterprise Manager Incoming Metric Data Incoming Event Data Pro-active measure to check increase in count of targets and agents Repository-side Metric Extension to alert if number of targets and agents increases above the recommended sizing guidelines for the installed configuration Set alert threshold to the installed configuration EVAL / SMALL / MEDIUM / LARGE Refer Appendix for configuration Re-evaluate Sizing configuration when alerted 25

26 Key Indicators of increasing inflow of Metric Data Uncontrolled metric data generation causes significant performance degradation of Enterprise Manager 26

27 Fine tuning incoming metric data Best practice recommendation : Collect only what you need Evaluate what the system is collecting and how frequently it is collecting Enable /Disable metric to prevent unwanted metrics from being collected Use monitoring templates to turn it off Identify out of date monitoring templates, metric extensions to reduce inflow Reduce the collection frequency of non-critical metrics Weighted cost of metric collection analyzed and provided out of box Daily analysis for each target and each metric Outlier targets and metrics violating upper bound identified daily SDK views available for configuring BIP report 27

28 Key indicators of increasing inflow of Event data Look out for consistent drastic increase in Metric alert backlog Metric Collection Errors Notification Delivery Backlog Setup -> Manage Cloud Control >Repository >Metrics 28

29 Fine tuning Incoming Event data Events are relatively expensive resource to load on repository Not expensive on agent but when loaded into repository, massive triggers to do lot of processing Notifications processing Callback processing Reduce the metric alerts by setting appropriate warning and critical threshold Set the number of occurrences to rule out sporadic alerts 29

30 Enterprise Manager Health Monitoring Diagnostic Workflow Is your system capacity sized appropriately? Yes Identify the sub-system needing attention No/Not Sure Follow the Sizing Recommendation for your configuration Small/Medium/Large Monitor-the-monitor (Self monitoring features) is the starting point of this investigation Yes Has the inflow of load increased in Enterprise Has Metric data or Event data inflow increased? Manager? Yes Analyze what has changed No More targets? Do you want the More metrics? changes? Yes Would you like to optimize performance? Yes Identify the target/metric and turn off the collection. Set appropriate alert threshold. Identify the sub-system and introduce RAC services 30

31 Tuning Critical Subsystems in Enterprise Manager Optimizing Performance and Improving Scalability with RAC Service Ping Service Repository Scheduler Jobs: Rollups Job Subsystem Events Subsystem 31

32 Tuning Critical Subsystems in Enterprise Manager Improving performance and scalability Ping Service Performance is crucial for determining Target Availability Not CPU heavy but processing is very frequent Every minute every agent pings OMS and updates row in EMD table Multiple fold increase in performance if Ping Service is pinned to single instance RAC node Single node service get rid of cluster wait events General Recommendation Anybody running RAC for Enterprise Manager repository should implement ping service and pin to single RAC node Configuration Steps in Appendix 32

33 Tuning Critical Subsystems in Enterprise Manager Improving performance and scalability Repository scheduler Jobs :Metric Rollups Aggregation mechanism for historical trending and capacity planning Background process and the data is not required immediately I/O intensive operation If this job is running for more than 6 hours, general rule to add another thread if the database can handle the increased load from these threads. General Recommendation Configure rollup service only if there are cluster wait events Check if single instance RAC node can handle large I/O volume. Pin rollup service to single instance RAC node Configuration Steps in Appendix 33

34 Tuning Critical Subsystems in Enterprise Manager Improving performance and scalability Job subsystem Jobs are comparatively resource-intensive at the database If there lot of running jobs, expect to see application locks wait events, transaction locking wait events in AWR report in repository Job system uses the locks to maintain sequence Normal if it consumes 5-8% of the waiting time. Problem if over 20-30%. General Recommendation Improve performance by pining Job Service to 2Node RAC instances. Events subsystem Events are comparatively resource-intensive at the database General Recommendation Improve performance by pining Events Service to single RAC node instances. Configuration Steps for Job Service and Event service in Appendix 34

35 Appendix 35

36 Configuring RAC services for Ping Service Create database service and set affinity to only run on single RAC node Create database service ping and set one of RAC instance as primary instance in -r srvctl add service -d <dbname>-s ping -r <primary instance> -a <the the other instances> -y automatic Execute the following DBMS_SCHEDULER jobs As sys user, execute DBMS_SCHEDULER.create_job_class( job_class_name => PING', service => ping') GRANT EXECUTE ON sys.ping TO sysman; As sysman user, execute DBMS_SCHEDULER.SET_ATTRIBUTE ( name => 'EM_PING_MARK_NODE_STATUS', attribute => 'job_class', value => PING') As sysman user, execute DBMS_SCHEDULER.SET_ATTRIBUTE ( name => 'EM_REPOS_SEV_EVAL', attribute => 'job_class', value => PING') As sysman user, execute GC_SCHED_JOB_REGISTRAR.SET_JOB_CLASS('EM_REPOS_SEV_EVAL', 'PING') As sysman user, execute GC_SCHED_JOB_REGISTRAR.SET_JOB_CLASS('EM_PING_MARK_NODE_STATUS', 'PING') Set the connect string with ping service name to emctl property "oracle.sysman.core.omsagentcomm.ping.connectionservice.connectdescriptor Sample : emctl set property -name "oracle.sysman.core.omsagentcomm.ping.connectionservice.connectdescriptor" -value "\(DESCRIPTION=\(ADDRESS_LIST=\(ADDRESS=\(PROTOCOL=TCP\)\(HOST=xxx.us.oracle.com\)\(PORT=1521\)\)\)\(CONNECT_DATA= \(SERVICE_NAME=ping\)\)\)" 36

37 Configuring RAC services for Rollups Create database service and set affinity to only run on single RAC node Create database service rollup and set one of RAC instance as primary instance in -r srvctl add service -d <dbname>-s rollup -r <primary instance> -a <the the other instances> -y automatic srvctl start service -d <dbname>-s rollup srvctl status service -d <dbname> As sys user, execute DBMS_SCHEDULER.create_job_class( job_class_name => 'ROLLUP', service => 'rollup') GRANT EXECUTE ON sys.rollup TO sysman; As sysman user, execute DBMS_SCHEDULER.SET_ATTRIBUTE ( name => 'EM_ROLLUP_SCHED_JOB', attribute => 'job_class', value => 'ROLLUP') As sysman user, execute GC_SCHED_JOB_REGISTRAR.SET_JOB_CLASS('EM_ROLLUP_SCHED_JOB', 'ROLLUP') 37

38 Configuring RAC services for Jobs subsystem Create database service and set affinity to run on 2 node RAC instance Create database service emjob and set two of the RAC instances as primary instance in -r srvctl add service -d <dbname>-s emjob -r <primary instances> -a <the the other instances> -y automatic Execute the following DBMS_SCHEDULER jobs As sys user, execute DBMS_SCHEDULER.create_job_class( job_class_name => EMJOB', service => emjob ') GRANT EXECUTE ON sys.emjob TO sysman; As sysman user, execute DBMS_SCHEDULER.SET_ATTRIBUTE ( name => ' EM_JOBS_STEP_SCHED ', attribute => 'job_class', value => EMJOB') As sysman user, execute DBMS_SCHEDULER.SET_ATTRIBUTE ( name => ' EM_JOB_PURGE_POLICIES ', attribute => 'job_class', value => EMJOB') As sysman user, execute GC_SCHED_JOB_REGISTRAR.SET_JOB_CLASS('EM_JOBS_STEP_SCHED', 'EMJOB') As sysman user, run GC_SCHED_JOB_REGISTRAR.SET_JOB_CLASS('EM_JOB_PURGE_POLICIES', 'EMJOB ) INSERT INTO MGMT_PARAMETERS(parameter_name, parameter_value) VALUES ('EM_jobs_step_sched_job_class', 'EMJOB') Set the connect string with ping service name to emctl property "oracle.sysman.core.omsagentcomm.ping.connectionservice.connectdescriptor Sample : emctl set property -name "oracle.sysman.core.jobs.conn.service" -value "\(DESCRIPTION=\(ADDRESS_LIST=\(ADDRESS=\(PROTOCOL=TCP\)\(HOST=xxx.us.oracle.com\)\(PORT=1521\)\)\)\(CONNECT_DATA= \(SERVICE_NAME=emjob\)\)\)" 38

39 Configuring RAC services for Events subsystem Create database service and set affinity to only run on single RAC node Create database service event and set one of RAC instance as primary instance in -r srvctl add service -d <dbname>-s event -r <primary instance> -a <the the other instances> -y automatic Set the connect string with ping service name to emctl property "oracle.sysman.core.events.connectdescriptor Sample : emctl set property -name "oracle.sysman.core.events.connectdescriptor" -value "\(DESCRIPTION=\(ADDRESS_LIST=\(ADDRESS=\(PROTOCOL=TCP\)\(HOST=xxx.us.oracle.com\)\(PORT=1521\)\)\)\(CONNEC T_DATA=\(SERVICE_NAME=event\)\)\)" 39

40 Repository-side Metric Extension to compute increase in target and agents count SELECT outer1.target_guid, CASE WHEN outer1.targetcount < 100 AND outer1.agentcount < 10 THEN 'EVAL' WHEN outer1.targetcount < 1000 AND outer1.agentcount < 100 THEN 'SMALL' WHEN outer1.targetcount < AND outer1.agentcount < 1000 THEN 'MEDIUM' ELSE 'LARGE' END AS EvnAlert FROM (SELECT inner2.target_guid, inner1.targetcount, inner1.agentcount From (SELECT Count(*) TargetCount, SUM(CASE WHEN target_type='oracle_emd' THEN 1 ELSE 0 END) AS AgentCount FROM mgmt$target) inner1, (SELECT Target_Guid FROM mgmt$target WHERE target_type='oracle_emrep') inner2 ) outer1 Monitoring Setting: 40

41 Proxy servers: Configuring High Availability Can configure multiple proxies for communication from OMS to Agents for High Availability Proxy Server is a target If Proxy Server is down, OMS uses the alternative proxy server Addition of proxies no longer require OMS bounce Can configure different proxy servers for agents in different networks Agents are associated to proxy servers by name or pattern during proxy server creation CORPORATE FIREWALL OMS1 CORPORATE OMS2 NETWORK C1 Proxy Server 3 C3 Proxy Server 1 Proxy Server 2 C2 AGENTS NOT IN ANY NETWORK A1 A2 A3 A4 AGENTS IN NETWORK1 B1 B3 B2 B4 AGENTS IN NETWORK2 41

42 Highly Available Enterprise Manager 12c implementation using Intel Zeon Based Servers Deepen Chakraborty Enterprise Architect Technology Manufacturing Group Intel Corporation Oct, 29th

43 Agenda Intel's DB Monitoring profile and challenges Enterprise Manager 10g Grid Control Vs Enterprise Manager 12c Cloud Control Centralized Enterprise Manager 12c Cloud Control Architecture Critical subsystems monitoring results Lessons Learned Conclusion 43

44 Intel s DB Monitoring Profile & Challenges Intel s Factory Automation Databases used for making critical Manufacturing decisions. (Operational and Planning, Engineering Analysis, Process control) Automation Databases include both Mission Critical OLTP and Mission Important DSS type systems (Ranging from few hundred Gigs up to 30 TB) spread across US and Asia subcontinents Applications clients include both vendor and homegrown apps (ODP.NET based) The monitoring of databases was done using grid control around 14 implementations It was a very hard to maintain 14 installations to have same setup for monitoring and alerting Patching and monitoring implementations took longer time which is resource intensive 44

45 Oracle 10g Grid Control Enterprise Manager 12c Cloud Control Site2 in China Site1 in Oregon Site3 in China Site1 in China Site3 in Malaysia Site2 in Malaysia Site1 in Vietnam Site4 in Malaysia Site1 in Malaysia Site2 in Oregon Site1 in Israel Site2 in Arizona Site1 in Arizona 45

46 Enterprise Manager 12c Architecture Box Diagram 46

47 Centralized Enterprise Manager 12c Capabilities Able to monitor or use all the features of 11gR2 and 12c databases using EM12c cloud control e.g. real time query performance, real time ADDM, reporting and metrics capabilities etc. High Availability (Database and Oracle Management Server) Fully Automated Target Setup using dynamic group feature Robust Security (super user, admin user, read-only user) Target control from target nodes instead of cloud control nodes No central dependency of the target databases high availability(fsfo capability for targets) and backup recovery 47

48 EM12c Reference Architecture Stack Component Technology Description OMR Oracle 11g R2 64 bit ( P11 ) HA/DR of OMR Broker Managed Oracle Data Guard (Max Availability Mode) in a Fast Start Failover configuration Oracle Restart (Part of Grid Infra Install) Observer For OMR Backup of OMR On Recovery Manager Node; Custom script (OBMAN) to automatically move the Observer away from the PRY data center upon role transition RMAN integration with Cloud Control ; Site Based RCAT for all OLTP Apps; Basic Compressed Backup sets Wkly Full and Daily Incrementals; Backups to run on PRY (with Block Change Tracking) OMS Oracle Enterprise Manager Cloud Control OMS HA Active/Passive Microsoft Failover Cluster for Windows 2008 R2 with HP Remote Copy OMS Backup Using Script for configuration OS Storage/File System 64 bit Windows 2008 R2 Server HP 3PAR SAN; OMR App specific Data/Index, Backup, Redo, Archive, Control ASM Oracle and Grid Infrastructure (ASM) Binaries NTFS OMS File system NTFS Network Teamed Private Network 1 GigE amongst the OMR Nodes ; Distance between Data centers typically < 1 Km; Network Latency 2ms. Public Network 10 GigE between the OMR and OMS Cluster Nodes Server Config Windows 2008 R2 Service Pack1; (BL460c Gen 8 w/ 2 Sockets - 8 core Intel Xeon processors ; 192G RAM, Hyper Threading Enabled 48

49 49 OMS Performance Capture : 1

50 50 OMS Performance Capture : 2

51 51 OMS Performance Capture : 3

52 52 OMS Performance Capture : 4 Top 25 Metric Data loading

53 53

54 54 OMR Performance Capture

55 Lessons Learned Intel created Homegrown custom scripts/utilities to: Auto Relocate the FSFO Observer (where the current PSB s Data Center is) upon role transition Upload OMA dynamic properties so that Cloud Control reflects real state of PRY/PSB If central OMS/OMR goes down a home grown script for site database monitoring Oracle Development Team helped in providing fix/patch to: Automatically transfer EM12c RMAN Jobs to the new primary upon Data Guard planned switchover and unplanned failover Helped to implement workaround for Recovery Catalog HA, enhancement will be available next EM Release Has provided custom OMS Graceful failover script on Microsoft Windows 2008 R2 Failover Cluster setup 55

56 Conclusion Oracle Enterprise Manager 12c provides seamless integration of monitoring of Oracle Stack includes dataguard management with FSFO, Backup and Recovery Oracle EM12c can monitor the databases geographically sparse locations HA and security implementations of Enterprise Manager 12c stack provides seamless monitoring of databases 56

57 Thanks to TMG Engg Team for the contributions. Thank You for attending the session.

58 58

59 59

60 Data Center 1 Public network Application Database Blue Print OBSERVER/OB-MAN on OEM GC Node OBSERVER/OB-MAN on OEM GC Node Data Center 2 DB Instance Grid Infra Primary Broker Enabled Data Guard/SYNC DB Instance Grid Infra Standby Storage Network Private network Storage Network ASM Data & FRA Mirrored LUN of REDO/Control/Arch logs for Double Failure Coverage ASM Data & FRA 60

61 Observer and Recovery Catalog Database In Each Site Data Center 1 OBSERVER/OB-MAN OBSERVER/OB-MAN Data Center 2 Public network DB Instance DB Instance Grid Infra Primary Broker Enabled Data Guard/SYNC Grid Infra Standby Storage Network Private network Storage Network ASM Data & FRA ASM Data & FRA Data Guard (Broker Enabled) Fast -Start Failover Zero Data Loss Configuration (SYNC/Max Availability Mode/Real Time Apply). For Different version of databases we will have multiple observers which will run from different ORACLE_HOME 61

62 Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 62

63 63

64