Transcription

1

2 Infrastructure at your Service. Oracle Grid Infrastructure Cold failover cluster

3 Infrastructure at your Service. About me Jérôme Witt Senior Consultant Mobile Page 2

4 Who we are dbi services Experts At Your Service > 40 specialists in IT infrastructure > Certified, experienced, passionate Based In Switzerland > 100% self-financed Swiss company > Over CHF 6 mio. turnover Leading In Infrastructure Services > More than 100 customers in CH, D, & F > Over 40 SLAs dbi FlexService contracted dbi services is hiring in Basel & Zürich (career@dbi-services.com) Page 3

5 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Oracle Clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover dbi-services 6. User experience 7. Core Message 8. Homework Page 4

6 Introduction > Products > Oracle cluster software stack > Licensing Page 5

7 Introduction - products Cold failover cluster > Very popular high availability solution > Database active only at one node at a time > Large choice of vendors > Service Guard > Veritas cluster > Open Source solutions Page 6

8 Introduction Oracle cluster software stack Oracle Grid Infrastructure, Clusterware do we speak the same language? > Oracle Grid Infrastructure is a suite of software which includes many components like ASM,ACFS, Database Qualify of service > Oracle Clusterware is a generic, general purpose clustering solution for all applications RAC Oracle ASM/ACFS Oracle Clusterware HA_App1 HA_App2 Consolidated pool of storage with Oracle Automatic Storage Management (ASM) Page 7

9 Introduction Licensing Oracle Clusterware is free of charge > Reference Database Licensing information 12.1 > For any kind of application J > Support: only if the server is running an Oracle product Database licensing > Reference: Licensing Data Recovery Environments > Method Failover data recovery > Multiple nodes have access to one single storage (SAN) > Includes the right to run the licensed program on an unlicensed spare computer for up to a total of 10 separate days > Downtime for maintenance purposes counts towards! > If failover period exceeds 10 days then the failover node must be licensed Page 8

10 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Oracle Clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover 6. User experience 7. Core Message 8. Homework Page 9

11 Oracle Grid Infrastructure main components > Network > Shared storage > Single Client Access Name > The Big picture Page 10

12 Oracle Grid Infrastructure main components - 1 Network - Each node must have at least two interfaces > Private network interconnect > Used for communication between all cluster nodes to maintain the integrity of the cluster (CSS deamon) and by database instances for global cache management (Oracle RAC) > Critical component which shall be fault tolerant > Oracle Clusterware support one to four interfaces (HAIP) > Public network node IPs, node VIPs.. > Required on each node by Oracle Clusterware for node applications Or applications managed by Oracle Clusterware Shared storage cluster quorum > OCR is a file that contains the configuration information and status of the cluster. (ie: DBCA uses the OCR for storing configuration) > Voting files are used to monitor cluster node status Page 11

13 Oracle Grid Infrastructure main components - 2 Shared storage - Oracle ASM/ACFS > ASM is the preferred storage manager for all database files > Volume manager and a file system > 3 levels of mirroring (EXTERNAL,NORMAL,HIGH) > ACFS is a file system on top of ASM for all non-database files Single Client Access Name (SCAN) > Single name for client to access any database running on the cluster jdbc:oracle:thin:ora-de-ch-scan.jew.local:1521/cdbutf8 > Default- composed of 3 IP addresses and 3 scan listener > Node assignment automatically controlled by Oracle Clusterware > Database(s) register with the SCAN listeners remotely remote_listener=ora-de-ch-scan.jew.local:1521 > INIT parameter local_listener is automatically updated by the agent process oraagent.bin Page 12

14 The Big picture jdbc/sqlnet/oci SCAN Address SCAN lsnr 1 SCAN lsnr 2 SCAN lsnr 3 tcp tcp DB11 +ASM1 +ASM2 VIP1 VIP2 VIP3 NIC1 NIC2 PUB NIC3 NIC4 HAIP NIC1 NIC2 PUB NIC3 NIC4 HAIP SCANVIP1 hba1 hba2 SCANVIP2 SCANVIP3 hba1 hba2 udp/infiniband/rds data a bunch of disks ocr/voting Page 13

15 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Oracle Clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover dbi-services 6. User experience 7. Core Message 8. Homework Page 14

16 Oracle Clusterware resource management > Types > Resources > Attributes Page 15

17 Oracle Clusterware resource management - 1 Oracle Clusterware use types to organize similar resources > Every resource must have a type > Manage (inherit) only necessary resource attributes Oracle Clusterware managed resource (CRSD) Resource Local listener(s), ASM Database(s),VIP, SCAN Resource Attributes Resource (base) Types generic_application RESTART_ATTEMPTS CHECK_INTERVAL Nx resource types ora.listener.type local_resource STOP_TIMEOUT START_DEPENDENCIES. Nx resource types ora.database.type cluster_resource Page 16

18 Oracle Clusterware resource management - 2 Resource base types > All base types are supplied by Oracle Local resource > Runs on each server of the cluster > CRS automatically extends local resources tied to the new server > They do not failover from one server to another Cluster resource > Subject to switchover and failover > Subject to resource attribute CARDINALITY Generic application > Deprecated > Pre-Oracle CRS 11.2 resources Page 17

19 Oracle Clusterware resource management - 3 Attributes > Default values can be used for some of them > Others must be specified depending on your needs J Resource Attributes Configuration Monitoring Read-Write Read-Write Read-Only - ACTION_SCRIPT - ACL - AUTO_START - PLACEMENT - HOSTING_MEMBERS - SERVER_POOLS - USR_ORA_ENV - ACTION_SCRIPT - START_DEPENDENCIES - CARDINALITY - aso - CHECK_INTERVAL - RESTART_ATTEMPS FAILURE_INTERVAL - STOP_TIMEOUT - START_TIMEOUT - UPTIME_THRESHOLD - aso - STATE - LAST_STATE_CHANGE - TARGET - RESTART_COUNT - aso Page 18

20 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Oracle Clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover dbi-services 6. User experience 7. Core Message 8. Homework Page 19

21 Cold failover cluster vs RAC One Node > Technology comparison > RAC One Node > Cold failover cluster Page 20

22 Cold failover cluster vs Oracle RAC One Node - 1 Oracle RAC One Node > RAC enabled Oracle Home > Easy conversion to RAC from RAC One Node and vice-versa > Online database relocation > Option for Oracle EE only (cost ± 20%) Cold Failover cluster > Single instance database > Officially replaced by Oracle RAC One Node starting with 11.2! L? > Available without restriction for Oracle SE,SE1 and EE! > Service disruption during service relocation Page 21

23 Cold failover cluster vs Oracle RAC One Node 2 Oracle RAC One node > Based on a Oracle supplied type ora.database.type (base type: cluster_resource) > Convenient Resource registration (instance suffixed by <OracleSID>_1) > CAUTION! DBCA creates the necessary RAC aware database settings > Undo, Redo Threads depending on the database management Administrator managed database (as oracle RDBMS software owner) # srvctl add database -db $ORACLE_SID \ -oraclehome $ORACLE_HOME -dbtype RACONENODE -server orabs,orazh Policy managed database (as oracle RDBMS software owner) # srvctl add database -db $ORACLE_SID \ -oraclehome $ORACLE_HOME dbtype SINGLE -serverpool my_pool Manage Rac One Node # srvctl start database db $ORACLE_SID # srvctl relocate database db $ORACLE_SID # srvctl add service s <ServiceName> -d $ORACLE_SID Page 22

24 Cold failover cluster vs Oracle RAC One Node 3 Addendum Oracle RAC One Node > SINGLE instance database type restricted for Oracle Restart # Administrator managed database PRCD-1146 : Database CDB1 is not a RAC One Node database # Policy managed database PRKO-2067 : The size of server pool should be one for single instance database > Oracle RAC One Node ACFS = BUG LL > Oracle Grid Infra + RDBMS GI - PSU Jan2015 > Failover capabilities are not affected! # srvctl relocate database -db $ORACLE_SID... PRKH-1001 : HASContext Internal Error PRKH-3005 : Entity ncdb_1 was not registered with CSS CLSS return code=12 Page 23

25 Cold failover cluster vs Oracle RAC One Node 4 Cold failover cluster > 2 variants are possible: > Using the deprecated resource (base) type generic_application Attribute START_PROGRAM STOP_PROGRAM CLEAN_PROGRAM PID_FILES Value /u01/app/oracle/admin/db1/dbstart.sh /u01/app/oracle/admin/db1/dbstop.sh /u01/app/oracle/admin/db1/dbclean.sh /u01/app/oracle/admin/db1/pid.file > Using a user defined resource and type with an action script Page 24

26 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Oracle clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover dbi services 6. User experience 7. Core Message 8. Homework Page 25

27 Cold failover dbi services > Setup > Resource dependencies > Monitoring > Placement Page 26

28 Cold failover dbi services - 1 Oracle Clusterware manage the resources using resource attributes cluster_resource clu_db.type STOP_TIMEOUT=600 RESTART_ATTEMPTS=2 attributes inherited or modified STOP_TIMEOUT=600 RESTART_ATTEMPTS=5 new attribute(s) ACTION_SCRIPT= User defined resource HOSTING_MEMBERS= node1,node2 Page 27

29 Cold failover dbi services - 2 The resource is managed by an Oracle Clusterware supplied agent which calls the action script (start/stop/clean/check) > $GRID_HOME/log/<Node>/agent/crsd/scriptagent_<Usr> > $GRID_BASE/diag/crs/<Node>/crs/trace/crsd_scriptagent<Usr> :28:21.302: [DB1.db][ ] {1:62596:2} [check] Executing action script: /u01/app/oracle/local/dmk/bin/db_crs.ksh[check] CRSD oraagent (ora.database.type) orarootagent (ora.cluster_vip.type) scriptagent (cluster_resource) Rac One Node/RAC VIP ACTION_SCRIPT DB1 user defined resource/type START/STOP/CHECK/ CLEAN Page 28

30 Cold failover dbi services - 3 Resource start/stop dependencies Databse resource weak() pullup() hard() ora.data.dg ora.fra.dg ora.scan_listener.type ora.listener.lsnr database resource ora.data.dg ora.fra.dg shutdown hard() ora.asm ACFS intermediate Page 29

31 Cold failover dbi services - 4 Step 1: Create the user defined type clu_db.type Run as Grid infrastructure software owner (usually O.S user grid) # crsctl add type clu_db.type -basetype cluster_resource \ -file [path to attributes file] Step 2: Create the database resource > Beware! The resource can t be prefixed by ora. (Oracle supplied res.) Run as the Oracle database O.S user (or add ACL explicitly) # crsctl add resource <OraSID>.db -type clu_db.type attr HOSTING_MEMBERS=orabs orazh START_DEPENDENCIES=hard(ora.DATA.dg,ora.FRA.dg) weak(type:ora.scan_listener.type) pullup(ora.listener.lsnr) STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutd own:ora.fra.dg,intermediate:ora.acfs.vol0_u01.acfs) ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--" That s it J Page 30

32 Cold failover dbi services - 5 Monitoring > Do not shutdown the database from SQL*Plus Run as the user referenced by the ACL type/resource attribute (usually O.S user oracle or grid) # crsctl status res NCDB.db -f grep ENABLED ENABLED=1 # crsctl modify res NCDB.db -attr "ENABLED=0" # crsctl status res NCDB.db -t Name Target State Server State details Cluster Resources NCDB.db 1 ONLINE ONLINE orabs STABLE # crsctl status res NCDB.db -f grep AUTO_START AUTO_START=restore Page 31

33 Cold failover dbi services - 6 Placement policies for Cold failover clusters > Administrator Managed (previous example) > Policy Managed server pools Cluster partitioning : Two (2) or N nodes, all active protecting single instance databases > Spread the the databases (evenly) across all nodes > Remain protected against node failures > Shutdown all non critical database in case of node failure Critical.pool ERP ECOM Low.pool DWH UAT Page 32

34 Cold failover dbi services - 7 Partitioned (Cold) failover cluster Modify type to spread the resource evenly (reset HOSTING_MEMBERS) crsctl modify type clu_db.type \ -attr "... dispersion(type:clu_db.type ) Add the serverpools for partitioning crsctl add srvpool HCRITICITY.sp -attr "IMPORTANCE=1, MIN_SIZE=1, MAX_SIZE=1" crsctl add srvpool LCRITICITY.sp -attr "IMPORTANCE=0, MIN_SIZE=1, MAX_SIZE=1 Adapt the resources crsctl modify res ECOM.db -attr "SERVER_POOLS=HCRITICITY.sp, PLACEMENT=favored ACTIVE_PLACEMENT=1..." crsctl modify res UAT.db -attr "SERVER_POOLS=LCRITICITY.sp, PLACEMENT=favored ACTIVE_PLACEMENT=1..." Page 33

35 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Cold failover cluster vs RAC One Node 4. Cold failover dbi services 5. User experience 6. Core Message 7. Homework Page 34

36 User experience > DNS flooding > ASYNC I/O > srvctl vs crsctl Page 35

37 User experience - 1 DNS flooding? (process mdnsd.bin) > Enable DNS cache on each nodes > Allows Oracle Clusterware to better tolerate network failures Page 36

38 User experience - 2 ASYNC I/O operations > Oracle recommends to limit the maximum number of allowable concurrent AIO requests within the database in Grid Infrastructure Installation guide: > fs.aio-max-nr = > Kernel parameters are automatically by installing the RPMs > oracle-rdbms-server-12cr1-preinstall > oracle-rdbms-server-11gr2-preinstall Fri Feb 27 10:50: Error requesting async resources falling back to sync, expect performance degradation Error requesting async resources falling back to sync, expect performance degradation UNABLE TO RESERVE KERNEL RESOURCES FOR ASYNCHRONOUS DISK I/O (Doc ID ) > Oracle s M.O.S recommendation fs.aio-max-nr= Page 37

39 User experience - 3 SRVCTL vs CRSCTL Oracle discourage to use CRSCTL to modify > Use SRVCL for Oracle supplied resources (Listener, network, SCAN IPs, VIPs, SAN listeners, and so...) > Use CRSCTL to manage Oracle Clusterware > Version,CSS, CRSD, cluster maintenance, aso > User defined resources and types EXCEPTION > User defined type/resource, user defined server pools, aso Run as the user referenced by the ACL type/resource attribute (usually O.S user oracle or grid) # crsctl start resource <OraSID>.db # crsctl stop resource <OraSID>.db # crsctl relocate resource <OraSID>.db # crsctl modify resource <OraSID>.db attr ENABLED=0 Page 38

40 User experience - 4 Never trust in any kind of hardware > CISCO USC everything is redundant > Are your sure that the system engineers are gurus? > Of course it s redundant but it must be correctly configured > Split-Brain because of a Fabric Interconnect failure > IBM SAN Storwize > Automatic LUN failover from a controller to another > Great! the Voting files are constantly accessed by all cluster nodes > Reboot less fencing (disktimeout 200s) because of a hanging failover Last one but not least, the interruptions were transparent for the end-users! J Page 39

41 Agenda 1. Introduction > Vocabulary > Oracle cluster software stack > Oracle Highly Available architectures 2. Oracle Grid Infrastructure main components 3. Oracle Clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover cluster 6. User experience 7. Core Message 8. Homework Page 40

42 Core Message > Features > Not only reserved for databases > Flexibility > Command-Line tools JJJ > No additional license fees (Data Recovery env. 10-days rule) > Single point of support > Knowledge > Complexity (less than RAC) > EM12c integration > ACFS O.S integration (ie: tail) dbi services Cold failover cluster works out of the box J Page 41

43 Agenda 1. Introduction 2. Oracle Grid Infrastructure main components 3. Oracle Clusterware resource management 4. Cold failover cluster vs RAC One Node 5. Cold failover cluster 6. User experience 7. Core Message 8. Homework Page 42

44 Homework build your own Grid Infra. cluster DNS Server Public network (vibr0) Private Interconnect (vboxnet0) iscsi (vboxnet1) Virtual Box Oracle ZFS storage appliance Page 43

45 Infrastructure at your Service. Any questions? Please do ask. Jérôme Witt Senior Consultant Mobile We look forward to working with you! Page 44