A Deep Dive into ASM Redundancy in Exadata

Size: px

Start display at page:

Download "A Deep Dive into ASM Redundancy in Exadata"

Imogen Simmons
6 years ago
Views:

1 Emre Baransel Advanced Support Engineer - Oracle

2 About Me Oracle ACE Alumni Advanced Customer Support Engineer Founding member of Turkish Oracle User Group Author of Data Guard Administration 11gR2 (Packt) Contributed to RMAN 11g &12c Backup and Recovery (Oracle Press) Open World, Oracle Days, UKOUG, DOAG, OUGF, SROUG, BGOUG, TROUG

3 Storage Servers Notation Storage Server 1 Storage Server 2 Storage Server 3 We ll consider 3 storage servers in examples

4 Disks on Storage Servers Storage Server 1 Storage Server 2 Storage Server

5 Physical Disks Storage Server 1 Storage Server 2 Storage Server 3 PHYSICAL DISC

6 Logical Partitions/Diskgroups Storage Server 1 Storage Server 2 Storage Server 3 SYSTEM PARTITIONS DBFS DG RECO DG DATA DG

7 Grid Disks Storage Server 1 Storage Server 2 Storage Server 3 GRID/ASM DISCS SYSTEM PARTITIONS DBFS DG RECO DG DATA DG

8 Disks Usage Notation Storage Server 1 Storage Server 2 Storage Server 3 SYSTEM PARTITIONS DBFS DG RECO DG DATA DG

9 Normal Redundancy Diskgroups NORMAL REDUNDANCY FAILGROUP 1 FAILGROUP 2 FAILGROUP 3

10 High Redundancy Diskgroups HIGH REDUNDANCY FAILGROUP 1 FAILGROUP 2 FAILGROUP 3

11 Types of Failures - Disk Failure - transient disk failure - physical disk failure - Storage Server Failure This presentation examines failures in groups, in order to provide clarity. There may be exceptional cases.

12 Transient Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 TRANSIENT FAILURE (OFFLINE) SYSTEM PARTITIONS DBFS DG RECO DG DATA DG

13 Transient Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 FAILURE CORRECTED or NEW DISK FAILURE CORRECTED or DISK REPLACED BEFORE DISK_REPAIR_TIME EXCEEDS

14 Transient Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 DISK IS RESYNCED WITH ASM FAST MIRROR RESYNC

15 Transient Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 IF DISK_REPAIR_TIME EXCEEDS THEN ASM DROPS THE DISKS AND REBALANCE DATA IF THERE IS ENOUGH SPACE

16 DISK_REPAIR_TIME Attribute DISK_REPAIR_TIME is a diskgroup attribute. Default is 3.6 hours. alter diskgroup data set attribute 'disk_repair_time' = '4.5h Altering the DISK_REPAIR_TIME attribute has no effect on offline disks

17 Physical Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 PHYSICAL DISC FAILURE SYSTEM PARTITIONS DBFS DG RECO DG DATA DG

18 Physical Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 Pro-Active Disk Quarantine (Exadata feature came with ) ASM DOESN T WAIT FOR DISK_REPAIR_TIME, DROPS THE DISK AND REBALANCE DATA IF THERE IS ENOUGH SPACE

19 Physical Disk Failures Storage Server 1 Storage Server 2 Storage Server 3 WHEN DISK IS REPLACED GRID DISCS ARE CREATED & 2. REBALANCE STARTS AUTOMATICALLY

20 Auto Disk Management AUTO DISK MANAGEMENT feature in EXADATA Exadata Automation Manager (XDMG) initiates automation tasks. Monitors all configured storage cells for state changes. Exadata Automation Worker (XDWK) performs automation tasks requested by XDMG. _AUTO_MANAGE_EXADATA_DISKS controls the auto disk management feature. To disable the feature set this parameter to FALSE. Range of values: TRUE [default] or FALSE. _AUTO_MANAGE_NUM_TRIES controls the maximum number of attempts to perform an automatic operation. Range of values: Default value is 2. _AUTO_MANAGE_MAX_ONLINE_TRIES controls maximum number of attempts to ONLINE a disk. Range of values: Default value is 3. NOTE: Auto disk management feature in Exadata

21 Storage Server Failures Storage Server 1 Storage Server 2 Storage Server 3 F A I L E D

22 Storage Server Failures WHEN A STORAGE SERVER FAILS IT MEANS THE FAILURE OF THE WHOLE FAILGROUP IN ASM ASM DOES NOT DROP DISKS BEFORE DISK_REPAIR_TIME EXCEEDS SAME WHEN REBOOTING THE STORAGE SERVER

23 Storage Server Failures Storage Server 1 Storage Server 2 Storage Server 3 IF SERVER IS ALIVE BEFORE DISK_REPAIR_TIME EXCEEDS, DISKS WILL BE SYNCED NO REBALANCE

24 Storage Server Failures Storage Server 1 Storage Server 2 Storage Server 3 F A I L E D IF DISK_REPAIR_TIME EXCEEDS, ASM WILL REBALANCE DATA IF THERE IS ENOUGH SPACE

25 Storage Server Failures Storage Server 1 Storage Server 2 Storage Server 3 WHEN STORAGE SERVER COMES BACK THERE WILL BE A SECOND REBALANCE

26 Second Failure / Bad Chance In Normal Redundancy; What happens at second failure, is first related with when it occurs. - If after rebalance/sync is completed, then procedure is same with the first failure. - If before rebalance/sync is completed, then what happens is related with which disk is failed. - If first & second failed disks are not partner disks, a new rebalance is in question, if there s enough space - If first & second failed disks are partner disks data loss occurs. This is a small possibility but needs consideration. Partner disks are on different storage servers (failgroups). First incident doesn t have to be a failure, storage server reboot causes the same. Exadata Database Machine : How to identify cell failgroups and Partner disks for a grid disk (Doc ID )

27 Second Failure / Bad Chance In High Redundancy; There are three copies of each extent So second failure never cause a data loss in High Redundancy

28 A Useful Feature MOUNT RESTRICTED FORCE FOR RECOVERY feature >= BP16 >= BP4 Applicable to NORMAL redundancy diskgroups only. Potential Use Cases that this procedure will be applicable to : 1. Exadata cell rolling upgrade/patching and a partner disk failure at the same time 2. Transient disk failure in a cell followed by a permanent partner disk failure before the first failed disk comes back online. NOTE: Recover from diskgroup failure using the mount restricted force for recovery feature - An Example

29 Example Procedure MOUNT RESTRICTED FORCE FOR RECOVERY example: o o o o o o o o Cell 1 CellCLI> Alter cell shutdown services all; Cell 2 alter physicaldisk <disk> simulate failuretype=failed; database crashes SQL> alter diskgroup datac1 mount restricted force for recovery; CellCLI> Alter cell start services all; SQL> alter diskgroup datac1 online disks in failgroup CELLFG1; o Wait until MODE_STATUS column in v$asm_disk for the disks being onlined changes to ONLINE from SYNCING. o Do NOT execute the subsequent steps if the mode_status column shows SYNCING. It will lead to data corruption. In resync, due to the second disk failure, Arb0 will not be able to read some of the required extents (which are in the failed second disk) and hence marks those missing extents with BADFDA7A. (arb0 trace file => WARNING: group 1, file 258, extent 100: filling extent with BADFDA7A during recovery) SQL> alter diskgroup datac1 dismount; SQL> alter diskgroup datac1 mount; Start database & Perform RMAN block media recovery

30 What kind of Usable Space? In an Exadata ASM Diskgroup, we can mention following disk spaces: Total Raw Size (TRS) Used Raw Size (URS) Free Raw Size (FRS) Total Allocatable Size (TAS) TRS / Redundancy Factor Used Allocatable Size (UAS) URS / Redundancy Factor Free Allocatable Size (FAS) FRS / Redundancy Factor Size Needed for Disk Failure Coverage (SNDFC) Largest Disk (or 2 Disks for High R.) Size Needed for Cell Failure Coverage (SNCFC) Total Disk Failure Safe Allocatable Size Total Cell Failure Safe Allocatable Size Largest Cell (or 2 Cells for High R.) (TRS - SNDFC) / Redundancy Factor (TRS - SNCFC) / Redundancy Factor Free Disk Failure Safe Allocatable Size (FRS - SNDFC) / Redundancy Factor Free Cell Failure Safe Allocatable Size (FRS - SNCFC) / Redundancy Factor

Calculations for Normal Redundancy Normal Redundancy Total Raw Size (TRS) 360 Used Raw Size (URS) 120 Free Raw Size (FRS) 240 Total Allocatable Size (TAS) TRS / 2 = 180 Used Allocatable Size (UAS)

31 Calculations for Normal Redundancy Normal Redundancy Total Raw Size (TRS) 360 Used Raw Size (URS) 120 Free Raw Size (FRS) 240 Total Allocatable Size (TAS) TRS / 2 = 180 Used Allocatable Size (UAS) URS / 2 = 60 Free Allocatable Size (FAS) FRS / 2 = 120 Size Needed for Disk Failure Coverage (SNDFC) 10 Size Needed for Cell Failure Coverage (SNCFC) 120 Total Disk Failure Safe Allocatable Size (TRS - SNDFC) / 2 = 175 Total Cell Failure Safe Allocatable Size (TRS - SNCFC) / 2 = 120 Free Disk Failure Safe Allocatable Size (FRS - SNDFC) / 2 = 115 Free Cell Failure Safe Allocatable Size (FRS - SNCFC) / 2 = 60

32 Calculations for High Redundancy Normal Redundancy High Redundancy Total Raw Size (TRS) Used Raw Size (URS) Free Raw Size (FRS) Total Allocatable Size (TAS) TRS / 2 = 180 TRS / 3 = 120 Used Allocatable Size (UAS) URS / 2 = 60 URS / 3 = 40 Free Allocatable Size (FAS) FRS / 2 = 120 FRS / 3 = 80 Size Needed for Disk Failure Coverage (SNDFC) Size Needed for Cell Failure Coverage (SNCFC) Total Disk Failure Safe Allocatable Size (TRS - SNDFC) / 2 = 175 (TRS - SNDFC) / 3 = Total Cell Failure Safe Allocatable Size (TRS - SNCFC) / 2 = 120 N/A for Quarter Rack Free Disk Failure Safe Allocatable Size (FRS - SNDFC) / 2 = 115 (FRS - SNDFC) / 3 = 73.3 Free Cell Failure Safe Allocatable Size (FRS - SNCFC) / 2 = 60 N/A for Quarter Rack

What we have in ASMCMD ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL N 512 4096 4194304 27942912

33 What we have in ASMCMD ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL N N DATAC1/ MOUNTED NORMAL N Y DBFS_DG/ MOUNTED NORMAL N N RECOC1/ Total_MB Total Raw Size (TRS) Free_MB Free Raw Size (FRS) Req_mir_free_MB & Size Needed for Disk Failure Coverage (SNDFC) < & < Size Needed for Cell Failure Coverage (SNCFC) Usable_file_MB & Free Disk Failure Safe Allocatable Size & Free Cell Failure Safe Allocatable Size

of Storage Servers o o o Check DISK_REPAIR_TIME and increase if required Check if ASM will be OK if the grid disks go

34 What to Pay Attention to Replacing Disks o Before replacement, Be sure that rebalance is completed and disk is ready for replacement o After replacement, Be sure that cell/grid disks are created and rebalance started Rolling Shutdown/Upgrade of Storage Servers o o o Check DISK_REPAIR_TIME and increase if required Check if ASM will be OK if the grid disks go OFFLINE Be sure that there s enough space for rebalance if something goes wrong (Free Cell Failure Safe Allocatable Size)

35 References Oracle Exadata Database Machine Maintenance Guide Automatic Storage Management Administrator's Guide NOTE: Auto disk management feature in Exadata NOTE: ASM Fast Mirror Resync - Example To Simulate Transient Disk Failure And Restore Disk NOTE: Exadata Database Machine : How to identify cell failgroups and Partner disks for a grid disk NOTE: Recover from diskgroup failure using the mount restricted force for recovery feature - An Example NOTE: How to Replace a Hard Drive in an Exadata Storage Server (Hard Failure) NOTE: Operational Steps for Recovery after Losing a Disk Group in an Exadata Environment NOTE: Understanding ASM Capacity and Reservation of Free Space in Exadata NOTE: ASM Usable Space Calculations in Exadata Environment along with cell failure considerations

36 End of the Presentation Q&As

See the disks inside ASM ASM> select name,label,path,header_status,os_mb,free_mb from v$asm_disk order by 2;

See the disks inside ASM ASM> select name,label,path,header_status,os_mb,free_mb from v$asm_disk order by 2; -- Demystifying ASM REQUIRED_MIRROR_FREE_MB and _FILE_MB -- See the available ASMLIB disks ASM>!oracleasm listdisks ACFS DATA JBOD01 JBOD02 JBOD03 JBOD04 JBOD05 JBOD06 RECO -- See the disks inside ASM