A Deep Dive into ASM redundancy in Exadata

Transcription

1 A Deep Dive int ASM redundancy in Exadata Emre Baransel DOAG 2016, Nurnberg Overview As yu prbably knw exadata bxes used t have several ptins like full, half, quarter and eighth. A full bx had 14 strage servers, half bx had 7, quarter and eighth bxes had 3 strage servers. Nw it s pssible t chse the number f strage servers and database servers in a bx. Yu can have frm 3 t 18 Strage Servers per rack We als call them cell servers. Minimum cnfiguratin has three cell servers, because each cell server is a failure grup in ASM and high redundacy diskgrups require at least 3 failure grups. We ll cnsider 3 strage servers in examples. Except the Extreme Flash ptin which is an all flash Exadata Machine, every strage server in Exadata physically have 12 equal size and equal perfrmance disks n them. Fr Exadata X6-2 these disks are 8 TB SAS disks. This is the High Capacity cnfiguratin f the Exadata machine. There used t be an ptin t chse high perfrmance r high capacity hard disks in an Exadata instalatin. But ur pint here is whatever ptin we used t chse, all the hard disks in the Exadata was the same type. Exadata machine als has Flash Acceleratr cards with flash memry but flash is almst always cnfigured as a flash cache (Exadata Smart Flash Cache) in frnt f disk strage. We ll als cnsider a cnfiguratin where harddisks are used as ASM disks and flashdisks are used fr flashcache. In Exadata language we call each f these disks Physical disk. In this case we have ttal 36 physical disks n ur Exadata. On the installatin f Exadata we create lgical partitins n the physical disks. Then use the same sized partitins t create ASM diskgrups.

2 Lk at the green bxes. These are the system partitins where strage server perating system resides. In every strage server first tw disks have the system partitin fr redundancy. As a best practice, n the ther disks f the cell servers, we create same sized lgical partitins with system partitin. Yu can see them with yellw clr. And use them t create DBFS diskgrup t prvide filesystem area and t keep clusterware files. Then we create tw mre lgical partitins n every disk and create RECO and DATA diskgrups. Yu can see the blue lgical partitins build the RECO diskgrup and RED lgical partitins build the DATA diskgrup. We left the DATA diskgrup at the end f the disk, because partitins start frm inner edge t uter and the uter edge f the disk prvides mre perfrmance than the inner edges. We call these lgical partitins Grid Disks. Exadata Grid disks frm the basis f ASM. ASM uses these disks in diskgrups s we als call them ASM disks. Then we start t put data n diskgrups. ASM stripes files accrss all the disks within the diskgrup. S we can see a unifrm distributin f data in ASM diskgrups. In Exadata, we can chse nrmal r high redundancy diskgrups, external redundancy is nt supprted. In a nrmal redundancy diskgrup a primary extent has ne mirrred extent (secndary extent). When reading a blck, database reads frm the primary extent and when writing, writes t bth. And if ne f the extents is crrupted, ASM can fix it by using the healty extent. Which is called bad blck remapping In a high redundancy diskgrup a primary extent has tw mirrred extents in different failgrups. In this case where there are 3 strage servers, each server has a cpy f extent. Failure Nw it s time t investigate failures. We shuld lk int tw types f failures: disk failures and cell server failures. A disk failure can be transient which can be caused by disk path malfunctins, such as cable failures, hst bus adapter failures, cntrller failures, r disk pwer supply interruptins. S the disk is actually healty but ASM cannt make I/O t the disk. Other disk failure type is physical where disk itself has hardware errrs. In a transient disk failure, exadata takes the physical disk int OFFLINE status and grid disks n it will be marked as INACTIVE. Then time starts ticking.

3 If the failure is crrected r the disk is replaced befre DISK_REPAIR_TIME value exceeds, Then the grid disks are resynced with ASM FAST MIRROR RESYNC feature. This is an ASM 11g feature andis very useful s that n cmplete rebalance f failed disk is required. ASM FAST MIRROR RESYNC keeps track f pending changes t extents n an OFFLINE disk during an utage. The extents are resynced when the disk is brught back nline. But if we dn t repair the failure r dn t replace the disk, befre DISK_REPAIR_TIME, then asm drps the disks frm diskgrups and rebalance the data in rder t prvide redundancy again. S in a transient failure, the value f DISK_REPAIR_TIME is very imprtant. It s als very imprtant when we manually ffline disks r shutdwn strage servers. Our maintenance task must finish befre DISK_REPAIR_TIME in rder t prevent an unnecessary rebalace. We shuld remember that DISK_REPAIR_TIME is a diskgrup attribute and its default is 3.6 hurs. We can change the default with alter diskgrup cmmand but changing it after a failure has n effect n ffline disks. In a physical disk failure case the stry is different. If a physicaldisk fails, if the disk itself has hardware failure, all griddisks n that physicaldisk will be immediately DROPPED with FORCE ptin frm ASM. Which is called Pr-Active Disk Quarantine ASM will nt wait DISK_REPAIR_TIME t drp disks in this case. After drpping disks, rebalance will start n diskgrups if there s enugh disk space n diskgrups. Then when we replace the disk, celldisk and griddisks will be recreated autmatically and thse griddisks will be autmatically added t ASM. Then a new rebalance will start autmatically and write extents t new disks. These autmatic peratins will ccur if the griddisk was previusly drpped by the aut disk managment. If the griddisk was manually drpped, we will need t add disks back int ASM manually. If we have a clser lk int Aut Disk Management feature in Exadata, we see tw prcesses wrking fr it n ASM instances. The Exadata Autmatin Manager and Exadata Autmatin Wrker. Manager prcess initiates the autmatin tasks and Wrker perfrms the tasks.

4 And there are sme hidden parameters which cntrls this feature. _AUTO_MANAGE_EXADATA_DISKS can enable and disable the feature. _AUTO_MANAGE_NUM_TRIES cntrls the maximum number f attempts t perfrm an autmatic peratin. _AUTO_MANAGE_MAX_ONLINE_TRIES cntrls maximum number f attempts t ONLINE a disk. These parameters are hidden and yu knw needs Oracle supprt advise. Nw let s lk int strage server failures. A strage server can fail fr sme reasn and ASM cntinues wrking in this case bth in nrmal and high redundancy. High redundancy can handle tw strage servers. A strage server failure is actually the failure f the whle failgrup in asm. In this case ASM des nt drp disks and it waits fr disk_repair_time. remember this behaviur is als same when rebting the strage server If the fault is crrected and server is alive befre DISK_REPAIR_TIME, then grid disks will be synched and there will be n rebalance peratin. But if the server is nt alive after DISK_REPAIR_TIME, ASM will drp grid disks frm diskgrups and a rebalance will start t build redundancy. The rebalance will start if there is enugh space in the diskgrups. Here nrmal redundancy nly guarantees that system will be alive after ne cell server failure, building the redundancy again is pssible nly if there s enugh disk space. We ll see hw t calculate the required space t build reduncy after a cell server fault in next slides. When the fault is crrected and cell server cmes back, there will be a secnd rebalance and data will be distributed back t all disks. Secnd Failure First failure desn t cause an ASM service lss in Exadata because there is nrmal r high redundancy. Nw let s lk what happens at secnd failure. In a nrmal redundancy diskgrup what happens at secnd failure is first related with when it ccurs. If it happens after rebalance r sync is cmpleted, then prcedure is same with the first failure. It ll be like a first failure case.

5 But if it happens befre the rebalance r sync is cmpleted, then what happens is related with which disk is failed. If first & secnd failed disks are nt partner disks, which means they dn t stre tw cpies f an extent, then a new rebalance will ccur, f curse if there s enugh space. But if first & secnd failed disks are partners then ASM will stp and failure will result with data lss. Ok, this is a small pssibility but we have t cnsider especially depending n the value f the data. Partner disks are n different strage servers, which means if the secnd failure is in the same cell, there s n data lss. Als we say first failure, secnd failure but first incident desn t have t be a failure, strage server rebt causes the same, if the sync peratin nt yet finished. In sme cases it s useful t find ut the partner disks f a specific ASM disk. In this MOS nte tw methds are given fr this purpse. It s pssible t use X$KFDPARTNER table jined with v$asm_disk view if the diskgrup is munted. If diskrup is nt munted it s pssible t use GMON trace which is n the backgrund_dump_dest f ASM Instance. A shell cmmand with grep and sed is given t extract the partnership infrmatin frm the trace file. In High Redundancy we knw there are three cpies f each extent. S we can say that a secnd failure never cause a data lss in High Redundancy Here i want t mentin a feature which came with BP16 and BP4. The feature is MOUNT RESTRICTED FORCE FOR RECOVERY and it s applicable t nrmal redundancy. This feature is useful in the fllwing cases. First, during an Exadata cell rlling upgrade/patching r any ther maintenance wrk, ne f the cell server is nt in use and a partner disk failure ccurs. Or a transient disk failure in a cell fllwed by a permanent partner disk failure befre the first failed disk cmes back nline. Hw we use this feature? Let s lk int an example. First we shutdwn all services n first cell and ASM takes these disks ffline. Just like we re ding maitenance n the cell. Then simulate a disk failure in secnd cell with this alter physicaldisk cmmand. Then ASM stps and database crashes. Nw we munt the diskgrup with restricted frce fr recvery ptin.

6 We start services n first cell and make the disks nline with this cmmand. Nw a SYNC peratin will start t update mdified extents in the first cell. We must wait until SYNC finishes. In this resync the prcess will nt be able t read sme f the required extents because f the secnd disk failure and mark them with BADFDA7A. We ll see this in rebalance prcess trace file. Nw we have a munted diskgrup with sme extents crrupted. We can remunt the diskgrup and perfrm RMAN blck recvery t fix thse crruptins. Usable Space Nw we ll lk at the usable space expressin n an Exadata diskgrup. Here the questin is what kind f usable space. What a persn wants t express with the term usable space in Exadata is generally different frm what peple understand. I listed sme disk size terms in Exadata t make the usable space expressin clear. Ttal Raw Size (TRS) Used Raw Size (URS) Free Raw Size (FRS) Ttal Allcatable Size (TAS) TRS / Redundancy Factr Used Allcatable Size (UAS) URS / Redundancy Factr Free Allcatable Size (FAS) FRS / Redundancy Factr Size Needed fr Disk Failure Cverage (SNDFC) Largest Disk (r 2 Disks fr High R.) Size Needed fr Cell Failure Cverage (SNCFC) Largest Cell (r 2 Cells fr High R.) Ttal Disk Failure Safe Allcatable Size (TRS - SNDFC) / Redundancy Factr Ttal Cell Failure Safe Allcatable Size (TRS - SNCFC) / Redundancy Factr Free Disk Failure Safe Allcatable Size (FRS - SNDFC) / Redundancy Factr Free Cell Failure Safe Allcatable Size (FRS - SNCFC) / Redundancy Factr

7 Let s make an example with a ttal raw size f 360. Let s say used raw size is 120 and free raw size is 240. If we dn t care abut building the redundancy after a failure then we can use 180. If we want ASM t be available t build redundancy after a disk failure, we shuld nt use mre 175 at ttal. If we want ASM t be available t build redundancy after a cell failure, we shuld nt use mre 120 at ttal. Nrmal Redundancy High Redundancy Ttal Raw Size (TRS) Used Raw Size (URS) Free Raw Size (FRS) Ttal Allcatable Size (TAS) TRS / 2 = 180 TRS / 3 = 120 Used Allcatable Size (UAS) URS / 2 = 60 URS / 3 = 40 Free Allcatable Size (FAS) FRS / 2 = 120 FRS / 3 = 80 Size Needed fr Disk Failure Cverage (SNDFC) Size Needed fr Cell Failure Cverage (SNCFC) Ttal Disk Failure Safe Allcatable Size (TRS - SNDFC) / 2 = 175 (TRS - SNDFC) / 3 = Ttal Cell Failure Safe Allcatable Size (TRS - SNCFC) / 2 = 120 N/A fr Quarter Rack Free Disk Failure Safe Allcatable Size (FRS - SNDFC) / 2 = 115 (FRS - SNDFC) / 3 = 73.3 Free Cell Failure Safe Allcatable Size (FRS - SNCFC) / 2 = 60 N/A fr Quarter Rack In the ASMCMD lsdg utput there are infrmatin like redundancy type, rebalance status, allcatin unit etc. What s imprtant fr us in terms f usable space is in the yellw backgrund. Ttal_mb, free_mb, req_mir_free_mb and usable_file_mb. Let s lk what they mean. Ttal_mb is what we said as ttal raw size and free_mb is free raw size

8 What Req_mir_free_mb means changes depending n the versin. With and after and it means Size Needed fr Disk Failure Cverage. Befre these versins it means Size Needed fr Cell Failure Cverage. In the same way, what usable_file_mb means als changed. It s Free Disk Failure Safe Allcatable Size in the new versins and Free Cell Failure Safe Allcatable Size in the lder versins. Oracle changed the calculatin frm cell t disk because usable_file_mb becmes negative much earlier when calculating cell failure. And this leads t sme misunderstandings. What t Pay Attentin t Replacing Disks Befre replacement, Be sure that rebalance is cmpleted and disk is ready fr replacement After replacement, Be sure that cell/grid disks are created and rebalance started Rlling Shutdwn/Upgrade f Strage Servers Check DISK_REPAIR_TIME and increase if required Check if ASM will be OK if the grid disks g OFFLINE Be sure that there s enugh space fr rebalance if smething ges wrng (Free Cell Failure Safe Allcatable Size)