AlwaysOn Availability Groups: Backups, Restores, and CHECKDB

Size: px

Start display at page:

Download "AlwaysOn Availability Groups: Backups, Restores, and CHECKDB"

Austin Singleton
5 years ago
Views:

2 What I m not covering Recovery models: full, simple, bulk logged Backup types: full, differential, transaction log AG internals, worker thread exhaustion issues Agenda Business requirements Scenario 1: oops-deletions Scenario 2: corruption Scenario 3: stale backups Scenario 4: replication breaks Scenario 5: long-term outage 2016 Brent Ozar Unlimited. All rights reserved. 2

3 We run a factory. The catch: We face financial penalties if we miss our RPO/RTOs. So when we fail over to DR environment, we want HA & Oops protection there too, just like we have in production. (We don t need a DR for DR, though.) 2016 Brent Ozar Unlimited. All rights reserved. 3

6 FIRE. Fail over to DR. Write down what you d do before you d let end users insert data again. Scenario #1: oops-delete The primary site goes down We manually fail over to DR Ten minutes later, user accidentally truncates a table How do you react? Do you have both full and log backups in DR? Including whole LSN chain from full backup to now? 2016 Brent Ozar Unlimited. All rights reserved. 6

7 Lessons learned from scenario 1 Before we let people in The moment we manually fail over to another DC: Take a differential backup (if not a full) Transaction log backups to meet my RPO/RTO Make sure these backups are copied offsite I may not be able to let people into the database until this first differential and log completes Brent Ozar Unlimited. All rights reserved. 7

8 Scenario #2: corruption As before, the primary site goes down We manually fail over to DR Ten minutes later, users report corruption errors How do you react? Scenario #2: corruption Have you CHECKDB d any of your DR replicas? Automatic page repair only works if: There s an available replica It has clean copies of the page Restores only work if: You have full & log backups available in DR 2016 Brent Ozar Unlimited. All rights reserved. 8

9 Lessons learned from scenario 2 High availability requirements Every day, in every data center, we need at least one: Replica that had a clean CHECKDB Full backup This impacts licensing: if we re backing up and DBCCing everywhere, these may be considered production servers Brent Ozar Unlimited. All rights reserved. 9

10 Monitoring impacts It s not enough to know that a database has had a successful backup and CHECKDB somewhere. I need to know if any data center is missing those. The instant I encounter corruption in any data center, it s an emergency for me, because it stops me from failing over. Or, I can build in protection 2016 Brent Ozar Unlimited. All rights reserved. 10

11 Adding a node In each DC, add one replica on local SSDs When we lose a DC, we still get automatic page repair Adding a node In each DC, add one replica on local SSDs When we lose a DC, we still get automatic page repair Now, where do we do backups and DBCCs? Should our design offload them to a secondary? 2016 Brent Ozar Unlimited. All rights reserved. 11

12 Let s test it: scenario 3 Scenario #3: primary down. At 10:00 AM, the primary goes down. How do you react? What are you expecting to happen automatically? What if that doesn t happen? 2016 Brent Ozar Unlimited. All rights reserved. 12

primary automatically. Write down what you d do. Go. https://msdn.microsoft.

13 You have sixty seconds. The primary is down. The cluster is up. You can contact all of the secondaries, but none of them will take over as primary automatically. Write down what you d do. Go Brent Ozar Unlimited. All rights reserved. 13

14 Gone in sixty seconds Even sync secondaries will get out of sync. Network issues Windows reboots SQL Server service goes down on secondary Redo thread falling behind SQL Server vnext has a fix In SQL Server , all of your replicas can go down and the primary keeps right on accepting data. SQL Server vnext adds Minimum Replica Commit: if you can t get at least X replicas to do a commit, take the primary down. (That obviously adds a new set of problems.) 2016 Brent Ozar Unlimited. All rights reserved. 14

15 But til then, no guarantees. Can we get closer today? Option A: SQL Server Seppuku Build an app that watches latency across the secondaries, and if none of them are keeping up, degrade the application. Option B: Continuous Log Backups While 1 = 1, take a log backup, and restore it to other SQL Servers outside of the AG Brent Ozar Unlimited. All rights reserved. 15

Read more: https://www.brentozar.com/archive/2017/01/always-isnt-handling-outages-application/ Option B: Continuous log backups Did we decide to offload backups to a secondary?

16 Read more: Option B: Continuous log backups Did we decide to offload backups to a secondary? If so, is that replica in sync? Secondary log backup jobs will run successfully for a period of time even when the replica isn t in sync Brent Ozar Unlimited. All rights reserved. 16

17 Your backup job logic Loop through the databases and check: If I m this database s primary replica, And if the replica that s supposed to do the backups is more than X minutes behind, Then take the transaction log backups here If I m this database s secondary that s supposed to do backups, And I m not more than X minutes behind, do the backup If I m behind, send an alert Your backup job logic Loop through the databases and check: If I m this database s primary replica, And if the replica that s supposed to do the backups is more than X minutes behind, Then take the transaction log backups here If I m this database s secondary that s supposed to do backups, And I m not more than X minutes behind, do the backup If I m behind, send an alert 2016 Brent Ozar Unlimited. All rights reserved. 17

18 Not from the DMVs, you can t. This gets harder. Loop through the databases and check: If I m this database s primary replica, And if the replica that s supposed to do the backups is more than X minutes behind, Then take the transaction log backups here If I m this database s secondary that s supposed to do backups, And I m not more than X minutes behind, do the backup If I m behind, send an alert 2016 Brent Ozar Unlimited. All rights reserved. 18

19 This stuff gets complex fast. If you have to guarantee RPO and RTO numbers, and there s real financial penalties involved, run your log backups on the primary replica. But even that isn t going to be enough. Let s test it: scenario Brent Ozar Unlimited. All rights reserved. 19

20 Scenario #4: slow replication The primary site goes down on Saturday at 10PM We manually fail over to DR, and find out that the DR replicas are ~90 minutes behind Index rebuilds were running on the primary, which queued up a lot of log data to be sent to DR Now what? Remember We face financial penalties if we miss our RPO/RTO Brent Ozar Unlimited. All rights reserved. 20

21 If your network pipe can t keep up Reduce changes: Defrag/rebuild less often (fix query perf issues in other ways, like RAM) Add long delay times between objects Move non-critical tables to non-ag databases Prioritize AG replication traffic using QoS Monitor how far behind each replica is: WAN can slow down at any time due to other servers, and cause you to miss your RPO despite planning Let s test it: scenario Brent Ozar Unlimited. All rights reserved. 21

22 Scenario #5: replica down A replica goes down. From this point forward, your primary s transaction log starts growing while you troubleshoot. Even your reporting replicas are mission-critical: when they fail, they can cause the primary to run out of space. Immediately disable optional jobs like index rebuilds. (Bonus points: code this into your index rebuild jobs.) Every hour: Assess the primary s log sizes, drive free space Ask: do you need to take the failed replica out of the AG so that the log file will stop growing? The gotcha with DR replicas All replicas get their log file changes from the primary server, not from the nearest replica. If you decide to take the DR replica out of the AG, you ll need to reseed it with a full backup and logs. You can use a full backup taken in DR, but remind me where are your log backups taken again? 2016 Brent Ozar Unlimited. All rights reserved. 22

23 Recovering DR3 We could take a full backup on the DR failover cluster instance, which is local to DR3 Then only copy the necessary transaction log files (or even switch to doing log backups in DR temporarily) This can speed up the DR3 recovery. Node failures have lots of options. Let it resync from the primary node if you don t have much log file data to copy over, and a fast connection, or Fail over to a closer node if you need to bring it up to speed faster, and can tolerate moving the DB, or Take it out of the AG entirely and reseed it from backups, possibly taken from closer nodes 2016 Brent Ozar Unlimited. All rights reserved. 23

25 Designing for limited data loss Sketch out the infrastructure including: Every server Network connections with tested average and peak throughput metrics Jobs for backups and CHECKDB, with schedules Then make a copy of this sketch for every server. Got 5 servers? Make 5 copies. On each copy, pick a primary. When one of the servers is a primary, what happens when each piece of infrastructure: Fails outright for a short period (like 30 minutes) Fails for an extended period (like 18 hours) Degrades for a short period (like slows down for 30 minutes due to workload, connectivity) Bonus exercises: corruption, intermittent network connectivity, related servers (app, domain controllers) 2016 Brent Ozar Unlimited. All rights reserved. 25

26 These are your recovery designs. Got financial penalties for missing RPO/RTO? Get: Full backups in every data center DBCC CHECKDB on every replica Log backups taken on the primary, written to a file share that is automatically replicated to all DCs Monitoring for failure of any of the above jobs, with a simple runbook so others can fix it Monitoring for AG replication latency 2016 Brent Ozar Unlimited. All rights reserved. 26

High Availability- Disaster Recovery 101

High Availability- Disaster Recovery 101 DBA-100 Glenn Berry, Principal Consultant, SQLskills.com Glenn Berry Consultant/Trainer/Speaker/Author Principal Consultant, SQLskills.com Email: Glenn@SQLskills.com