Patient C SQL Critical Care

Transcription

1 Background This patient called us because they had outages lasting for several hours. Big complaints from users. They said 1 hour of downtime would be okay for downtime, but not great. They were using Merge Replication to copy the data to another data center but wanted to move away from it as they were experiencing recurring problems with it. The patient was using VMWare, which gave them some HA. They were using Enterprise Edition but wanted to move to Standard Edition for cost reasons. They had sysadmins and developers. One of the developers was the acting DBA/Accidental DBA. Let s see what we found!

2 Patient C SQL Critical Care SQL Server High Availability and Disaster Recovery Therapy 2017 Brent Ozar Unlimited. All Rights Reserved. For details:

3 Session agenda Your current setup and your goals Option 1 Synchronous Database Mirroring + Log Shipping Option 2 Failover Clustering + Log Shipping Factors that would change our recommendation

4 Your current setup and your goals

5 What instance did we look at? Instance: [REDACTED] Applications involved: [REDACTED] Total data size: 180GB Memory Size: 96GB Number of logical cores: 8 SQL Server version and edition: 2012 SP3, Enterprise Edition (but should be Standard Edition) Virtualized: Yes, VMWare

6 Edition and Software Assurance You are using Enterprise Edition but are licensed for 2008 Standard Edition. We re going to assume Standard Edition is in place as you indicated Enterprise Edition licenses are not financially feasible.

7 Virtualization VMware does give you some High Availability, but it doesn t detect when SQL Server crashes If the OS goes down, the OS has to come online on another host; this is slower than just SQL Server starting on another host This is why you need an HA solution for Windows and SQL Server. You can use VMware in conjunction with a High Availability feature to get a really amazing HA solution.

8 Your RPO and RTO Goals Recovery Point Objective (RPO): How many minutes of data can you lose in a worst case scenario? Recovery Time Objective (RTO): How many minutes can you be offline in a worst case scenario? Server Offline Corrupt data Oops deletes Datacenter offline Minutes of data loss allowed Minutes of downtime allowed

9 Synchronous Database Mirroring

10 Synchronous Database Mirroring with Automatic Failovers If we add a Witness server, which can use Express Edition, we can also set it up to have automatic failovers. Failovers happen in under a minute.

11 Synchronous Mirroring Database Mirroring can be used for High Availability or Disaster Recovery. But it can t be used for both. You only get one mirror. If you are using it for HA, then it would be Synchronous Database Mirroring. If you are using it for DR, then it would be Asynchronous Database Mirroring which is an Enterprise Edition-only feature. The highlights and lowlights: Databases fail over individually: dangerous in multi-database applications Works well with single-database 3 rd party apps Only user databases fail over - not jobs, logins anything external to the user database needs to be handled by you Requires blazing fast networking and storage it s a two-phase commit across both servers Usually slows down write throughput (inserts/updates/deletes) but not selects Secondary instance generally doesn t have to be licensed if you have Software Assurance unless you query it once you query from it: $$$$$

12 Does Sync Mirroring meet our goals? A synchronous mirror with automatic failover configured gives you 0 minutes of data loss and less than 1 minute of downtime for the failovers. It beats your goals! No data loss, very little downtime for failovers. Management will love that, except for the fact that you need double the storage.

13 More Highlights Works in Standard Edition Synchronous only Much easier to manage than Availability Groups No Windows Failover Cluster required Much simpler to document Management can all be done using SSMS/TSQL Perks Automatic page repair Doesn t require a domain / can be used with multiple domains (using certificates)

14 Mirroring is deprecated but this doesn t mean you shouldn t use mirroring It will be removed in a future version of SQL Server What version that is hasn t been disclosed. Features may stay deprecated for many many versions without being removed. Fully supported in SQL Server 2012, 2014, and 2016 This is a well known, widely documented, established feature in SQL Server Supported by many vendors Typically stable

15 Mirroring modes You must choose between safety and performance High Safety mode Synchronous Automatic failover is optional Witness definitely use this if you re using automatic failover! Can be Express Edition. High performance mode Asynchronous Enterprise Edition only No automatic failover Do not use a witness with this mode

16 Problems mirroring doesn t solve Databases are mirrored individually, and fail over individually Distributed transactions don t survive this If you want them to fail over as a group, you have to write custom code and make sure it happens You have to keep SQL Agent Jobs and Logins in sync (and make sure the right jobs run after failover)

17 Gotchas You have to keep logins and jobs synched Index maintenance may need adjustment; delay in between commands

18 Setting up logins You can script these all out from SQL Server Management Studio s GUI, but you ll need to know the passwords if you have any SQL logins. SQL logins have a second issue, and that s with the SID. If you create the login on the secondary without specifying the primary s SID, you end up with an orphaned user. The login at the instance level won t match up to the user at the database level. The SIDs don t script automatically from the GUI either. Robert Davis has published a free script that helps you automate this and solves both of these problems:

19 Orphaned Users If you do have a login where the SID wasn t transferred over to the mirror, the users associated with the login will be orphans and they won t work You can detect and fix this without recreating the login if it happens using sp_change_users_login: The next time you fail over to this server, you ll again have to unorphan the login. It s best to fix the login so that it has the same SID on both servers.

20 Setting up SQL Server Agent jobs The easy way to script out all jobs in SSMS: View: Object Explorer Details In the Object Explorer Details pane, you highlight all jobs and script them out in a single script But beware If job steps use log files, make sure the same path exists where you re installing it Make sure you enable/disable the right jobs on the mirror

21 Making jobs work auto-magically after failover You need custom code to help your jobs out: If an automatic failover occurs (sync mirroring with a witness only), the right jobs need to run without an administrator having to manually enable / disable them You need to accomplish this without having jobs fail all the time if they re running against the mirror because that will destroy monitoring for failed jobs This means each job needs to check Am I the principal? and act accordingly Ola Hallengren s maintenance scripts have this functionality built in for index maintenance, backups, and CHECKDB:

22 CLR Assemblies: Trustworthy? CLR Assemblies are part of the database and are mirrored But they may not work when you first fail over without extra steps The TRUSTWORTHY database property isn t automatically restored with the database. If you need this, it must be enabled the first time you fail over Not all CLR assemblies require this-- you can see your current setting in the sys.databases DMV

23 Split brain in High Safety Mode Split brain is as bad as it sounds: everything shuts down Let s say you re using high safety mode with a principal, a mirror, and a witness If the principal can t see either the mirror or the witness, it can t know if the database is online at the witness. It will take the database offline. If the mirror can t see the principal or the witness, it freaks out too, and puts the database offline. The database is offline everywhere. We ve lost quorum! Constant communication between the principal, mirror, and witness is critical

24 Suspended mirror Suspended status can happen after things go wrong Forced service SQL Server bugs in some versions (Examples: ) Suspended means that data is not being transferred from the principal to the mirror The log will grow on the principal You re no longer protected by mirroring To fix, on either partner run: ALTER DATABASE YourMirrorDB SET PARTNER RESUME;

25 Disconnected witness In some cases the witness may become disconnected We ve seen this if there have been network interruptions to the witness, or problems in DNS Mirroring may keep running (depending on whether or not the communication between the primary and the mirror is interrupted) If you have a disconnected witness, you may fix this by restarting the endpoints on the witness To prevent unplanned failovers, I prefer to: Switch to high performance mode and remove the witness Repair the witness Add the witness back and switch to high safety mode

26 Forcing up the mirror Example: you re using asynchronous mirroring ( high performance mode). This doesn t have automatic failover. The principal server is lost! You need to bring things up on the mirror. To do this, you run: ALTER DATABASE YourMirrorDB SET PARTNER FORCE_SERVICE_ALLOW_DATA_LOSS; Discussion: 1. What do you do to identify how much data was lost? 2. What other steps need to be done immediately if you have to do this?

27 Your biggest takeaways Monitoring is critical If you have automatic failover, you need to make sure the witness and mirror are healthy If you ignore failures, the likelihood of split brain taking everything down goes up Placement of the witness is important If you re using multiple datacenters, the witness should be in the primary datacenter

28 sys.database_mirroring This DMV helps you quickly see which databases are mirrored For each database that s mirrored, you can see: Mirroring mode Who s the partner and what their state is Who s the witness and what their state is Endpoint address

29 Database Mirroring Monitor Job This is created automatically if you use SQL Server Management Studio to set up mirroring Runs every one minute by default Calls the built in sp_dbmmonitorupdate procedure (which you can use in your own code, if you want) This job allows you to see the current mirroring status by using: The Database Mirroring Monitor tool The sp_dbmmonitorresults procedure

30 Launching the monitor Right click on the database name in SSMS Snoop around under Tasks

31

32 Thresholds and alerts You can setup alerts via the Database Mirroring Monitor (under the Warnings tab) or via sp_dbmmonitorchangealert. The alerts are per database. PER DATABASE. You can script out the alert for each database, but you ll need to schedule a job to catch any new databases. Pseudocode: For each database in (SELECT DB_NAME(database_id) AS database_name FROM sys.database_mirroring WHERE mirroring_role_desc = 'PRINCIPAL') { EXEC = = = 1 } --@alert_id = 1 is the alert for is number of minutes of latency before an alert is raised

33 It s better to buy monitoring You can build your own, but: Complex and time consuming to set up alerts for state changes and performance alerts yourself You ll get lots of duplicate alerts (especially if you re mirroring multiple databases) The Database Mirroring Monitor shows a limited amount of history troubleshooting past failovers can be tricky Try a free trial of a vendor product that does it for you and see if you like their interface: SQL Sentry Performance Advisor Idera Diagnostic Manager

34 Become a mirroring pro Create a test environment Run through setup multiple times Practice planned failover, switching modes Implement and test monitoring Notification of state changes (pausing, failovers) Create a delay by performing multiple index rebuilds on the primary do you get notifications of latency? Shut the witness down do you get a notification Make it fail Practice forcing service on a mirror Practice recovering afterwards

35 Failover Clustering

36 Q L Cluster Nodes: 2 Instances: 1 Single-instance cluster, also known as an active/passive cluster. F We ve got a hot standby ready to go, but we only have one running instance of SQL Server at any time. Q ueried. L icensed. F ree w/sa As long as we have Software Assurance, we don t have to license our one hot standby node.

37 Failover Clustering Failover Clustering is a High Availability solution. It can be used for Disaster Recovery, but $$$$$$$. The highlights and lowlights: Everything fails over together: all databases, logins, jobs, server name-the entire SQL instance Automated failover in 1-2 minutes (ideally) Works great with 3 rd party apps and other HA/DR features Lowest license is Standard Edition on primary node, no cost for passive node Requires shared storage: $$$ and admin time Zero data loss as long as shared storage doesn t fail Shared storage is a single point of failure

38 Does an FCI meet our goals? FCI gives you 0 minutes of data loss and 1 minute of downtime for the failovers. It beats your goals! No data loss, very little downtime for failovers. Management will love that.

39 Pros and Cons of an FCI Pros Shared storage one copy of the data The entire instance fails over You don t have to keep jobs, logins or anything external to the user database in sync No data loss on failovers Failovers take about a minute Cons Shared storage single point of failure, make sure your backups are saved somewhere else such as a NAS Not trivial to setup

40 Quorum is critical

41 Failover Cluster Quorum Get Quorum wrong, and you could end up with unexpected downtime. What good is an HA feature if it s not highly available due to how it was configured?

42

43 You set the voting rules You may choose ONE option You generally want an odd number Node majority Node + quorum disk majority Node + file share majority No majority (disk witness) A single point of failure by definition.

44 Clusters fear isolation Failover Cluster Service Node1 Network Partition Node2 Are you doing something without me? Are YOU doing something without ME?

45 Obviously not good Node1 You might be doing something. Network Partition Node2 Oh no! This is bad. * Windows Server helps with this scenario more soon.

46 Can I get a witness? Node1 Glad to see you! Glad to be seen. Network Partition Node2 Not again.

47 Enhancement NodeWeight may be manually set to zero Windows Server 2008R2 Windows Server 2012 Windows Server 2012 R2 Hotfix Yes Yes File Share Witness Yes Yes Yes Dynamic Quorum Yes Yes Dynamic Witness Tie Breaker on 50% node loss Force Quorum Resiliency Yes Yes Yes Bonus: Has a Start Menu? Yes Yes

48 Disabling voters (optional) Node1 Node2 Primary Datacenter (configured for odd number) I get updates about what s happening in the cluster, but I don t count Node3 SAN Repl DR Datacenter NodeWeight set to 0

49 Problems can look like this

50 (checks watch)

51 Or like this

52 Recovery isn t always easy I fixed this by starting the cluster service on CL02 the message just didn t list that option.

53 Witness rules With Windows 2012 R2 or greater, you always want a witness Prior to that you want a witness to make an odd number of votes Monitor your witness disk or fileshare!

54 Cluster validation

55 View validation

56 Ooops Who wants to be fumbling trying to find this when a node is offline?

57 Who set this thing up?

58 Don t ignore warnings You don t need a dedicated heartbeat network anymore, but you DO need multiple physical NICs

59 Validation Not all tests are online Stay current Validate after configuration changes Keep copies of your validation reports If you have to escalate, you want these handy You may run a subset of tests, but in that case the last report only shows the tests you ran (it doesn t include any prior tests) Don t ignore warnings.

60 Living well with clustering

61 Practice failures Practice causing quorum failures Discovering the error Bringing resources back up Recovering other resources afterward You need a safe place to do this outside of production

62 Avoid single points of failure Example: Multiple physical NICs in clustered servers Have you tested what happens when a NIC fails? (Does the teaming really work?) Example: What happens if quorum is lost when you re on vacation? Documenting how to recover from all possible points of failure is very complex you must have multiple staff members trained up

63 Validation is critical Keep cluster validation up to date Run all tests after you change any quorum configuration Keep copies of your validation report off the production server in case you need to escalate

64 Monitoring is key If components fail, you need to respond promptly even if the clustered resources are still online Example: File share witness fails This may put you in a vulnerable state You need to reconfigure the witness or remove it (depending on your windows version) Don t just monitor the SQL Server! Monitor and create troubleshooting guides for all the components that could fail

65 Log Shipping

66 Q L Log Shipping Nodes: 2 Instances: 2 Two standalone SQL Servers. Our production instance writes log backups to a shared folder, and DR instance restores them. F If we have SA and no one queries the DR instance, we don t have to license it. Q ueried. L icensed. F ree w/sa

67 Multiple Destinations

68 Good ol Log Shipping Log Shipping is a solid Disaster Recovery option. It has been around since SQL Server It is tried and true. It is not deprecated, unlike Database Mirroring. One of the benefits of Log Shipping over Database Mirroring, Basic Availability Groups and Availability Groups is that if any of the secondary servers fall behind it does not impact the primary server. Latency can cause a production outage in those technologies.

69 Does LS meet our goals? The 1-hour RTO goal is achievable with Log Shipping. With a lot of practice and scripts, you could failover in under 5 minutes. The 15-minute RPO goal is usually achievable with Log Shipping, but it could slip a bit, depending on the transactions. If a large transaction runs, such as the case with index rebuilds of a large index or nightly processes, it ll create big transaction log backups. Those have to be copied and restored. The copy time is usually where the delay occurs and that s what slips the RPO.

70 Basic log shipping recipe There s a GUI, but the basics steps are: Prepare the secondary Full database backup on ServerA Copy full backup to ServerB Restore full backup with NORECOVERY to ServerB Ship the logs Take transaction log backups on ServerA Copy transaction log backups to ServerB Restore transaction log backups with NORECOVERY in chronological order to ServerB

71 Can you tell I m log shipped?

72 On the log shipping primary SQL Server Agent Jobs Database Properties

73 NORECOVERY mode Most common option for secondary You can t read from the secondary database It s just there for failover

74 Standby mode Option for secondaries Cool: Allows point-in time reads Gotchas: You have to kick everyone out when you restore logs again Performance impact on the log restore time

75 What s awesome Relatively inexpensive Works with Standard edition If you have active Software Assurance and are not reading from the secondary, the secondary doesn t have to be licensed Very minimal overhead added to the primary (if you re already taking transaction log backups) Can have multiple log shipping secondaries Standby mode makes the secondary readable Does NOT require Primary and Secondary instances to: Be in the same domain (or be on-premises) Run the same Edition of SQL Server

76 What s not awesome There s a GUI, but it s clumsy Many people have a hard time using the monitoring Clean failover to a secondary and failback are not easy to do (and no magical connection strings like mirroring and AGs have) You have to keep logins and jobs in sync between instances Restoring logs to the secondary can be slow (and it can get very far behind) Not all databases fail over at the exact same point in time

77 Gotchas SQL Agent Service account not having permission to the share You have to keep logins and jobs synched Custom scripting to have all databases failover at once Failover isn t automatic Index maintenance may need adjustment; delay in between commands Make sure monitoring instance (can be SQL Express) is ready Make sure jobs send failure s

78 Keepin things in sync Moving stuff over initially is pretty easy! Microsoft has a couple stored procedures to help you with logins: Scripting jobs is pretty easy with the GUI keep in mind that you may need to change drive letters/path and that you may need to disable them on the secondary Keeping them synchronized is a little more difficult Depending on volume, you may just need to add a manual step to any configuration change, that it also has to be done on your Log Shipping Secondary If this doesn t fit, you ll have to script out a process to keep them synchronized

79 Application Awareness Set up a CNAME alias in DNS points to an IP Do a one-time change to the application connection strings to point to this alias After a failover, you can just update the IP in DNS, instead of in every single connection string

80 What does a failover entail? When you are ready to do the failover, run the LOG backup job. Once that job finishes, run the copy job. And when that finishes, run the restore job. You may want to disable these jobs after each finishes. Now you ve got to recover the database as it is in a recovery state: RESTORE DATABASE DatabaseName WITH RECOVERY Run any necessary post-failover scripts. Do you need to unorphan any users (not necessary if you created the logins with the same sid as the old primary server)? Enable jobs on the new primary, if necessary, and disable jobs on the old primary, if necessary. Make sure users and applications have access. If you plan on failing back to the old primary server, you need to setup Log Shipping in the reverse.

81 Practice makes perfect Practice the failovers. Some organizations fail over 1-2 times per year to ensure they can, to prove to their customers that they can, and to make sure their applications work at the other site. The more you practice, the more you are going to want to script it. The more you script, the faster it ll complete. Practice makes perfect. If ever the time comes to do an unplanned failover, you ll be ready.

82 We d change our tune if

83 If you said Enterprise Edition is an option If Enterprise Edition was an option, you could use an Availability Group for your HA, DR and reporting needs. All in one neat little package! There s synchronous replicas for HA, and asynchronous replicas for DR. There s also readable secondaries, either sync or async. Except Availability Groups are complex and require Failover Clustering knowledge. It s easy to setup, but if you get it wrong (most do initially) you end up with unexpected downtime. We d recommend training or consulting+training on Availability Groups if you decide to use an AG.

84 If you said we want to offload reports There are a few solutions to offload reports: Log Shipping with a delay in the restores Database snapshot on a mirrored server Transaction Replication Availability Groups

85 Log Shipping with a Delay Log Shipping with a delay is a good solution when the reports can have stale data. Usually Log Shipping is used just for Disaster Recovery. If you are using it for reporting as well, you need to be aware the delay in the restores could impact your RTO goal. The secondary server where reports are reading data from has to be licensed. It does not have to be licensed if you are only using it for DR and have Software Assurance.

86 Database Snapshot on a Mirror Database snapshots require Enterprise Edition. If you were to use Enterprise Edition, I d instead recommend Availability Groups.

87 Transactional Replication I used Transactional Replication for years for the reporting solution. Large transactions can cause excessive latency, but it s worse with Transactional Replication since you are replicating transactions and not just log records or backup files. I ve seen several hours of latency. Troubleshooting can be a nightmare. Transactional Replication requires primary key constraints on every table that will be replicated.

88 Availability Groups When SQL Server 2012 was released, I couldn t wait to upgrade to it so that I could have HA, DR and reporting all in one package. Enter Availability Groups! It has synchronous and asynchronous replicas, up to 4 of them in SQL Server And you can read from them. It has readable secondaries! But it requires Enterprise Edition, which is ~$7000 per core. And you have to license the primary server and all replicas you read from. $$$$$$

89 Recap

90 The two recommendations We recommend Log Shipping for your Disaster Recovery needs. But should you use Synchronous Database Mirroring or Failover Clustering for your High Availability needs? Database Mirroring is a much simpler feature, but it requires double the storage. Failover Clustering is much more complex. You need training or consulting+training in order to set it up or support it. We recommend starting with Synchronous Database Mirroring while you work towards getting training on Failover Clustering.

91 Homework

92 You ve got homework We ve got a great HA/DR course in our online training: You don t have to watch all of the modules, just focus on the ones you are interested in: Database Mirroring, Log Shipping and Failover Clustering. These should help you decide which is best for you, besides my recommendations, and also help you implement them.

93 Questions?

94 We believe these will help you accomplish your RPO/RTO goals. Which one are you leaning toward, and why?