PBO3350BUS: Snapshots and SQL Server - Technical Deep Dive & Detailed Lab Findings Rob Girard Sr. Technical Marketing Engineer VMworld 2017 Content: N

Size: px

Start display at page:

Download "PBO3350BUS: Snapshots and SQL Server - Technical Deep Dive & Detailed Lab Findings Rob Girard Sr. Technical Marketing Engineer VMworld 2017 Content: N"

Garry Sullivan
5 years ago
Views:

1 PBO3350BUS Snapshots and SQL Server - Technical Deep Dive & Detailed Lab Findings Rob Girard Principal Technical Marketing Engineer Shawn Meyers SQL Server Principal Architect VMworld 2017 Content: Not for publication #VMworld #PBO3350BUS

2 PBO3350BUS: Snapshots and SQL Server - Technical Deep Dive & Detailed Lab Findings Rob Girard Sr. Technical Marketing Engineer VMworld 2017 Content: Not for publication Shawn Meyers SQL Server Principal Architect

About Shawn Shawn Meyers @1dizzygoose linkedin.

3 About Shawn Shawn linkedin.com/in/shawnmeyers42 SQL Server Principal Architect, practice lead Experience in VMware, Microsoft, SQL Server, storage infrastructure, performance tuning. Working in IT since 1992, SQL Server since 1996, VMware since Tintri, Inc. All Rights Reserved. 3

4 About Rob Rob Principal Technical Marketing Tintri as of Jan, 2014 Working in IT since 1997 with >10 years of VMware experience vexpert, VCAP4/5-DCA, VCAP4-DCD, VCP2/4/5, MCSE, CCNA AND TCSE 2017 Tintri, Inc. All Rights Reserved. 4

VAP6433 Group Discussion session @ VMworld 2015 This session summarizes the

5 Introduction Met at SQL Elite Workshop, hosted by VMware and Tintri [April 2015] Partnered to share expertise with different aspects of virtualization Delivered VAP6433 Group Discussion VMworld 2015 This session summarizes the research & lab behind that session For those who want a closer look under the covers 5

6 Scope of session & Testing Scope-creep was winning. 6

7 The Great Debate Application vs Crash Safe? Not Safe! Not Safe? Safe!.. 7

9 Crash consistent vs application consistent Crash Consistent Same concept as if pulling the power plug out of the back of the server SQL Server recovery can take longer depending upon what the server was doing when the crash occurred Application Consistent SQL Server will be in the same state as an OS reboot SQL Server startup will be the same every time Flushes completed transactions 2016 Tintri, Inc. All Rights Reserved. 43

10 Snapshot 101: Anatomy of a Snapshot Think of a snapshot as layers within PhotoShop Let s use a quick visual aid to get us started 10

Snapshot Definition A snapshot is a point-in-time copy of the data that represents an image Can be used to recover individual items to a full server recovery and everything in between Basics is

28 Snapshot Definition A snapshot is a point-in-time copy of the data that represents an image Can be used to recover individual items to a full server recovery and everything in between Basics is a metadata file and a collection of pointer records for a point in time VMworld 2017 Content: Not for Warning!!! A snapshot is not a backup publication 2016 Tintri, Inc. All Rights Reserved. 25

29 Snapshots What can I use them for? Their original intended purpose: Recovery Patching Cloning for various reasons: New server with a similar function Test/Dev/Staging Environments Non-Intrusive Recovery Verification Troubleshooting Moving Data 2016 Tintri, Inc. All Rights Reserved. 29

VMware snapshots 01 02 03 04 Done at the hypervisor level Includes the state and data of a virtual machine (if memory is snapped) Can have snapshot chains Can have snapshot

30 VMware snapshots Done at the hypervisor level Includes the state and data of a virtual machine (if memory is snapped) Can have snapshot chains Can have snapshot consolidation, orphan snapshots, removal issues Tintri, Inc. All Rights Reserved. 27

31 Microsoft SQL Server Support Statement on Snapshots SQL Server supports virtualization-aware backup solutions that use VSS (volume snapshots). For example, SQL Server supports Hyper-V backup. Virtual machine snapshots that do not use VSS volume snapshots are not supported by SQL Server. Any snapshot technology that does a behind-the-scenes save of a VM s point-in-time memory, disk, and device state without interacting with applications on the guest using VSS may leave SQL Server in an inconsistent state. VMworld 2017 Content: Not for Short version you need to quiesce for support from Microsoft publication 2016 Tintri, Inc. All Rights Reserved. 28

32 Storage array based snapshots Known as Crash-Consistent snapshots within Tintri VMstore Don t preserve VM state Chaining is efficient, automatic, and user-proof Very quick and imperceptible by the applications/users Superior to a LUN snapshot as it is only the VM you want, not the churn of everything else in the LUN 2016 Tintri, Inc. All Rights Reserved. 29

34 To quiesce or not to quiesce? That is the question! Windows servers use VSS Process for bringing the OS into alignment for a proper backup Flushes dirty pages (buffers) to disk Primarily used for backups not for rollback snapshots Can be used to call custom scripts Freeze & Thaw Performance & other implications? Wait for test results! Extremely minor risk of snapshot having database corruption 2016 Tintri, Inc. All Rights Reserved. 31

$bat can be placed in C:\Program Files\VMware\VMware Tools\backupScripts.$

36 Testing Environment 1 Hypervisor: vsphere 6.0U2 Physical Servers: Cisco UCS B200 Blades (12-core Intel E s) Storage: Tintri T5080 VMstore Virtual Machine Config: Windows Server 2012R2 Microsoft SQL vcpu + 16 GB RAM 7+ vdisks Client VMs: Windows 2012R2 w/ HammerDB 36

37 Testing methodology HammerDB to populate databases and generate load Take a variety of snapshots (vsphere & storage) Observe and Measure: Impact to VM during snapshot process Recovery impact Recovery Assessment Clone snapshots into VMs & power up into isolation (disconnect NIC) Review logs. Recovery times use time stamps from Starting Master DB until Recovery Complete 2016 Tintri, Inc. All Rights Reserved. 47

38 Testing methodology Taking snaps 38

39 Testing methodology Cloning Snaps to Clones 39

40 Testing methodology: Start to End Timing 40

41 Test results assumptions We try to go in with a clean slate, but we all have ideas about what the results are going to be prior The assumptions can pre judge the results Sharing so you know where we are coming from and you can judge us These were written down prior to testing occurring VMworld 2017 Content: Not for publication 41

42 The assumptions Quiesce will take longer but will provide for better SQL Server recovery VAAI snapshots will be faster Managing VM Stun time is key Memory snapshot is not needed; offer no added value VMworld 2017 Content: Not for publication 42

44 Test #1 Fill er Up! What are we testing, and under what conditions? Crash-consistent vs Quiesced Does the size of the DB make a difference? Do multiple databases on the same disk make a difference? What about a dataspace spanning many disks? Snapping the SQL VM under varying levels of stress (up to & including 100% CPU) 2016 Tintri, Inc. All Rights Reserved. 54

45 Test #1 Fill er Up! The Setup: 1 SQL 2016 VM w/ 11 vdisks: OS, Page, Data1, Logs1, TempDB, TempLogs, Backup, Data2, Logs2, Data3, Logs3 9 x HammerDB Client VMs to create databases of various sizes Primarily Write operations 2016 Tintri, Inc. All Rights Reserved. 55

46 Test #1 Fill er Up! 46

47 Test #1 Fill er Up! 47

48 Test #1 Fill er Up! 48

49 Test #1 Fill er Up! [DURING] Observations: No errors observed on the HammerDB clients Screenshot of SQL Logs at time of quiesce. 49

50 Test #1 Fill er Up! [DURING] 50

51 Test #1 Fill er Up! [DURING] 51

52 Test #1 Fill er Up! [DURING] 52

53 Test #1 Fill er Up! (RECOVERY) Observations : Review of SQL Logs Screenshots 53

54 Test #1 Fill er Up! [RECOVERY] 54

55 Test #1 Fill er Up! [RECOVERY] 1 second recovery observed 55

56 Test #1 Fill er Up! [RECOVERY] Observations: Disk I/O observed (VSS vs Crash-Consistent) Not much to report, but there was some I/O incurred on crash-consistent snap not seen on VSS 56

57 Test #1 Fill er Up! [CONCLUSION] No stuns observed for Crash-Consistent at time of snap VSS stuns in these tests were minimal Recovery: 1 second for VSS 11 seconds or crash-consistent 57

58 What about my memory?..i forget You can snap the memory of the VM within a VMware snapshot (enabled by default) VMworld 2017 Content: Not for publication 58

59 VMware Snaps w/ Memory Makes for longer stun times Not typically needed for backups but needed for recovering a VM to an exact state, including inflight transactions. Revert snapshot rolls back VM into a running state Can t natively clone the snapped state, BUT, you can use Storagebased snapshots to create a clone, and then revert. Handy to troubleshoot a condition that would otherwise clear with a reboot Tintri, Inc. All Rights Reserved. 37

60 VMware Snaps w/ Memory con t Takes longer to create & requires Disk I/O - all provisioned VM memory needs to be dumped to disk VMworld 2017 Content: Not for publication 60

61 Snapshot consolidation Using agents tends to take snaps, attach to different guest for backups, then revert the snapshot back When error occur you have chain issues Snapshot consolidation can be a smooth process or can take forever Many factors VMworld 2017 Content: Not for publication 2016 Tintri, Inc. All Rights Reserved. 39

62 Recovery time Restore of a large database can take a long time with native backups Snaps can be back online in minutes if not seconds Snaps can be mounted to recover individual objects 2016 Tintri, Inc. All Rights Reserved. 44

63 Snapping SQL Server drives All data and log drives need to be snapped at the same instant Not an issue with Tintri Snaps! Many times data and log are on different datastores due to different IO patterns or to create multiple queues on vsphere host When data and log are on different datastores, the snapshots must keep these consistent or there can be database corruption Not all SAN vendors offer a way to snap multiple datastores at the same instant 2016 Tintri, Inc. All Rights Reserved. 45

64 Test #2 Long Time, No See! Does the time elapsed since the last VSS-quiesced snapshot make an impact on crash-consistent snaps, or future VSS snaps? (ie. database growth) 12 hours since the last VSS snapshot or distribution 65

65 Test #2 Long Time, No See! [DURING] No noticeable changes observed at time of snap compared to 12+ hours earlier, when databases first began populating VMworld 2017 Content: Not for publication 66

66 Test #2 Long Time, No See! [RECOVERY] Recovery of Crash-Consistent snapshot, 12 hours earlier: 11 seconds 67

67 Test #2 Long Time, No See! [RECOVERY] Inspect SQL logs from Crash-consistent, 12 hours since last VSS operation VMworld 2017 Content: Not for publication 68

68 Test #2 Long Time, No See! [RECOVERY] Compare to SQL Log of VSS-queisced snap 69

69 Test #2 Long Time, No See! [RECOVERY] 13 seconds crash-consistent versus 11 seconds crash-consistent 12 hours earlier (ie. within <15 mins of last VSS-quiesed snap) 13 seconds (crash-con) vs 1 second VSS VMworld 2017 Content: Not for publication 70

70 Test #2 Long Time, No See! Let s compare disk activity 71

71 Test #2 Long Time, No See! 72

72 Test #2 Long Time, No See! 73

73 Test #2 Long Time, No See! (CONCLUSION) The longer you go between backups/vss, the more work that is required at recovery time VMworld 2017 Recovery time difference is minimal assuming decent performance Content: Not for publication 74

74 SQL Server backup and recovery SQL Server Native backup when structured properly will allow for point in time recovery to the nearest transaction or sub second Only completed transactions will be part of the backup Recovering from a snapshot will either be backup consistent or crash consistent Many SQL Server experts warn about all snapshots VMworld 2017 Content: Not for publication Snapshots are not backups!!! 2016 Tintri, Inc. All Rights Reserved. 40

75 SQL Server Native Backups Usually stored on separate disk subsystem Storage corruption will not cause data loss DBA has the control they desire to manage their risk Using a mixture of both snapshots and backups provide the most flexibility Think about as multiple layers of protection one does not replace the other Depends upon SLA (business rules) 2016 Tintri, Inc. All Rights Reserved. 41

76 Recovery models SQL Server has three recovery models Simple Can only be restored to last full backup Bulk logged Can be restored to point in time, but bulk logged processes are not in the restore and have to be repeated. Full Can recover to any point in time, down to a single transaction or even a certain millisecond VMworld 2017 Content: Not for publication 2016 Tintri, Inc. All Rights Reserved. 42

Set your phasers for VM stun time Snapshots can pause a virtual machine in order to quiesce Have seen hourly 5 minute stuns of IO (bad config) Proper setup can make these manageable

77 Set your phasers for VM stun time Snapshots can pause a virtual machine in order to quiesce Have seen hourly 5 minute stuns of IO (bad config) Proper setup can make these manageable stun Some SQL Server databases can never be stunned Even worse in a LUN-based datastore where ALL VMs need to be stunned prolonging the pain 2016 Tintri, Inc. All Rights Reserved. 38

78 Test #3 The Never-ending Transaction Image Credit: neverendingstory.com 79

79 Test #3 The Never-ending transaction What happens during quiesce if there s no clean break in active I/O? Start with an ugly query that never commits Run it against a decent-sized database. We used a 5,000 warehouse HammerDB 500 GB Sit back and wait. And wait. And wait. While query is running, take snaps after 20 mins, 1.5 hours and ~21 hours (7.5 million row affected) Finally terminated the process after 17+ million rows were affected, 2.5 days later 80

80 Test #3 The Never-ending transaction 81

81 Test #3 The Never-ending transaction [During] 82

82 Test #3 The Never-ending transaction (During) Increased snapshot times (LONG!) VMware snapshot: 02:40 (compared to ~40 seconds) Removal: 34:43 (compared to ~20 second removals) 83

83 Test #3 The Never-ending transaction [During] SQL log during snap removal I/O requests taking longer than 15s 84

84 Test #3 The Never-ending transaction [Recovery] Both VSS quiesced snapshot AND crash-consistent snap entered into recovery mode after boot up SQL database was not ready, and relatively high I/O was observed on storage VMworld 2017 Content: Not for publication 85

85 Test #3 The Never-ending transaction [Recovery] 86

86 Test #3 The Never-ending transaction [Recovery] 87

87 Test #3 The Never-ending transaction [Recovery] Analysis phase took 290x (!!!) LONGER on Crash-consistent snap Advantage: VSS? Not quite VSS = 7ms, crash-consistent = 2037ms Only 2 seconds added to recovery time Out of 16+ minute recoveries at 30K 40K IOPS 88

88 Test #3 The Never-ending transaction [Recovery] 89

89 Test #3 The Never-ending transaction 90

90 Test #4 vsstrace Available in the Windows SDK Information overload! ~48,600 events for idle SQL Server w/ 9 databases on 11 vdisks 37 seconds logged ~35,500 events for an idle SQL Server with NO custom DBs 35.5 seconds logged What to use it for? Deep inspection Beware the observer effect Debugging isn t free What can be found? 91

91 Test #4 vsstrace Interesting Findings DB names 92

92 Test #4 vsstrace Interesting Findings DB names 93

93 Test #4 vsstrace Interesting Findings DB names 94

Test #4 vsstrace As it relates to SQL References to SQL-specific use the SQLServerWriter by default Filter by the WRITER module to narrow down logging results

94 Test #4 vsstrace As it relates to SQL References to SQL-specific use the SQLServerWriter by default Filter by the WRITER module to narrow down logging results (~10% of events in our sample: 4,841 / 48,647 total events vssadmin list writer Many writers may be listed, vsstrace will tell you which is actually being used 95

95 Ask Again: To quiesce or not to quiesce? Stun Times Summary of results on whether it s worthwhile or not Recovery from snapshots Data loss (transactions rolled back) Supportability or distribution 96

96 Recommendations Snapshots do not replace backup but are another tool for improved recovery Snapshots offer many great benefits beyond just recovery Quiesce if supportability is key Suggest frequent crash-consistent with occasional VM-consistent or VSS-enabled backup job It is a business tradeoff snapshots are for every situation 97

Tintri: Best Practices for Snapshot Heavy Environments

Tintri: Best Practices for Snapshot Heavy Environments Tomer Hagay, Tintri #vmworld VAP3915BUS #VAP3915BUS About Myself Tomer Hagay https://www.linkedin.com/in/tomerhagay Director, Technical Marketing