VIRT1630BU Wrangling and Taming your Databases, Storage, Availability, and Disaster Recovery Rawlinson Rivera, Cohesity Deji Akomolafe, VMware VMworld 2017 Content: Not for publication #VMworld #VIRT1630BU
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #VIRT1630BU CONFIDENTIAL 2
Agenda Why Everyone Hates Loves You Your Data Herd What Ails Them Wrangling the Data Beast Database The ESXi Storage Stack Improving Storage IO Performance in vsphere Taming the Data Beast Database Training the Data Beast Database Case Study: Modern Data Wrangling BONUS SLIDES: Storage Performance Troubleshooting Tools Resources
The ACTUAL You The Multiple YOU: When your MDF agrees with your NDF And your LDF is not jealous 4
Your Database is. Just a Bunch of Files 5.AIF.AIFF.AU.AVI.BAT.BMP.CLASS.CSV.CVS.DBF.DIF.DOC.DOCX.EPS.EXE.FM3.GIF.HQX.HTM.HTML.JAVA.JPG.JPEG.LDF.MAC.MAP.MDB.MDF.MID.MIDI.MOV.NDF.QT.MTB.MTW.PDF.P65.PNG.PPT.PPTX.PSD.PSP.QXD.RA.RTF.SIT.T65.TAR.TIF.TXT.WAV.WK3.WKS.WPD.WP5.XLS.XLSX.ZIP.BLAH.XYZ VMworld 2017 Content: Not for publication or distribution
Why Why Everyone Everyone Loves Hates YOU YOU Queries / Order Processing Reporting Data Warehouse / Lake Backup / Restore Maintenance Databases / Data files / Filegroups = Line of Business Applications + Business Objectives (Profit) Archiving / Retrieval
Your Data Herd What Ails Them VMworld 2017 Content: Not for publication CONFIDENTIAL 7
Factors Impacting I/O Performance SQL Server Data Type: OLTP OLAP Data Warehouse Read / Write Ratio Sequential or Random Storage Fabric: Array Type Disk Type (Spindle vs SSD, All-Flash vs Tiered) Disk Presentation Platform: OS Constraints Array Management Overlay Hypervisor (Yaaaay.Getting Warmer) Disk Partition Disk format (block size) Disk Alignment
What You Desire The Fried Ice 1mm IOPS, >2ms latency, 8kb block, 32 OIO s VMworld 2017 Content: Not for Yes You Can Have Both Really publication
Storage Choices Impacting Performance, Manageability, Serviceability primary storage platforms secondary storage platforms Storage platform requirements and considerations for SQL Server: Performance IOPS, Latency primary storage platform Availability suitable architecture of primary storage platforms SLA s mission critical applications Simplified Manageability Faster Backups and Restores Efficient Storage features and Capabilities Scalability and High Availability Protection against cyberattacks (Ransomware) Data and Information visibility 10
Data Management Challenges with Line of Business Applications NFS/VMFS DATA PRTOTECTION & RECOVERY Tape Library VMware vsphere + vsan Media Servers Master Servers Challenges App performance impact due to backup jobs High volume of jobs with Use of primary storage as dump for temp backups Contending with other applications increased latency Complex and fragmented data protection environments Difficult to scale backup software and storage for growth Long RTOs leading to revenue and productivity losses Long RPOs constrained to 24 hour backup cycles Target Storage Cloud Gateway Cloud
So, The Data Architect Went to The Genie Performance Resilience Availability VMworld 2017 Content: Not for It s the Storage, Stupid publication 12
Wrangling the Data Beast Database CONFIDENTIAL 13
First Rule: Know Your Beast The VMware ESXi Architecture Virtual Machine Guest OS ESXi Physical Hardware Scheduler Memory Allocator VMworld 2017 Content: Not for TCP/IP Monitor (BT, HW, PV) Virtual NIC Virtual Switch NIC Drivers File System I/O Drivers publication Virtual SCSI File System I/O Drivers
Second Rule: Know Where it Hides Disk I/O Latencies Application Guest OS VMM vscsi ESX Storage Stack Driver HBA Fabric Array SP QAVG DAVG Time spent in ESX storage stack is minimal *KAVG ~= QAVG For properly-configured systems, QAVG = 0 * KAVG = GAVG DAVG KAVG GAVG
Third Rule: Know Why it Hides Disk I/O Queues Application Guest OS VMM vscsi ESX Storage Stack Driver HBA Fabric Array SP GQLEN WQLEN AQLEN DQLEN SQLEN GQLEN Guest Queue WQLEN World Queue AQLEN Adapter Queue DQLEN Device / LUN Queue SQLEN Array SP Queue Reported in esxtop DQLEN can change dynamically when SIOC is enabled
Taming the Data Beast Database CONFIDENTIAL 17
Storage Choices Impacting Performance Queue Depth vscsi Adapter Be aware of per device/adapter queue depth maximums (KB 1267) LSI Logic SAS = 32 PVSCSI = 64 Configurable to 254 per device and 1024 per adapter Smaller or Larger Datastores? Datastores have queue depths, too. Always Remember THAT Determined by the LUN queue depth Smaller or Larger VMDK? 62 TB is Awesome, but Queue. Depth
I Loooooove Long Queues Nobody, 2017 Arriving Customers Queue Checkout input server output queue time service time Utilization = busy-time at server / time elapsed response time
Additional vscsi Controllers Improves Concurrency Guest Device vscsi Device Storage Subsystem
Increase PVSCSI Queue Depth Just increasing LUN, HBA queue depths is NOT ENOUGH - http://kb.vmware.com/kb/2053145 Increase PVSCSI Default Queue Depth (after consultation with array vendor) Linux: Add following line to /etc/modprobe.d/ or /etc/modprobe.conf file: options vmw_pvscsi cmd_per_lun=254 ring_pages=32 OR, append these to the appropriate kernel boot arguments (grub.conf or grub.cfg) vmw_pvscsi.cmd_per_lun=254 vmw_pvscsi.ring_pages=32 Windows: Key: HKLM\SYSTEM\CurrentControlSet\services\pvscsi\Parameters\Device Value: DriverParameter Value Data: "RequestRingPages=32,MaxQueueDepth=254
Disk Partition Alignment Configure VM s VMDK with vcenter to ensure VMFS partition alignment Double-check NTFS alignment http://msdn.microsoft.com/en-us/library/dd758814.aspx Unaligned partitions result in additional I/O Aligned partitions reduce I/O
Example Best Practices for VM Disk Layout Characteristics: OS on shared DataStore/LUN 1 database; 4 equally-sized data files across 4 LUNs 1 TempDB; 4 (1/vCPU) equally-sized tempdb files across 4 LUNs Data, TempDB, and Log files spread across 3 PVSCSI adapters Data and TempDB files share PVSCSI adapters Virtual Disks could be RDMs Advantages: Optimal performance; each Data, TempDB, and Log file has a dedicated VMDK/Data Store/LUN OS C:\ D:\ H:\ E:\ I:\ F:\ J:\ G:\ K:\ L:\ T:\ OS NTFS Partition: 64K cluster size LSI1 OS VMDK Can be placed on a DataStore/LUN with other OS VMDKs DataFile1.mdf VMDK1 Data Store 1 TmpFile1.mdf VMDK2 PVSCSI1 DataFile5.ndf VMDK3 TmpFile2.ndf VMDK4 Data Store 2 Data Store 3 Data Store 4 SQL Server Can be Mount Points under a drive as well. DataFile3.ndf ESX Host VMDK5 TmpFile3.ndf VMDK6 Data Store 5 Data Store 6 PVSCSI2 DataFile7.ndf VMDK5 TmpFile4.ndf VMDK6 Data Store 5 Data Store 6 LogFile1. ldf TmpLog1.ldf Can also be a shared LUN since TempDB is usually in Simple Recovery Mode VMDK5 PVSCSI3 VMDK6 Data Store 5 Data Store 6 I/O spread evenly across PVSCSI adapters Log traffic does not contend with random Data/TempDB traffic LUN1 LUN2 LUN3 LUN4 LUN5 LUN6 Disadvantages: You can quickly run out of Windows driver letters! More complicated storage management LUN5 LUN6 LUN5 LUN6
Alternative VM Disk Layout Characteristics: OS on shared DataStore/LUN 1 database; 8 Equally-sized data files across 4 LUNs 1 TempDB; 4 files (1/vCPU) evenly distributed and mixed with data files to avoid hot spots Data, TempDB, and Log files spread across 3 PVSCSI adapters Virtual Disks could be RDMs Advantages: Fewer drive letters used I/O spread evenly/tempdb hot spots avoided Log traffic still does not contend with random Data/TempDB traffic OS C:\ D:\ E:\ F:\ G:\ L:\ T:\ OS NTFS Partition: 64K cluster size VMworld 2017 OS VMDK Can be placed on a DataStore/LUN with other OS VMDKs DataFile1.mdf DataFile2.ndf TmpFile1.mdf VMDK1 Data Store 1 DataFile3.ndf DataFile4.ndf TmpFile2.ndf VMDK2 Data Store 2 DataFile5.ndf DataFile6.ndf TmpFile3.ndf ESX Host VMDK3 Data Store 3 SQL Server Can be Mount Points under a drive as well. DataFile7.ndf DataFile8.ndf TmpFile4.ndf VMDK4 Data Store 4 LogFile.ldf VMDK5 Data Store 5 TmpLog.ldf Can also be a shared LUN since TempDB is usually in Simple Recovery Mode Content: Not for publication LSI1 PVSCSI1 PVSCSI2 PVSCSI3 VMDK6 Data Store 6 LUN1 LUN2 LUN3 LUN4 LUN5 LUN6
Training the Data Beast Database CONFIDENTIAL 25
Protecting the (Data) Castle SQL Server Data Protection Options in a vsphere Environment SQL Adapter for VMware Agentless Approach simplifies management Leverages VMware CBT for optimized backups CBT based approach is superior to processing entire DB every full backup SQL Adapter for Physical Servers Eliminates concerns around VM Stun issues Leverages Cohesity CBT for optimized backups Offers same CBT advantages Native SQL Adapter VMworld 2017 Content: Not for Leverage Microsoft VSS and VDI for Instant Restores SQL Maintenance Plan publication Leverage storage platform as target for SQL Dumps 26
Yes, The 90s Was Hip, But it s now 2017 Your backup/restore processes and solutions should: Mama I wanna go fast! Backup and Restore Do so predictably Do so successfully and efficiently Do so with minimal administrative effort Provide lower RTO Provide achievable and configurable RPO Provide Hypervisor CBT-based backups Provide Changed-block-triggered Full backups Provide Any Point-in-Time Recovery Support transaction logs forward roll
Case Study: Modern Data Wrangling 28
Optimized Performance Optimized SQL Backup performance Parallel ingest via storage platform VIPs Improved streaming performance with BUFFERCOUNT Default 1 MB block size ideal for fast ingest 29
CBT Based Protection and Recovery DATA PROTECTION & RECOVERY MS SQL VM Snapshots MS SQL Transaction log logs logs VADP snapshots changed blocks helper VSS CBT SQL Server Host SQL VDI Application Consistent Backups of SQL Servers Optimized SQL Backups First Full Backup Optimized Full Backups with CBT eliminates the need for Differential Backups Log Backups Leverages Microsoft VSS for Full Backups Leverages SQL VDI for Log Backups Agentless Backups for VMware 30
31
Simplified Manageability Single pane of glass management Agentless Backup and Restores Policy Driven Operations Backup Workflows Recovery Workflows 32
Modern Data Management Solution for Line of Business Applications VMware vsphere + vsan NFS/VMFS DataPlatform Broad app and workload support Sub-minute RPO Instantaneous RTO Policy-based Management API-first data protection Integration
Architected for High Availability Rack 1 Rack 2 Node 1 Node 2 Node 3 Node 4 Backup SW, file clients, UI sessions, Virtual IPs Node 5 Node 6 Node 7 Node 8 Always On Availability with no single point of failure Distributed Scale-Out platform System stays available and consistent in case of node failure Data replicated across 2 nodes Rack-awareness to replicate data across racks when available Virtual IPs to provide uninterrupted service in case of node failure 34
Storage Efficiency Global implementation of storage space efficiency features to reduce the storage footprint Variable length blocks In-line and Post-process SnapTree Clones Retain data for longer retention periods VMworld 2017 Content: Not for publication 35
! Intel Inside XEON UCS C240 M4 Console! Intel Inside XEON UCS C240 M4 Console! Intel Inside XEON UCS C240 M4 Console MS SQL Server Recovery Options DATA PROTECTION & RECOVERY Tape Library VMware vsphere + vsan NFS/VMFS VMworld 2017 DataProtect DataPlatform Web-scale platform for secondary data Cloud MS SQL Server Recovery Options MS Host SQL level database recovery recovery Recover SQL entire restore application with NORECOVERY Recovery SQL application NDF files granular to any-point-in-time restore Recovery VM recovery application database to any-point-in-time and binaries File and Capture folder recovery tail logs to with eliminate VM availability gaps Content: Not for publication Instant access to SQL Server via cloning capabilities Copy-attach specific database to any SQL Server instance
! Intel Inside XEON UCS C240 M4 Console! Intel Inside XEON UCS C240 M4 Console! Intel Inside XEON UCS C240 M4 Console MS SQL Server Recovery Options SQL NORECOVERY & NDF DATA PROTECTION & RECOVERY Tape Library VMware vsphere + vsan NFS/VMFS Single UI DataProtect DataPlatform Web-scale platform for secondary data SQL Server NORECOVERY & SQL NDF File Restores Cloud
Primary Storage Integrations Snapshot Integration primary storage secondary storage VMware vsphere DataProtect DataPlatform Advantage of Pure Snap Integration Faster backups and lower RTO & RPO Lower impact on primary production server Ability to perform Application consistent backups Snapshot retention management done using Cohesity DataProtect Efficiently use primary storage by moving older snapshots to the secondary storage No impact on RTO as latest snapshots will still be available on primary Pure storage Optimal Snapshot movement using Snapshot API Registered Pure array or an Alternate Pure array
Server Business Continuity / Disaster Recovery Options Infrastructure admins and DBA s can leverage native integration with all the leading public cloud providers to support the following use cases: Cloud based BCDR: Cloud based Backup Cloud based DR Business Continuity Cloud based Test/Dev Performance, R&D VMware vsphere + vsan NFS/VMFS DataPlatform Self-Service use cases: DRaaS Disaster Recovery-as-a-Service DPaaS Data Protection-as-a-Service DevaaS Test/Dev-as-a-Service
BONUS SLIDES: Storage Performance Troubleshooting Tools
Performance Needs Monitoring at Every Level Application Guest OS ESXi Stack Physical Server Connectivity Peripherals Application Level App Specific Perf tools/stats Guest OS CPU Utilization, Memory Utilization, I/O Latency Virtualization Level vcenter Performance Metrics /Charts Limits, Shares, Virtualization Contention Physical Server Level CPU and Memory Saturation, Power Saving Connectivity Level Network/FC Switches and data paths Packet loss, Bandwidth Utilization Peripherals Level SAN or NAS Devices Utilization, Latency, Throughput START HERE
Bonus: Key Metrics to Monitor for vsphere Resource CPU Memory Disk Network Metric Host / VM Description %USED Both CPU used over the collection interval (%) %RDY VM CPU time spent in ready state %SYS Both Percentage of time spent in the ESX Server VMKernel Swapin, Swapout Both Memory ESX host swaps in/out from/to disk (per VM, or cumulative over host) MCTLSZ (MB) Both Amount of memory reclaimed from resource pool by way of ballooning READs/s, WRITEs/s Both Reads and Writes issued in the collection interval DAVG/cmd Both Average latency (ms) of the device (LUN) KAVG/cmd Both Average latency (ms) in the VMkernel, also known as queuing time GAVG/cmd Both Average latency (ms) in the guest. GAVG = DAVG + KAVG MbRX/s, MbTX/s Both Amount of data transmitted per second PKTRX/s, PKTTX/s Both Packets transmitted per second %DRPRX, %DRPTX Both Drop packets per second
Logical Storage Layers: from Physical Disks to vmdks Guest OS disk.vmdk file Virtual Machine VMware Data store (VMFS Volume) Storage LUN GAVG Tracks latency of I/O in the guest VM Investigation Threshold: 15-20ms KAVG Tracks latency of I/O passing thru the Kernel Investigation Threshold: 1ms DAVG Tracks latency at the device driver; includes round-trip time between HBA and storage Investigation Threshold: 15-20ms, lower is better, some spikes okay Physical Disks Storage Array Aborts (ABRT/s) # commands aborted / sec Investigation Threshold: 1
Bonus: Key Performance Indicators (Storage) Storage Kernel Latency Average (KAVG) This counter tracks the latencies of IO passing thru the Kernel Investigation Threshold: 1ms Device Latency Average (DAVG) This is the latency seen at the device driver level. It includes the roundtrip time between the HBA and the storage. Investigation Threshold: 15-20ms, lower is better, some spikes okay Aborts (ABRT/s) The number of commands aborted per second. Investigation Threshold: 1 Size Storage Arrays appropriately for Total VM usage > 15-20ms Disk Latency could be a performance problem > 1ms Kernel Latency could be a performance problem or a undersized ESX device queue
Bonus: Key Performance Indicators (CPU) CPU Ready (%RDY) % time a vcpu was ready to be scheduled on a physical processor but couldn't t due to processor contention Investigation Threshold: 10% per vcpu Co-Stop (%CSTP) % time a vcpu in an SMP virtual machine is stopped from executing, so that another vcpu in the same virtual machine could be run to catch-up and make sure the skew between the two virtual processors doesn't t grow too large Investigation Threshold: 3% Max Limited (%MLMTD) % time VM was ready to run but wasn t scheduled because it violated the CPU Limit set ; added to %RDY time Virtual machine level processor queue length
Bonus: Key Performance Indicators (Others) Memory Balloon driver size (MCTLSZ) the total amount of guest physical memory reclaimed by the balloon driver Investigation Threshold: 1 Swapping (SWCUR) the current amount of guest physical memory that is swapped out to the ESX kernel VM swap file. Investigation Threshold: 1 Swap Reads/sec (SWR/s) the rate at which machine memory is swapped in from disk. Investigation Threshold: 1 Swap Writes/sec (SWW/s) The rate at which machine memory is swapped out to disk. Investigation Threshold: 1 Network Transmit Dropped Packets (%DRPTX) The percentage of transmit packets dropped. Investigation Threshold: 1 Receive Dropped Packets (%DRPRX) The percentage of receive packets dropped. Investigation Threshold: 1
Monitoring Disk Performance with esxtop Rule of thumb GAVG/cmd > 20ms = high latency! What does this mean? When command reaches device, latency is high very large values for DAVG/cmd and GAVG/cmd Latency as seen by the guest is high Low KAVG/cmd means command is not queuing in VMkernel
Tools: vscsistats DEEP Storage Diagnostics vscsistats characterizes IO for each virtual disk Allows us to separate out each different type of workload into its own container and observe trends Histograms only collected if enabled; no overhead otherwise Metrics I/O Size Seek Distance Outstanding I/Os I/O Interarrival Times Latency VMworld 2017 Content: Not for publication
Tools: Iometer An I/O subsystem measurement and characterization tool for single and clustered systems. Supports Windows and Linux Windows and Linux Free (Open Source) Single or Multi-server capable Multi-threaded Metrics Collected Total I/Os per Sec. Throughput (MB) CPU Utilization Latency (avg. & max)
Tools: Diskspd Utility (SQLIO) https://gallery.technet.microsoft.com/diskspd-a-robust-storage-6cd2f223 Windows-based feature-rich synthetic storage testing and validation tool Replaces SQLIO and effective for baselining storage for MS SQL Server workloads Fine-grained IO workload characteristics definition Configurable runtime and output options Intelligent and easy-to-understand tabular summary in text-based output VMworld 2017 Content: Not for publication
Tools: I/O Analyzer A virtual appliance solution Provides a simple and standardized way of measuring storage performance. http://labs.vmware.com/flings/io-analyzer Readily deployable virtual appliance Easy configuration and launch of I/O tests on one or more hosts I/O trace replay as an additional workload generator Ability to upload I/O traces for automatic extraction of vital metrics Graphical visualization
Tools: IO Blazer Multi-platform storage stack micro-benchmark. Supports Linux, Windows and OSX. http://labs.vmware.com/flings/ioblazer Capable of generating a highly customizable workloads Parameters like: IO size, number of outstanding Ios, interarrival time, read vs. write mix, buffered vs. direct IO IOBlazer is also capable of playing back VSCSI traces captured using vscsistats. Metrics reported are throughput and IO latency.
Storage Profiling Tips and Tricks Common IO Profiles (database, web, etc): http://blogs.msdn.com/b/tvoellm/archive/2009/05/07/useful-io-profiles-for-simulatingvarious-workloads.aspx Make Sure to Check / Try: Load balancing / multi-pathing Queue depth & outstanding I/Os pvscsi Device Driver Look out for: I/O contention Disk Shares SIOC & SDRS IOP Limits
Resources
The Links are Free. Really Virtualizing Business Critical Applications http://www.vmware.com/solutions/business-critical-apps/ http://blogs.vmware.com/apps Everything About Clustering Windows Applications on VMware vsphere http://kb.vmware.com/kb/1037959 http://vmw.re/1m9hnzl VMware s Performance Technical Papers http://www.vmware.com/files/pdf/solutions/sql_server_on_vmware-best_practices_guide.pdf https://www.vmware.com/files/pdf/solutions/sql_server_on_vmware http://www.vmware.com/files/pdf/solutions/vmware-sql-server-vsphere6-performance.pdf http://www.vmware.com/files/pdf/techpaper/vmware-sql-server-vsphere55-perf.pdf http://www.vmware.com/files/pdf/techpaper/vmw-tuning-latency-sensitive-workloads.pdf http://www.running-system.com/vsphere-6-esxtop-quick-overview-for-troubleshooting/ - ESXTop Cheat Sheet Performance Best Practices http://www.vmware.com/files/pdf/techpaper/vmware-perfbest-practices-vsphere6-0.pdf http://www.vmware.com/pdf/perf_best_practices_vsphere5.5.pdf http://www.vmware.com/pdf/perf_best_practices_vsphere5.1.pdf Something for the DBA in You http://www.vmware.com/files/pdf/solutions/dba_guide_to_databases_on_vmware-wp.pdf