davidklee.net heraflux.com linkedin.com/in/davidaklee

Transcription

1 @kleegeek davidklee.net heraflux.com linkedin.com/in/davidaklee Specialties / Focus Areas / Passions: Performance Tuning & Troubleshooting Virtualization Cloud Enablement Infrastructure Architecture Health & Efficiency Capacity Management High Availability Disaster Recovery PASS Virtual Chapters Virtualization HA / DR Performance Heraflux Technologies 2 1

2 It s running slower but nothing changed on my SQL Server!. becomes How can I tell if my infrastructure changed and is slowing it down? Heraflux Technologies 3 Where to start? Bottleneck analysis Storage & interconnects Compute hardware Virtualization Operating system SQL Server instance and database Networking Application Heraflux Technologies 4 2

3 Our business critical application is running 80x slower than yesterday, and our end users are complaining. NOTHING changed on my production SQL Server. What do you do? Most dive into queries. Start holistically before you dive into the queries because. Heraflux Technologies 5 App request Firewall Load Balancer Database FG1 FG2 Log Web Server Web Server Web Server MDF NDF NDF NDF LDF App Server Report Server Desktop App Desktop App Desktop App Desktop App SQL Server Instance E: F: G: L: SCSI Controller Queue Windows Operating System SCSI Controller Queue Virtualization Hypervisor vdisk0 vdisk1 vdisk2 vdisk3 HBA pswitch SAN Controller Disk Pool Heraflux Technologies 6 3

4 Application SQL Server DB SQL Server Instance Operating System Virtualization Networking Interconnects Physical Server Storage Heraflux Technologies 7 SQL Server tells you what it s waiting on! Use SQL Server wait statistics as a guide Don t jump on any one statistic Shift the bottleneck Glenn Berry diagnostic scripts available at: Heraflux Technologies 8 4

5 Performance Gains Knowledge is everything Your sweet spot! Just focus on data handling OS Virtualization SQL Server Instance Hardware SQL Server Database(s) Application Heraflux Technologies 9 (Adapted from MSDN Source: It always comes back to benchmarking and baselines Methods: Third party tools Do it yourself solutions Infrastructure baselines: Storage performance per test type Network throughput CPU test SQL Server baselines: Long-running queries Reports ETL / backups / maintenance Heraflux Technologies 10 5

6 Start with SQL Server wait statistics analysis Paul Randal Tell me where it bit.ly/1mibfxl Tom Davidson Waits and queues bit.ly/1e1i38f Enrico van de Laar - Wait stat bit.ly/1jvqhmf Glenn Berry diag query top waits Heraflux Technologies 11 Usually the slowest and least responsive part of the stack Perfmon Physical Disk counters Disk sec/read & Write = latency in ms Disk Reads/sec & Writes/sec = IOPs sys.dm_io_virtual_file_stats Cumulative since the last service startup Check the error log Heraflux Technologies 12 (Image from Henk van der bit.ly/1qa6qwk) 6

7 Network Paths & Switches Storage Paths & Switches LUN 1 LUN 2 LUN 3 Storage Controllers & Disk Groups Heraflux Technologies 13 DiskSpd Batch: heraflux.com/go/diskspd Graph averages by operations per thread IOps Per Operations by Intensity IOps Sequential Read Random Read Sequential Write Random Write Thread Intensity Heraflux Technologies 14 7

8 Work with storage admins Add more SQL Server memory Use storage interconnect multipathing Faster individual interconnects Block size alignment with SAN / architecture HBA queue depth settings Improve disk configuration for performance Faster RAID configuration SSD caching / tier Replace storage with faster array Spread out workload Heraflux Technologies 15 SAN DB Primary FG DF1 E: F: G: Windows Server OS x Virtualization Hardware HBA4 HBA3 HBA2 Interconnect Switch Controller 1 Controller 2 LUN 2 HBA1 LUN 1 SAN Disk Group Heraflux Technologies 16 8

9 SAN DB FG1 FG2 DF1 DF2 DF3 DF4 E: F: G: Windows Server OS x Virtualization Hardware HBA4 HBA3 HBA2 HBA1 Interconnect Switch Controller 1 Controller 2 LUN 2 LUN 1 SAN Disk Group Heraflux Technologies 17 Have you overwhelmed the hardware itself? CPU? Memory? Heraflux Technologies 18 9

10 Waits CXPacket, SOS_Scheduler_Yield, ThreadPool, LCK_XX MaxDOP Cost Threshold for Parallelism Not enough CPUs? Is it really SQL Server? Virtualization CPU scheduling & background contention Heraflux Technologies 19 Waits - Resource_Semaphore Low page life expectancy & low buffer cache hit ratio Very high I/O as a result High CPU consumption Virtual machine memory ballooning RAM is cheap! Artificial licensing limitations are not Heraflux Technologies 20 10

11 Background activity on the same host can cause performance variations 32 vcpu 256 GB vram 8 vcpu 64 GB vram 2 vcpu 16 GB 2 vcpu 16 GB 2 vcpu 16 GB 2 vcpu 16 GB 2 vcpu 16 GB 2 vcpu 16 GB V I R T U A L I Z A T I O N 40 CPU 512GB RAM Heraflux Technologies 21 Get access! Host and in-guest-level metrics. Read-only! Heraflux Technologies 22 11

12 Validate your -level CPU configuration Generate a HammerDB or sample workload that best represents your average SQL Server workload Adjust vnuma config / vcpu counts & retest Transactions / min vnuma SQL Server Scalability - 16 vcpus 31% Difference! Concurrent HammerDB Users 4socket x 4CPU 8socket x 2CPU 2socket x 8CPU Heraflux Technologies 23 Background tasks Antivirus real-time scanning Sysadmin tools OS-level disk and I/O queueing More disks / drive letters / mount points Spread out the workload Licensing limitations Artificial limits on CPU & memory utilization Heraflux Technologies 24 12

13 Reduce I/O Backup compression Instant file initialization Multiple tempdb data files Large memory pages Min and max memory SQL Server 2014 Buffer Pool Extensions CPU Affinity MaxDOP & Cost Threshold for Parallelism Stagger job timings Heraflux Technologies 25 Spread out the workload File groups, data files, partitions Slow I/O? Reduce footprint with table & index compression In-memory constructs such as In-Memory OLTP and updatable Columnstore indexing Workload analysis for Cost Threshold script at bit.ly/1nxrxin SSD read caching Heraflux Technologies 26 13

14 Waits - ASYNC_NETWORK_IO Application might not be able to consume data quickly OR Application has to communicate with the DB server End-user application has to communicate to the app server or DB Validate network throughput and stability between application and database servers Heraflux Technologies 27 Verify and test end-to-end performance Use iperf3 to test network throughput bit.ly/1lzuyry How-To Guide: tinyurl.com/ost9gll Heraflux Technologies 28 14

15 Heraflux Technologies 29 SQL Server performance might not be a code problem Opportunities to tune the infrastructure are all over the place Use the tools that you have available to you to find the infrastructure bottlenecks to pursue Shift the bottleneck until it s acceptable Heraflux Technologies 30 15

16 @kleegeek davidklee.net heraflux.com linkedin.com/in/davidaklee Heraflux Technologies 31 16