Monitoring Linux Performance for the SQL Server Admin. Anthony E. Nocentino, Enterprise Architect, Centino Systems

Transcription

1 Monitoring Linux Performance for the SQL Server Admin Anthony E. Nocentino, Enterprise Architect, Centino Systems

2 Consultant and Trainer Founder and President of Centino Systems Specialize in system architecture and performance Computer Science, M.S. and B.S. Microsoft MVP - Data Platform Friend of Redgate Linux Foundation Certified Engineer Microsoft Certified Professional Anthony E. Nocentino Enterprise Architect, Centino Systems aen@centinosystems.com Other places online Blog - Pluralsight Author

3 Agenda Linux System Architecture SQL on Linux Architecture System Components CPU/Processes Memory/Pages Disk/File Systems Monitoring Tools

4 Things we re going to cover Linux OS concepts, how it works! Tools to view performance data What s good and what s bad Things we re NOT going to cover SQL Server internals Performance troubleshooting

5 Linux Architecture User Space Users Interact with the Shell Cause Problems :) Shell Executes Your Commands Your Interface to the Kernel Commands, Editors any User Program Kernel Space Kernel Resource Management and Access Process, Pages and File Systems Hardware Physical Resources CPU, Memory and Disk

6 SQLOS Scheduling Synchronization I/O Memory Management Placing tasks into workers and getting access to the CPU Controlling access to system resources Scheduling of I/O both network and disk Allocation of memory to various system objects Primary function is resource management specific to RDBMS A new platform layer in SQL Server 2005 to exploit new hardware capabilities and their trends S. Oks Operating System support for Database Management M. Stonebraker

7 SQL on Linux Architecture - Drawbridge From:

8 SQL on Linux Architecture - SQLPAL From:

9 SQL on Linux Architecture - Process Layout From:

10 SQL on Linux Architecture - SQLPAL All other Code Performance Sensitive Code Win32 Hosted APIs (LibOS) SQL OS API SQL OS v2 Host Extension (Memory, Threads/Scheduling, I/O) Linux Kernel via System Calls From:

11 SQL on Linux Architecture - Host Extensions Call table maps Win32 API semantics to Linux System calls ~45 ABI Calls Memory Management Threads and Scheduling Synchronization Primitives I/O Network and Disk We care a lot about host extensions it s more code

12 Shhhhhh - SQLPAL is Virtualization ;) Process virtualization (not machine) Presenting another environment inside the process context that s different than that of the hardware s operating environment But the environment is purpose built for SQL Server We need to understand that this is a hybrid Win32/Linux process and have a firm grasp of Resource allocation and management in SQLPAL How that turns into Linux OS performance Debugging

13 CPU and Processes

14 What is a Process Process Executing program, program code, memory and resources Thread (LWP) Shared access to resources Process and Thread Creation fork/exec - parent process yields a child process with a PID clone - same address space as thread creator, cheap and fast! Process Tree The hierarchy of parent and it s child processes

15 What is a Process (con t) Process States Running Stopped Zombie Runnable Sleeping

16 Controlling Processes Signals Methods of process control kill and killall Niceness Set the execution priority nice and renice Default 20, lower is less nice

17 More on Processes Context switching Kernel versus User Mode CPU Scheduling How is a SQLOS Worker scheduled onto the CPU? Creates a thread via pthread and that s pushed into the scheduler pthreads?

18 Process/Thread Scheduling Unit of scheduling is the thread Default scheduler is SCHED_OTHER/SCHED_NORMAL Time sharing scheduler Preemptive Dynamic priority list, based on niceness Calculated quantum length based on priority kernel.sched_min_granularity_ns = (10ms) - default kernel.sched_wakeup_granularity_ns = (15ms) - default NUMA Aware, but kernel.numa_balancing = 0 - default

19 CPU - What to look for? Percentage of what? Load average Run queue length and I/O waits Spikes aren t bad Long waits User I/O - disk latency will effect access to the CPU System

20 Tools to use for process monitoring top/htop ps mpstat/pidstat dstat procfs

21 Demos Processes and threads Run load average under CPU saturation Exploring procfs

22 Memory and Pages

23 Memory Memory Layout and Architecture Physical and Virtual Memory NUMA - free lists per node Pages (Anonymous) Demand Paging Swap out Time and Pressure Swap in, Major Page Fault Allocation, Minor Page Fault File System Cache and swappiness - vm.swappiness = 10 (default 30, 0 disables swapping) vm.dirty_ratio = 40 (default 30) vm.max_map_count = (default 65530)

24 Pages Regular pages - 4KB Transparent huge pages - 2MB Increases memory I/O by decreasing TLB cache misses SQLOSv2 Can request large pages inside SQL Server with trace flag 834 SQL will allocate memory on start up When SQLPAL exposes 8GB+ to SQL Server As of today, no locked pages but TF 835 is on?

25 Hello Old Friend AWE On Windows Lock Pages in Memory is Address Window Extensions (AWE) Allocates contiguous mappings to PFNs. Logically contiguous, but not guaranteed contiguous Linux will try to make the THPs contiguous Then those PFNs are mapping into the process virtual address space Why use AWE? How are they unpageable?

26 Memory - What to look for? High consumers of space Physical Virtual External memory pressure on SQL Server Excessive swapping swapping in/out

27 Tools to use for memory monitoring /proc/meminfo free top/htop ps vmstat pidstat

28 Demos Memory layout Isolating a memory hog Identifying external memory pressure External memory pressure on SQL Server Excessive swapping Swapping in/swapping out

29 Disks and File Systems

30 Disks Sectors (physical) Actual storage unit of the disk, 512B or 4KB Blocks (logical) Fundamental unit of I/O, allocation Disks have finite performance characteristics Bandwidth - how much data Latency - how fast Storage Interconnects Internal External

31 File Systems XFS Default file system - EXT4 Block size Impact utilization and performance nominally 4KB default block size Mount time options Access times - noatime

32 Block Allocation in Linux XFS and EXT4 essentially the same Files i-nodes Extents Blocks

33 I/O under SQLPAL Stream I/O via NTUM Fast I/O via the host extension Kernel asynchronous IO (kaio) io_submit() Returns to caller immediately, completion polling is in user space O_DIRECT - bypasses page cache and I/O stays in user mode fsync() probably designed by a deranged monkey on some serious mind-controlling substances. - Linus man 2 open

34 Disks - What to look for? This is the slowest thing in your computer, sorry Argenis! :) Saturated disks and I/O subsystems Swapping Caching is your friend (generally, but not in an RDBMS) Baseline!

35 Tools to use for disk monitoring iostat iotop pidstat dstat

36 Demos Finding high I/O processes Measuring disk latency (DMVs and cmd line tools) sys.dm_io_virtual_file_stats

37 Monitoring Tools

38 Baselining Tools Nearly everything we ve talked about so far has been point in time what about baselining? sar - System Activity Reporter dstat - writes to CSV

39 Tools for Monitoring SQL Server You have all of the same tools you re used to for SQL Server Because of SQLOS we get DMVs Extended Events

40 New Tools Available for SQL on Linux New DMVs PSSDiag DBFS Grafana

41 Metrics Captured by PSSDiag Don t just listen to me here s what Microsoft is interested in CPU - mpstat, pidstat Disk - iostat, iotop Memory - free, sar Network - sar DMV Data System log information

42 Review Linux System Architecture SQL on Linux Architecture System Components CPU/Processes Memory/Pages Disk/File Systems Monitoring Tools

43 Need more data? Blog Pluralsight Understanding and Using Essential Tools for Enterprise Linux 7 Linux basics, system architecture, file and directory management LFCE: Advanced Network and System Administration systemd, Performance and Tools

44 References Many of the man pages

45 Session evaluations Your feedback is important and valuable. Submit by 5pm Friday, November 10 th to win prizes. 3 Ways to Access: Go to passsummit.com Download the GuideBook App and search: PASS Summit 2017 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide

46 Thank You Join me for the BOF lunch from