A Support Engineer Walkthrough on ptstalk

Transcription

1 A Support Engineer Walkthrough on ptstalk Marcos Albe Principal Support Engineer - Percona 1

2 Who is Speaking? Marcos Albé - Principal Support Percona - 7 years here - Focus on performance and observability 2

3 Agenda 3

4 Agenda 1. What do we USE 2. Who s stalkin? 3. How pt-stalk can be USEd 4. Walkthrough 4

5 What do we USE 5

6 What do we USE We use one of these... 6

7 What do we USE Utilization 7

8 What do we USE Utilization Saturation 8

9 What do we USE Utilization Saturation Errors 9

10 What do we USE 10

11 Who s stalkin? 11

12 Who s stalkin? check out those resources! 12

13 Who s stalkin? Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron) # The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun. Now part of pt-toolkit Monitors variable or a script s single-valued output Triggers capture when set threshold is reached - files from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat. - vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof). - SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL PROCESSLIST and SHOW OPEN TABLES. Production safe! Custom triggers and custom captures Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection 13

14 Who s stalkin : common parameters Stalking part - --daemonize --no-stalk - --function=status - --interval=1 - --variable=threads_running - --threshold= cycles=5 Collecting part - --run-time= sleep= iterations= - --plugin= - --dest=/var/lib/pt-stalk Values shown are the defaults for each (where given) 14

15 Who s stalkin : work flow 15

16 Who s stalkin : working examples I Collect data during 60 seconds for 3 consecutive times, starting immediately. pt-stalk --no-stalk \ --dest=$home/percona \ --user=root \ --ask-pass \ --run-time=60 --sleep=60 --iterations=3 16

17 Who s stalkin : working examples II Run as a daemon and trigger collection if InnoDB has more than 5 pending fsyncs. pt-stalk --daemonize \ --function=status \ --variable=innodb_data_pending_fsyncs \ --threshold=5 \ --cycles=2 \ --interval=1 \ --iterations=5 \ --plugin=/usr/local/bin/raid_card_status.sh 17

18 Who s stalkin : custom triggers #!/bin/bash # Save this bash snippet as /root/delayed_slave_trg.sh; trg_plugin() { mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" \ grep "Seconds_Behind_Master" \ awk "{ print $2 }"; } # use like pt-stalk --function=/root/delayed_slave_trg.sh - -etc --etc 18

19 Who s stalkin : custom capture #!/bin/bash # save as /root/collect_perf.sh before_collect() { # run perf stat; match sleep length to --runtime /usr/bin/perf stat --all-cpus \ --output=${d}/${p}-stat_a \ -- sleep ${OPT_RUN_TIME}; } # use like pt-stalk --plugin=/root/collect_perf.sh --etc --etc 19

20 How pt-stalk can be USE d 20

21 How pt-stalk can be USE d Applications System libraries Linux Kernel System Call Interface VFS Sockets File Systems TCP/UDP Volume Manager IP Block Device Interface Ethernet Scheduler Virtual Memory CPU Interconnect CPU Memory Bus Device Drivers DRAM I/O Bus Expander Interconnect I/O Bridge I/O Controller Network Controller 21 Interface Transports Disk Disk Swap Port Port

22 SHOW ENGINE INNODB STATUS SHOW ENGINE INNODB MUTEX oprofile SHOW OPEN TABLES SHOW FULL PROCESSLIST SHOW ENGINE TOKUDB STATUS Applications (MySQL!) System libraries /proc/interrupts /proc/procstat mpstat SHOW GLOBAL VARIABLES SHOW SLAVE STATUS mysqladmin ext gdb (backtraces) strace lsof df iostat /proc/diskstats sysctl Linux Kernel System Call Interface VFS Sockets File Systems TCP/UDP Volume Manager IP Block Device Interface Ethernet Device Drivers I/O Bus Expander Interconnect I/O Bridge Scheduler Virtual Memory netstat tcpdump CPU Interconnect ps top vmstat /proc/meminfo /proc/slabinfo /proc/vmstat pmap CPU DRAM Memory Bus I/O Controller Network Controller Disk Disk Swa p Interface Transports Port Port 22

23 How pt-stalk can be USE d: Here Be Dragons! gdb (backtraces) oprofile strace tcpdump 23

24 A walkthrough 24

25 A walkthrough Using pt-stalk to solve problems with......high CPU...High IO...Network 25

26 Demo 26

27 Get Your Tickets for Percona Live Europe! Championing Open Source Databases MySQL, MongoDB, Open Source Databases Time Series Databases, PostgreSQL, RocksDB Developers, Business/Case Studies, Operations September 25-27th, 2017 Radisson Blu Royal Hotel, Dublin, Ireland Last Year s Conference Sold Out! Reserve your spot ASAP. 27

28 Talk to Percona Experts at AWS Re:Invent! Database Performance for Cloud Deployments Percona Support and Managed Services Amazon RDS, Aurora, Roll Your Own MySQL/MariaDB/MongoDB Reduce costs and optimize performance Percona Monitoring and Management Demos Point-in-time visibility and historical trending of database performance Detailed query analytics Booth #