A Support Engineer Walkthrough on pt-stalk

Transcription

1 A Support Engineer Walkthrough on pt-stalk Marcos Albe Principal Support Engineer - Percona Marcelo Altmann Senior Support Engineer - Percona

2 Thank You Sponsors! 2

3 SAVE THE DATE! April 23-25, 2018 Santa Clara Convention Center CALL FOR PAPERS OPENING SOON! 3

4 Who is speaking?

5 Who is Speaking? Marcelo Altmann - Senior Support Percona - Resident specialist on Surf & Brazil leisure Marcos Albé - Principal Support Percona - Chief Barbeque Officer & Designated Beer Analyst 5

6 Agenda

7 Agenda What do we USE Who s stalkin? How pt-stalk can be USEd Walkthrough 7

8 What do we USE

9 We use one of these... 9

10 Utilization 10

11 Utilization Saturation 11

12 Utilization Saturation Errors 12

13 13

14 Who s stalkin?

15 check out those resources! 15

16 Who s stalkin? Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron) # The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun. Now part of pt-toolkit Monitors variable or a script s single-valued output Triggers capture when set threshold is reached - files from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat. - vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof). - SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL PROCESSLIST and SHOW OPEN TABLES. Production safe! Custom triggers and custom captures Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection 16

17 Who s stalkin : common parameters Stalking part - --daemonize --no-stalk --function=status --interval=1 --variable=threads_running --threshold=25 --cycles=5 Collecting part run-time=30 --sleep=300 --iterations= --plugin= --dest=/var/lib/pt-stalk Values shown are the defaults for each (where given)

18 Who s stalkin : work flow 18

19 Who s stalkin : working examples I Collect data during 60 seconds for 3 consecutive times, starting immediately. pt-stalk --no-stalk \ --dest=$home/percona \ --user=root \ --ask-pass \ --run-time=60 --sleep=60 --iterations=3 19

20 Who s stalkin : working examples II Run as a daemon and trigger collection if InnoDB has more than 5 pending fsyncs. pt-stalk --daemonize \ --function=status \ --variable=innodb_data_pending_fsyncs \ --threshold=5 \ --cycles=2 \ --interval=1 \ --iterations=5 \ --plugin=/usr/local/bin/raid_card_status.sh 20

21 Who s stalkin : custom triggers #!/bin/bash # Save this bash snippet as /root/delayed_slave_trg.sh; trg_plugin() { mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" \ grep "Seconds_Behind_Master" \ awk "{ print $2 }"; } # use like pt-stalk --function=/root/delayed_slave_trg.sh --etc --etc 21

22 Who s stalkin : custom capture #!/bin/bash # save as /root/collect_perf.sh before_collect() { # run perf stat; match sleep length to --run-time /usr/bin/perf stat --all-cpus \ --output=${d}/${p}-stat_a \ -- sleep ${OPT_RUN_TIME}; } # use like pt-stalk --plugin=/root/collect_perf.sh --etc --etc 22

23 How pt-stalk can be USE d

24 How pt-stalk can be USE d Applications System libraries System Call Interface VFS CPU Interconnect Sockets Linux Kernel Scheduler File Systems TCP/UDP Volume Manager IP Block Device Interface Ethernet CPU Virtual Memory Memory Bus DRAM Device Drivers I/O Bus I/O Bridge Expander Interconnect I/O Controller Network Controller Interface Transports Disk 24 Disk Swap Port Port

25 SHOW ENGINE INNODB STATUS SHOW ENGINE INNODB MUTEX oprofile SHOW OPEN TABLES /proc/interrupts Applications (MySQL!) SHOW FULL PROCESSLIST /proc/procstat System libraries SHOW ENGINE TOKUDB STATUS mpstat System Call Interface SHOW GLOBAL VARIABLES VFS SHOW SLAVE STATUS CPU Interconnect Sockets Scheduler Linux Kernel mysqladmin ext gdb (backtraces) strace lsof File Systems TCP/UDP Volume Manager IP Block Device Interface Ethernet ps top df Virtual Memory iostat I/O Bus /proc/diskstats sysctl I/O Bridge vmstat /proc/meminfo /proc/slabinfo Device Drivers netstat tcpdump /proc/vmstat pmap Expander Interconnect I/O Controller Network Controller Interface Transports Disk 25 CPU Disk Swap Port Port Memory Bus DRAM

26 How pt-stalk can be USE d: Here Be Dragons! gdb (backtraces) strace 26 oprofile tcpdump

27 A walkthrough

28 A walkthrough Using pt-stalk to solve problems with High IO...High CPU...Networking

29 Useful links 29 grep commands 203d Check List -

30 Questions? Marcos Albe Marcelo