Monitoring MySQL Performance with Percona Monitoring and Management

Transcription

1 Monitoring MySQL Performance with Percona Monitoring and Management Santa Clara, California April 23th 25th, 2018 MIchael Coburn, Product Manager

2 Your Presenter Product Manager for PMM (also Percona Toolkit and Percona XtraBackup) With Percona for 6 years through 6 different roles Consultant, Managing Consultant, Principal Architect, Technical Account Manager, Principal Support Engineer #monitoring-mysql-perf Percona Live App 2

3 Agenda Introductions - 1:30pm Part 1 - Installation and Configuration - 1:45pm Break ~3:00pm Part 2 - Query Analytics - 3:15pm Part 3 - Metrics Monitor - 3:45pm Questions - 4:15pm 3

4 Goals of Today's Tutorial 1. Understand the components of PMM pmm-client - Client tools & agents you install on each server PMM Server i. Prometheus, Grafana, Consul, QAN 2. Able to install PMM Server at your site OVF - Open Virtualization Format (VMware, Microsoft Systems Center, Virtualbox) docker AWS Marketplace 3. Review MySQL queries using Query Analytics 4. Analyze MySQL performance using Metrics Monitor 4

5 What is PMM Free, Open Source database troubleshooting and performance optimization platform for MySQL and MongoDB We also support: ProxySQL Amazon RDS MySQL Amazon Aurora MySQL Runs in your secure environment (this is not a SaaS product!), on your equipment Secured with SSL between client and server 5

6 PMM Distribution Methods docker docker pull percona/pmm-server:latest Virtual Appliance Supports VMware, RedHat Virtualization, Microsoft Systems Center and VirtualBox! AWS Marketplace Production-ready AMI running in EC2 Available since November

7 AWS Marketplace Deploy directly to EC2 Running CentOS 7 Search for "pmm" or "Percona Monitoring and Management" 7

8 PMM Architecture pmm-client mysqld_exporter node_exporter qan-agent PMM Server Query Analytics QAN API QAN App Metrics Monitor Prometheus Grafana Consul 8

9 PMM Server Components Metrics Monitor Prometheus Timeseries database Powerful PromQL query language Grafana Visualization platform Consul Tracks which services are available to be scraped by Prometheus Query Analytics View query performance in real-time Aggregated for queries consuming most amount of time in MySQL Query drill-down for individual query performance Rows read, Rows scanned, Query time, Query count InnoDB statistics (Percona Server for MySQL only 9

10 pmm-client Components pmm-admin Command-line tool for client management node_exporter Agent that exports Linux metrics mysqld_exporter Agent that exports MySQL server metrics qan-agent Agent that collects query metrics from MySQL Slow Log or PERFORMANCE_SCHEMA 10

11 Prometheus Data Collection Prometheus server asks Consul for which services & instances to query by IP address and port Example: curl Prometheus exporter performs data collection upon curl request Exporter generates text exposed via web server at :42002/metrics ~]# curl -s -k grep mysql head -8 # HELP mysql_exporter_collector_duration_seconds Collector time duration. # TYPE mysql_exporter_collector_duration_seconds gauge mysql_exporter_collector_duration_seconds{collector="collect.global_status"} mysql_exporter_collector_duration_seconds{collector="collect.info_schema.innodb_metric s"} mysql_exporter_collector_duration_seconds{collector="connection"} e-05 # HELP mysql_exporter_hr_last_scrape_error Whether the last scrape of metrics from MySQL resulted in an error (1 for error, 0 for success). # TYPE mysql_exporter_hr_last_scrape_error gauge mysql_exporter_hr_last_scrape_error 0 11

12 Part 1 Installation and configuration

13 Environment Notes Authentication: percona / percona client ssh percona@ server ssh admin@ ~/.ssh/config Host client User percona HostName IdentityFile ~/.ssh/pmm 13

14 PMM Installation Steps High level steps: Import Appliance into VirtualBox Configure VirtualBox Host-Only Networking Validate client and server can communicate Start up sysbench test (optional) Configure pmm-admin to start: linux:metrics mysql:metrics mysql:queries 14

15 Configuring VirtualBox Import Appliance File > Import Appliance Configure Network VirtualBox > Preferences > Network > Host-only Networks Create new host-only network via button on right Adapter Configure IPv4 Address: DHCP Server Enable Server Configure Server Address: Configure Lower Address Bound: Configure Upper Address Bound: Consider setting Execution Cap at 50%, 2 cores Per instance 15

16 VirtualBox Host Only Network < v5 - OSX 16

17 VirtualBox > v5 - Windows 17

18 VirtualBox Network Interfaces - OSX 18

19 VirtualBox Network Interfaces - Windows 19

20 VirtualBox Network Interfaces - Windows Start the two instances via VirtualBox console Connect using ssh Optional: ssh-agent bash ssh-add ~/.ssh/pmm ssh -l percona [client $] ssh admin@ Ensure the nodes can see each other [percona@client ~]$ ping pmm-server [admin@pmm-server ~]$ ping client ~/.ssh/config Host client User percona HostName IdentityFile ~/.ssh/pmm Configure pmm-server for correct time [pmm-server $] date --set='timefromclient' 20

21 Starting Sysbench (optional) We use sysbench to generate load in MySQL Start the script: client$./sysbench.sh Configuration is for 1 trx/sec of 20 events on a 10k row InnoDB table 21

22 Configuring PMM FINALLY! Connect client to pmm-server sudo pmm-admin config --server server-user percona --server-password percona Start data collection sudo pmm-admin add mysql --create-user --socket /var/lib/mysql/mysql.sock --user percona --password percona Generates a MySQL user account with bare minimum privileges Creates three services pmm-linux-metrics pmm-mysql-metrics pmm-mysql-queries 22

23 Confirming it all Works PMM Server: Prometheus Do they work? Great - take a break! No? Let's Troubleshoot (next slide ) 23

24 Troubleshooting PMM Check for any red fields: sudo pmm-admin list sudo pmm-admin check-network Time drift? pmm-server$ sudo date --set='<timefromclient>' Restarting one or all components sudo pmm-admin restart linux:metrics pmm-client sudo pmm-admin restart --all Logs are in /var/log/pmm-*.log Check targets status in Prometheus 24

25 Confirming IP address of PMM Server Confirm the IP address the server obtained from DHCP 25

26 If all else fails... I have a few PMM Server instances available in EC2: To obtain a shell: ssh -i ~/.ssh/pmm ec2-user@<ip> 26

27 Break! We'll resume in 15 minutes

28 Using Query Analytics (QAN) Examining queries in depth

29 Query Analytics Dashboard 29

30 Query Analytics Overview Query Abstract Query pattern with placeholders ID Unique fingerprint, used for query group by Load Grand Total Time - percentage of time that MySQL server spent executing the query Count QPS, total count during window, % of total Latency Min, Med, Avg, P95, Max 30

31 PERFORMANCE_SCHEMA 31

32 Slow Log - Percona Server Enhanced 32

33 EXPLAIN - Table and JSON 33

34 CREATE TABLE, TABLE STATUS, and INDEXES 34

35 Server Summary Information Collects and displays per Server: pt-summary pt-mysql-summary _PMM System Summary Summary can be downloaded from the UI 35

36 Part 3 - Using Metrics Monitor Eye candy

37 Grafana in a Nutshell Open Source data visualisation tool Popular datasources Prometheus CloudWatch Graphite Elasticsearch Templated Variables Define your graph metrics, and let the hosts get filled in automatically GREAT for large, dynamic environments where hosts are considered ephemeral 37

38 Prometheus revisited Timeseries database - metric name + key/value pairs mysql_global_variables_innodb_buffer_pool_instances{instance= "ps57",job="mysql"} = 8 mysql_slave_status_slave_io_running{instance="ps57r",job="mys ql",master_host=" ",master_uuid=" d-4d97-11e 6-b85e-0007cb03dc86"} = 1 Flexible query language - PromQL Collection of metrics based on HTTP pull Targets identified via service discovery or static configuration files We're using consul in PMM for service discovery 38

39 How can I... Compare servers to each other Cross Server graphs Show behaviour now() compared to past period Trends Overview dashboard At a glance MySQL + indepth MySQL Overview, InnoDB, InnoDB Advanced Table statistics* Largest tables by rows and size, total DB size, tables by rows read and changed, auto_increment usage (about to hit the limit?) User statistics* Top users by connection count, network usage, rows read/changed 39

40 Annotations Visualize Application Events in PMM pmm-admin annotate "Application deployment v1.3" 40

41 Alerting Alerting Cannot use Templated Variables Instead, replace with string constants for instance name 41

42 Almost the end Parting thoughts

43 Advice PMM Metrics retention is 30 days We are looking at options to present a longer history mysql:metrics are polled at 1s, 5s, and 60s resolutions, and linux:metrics is every 1s On high-latency links you might need to tune scrape_interval up Don't skimp on resources Prometheus in particular needs a lot of CPU cores and fast disks, in order to sequence scrape data before writing chunks to disk Consider disabling some mysqld_exporter features to minimise performance impact --disable-tablestats, --disable-processlist Keep queries in the database (security) --disable-queryexamples 43

44 PMM Roadmap Prometheus 2.x - faster, more instances per PMM Server PostgreSQL Support MySQL -> ClickHouse for QAN datastore faster, aggregation across all servers, new filtering and sorting options Long term metrics storage (past 30 days) One-click ticket submission* Standardised data collection for tickets* Any feedback of what you'd like to see in PMM? * for Percona Subscribers (Customers) only 44

45 Thank You Sponsors!! 45

46 Questions? Michael Coburn Percona is looking for MongoDB and MySQL rockstars! Be sure to stop by Percona s booth. Do you have any areas or benchmarks you want Percona to talk about in blogs together? Any features or tools you think we should focus on to meet the community's needs? 46

47 Rate My Session 47

48 Thank You!