Monitoring MySQL Performance with Percona Monitoring and Management

Monitoring MySQL Performance with Percona Monitoring and Management

Your Presenters Michael Coburn - PMM Product Manager Working at Percona for almost 5 years Consultant, Manager, TAM, now Product Manager Canadian living in México Roma Novikov - PMM Development Manager Joined Percona in January 2017 Lviv, Ukraine http://bit.ly/joinperconaliveslack #monitoring-mysql-perf Percona Live App 2

Agenda Introductions - 1:30pm Part 1 - Installation & Configuration - 1:45pm Refreshments break ~2:30pm Part 2 - Using Query Analytics (QAN) - 2:45pm Part 3 - Using Metrics Monitor - 3:30pm Questions - 4:15pm 3

Goals of this tutorial 1. 2. 3. 4. 5. Build awareness of PMM and why it exists Understand the components of PMM Be able to install PMM at your own site Interpret and action slow queries in QAN Analyse MySQL metrics using Metrics Monitor 4

Percona Monitoring and Management (PMM) Free, Open Source database monitoring and management platform for MySQL, MariaDB, ProxySQL, and MongoDB Runs in your secure environment (not a SaaS), on your equipment Secured with SSL encryption between pmm-client and pmm-server 5

PMM distribution methods Docker Most common deployment method for PMM today VirtualBox Designed for non-production environments Can be used for one-off troubleshooting tasks What we'll be using today AWS AMI Production-ready Available in all 14 regions Coming to AWS Marketplace during Q2 2017 6

PMM architecture 7

pmm-server components Prometheus - timeseries data store Grafana - visualisation system on top of Prometheus Percona Dashboards Query Analytics (QAN) - individual and aggregated query performance Orchestrator - MySQL replication topology manager and visualisation tool We won't be discussing Orchestrator during this talk (sorry Shlomi) 8

pmm-client components pmm-admin Command-line tool for managing clients percona-qan-agent QAN service that collects query performance metrics from Slow Log or PERFORMANCE_SCHEMA node_exporter Prometheus exporter of Linux server metrics mysqld_exporter Prometheus exporter of MySQL server metrics 9

Part 1 Installation & Configuration

Environment notes Authentication percona / percona Or use the ssh key provided pmm-client - 192.168.56.2 ssh 192.168.56.2 -l percona CentOS 7 with Percona-Server-server-57 firwalld disabled ~/.ssh/config Host pmm-client User percona HostName 192.168.56.2 IdentityFile ~/.ssh/monitoring_mysql_performance pmm-server - 192.168.56.3 ssh 192.168.56.3 -l admin 11

PMM installation steps 1. 2. 3. 4. 5. Import Appliance into VirtualBox Configure VirtualBox networking Confirm pmm-client and pmm-server hosts can communicate Configure sysbench load tests (optional) Configure pmm-admin for MySQL + Linux statistics, and queries 12

Configuring VirtualBox Import Appliance File > Import Appliance... Configure Private Network VirtualBox > Preferences > Network > Host-only Networks Create new host-only network via button on right Adapter Configure IPv4 Address: 192.168.56.1 DHCP Server Enable Server Configure Server Address: 192.168.56.1 Configure Lower Address Bound: 192.168.56.2 Configure Upper Address Bound: 192.168.56.10 Consider setting Execution Cap at 50%, 2 cores Per instance 13

VirtualBox host only network 14

Configure network interfaces 15

Connecting to instances Start the two instances via VirtualBox console Connect using ssh ~/.ssh/config Optional: ssh-agent bash ssh-add monitoring_mysql_performance ssh -l percona 192.168.56.2 [pmm-client$] ssh pmm-server Host pmm-client User percona HostName 192.168.56.2 IdentityFile ~/.ssh/monitoring_mysql_performance Ensure the nodes can see each other [percona@pmm-client ~]$ ping pmm-server [admin@pmm-server ~]$ ping pmm-client Configure pmm-server for correct time [pmm-server $] date --set='timefromclient' 16

Starting sysbench (optional) Fire off the launcher script pmm-client$ cd ~/tests pmm-client$ sh start_all.sh Will take approximately 5 minutes to spin up Laptop too slow? Pause the pmm-client instance and configure Execution Cap 17

Configuring PMM FINALLY! Connect pmm-client to pmm-server sudo pmm-admin config --server 192.168.56.3 --server-user percona --server-password percona Start data collection sudo pmm-admin add mysql --create-user --socket /var/lib/mysql/mysql.sock --user percona --password percona Generates a MySQL user account with bare minimum privileges Creates three services pmm-linux-metrics pmm-mysql-metrics pmm-mysql-queries 18

Checking if it all works Load these pages: Landing page QAN Metrics Monitor Prometheus http://192.168.56.3 http://192.168.56.3/qan http://192.168.56.3/graph http://192.168.56.3/prometheus Do they work? Great - take a break, refreshments are available at 2:30pm No? Let's Troubleshoot (next slide ) 19

Troubleshooting PMM Check for any red: sudo pmm-admin list sudo pmm-admin check-network Time drift? pmm-server$ sudo date --set='<timefromclient>' Restarting one or all components sudo pmm-admin restart linux:metrics pmm-client sudo pmm-admin restart --all Logs are in /var/log/pmm-*.log Check targets status in Prometheus http://192.168.56.3/prometheus/targets 20

Refreshments Break Come back in ~15 minutes!

Part 2 - Using Query Analytics (QAN) What are my queries doing?

QAN dashboard 23

Overview Per instance view of queries consuming most time Fields: Query Abstract Query pattern with placeholders ID Unique fingerprint, used for query group by Load Grand Total Time - percentage of time that the MySQL server spent executing a specific query, compared to the total time it spent executing all queries during the selected period of time Count QPS, total count during window, % of total Latency Min, Med, Avg, P95, Max 24

Query fingerprint + EXPLAIN 25

PERFORMANCE_SCHEMA or Slow Log 26

Slow Log (Percona Server) 27

CREATE TABLE 28

TABLE STATUS 29

INDEXES 30

Server summary info Click the icon: pt-summary Instance-specific information pt-mysql-summary Facts about the running MySQL instance 31

Part 3 - Using Metrics Monitor Eye candy

Grafana in a nutshell Open Source data visualisation tool Popular datasources Prometheus CloudWatch Graphite Elasticsearch Templated Variables Define your graph metrics, and let the hosts get filled in automatically GREAT for large, dynamic environments where hosts are considered ephemeral NOTE: Isn't supported for Alerting 33

Prometheus real quick Timeseries database - metric name + key/value pairs Flexible query language PromQL (See Björn's talk on Thursday for more!) Collection of metrics based on HTTP pull Targets discovered via service discovery or static configuration files We're using consul in PMM 34

How can I... Compare servers' to each other Cross Server graphs Show behaviour now() compared to past period Trends Overview dashboard At a glance MySQL + indepth MySQL Overview, InnoDB, InnoDB Advanced View Prometheus performance data Prometheus dashboard MariaDB - Aria, Online DDL, ICP, and Defragmentation MariaDB dashboard Table statistics* Largest tables by rows and size, total DB size, tables by rows read and changed, auto_increment usage (about to hit the limit?) User statistics* Top users by connection count, network usage, rows read/changed 35

Alerting Define alerts while "you're in the data" Template variables not supported Workaround steps Add new Metrics Query using instance name Add Alert condition using new Query row Save dashboard 36

Almost the end Parting thoughts

Advice PMM Metrics retention as packaged is 30 days We are looking at options to present a longer history mysql:metrics are polled at 1s, 5s, and 60s resolutions, and linux:metrics is every 1s On high-latency links you might need to tune scrape_interval up Don't skimp on resources Prometheus in particular needs a lot of RAM to keep reads fast, and to sequence scrape data before writing chunks to disk Consider disabling some mysqld_exporter features to minimise performance impact --disable-tablestats, --disable-processlist Keep queries in the database (security) --disable-queryexamples 38

PMM Roadmap Alerting QAN for MongoDB MySQL -> ClickHouse for QAN datastore Plugins and Integrations Long term metrics storage (past 30 days) One-click ticket submission* Standardised data collection for tickets* Any feedback of what you'd like to see in PMM? * for Percona Subscribers (Customers) only 39

Questions? Michael Coburn michael.coburn@percona.com Roma Novikov roma.novikov@percona.com Percona is looking for MongoDB and MySQL rockstars! Be sure to talk to Jennifer @ Percona s booth. Do you have any areas or benchmarks you want Percona to talk about in blogs together? Any features or tools you think we should focus on to meet the community's needs? 40

Rate Our Session 41