Monitoring MySQL with Prometheus Ben Kochie - Prometheus Lead - GitLab

Transcription

1 Monitoring MySQL with Ben Kochie - Lead - GitLab

2

3 About Metrics collection Time-series database Graphing Alerting

4 Performance

5 Performance Millions of Timeseries 800k samples per second 1.3 bytes per sample (varbit) Thousands of targets

6 In Production Tables 160+ Databases 30+ Clusters 250+ Servers Events Digests

7 Real-World Load 1.5 million timeseries 80k samples per second 400GB (28 days, old encoding) 10 cores (2.0ghz Ivy-Bridge)

8 Basics

9

10 mysqld_exporter Basics

11 CREATE USER IDENTIFIED BY 'pass'; GRANT PROCESS, REPLICATION CLIENT ON *.* TO GRANT SELECT ON performance_schema.* TO DATA_SOURCE_NAME=

12 Start mysqld_exporter./mysqld_exporter \ -collect.info_schema.innodb_metrics \ -collect.info_schema.userstats \ -collect.perf_schema.eventsstatements \ -collect.perf_schema.indexiowaits \ -collect.perf_schema.tableiowaits

13 Exporter Output ( # HELP mysql_global_status_commands_total Total number of executed MySQL commands. # TYPE mysql_global_status_commands_total counter mysql_global_status_commands_total{command="admin_commands"} mysql_global_status_commands_total{command="alter_db"} 0 # HELP mysql_global_variables_read_only Generic gauge metric from SHOW GLOBAL VARIABLES. # TYPE mysql_global_variables_read_only gauge mysql_global_variables_read_only 1 # HELP mysql_slave_status_seconds_behind_master Generic metric from SHOW SLAVE STATUS. # TYPE mysql_slave_status_seconds_behind_master untyped mysql_slave_status_seconds_behind_master 0

14 Configuration

15 global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'mysql' target_groups: - targets: - localhost:9104

16 PromQL

17 rate(mysql_global_status_connections[2m]) sum( rate(mysql_global_status_commands_total[2m]) )

18 sum( rate(mysql_global_status_commands_total[2m]) )

19 sum by (instance) ( rate(mysql_global_status_commands_total[2m]) )

20 sum by (command) ( rate(mysql_global_status_commands_total[2m]) )

21 sum by (command) ( rate(mysql_global_status_commands_total{ command=~"(insert update delete)" }[2m]) )

22 mysql_slave_status_seconds_behind_master

23 mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

24 Rules Files

25 mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

26 mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

27 mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay ALERT MySQLSlaveLag IF mysql_slave_lag_seconds > 300 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }

28 mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay ALERT MySQLSlaveLag IF (mysql_slave_lag_seconds > 60 AND predict_linear(mysql_slave_lag_seconds[2m],60*60)) > 0 FOR 2m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }

29 Examples/Graphs

30 Basic Traffic Overview

31 sum by (digest_text) (rate( mysql_perf_schema_events_statements_tmp_disk_tables_total[2m]))

32 topk(5, sum by (schema,digest_text) (rate(mysql_perf_schema_events_statements_total[5m])))

33 The Future

34 Demo Time