Advanced MySQL Query Tuning

Size: px

Start display at page:

Download "Advanced MySQL Query Tuning"

Eric Henry
5 years ago
Views:

1 Advanced MySQL Query Tuning Alexander Rubin August 6, 2014

2 About Me My name is Alexander Rubin Working with MySQL for over 10 years Started at MySQL AB, then Sun Microsystems, then Oracle (MySQL Consulting) Joined Percona 1 year ago Helping customers improve MySQL performance performance tuning full text search high availability

3 Agenda Indexes How B-tree works Queries Temporary Tables and Filesort in MySQL GROUP BY Optimizations ORDER BY Optimizations Subqueries New in MySQL 5.6 Geo/Spatial Search

4 How to Deal with Slow Performance Indexes

5 MySQL Index Types: B-Tree Default index type (except for MEMORY tables) When you add index (except for MEMORY) MySQL will use B-Tree Support equality and range operations

6 MySQL Index Types: B-Tree Equality search: select * from table where id = 12 Scan thru the tree and go directly to 1 leaf Stop

7 MySQL Index Types: B-Tree Range: select * from table where id in (6, 12, 18) Scan thru the tree and visit many leafs/nodes

8 CREATE TABLE City ( ID int(11) NOT NULL AUTO_INCREMENT, Name char(35) NOT NULL DEFAULT '', CountryCode char(3) NOT NULL DEFAULT '', District char(20) NOT NULL DEFAULT '', Population int(11) NOT NULL DEFAULT '0', PRIMARY KEY (ID), KEY CountryCode (CountryCode) ) Engine=InnoDB;

9 MySQL will use 1 (best) index mysql> explain select * from City where ID = 1; table type possible_keys key key_len ref rows Extra City const PRIMARY PRIMARY 4 const mysql> explain select * from City where CountryCode = 'USA'; table type possible_keys key key_len ref rows Extra City ref CountryCode CountryCode 3 const 274 Using where

10 Leftmost part of combined index mysql> alter table City add key comb(countrycode, District, Population), drop key CountryCode;

$Leftmost part of combined index mysql> explain select * from City where CountryCode = 'USA'\G ********************** 1.$

11 Leftmost part of combined index mysql> explain select * from City where CountryCode = 'USA'\G ********************** 1. row ****************** table: City type: ref possible_keys: comb key: comb key_len: 3 ref: const rows: 273 Uses first field from the comb key

12 Key_len = total size (in bytes) of index parts used Index: comb(countrycode, District, Population) Explain: key: comb key_len: 3 Fields: CountryCode char(3) District char(20) Population int(11) 3 -> Char(3) -> First field is used

$2 Leftmost Fields mysql> explain select * from City where CountryCode = 'USA' and District = 'California'\G ********************** 1.$

13 2 Leftmost Fields mysql> explain select * from City where CountryCode = 'USA' and District = 'California'\G ********************** 1. row ****************** table: City type: ref possible_keys: comb key: comb key_len: 23 ref: const,const rows: 68 Uses 2 first fields from the comb key CountryCode = 3 chars District = 20 chars Total = 23

$3 Leftmost Fields mysql> explain select * from City where CountryCode = 'USA' and District = 'California and population > 10000\G ********************** 1.$

14 3 Leftmost Fields mysql> explain select * from City where CountryCode = 'USA' and District = 'California and population > 10000\G ********************** 1. row ****************** table: City type: range possible_keys: comb key: comb key_len: 27 ref: NULL rows: 68 Uses all fields from the comb key CountryCode = 3 chars/bytes District = 20 chars/bytes Population = 4 bytes (INT) Total = 27

15 Can t use combined index not a leftmost part mysql> explain select * from City where District = 'California' and population > 10000\G ********************** 1. row ****************** table: City type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3868 Does not have the CountryCode in the where clause = can t use comb index

Covered index = cover all fields in query select name from City where CountryCode = 'USA' and District = 'Alaska' and population > 10000 mysql> alter table City add key

16 Covered index = cover all fields in query select name from City where CountryCode = 'USA' and District = 'Alaska' and population > mysql> alter table City add key cov1(countrycode, District, population, name); Uses all fields in the query in particular order: 1. Where part 2. Group By/Order (not used now) 3. Select part (here: name)

$Explain mysql> explain select name from City where CountryCode = 'USA' and District = 'Alaska' and population > 10000\G *************************** 1.$

17 Explain mysql> explain select name from City where CountryCode = 'USA' and District = 'Alaska' and population > 10000\G *************************** 1. row *********** table: City type: range possible_keys: cov1 key: cov1 key_len: 27 ref: NULL rows: 1 Extra: Using where; Using index Using index = covered index is used MySQL will only use index Will not go to the data file

18 Order of Fields in Index Range and const conditions select * from City where district = 'California' and population > Index (district, population) in this order Rule of thumb: Const first, Range second Depends on query

19 How to Deal with Slow Performance Queries

20 Complex Slow Queries Group By Order By Select distinct Temporary tables Filesort The World s Most Popular Open Source Database

21 GROUP BY Queries

$How many cities in each country? mysql> explain select CountryCode, count(*) from City group by CountryCode\G id: 1 Temporary tables are slow!$

22 How many cities in each country? mysql> explain select CountryCode, count(*) from City group by CountryCode\G id: 1 Temporary tables are slow! select_type: SIMPLE table: City type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 4079 Extra: Using temporary; Using filesort

23 Temporary Tables: Theory

24 Main performance issues MySQL can create temporary tables when query uses: GROUP BY Range + ORDER BY Some other expressions 2 types of temporary tables MEMORY On-disk

25 First, MySQL tries to create temporary table in memory MySQL configuration variables: tmp_table_size maximum size for in Memory temporary tables max_heap_table_size Sets the maximum size for MEMORY tables

26 MySQL temp table > tmp_table_ size OR MySQL temp table > max_heap_ table_size convert to MyISAM temporary table on disk

27 MEMORY engine does not support BLOB/TEXT select blob_field from table group by field1 select concat(...string>512 chars) group by field1 Create on-disk temporary table right away

28 Temporary Tables: Practice

29 6M rows, ~2G in size CREATE TABLE ontime_2012 ( YearD int(11) DEFAULT NULL, MonthD tinyint(4) DEFAULT NULL, DayofMonth tinyint(4) DEFAULT NULL, DayOfWeek tinyint(4) DEFAULT NULL, Carrier char(2) DEFAULT NULL, Origin char(5) DEFAULT NULL, DepDelayMinutes int(11) DEFAULT NULL,... ) ENGINE=InnoDB DEFAULT CHARSET=latin1 _Name=On-Time

30 Find maximum delay for flights on Sunday Group by airline SELECT max(depdelayminutes), carrier, dayofweek FROM ontime_2012 WHERE dayofweek = 7 GROUP BY Carrier

31 select max(depdelayminutes), carrier, dayofweek from ontime_2012 where dayofweek = 7 group by Carrier type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: Full table scan! Temporary table! Extra: Using where; Using temporary; Using filesort

$mysql> alter table ontime_2012 add key (dayofweek); explain select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek =7 group by Carrier\G type: ref$

32 mysql> alter table ontime_2012 add key (dayofweek); explain select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek =7 group by Carrier\G type: ref Index is used = better possible_keys: DayOfWeek key: DayOfWeek BUT: Large temporary table! key_len: 2 ref: const rows: Extra: Using where; Using temporary; Using filesort

$Carrier\G... possible_keys: DayOfWeek,covered key: covered key_len: 2 ref: const rows: 905138 No temporary table!$

33 mysql> alter table ontime_2012 add key covered(dayofweek, Carrier, DepDelayMinutes); explain select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek =7 group by Carrier\G... possible_keys: DayOfWeek,covered key: covered key_len: 2 ref: const rows: No temporary table! MySQL will only use index Extra: Using where; Using index Called tight index scan

34 mysql> explain select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek > 3 group by Carrier, dayofweek\g... type: range Range scan possible_keys: covered key: covered key_len: 2 ref: NULL rows: Extra: Using where; Using index; Using temporary; Using filesort

35 (select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek = 3 group by Carrier, dayofweek) union (select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek = 4 group by Carrier, dayofweek)

36 *************************** 1. row *************************** table: ontime_2012 key: covered... Extra: Using where; Using index *************************** 2. row *************************** table: ontime_2012 key: covered Extra: Using where; Using index *************************** 3. row *************************** id: NULL select_type: UNION RESULT table: <union1,2> type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: NULL Extra: Using temporary

37 GROUP BY: Loose index scan Loose index scan: considers only a fraction of the keys in an index Following rules apply: The query is over a single table. The GROUP BY columns should form a leftmost prefix of the index The only aggregate functions = MIN() and MAX(), same column

$where dayofweek > 5 group by Carrier, dayofweek\g... table: ontime_2012 type: range possible_keys: NULL key: loose_index_scan key_len: 5 ref: NULL rows: 201 Loose index scan Very fast Range works!$

38 Loose index scan example mysql> alter table ontime_2012 add key loose_index_scan (Carrier, dayofweek, DepDelayMinutes); mysql> explain select max(depdelayminutes), Carrier, dayofweek from ontime_2012 where dayofweek > 5 group by Carrier, dayofweek\g... table: ontime_2012 type: range possible_keys: NULL key: loose_index_scan key_len: 5 ref: NULL rows: 201 Loose index scan Very fast Range works! Extra: Using where; Using index for group-by

39 Loose index scan vs. tight index scan Table: ontime_2012, 6M rows, data: 2G, index: 210M CREATE TABLE ontime_2012 ( YearD int(11) DEFAULT NULL, MonthD tinyint(4) DEFAULT NULL, DayofMonth tinyint(4) DEFAULT NULL, DayOfWeek tinyint(4) DEFAULT NULL, Carrier char(2) DEFAULT NULL, Origin char(5) DEFAULT NULL, DepDelayMinutes int(11) DEFAULT NULL,... KEY loose_index_scan (Carrier,DayOfWeek,DepDelayMinutes), KEY covered (DayOfWeek,Carrier,DepDelayMinutes) ) ENGINE=InnoDB DEFAULT CHARSET=latin1

40 Loose index scan vs. tight index scan Loose index scan select max(depdelayminutes) as ddm, Carrier, dayofweek from ontime_2012 where dayofweek = 5 group by Carrier, dayofweek table: ontime_2012 type: range possible_keys: covered key: loose_index_scan key_len: 5 ref: NULL rows: 201 Extra: Using where; Using index for group-by Carrier, DayOfWeek, DepDelayMinutes mysql> select ddm Carrier dayofweek AA rows in set (0.00 sec)

41 Loose index scan vs. tight index scan Results Loose index scan Tight Index Scan Temporary table

$group by Carrier, dayofweek \G table: ontime_2012 type: range key: covered key_len: 2 ref: NULL rows: 2961617 Extra: Using where; Using index; Using temporary; Using filesort mysql> select.$

42 Where loose index scan is not supported AVG() + Group By loose index scan is not supported mysql> explain select avg(depdelayminutes) as ddm, Carrier, dayofweek from ontime_2012 where dayofweek >5 group by Carrier, dayofweek \G table: ontime_2012 type: range key: covered key_len: 2 ref: NULL rows: Extra: Using where; Using index; Using temporary; Using filesort mysql> select ddm Carrier dayofweek AA rows in set (1.39 sec) 1. No loose index scan 2. Filter by key 3. Group by filesort

43 ORDER BY and filesort

44 mysql> explain select district, name, population from City where CountryCode = 'USA' order by population desc limit 10\G table: City type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 4079 Extra: Using where; Using filesort

$CountryCode = 'USA' order by population desc limit 10\G table: City type: ref key: my_sort2 key_len: 3 ref: const rows: 207 Extra: Using where$

45 mysql> alter table City add key my_sort2 (CountryCode, population); mysql> explain select district, name, population from City where CountryCode = 'USA' order by population desc limit 10\G table: City type: ref key: my_sort2 key_len: 3 ref: const rows: 207 Extra: Using where No filesort

$68 sec) mysql> explain select * from ontime_2012 where dayofweek in (6,7) order by DepDelayMinutes desc limit 10\G type:$ index possible_keys: DayOfWeek,covered key: DepDelayMinutes key_len: 5 ref: NULL rows: 24 Extra: Using where 10 rows in

index possible_keys: DayOfWeek,covered key: DepDelayMinutes key_len: 5 ref: NULL rows: 24 Extra: Using where 10 rows in

46 mysql> alter table ontime_2012 add key (DepDelayMinutes); Query OK, 0 rows affected (38.68 sec) mysql> explain select * from ontime_2012 where dayofweek in (6,7) order by DepDelayMinutes desc limit 10\G type: index possible_keys: DayOfWeek,covered key: DepDelayMinutes key_len: 5 ref: NULL rows: 24 Extra: Using where 10 rows in set (0.00 sec) 1. Index is sorted 2. Scan the whole table in the order of the index 3. Filter results 4. Stop after finding 10 rows matching the where condition

47 If Index points to the beginning of the table (physically) = fast As it stops after 10 rows (LIMIT 10)

48 If Index points to the end of table (physically) or random = slower Much more rows to scan (and skip)

49 Subqueries Optimizations

50 Subqueries, I Subquery in FROM and joins SELECT sb1,sb2,sb3 FROM (SELECT s1 AS sb1, s2 AS sb2, s3*2 AS sb3 FROM t1) AS sb WHERE sb1 > 1; Subquery inside where SELECT * FROM t1 WHERE column1 in (SELECT column2 FROM t2);

51 Subqueries, II MySQL 5.6: Resolved Adds an index on the derived table Subquery in FROM and joins SELECT sb1,sb2,sb3 FROM (SELECT s1 AS sb1, s2 AS sb2, s3*2 AS sb3 FROM t1) AS sb WHERE sb1 > 1; MySQL will create a temporary table for (SELECT s1 AS sb1, s2 AS sb2, s3*2 AS sb3 FROM t1) with no indexes (fixed in MySQL 5.6) Rewrite as join

52 Subqueries, III MySQL 5.6: resolved, semi-join subquery optimization strategies Subquery inside where SELECT * FROM t1 WHERE column1 in (SELECT column2 FROM t2 where..); Will not use index on column1 Can rewrite query as join

53 Collect system variables every minute with agent Find the latest version of each config per each server CREATE TABLE `sysconfigs` ( `service_id` tinyint(3) unsigned NOT NULL, `instance_id` smallint(5) unsigned NOT NULL,... `var` varchar(100) NOT NULL, `val` varchar(500) CHARACTER SET utf8 NOT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`service_id`,`instance_id`,`var`,`ts`), KEY `instance_id` (`instance_id`) ) ENGINE=InnoDB

54 Query with subquery SELECT s.instance_id, s.ts, s.instance_id, s.var, s.val FROM sysconfigs s JOIN (select instance_id, var, max(ts) as last_ts from sysconfigs s1 where service_id = 2 group by instance_id, var) s1 on s.instance_id = s1.instance_id and s.var=s1.var and s.ts = s1.last_ts

55 Explain plan. Actual rows examined: 65 million select_type table type possible_keys key key_len ref rows Extra PRIMARY <derived2> ALL NULL NULL NULL NULL 1268 PRIMARY s ref instance_id instance_id 2 s1.instance_id Using where DERIVED s1 ref PRIMARY PRIMARY Using where; Using index Query time instance_id ts instance_id var val :32:29 1 autocommit ON :32:29 1 automatic_sp_privileges ON :32:29 1 auto_increment_increment rows in set (2 min sec)

56 -- explicitly create temp table create temporary table t engine=memory as select instance_id, var, max(ts) as last_ts from sysconfigs s1 where service_id = 2 group by instance_id, var; -- add indexes alter table t add primary key (instance_id, var, last_ts); SELECT s.instance_id, s.ts, s.instance_id, s.var, s.val FROM sysconfigs s join t as s1 on s.instance_id = s1.instance_id and s.var=s1.var and s.ts = s1.last_ts; -- clean up drop temporary table t;

57 create temporary table t engine=memory as select instance_id, var, max(ts) as last_ts from sysconfigs s1 where service_id = 2 group by instance_id, var Query OK, 1268 rows affected (0.63 sec) Records: 1268 Duplicates: 0 Warnings: alter table t add primary key (instance_id, var, last_ts) Query OK, 1268 rows affected (0.00 sec) Records: 1268 Duplicates: 0 Warnings: select SQL_CALC_FOUND_ROWS s.instance_id, s.ts, s.instance_id, s.var, s.val from sysconfigs s join t as s1 on s.instance_id = s1.instance_id and s.var=s1.var and s.ts = s1.last_ts instance_id ts instance_id var val rows in set (0.26 sec) drop temporary table t New: 0.89 seconds Original: 2 min sec Query OK, 0 rows affected (0.00 sec)

58 MySQL id select_type table type possible_keys key key_len ref rows Extra PRIMARY s ALL instance_id NULL NULL NULL NULL 1 PRIMARY <derived2> ref <auto_key0> <auto_key0> Using index 2 DERIVED s1 ref PRIMARY,instance_id PRIMARY 1 const Using where; Using index MySQL select_type table type possible_keys key key_len ref rows Extra PRIMARY <derived2> ALL NULL NULL NULL NULL 1268 PRIMARY s ref instance_id instance_id 2 s1.instance_id Using where DERIVED s1 ref PRIMARY PRIMARY Using where; Using index

59 MySQL 5.6 mysql> SELECT s.instance_id, s.ts, s.instance_id, s.var, s.val FROM sysconfigs s JOIN (select instance_id, var, max(ts) as last_ts from sysconfigs s1 where service_id = 2 group by instance_id, var) s1 on s.instance_id = s1.instance_id and s.var=s1.var and s.ts = s1.last_ts; instance_id ts instance_id var val rows in set (0.75 sec) MySQL instance_id ts instance_id var val rows in set (2 min sec)

60 mysql> explain select * from metric_data where instance_id in (select mysql_server_id from mysql_servers where deleted is not null)\g *************************** 1. row *************************** id: 1 select_type: PRIMARY table: metric_data type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: Extra: Using where *************************** 2. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: mysql_servers type: unique_subquery possible_keys: PRIMARY key: PRIMARY key_len: 2 ref: func rows: 1 Extra: Using index 2 rows in set (0.00 sec) CREATE TABLE metric_data( KEY instance_id(instance_id) ) ENGINE=InnoDB;

61 mysql> explain select * from metric_data md join mysql_servers s on (s.mysql_server_id=md.instance_id) where deleted is not null\g *************************** 1. row *************************** id: 1 select_type: SIMPLE table: s type: ALL possible_keys: PRIMARY,deleted key: NULL key_len: NULL ref: NULL rows: 4 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: md type: ref possible_keys: instance_id key: instance_id key_len: 2 ref: o2004.s.mysql_server_id rows: Extra: 2 rows in set (0.00 sec)

62 mysql> explain select * from metric_data where instance_id in (select mysql_server_id from mysql_servers where deleted is not null)\g *************************** 1. row *************************** id: 1 select_type: SIMPLE table: mysql_servers type: index possible_keys: PRIMARY,deleted key: deleted key_len: 4 ref: NULL rows: 4 Extra: Using where; Using index *************************** 2. row *************************** id: 1 select_type: SIMPLE table: metric_data type: ref possible_keys: instance_id key: instance_id key_len: 2 ref: o2004.mysql_servers.mysql_server_id rows: 4023 Extra: NULL 2 rows in set (0.00 sec)

63 Geo/Spatial Search

64 MySQL with Spatial Extensions mysql> CREATE TABLE `points` ( `OGR_FID` int(11) NOT NULL AUTO_INCREMENT, `SHAPE` geometry NOT NULL, `osm_id` text, `name` text,... `other_tags` text, UNIQUE KEY `OGR_FID` (`OGR_FID`), SPATIAL KEY `SHAPE` (`SHAPE`) ) ENGINE=MyISAM AUTO_INCREMENT= DEFAULT CHARSET=latin1

65 Spatial Data types in MySQL MyISAM Supports both SPATIAL types and indexes InnoDB Supports SPATIAL types only, not indexes added SPATIAL index support in labs version of MySQL 5.7

66 MySQL with Spatial Extensions `SHAPE` geometry mysql> select astext(shape) from points limit 1; astext(shape) POINT( ) row in set (0.00 sec) World Known Text (WKT)

$MySQL with Spatial Extensions `SHAPE` geometry mysql> select astext(shape) from tl_2013_us_zcta510 where zcta5ce10 = '27701'\G ********************** 1.$

67 MySQL with Spatial Extensions `SHAPE` geometry mysql> select astext(shape) from tl_2013_us_zcta510 where zcta5ce10 = '27701'\G ********************** 1. row ************************ astext(shape): POLYGON(( , , , , , )) Polygon for ZIP = (Durham, NC)

68 ST_CONTAINS example mysql> SELECT zcta5ce10 as ZIP FROM tl_2013_us_zcta510 WHERE st_contains(shape, POINT( , )); ZIP row in set (0.00 sec)

69 ST_DISTANCE: all restaurants around (lat, lon) mysql> SELECT name, st_distance(shape, POINT( , )) as distance FROM points WHERE ["amenity"=>"restaurant >] ORDER BY distance LIMIT 5; name distance The Pit Piedmont Torero's Fishmonger's Pop's

70 More Info 4/creating-geo-enabled-applications-mysql-5-6/ Labs.mysql.com MySQL GIS/RTREE support SPATIAL Indexes for InnoDB

71 Performance Schema: Webinar Join me next week (Aug 13, 10 a.m. PDT) for a Performance Schema webinar. Example: find usage (IOPS) per user mysql sys> select * from user_summary_by_file_io where user like 'a%'; user ios io_latency api h api h api h app :01:06.33 app s app s rows in set (0.06 sec)

72 Questions? Thank you!

Advanced MySQL Query Tuning

Advanced MySQL Query Tuning Alexander Rubin July 21, 2013 About Me My name is Alexander Rubin Working with MySQL for over 10 years Started at MySQL AB, then Sun Microsystems, then Oracle (MySQL Consulting)