Writing High Performance SQL Statements. Tim Sharp July 14, 2014

Size: px

Start display at page:

Download "Writing High Performance SQL Statements. Tim Sharp July 14, 2014"

Bryce Lee
6 years ago
Views:

1 Writing High Performance SQL Statements Tim Sharp July 14, 2014

2 Introduction Tim Sharp Technical Account Manager Percona since years working with Databases

3 Optimum SQL Performance Schema Indices Queries

4 Explain Tool Provides query optimization info Great for understanding and optimizing troublesome MySQL queries MySQL 5.6: SELECT, DELETE, INSERT, REPLACE & UPDATE Estimate on how MySQL will execute query

5 Explain Tool id select_type table type possible_keys key key_len ref rows filtered Extra The SELECT identifier The SELECT type The table for the output row The join type possible indexes to choose The index actually chosen The length of the chosen key The columns compared to the index Estimate of rows to be examined Percentage of rows filtered by table condition Additional information

6 Explain Tool id select_type table type possible_keys key key_len ref rows filtered Extra The SELECT identifier The SELECT type The table for the output row The join type possible indexes to choose The index actually chosen The length of the chosen key The columns compared to the index Estimate of rows to be examined Percentage of rows filtered by table condition Additional information

7 Explain Tool id select_type table type possible_keys key key_len ref rows filtered Extra The SELECT identifier The SELECT type The table for the output row The join type -> const, ref, range, index_merge, all possible indexes to choose The index actually chosen The length of the chosen key The columns compared to the index Estimate of rows to be examined Percentage of rows filtered by table condition Additional information

8 Explain Tool id select_type table type possible_keys key key_len ref rows filtered Extra The SELECT identifier The SELECT type The table for the output row BEST! WORST! The join type -> const, ref, range, index_merge, all possible indexes to choose The index actually chosen The length of the chosen key The columns compared to the index Estimate of rows to be examined Percentage of rows filtered by table condition Additional information

9 Explain Tool id select_type table type possible_keys key key_len ref rows filtered Extra The SELECT identifier The SELECT type The table for the output row The join type -> const, ref, range, index_merge, all possible indexes to choose -> NULL The index actually chosen -> NULL The length of the chosen key The columns compared to the index Estimate of rows to be examined Percentage of rows filtered by table condition Additional information BAD

10 Explain - Examples CREATE TABLE `film` ( `film_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT, `title` varchar(255) NOT NULL, `description` text, `release_year` year(4) DEFAULT NULL, `language_id` tinyint(3) unsigned NOT NULL, `original_language_id` tinyint(3) unsigned DEFAULT NULL, `rental_duration` tinyint(3) unsigned NOT NULL DEFAULT '3', `rental_rate` decimal(4,2) NOT NULL DEFAULT '4.99', `length` smallint(5) unsigned DEFAULT NULL, `replacement_cost` decimal(5,2) NOT NULL DEFAULT '19.99', `rating` enum('g','pg','pg-13','r','nc-17') DEFAULT 'G', `special_features` set('trailers','commentaries','deleted Scenes','Behind the Scenes') DEFAULT NULL, `last_update` timestamp NOT NULL, PRIMARY KEY (`film_id`), KEY `idx_title` (`title`), KEY `idx_fk_language_id` (`language_id`), KEY `idx_fk_original_language_id` (`original_language_id`), )

11 Explain - Examples CREATE TABLE `actor` ( `actor_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT, `first_name` varchar(45) NOT NULL, `last_name` varchar(45) NOT NULL, `last_update` timestamp NOT NULL, PRIMARY KEY (`actor_id`), KEY `idx_actor_last_name` (`last_name`) ) CREATE TABLE `film_actor` ( `actor_id` smallint(5) unsigned NOT NULL, `film_id` smallint(5) unsigned NOT NULL, `last_update` timestamp NOT NULL PRIMARY KEY (`actor_id`,`film_id`), KEY `idx_fk_film_id` (`film_id`), ) Many to Many relationship: actor film_actor film

12 Explain - Examples explain SELECT actor.first_name FROM actor WHERE first_name like a%'\g; id: 1 select_type: SIMPLE table: actor type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 200 Extra: Using where

13 Explain - Examples Let s add an index! CREATE INDEX FN_idx on actor(first_name); EXPLAIN SELECT actor.first_name FROM actor WHERE first_name like 'a%'\g; id: 1 select_type: SIMPLE table: actor type: range possible_keys: FN_idx key: FN_idx key_len: 137 ref: NULL rows: 13 Extra: Using where; Using index

14 Explain - Examples Let s add an index! CREATE INDEX FN_idx on actor(first_name); EXPLAIN SELECT actor.first_name FROM actor WHERE first_name = 'allan'\g; id: 1 select_type: SIMPLE table: actor type: const possible_keys: FN_idx key: FN_idx key_len: 137 ref: NULL rows: 1 Extra: Using where; Using index

15 Explain - Examples EXPLAIN SELECT actor.first_name, actor.last_name FROM actor WHERE actor_id = ANY (SELECT actor_id FROM film_actor WHERE film_id= (SELECT film.film_id FROM film WHERE title='dinosaur Secretary )); 1. row id: 1 select_type: PRIMARY table: film_actor type: ref possible_keys: PRIMARY,idx_fk_film_id key: idx_fk_film_id key_len: 2 ref: const rows: 6 Extra: Using where; Using index 2. row id: 2 select_type: PRIMARY table: actor type: eq_ref possible_keys: PRIMARY key: PRIMARY key_len: 2 ref: sakila.film_actor.actor_id rows: 1 Extra: NULL 3. row id: 3 select_type: SUBQUERY table: film type: ref possible_keys: idx_title key: idx_title key_len: 767 ref: const rows: 1 Extra: Using where; Using index

16 Optimum Schema Schema Indices Queries

17 Schema Design Smaller is better Use smallest datatype to represent data Uses less resources: disk, memory, CPU cache Performance: comparing terms Integers > characters MySQL built-ins > strings Store IP addresses as integers instead of strings

18 Schema Design Avoid NULL Specify columns as NOT NULL Use ENUM instead of String Join like columns converting columns -> performance hit

19 Schema Normalization Don t over-normalize Some data redundancy is not bad Normalized Smaller tables Updates/inserts faster Smaller footprint More JOINS lowers performance Best for WRITE intensive Denormalized No need for JOINS SELECTS are fast Efficient use of indices Larger footprint Slower inserts/updates Potential inconsistency Best for READ intensive

20 Optimum Indices Indices queries Schema

21 Indexing Indexing optimization Powerful way to improve query performance Optimal indices can provide 2x improvement May require rewriting queries

22 Indexing Why? Reduces amount of data that needs to be examined Does pre-sorting of data Turns random I/O to sequential I/O When? Small tables - unnecessary Medium/Large tables - can be effective Enormous - consider sharding

23 Good vs Optimum Index places relevant rows next to each other

24 Good vs Optimum Index places relevant rows next to each other Rows are sorted in the order needed by the query

25 Good vs Optimum Index places relevant rows next to each other Rows are sorted in the order needed by the query Contains all the columns needed for the query

26 Good vs Optimum SELECT a FROM <table> WHERE b = const ORDER BY c;

27 Good vs Optimum SELECT a FROM <table> WHERE b = const ORDER BY c; KEY Good b ; Index places relevant rows next to each other

28 Good vs Optimum SELECT a FROM <table> WHERE b = const ORDER BY c; KEY Better b,c ; Index places relevant rows next to each other Rows are sorted in the order needed by the query

29 Good vs Optimum SELECT a FROM <table> WHERE b = const ORDER BY c; KEY Optimal a, b, c ; Index places relevant rows next to each other Rows are sorted in the order needed by the query Contains all the columns needed for the query

30 Good vs Optimum Covered Index SELECT a FROM <table> WHERE b = const ORDER BY c; KEY Optimal a, b, c ; Index places relevant rows next to each other Rows are sorted in the order needed by the query Contains all the columns needed for the query

31 Indexing - Multi-Column Index on 2 columns 2 separate single column indices. Common mistake: index all the individual columns MySQL can merge individual indices Explain: type = merged index Indicative of poorly indexed table Solution: 1 multi-column index

32 Indexing - Multi-Column Column order matters! Correct order depends on the queries Rule of thumb: place most selective to the left

33 Indexing - Multi-Column key (last_name, first_name, DOB) GOOD Match full value Match left most prefix Match a range of values Match one part exactly, match a range on another BAD No good if lookup doesn t start with left most column Can t skip columns in index Can t match after 1st range search

34 Covering Index An index that contains all the required fields of a query Does not need to access any data (it s all in the index) Index entries are usually small (can cache more) Range searches do less I/O

35 Hash index Can be lightening fast has many limitations, but great when appropriately Limitations no sorting no partial key match only equality comparisons MyISAM - Explicit Hash Index InnoDB - Implicit Hash Index Adaptive Hash index

36 Optimum Queries Queries Indices Queries

37 Query Optimization Queries are tasks composed of sub tasks Optimize query = optimize subtask Make subtask happen faster Eliminate subtask Reduce frequency of subtask

38 Basics Are you retrieving data you don t need?

39 Basics Are you retrieving data you don t need? Fetching more rows than necessary

40 Basics Are you retrieving data you don t need? Fetching more rows than necessary LIMIT n

41 Basics Are you retrieving data you don t need? Fetching more rows than necessary Fetching more columns than necessary LIMIT n

42 Basics Are you retrieving data you don t need? Fetching more rows than necessary Fetching more columns than necessary LIMIT n Choose carefully!

43 Too many columns Select * from

44 Multi-JOIN SELECT SELECT * FROM sakila.actor INNER JOIN sakila.film_actor USING(actor_id) INNER JOIN sakila.film USING(film_id WHERE sakila.film.title = Zorro Ark ;

45 Multi-JOIN SELECT BAD! SELECT * FROM sakila.actor INNER JOIN sakila.film_actor USING(actor_id) INNER JOIN sakila.film USING(film_id WHERE a = Zorro Ark ;

46 Multi-JOIN SELECT Better! SELECT sakila.actor.* FROM sakila.actor INNER JOIN sakila.film_actor USING(actor_id) INNER JOIN sakila.film USING(film_id WHERE sakila.film.title = Zorro Ark ;

47 Negating left joins with a WHERE clause Table1 id1 value1 1 AA 2 BB 3 CC Table2 id2 value2 1 DD 2 EE SELECT * FROM table1 LEFT JOIN table2 ON table1.id=test2.id id1 value1 id2 value2 1 AA 1 DD 2 BB 2 EE 3 CC NULL NULL

48 Negating left joins with a WHERE clause Table1 id1 value1 1 AA 2 BB 3 CC Table2 SELECT * FROM table1 LEFT JOIN table2 ON table1.id = test2.id WHERE table2.value2 = DD id1 value1 id2 value2 1 AA 1 DD id2 value2 1 DD 2 EE

49 Functions & Expressions Functions and expressions disable indices: SELECT * FROM your_table WHERE SOME_FUNCTION(indexed_field) > value; SELECT * FROM your_table WHERE indexed_field + const > value;

50 Functions & Expressions Example: using a key field inside an expression mysql> explain select * from actor where actor_id+1 > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where

51 Functions & Expressions Example: using a key field inside an expression mysql> explain select * from actor where actor_id+1 > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where

52 Functions & Expressions Example: using a key field inside an expression mysql> explain select * from actor where actor_id+1 > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where BAD!

53 Functions & Expressions Example: using a key field inside an expression mysql> explain select * from actor where actor_id+1 > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where BAD! mysql> explain select * from actor where actor_id > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor range PRIMARY PRIMARY 2 NULL 3 Using where

54 Functions & Expressions Example: using a key field inside an expression mysql> explain select * from actor where actor_id+1 > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where BAD! mysql> explain select * from actor where actor_id > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor range PRIMARY PRIMARY 2 NULL 3 Using where

55 Functions & Expressions Example: using a key field inside an expression mysql> explain select * from actor where actor_id+1 > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where BAD! mysql> explain select * from actor where actor_id > 197; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor range PRIMARY PRIMARY 2 NULL 3 Using where Better!

56 Functions & Expressions Example: using a key field inside a function mysql> EXPLAIN SELECT * FROM actor WHERE SQRT(actor_id)>14; id select_type table type possible_keys key key_len ref rows Extra SIMPLE actor ALL NULL NULL NULL NULL 200 Using where BAD!

57 Aggregate functions Performance killers on large data sets How much accuracy do you really need? Instead of: SELECT AVG(actor_id) FROM actor; How about: SELECT AVG(actor_id) FROM (SELECT actor_id FROM actor LIMIT 10);

58 Query Cache have_query_cache = YES Stores SELECT statement as well as resultant set Best for READ intensive applications Warning: Global mutex serializes access to query cache - possible bottleneck!

59 Know Your Engine InnoDB Row level locking Great for WRITE intensive apps MyISAM Table level locking Great for READ intensive apps TokuDB Row level locking Compression

60 Miscellaneous Purge old data Break up delete operations Are all BUFFER sized correctly? InnoDB Buffer Pool: Hot data set Use ANALYZE TABLE statement periodically Turn on slow query log file Run alter table: rebuilds tables - defrag ALTER TABLE tbl_name ENGINE=INNODB JOINS better than SubSelects

61 Percona Tools pt-query-digest pt-duplicate-key-checker pt-index-usage pt-visual-explain pt-table-usage

62 Percona Cloud Tools: PCT Percona Cloud Tools Performance optimization Troubleshooting Capacity planning Query review Nice graphs Great for multi-server environments

63 Thank you!

Covering indexes. Stéphane Combaudon - SQLI

Covering indexes. Stéphane Combaudon - SQLI Covering indexes Stéphane Combaudon - SQLI Indexing basics Data structure intended to speed up SELECTs Similar to an index in a book Overhead for every write Usually negligeable / speed up for SELECT Possibility