Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Transcription

1

2 Basic MySQL Troubleshooting for Oracle DBAs Sveta Smirnova Senior Principal Technical Support Engineer MySQL Support September 29, 2014

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

4 Program Agenda 1 Introduction 2 Basic troubleshooting techniques 3 High concurrency issues 4 High availability solutions 5 More information

5 Introduction

6 MySQL architecture Click icon to add picture Base - Installation layout - Log files Connectors - Clients - APIs Optimizer Caches & buffers Storage engines Management

7 Typical installation layout datadir Schema directory Table files: *frm, *ibd, *MYD, *MYI, *par, etc. Trigger files Schema InnoDB shared tablespace Log files InnoDB log files Binary, relay logs Error log Slow query log General query log Configurable You can setup custom path for each component Including custom path for tables

8 Connectors The way you connect to MySQL Server Clients MySQL CLI MySQL Workbench MySQL Enterprise Monitor (MEM) Oracle Enterprise Manager with MEM plugin APIs Exist for most popular programming languages C, C++, JDBC, PHP, Python, Net, ODBC

9 Plugins MySQL Server is highly configurable via plugins Storage engines Full-text parsers Daemon INFORMATION_SCHEMA Semisynchronous Replication Audit Authentication Password-validation Protocol Trace 5.7

10 Storage engines From troubleshooting point of view Own data Own index format Own locking model Own diagnostic Own log files CHECK TABLE

11 Basic troubleshooting techniques

12 MySQL Access Privilege System Typical issues Privileged client cannot connect Unprivileged client can connect Privileged user cannot perform operation Unprivileged user has undesired access

13 MySQL Access Privilege System Overview No roles by default, limited user limits All records are in the mysql database (schema) Pluggable authentication since version 5.5 Connections TCP/IP with login-password Socket (Unix) Named pipe (Windows)

14 MySQL Access Privilege System Where to find out why you have connection issues? select user, host from mysql.user order by user desc, host desc; Most descriptive host first, then wildcard Socket connection by default on Unix

15 MySQL Access Privilege System Where to find out why you have connection issues? mysql> select user, host from mysql.user order by user desc, host desc; user host root localhost root delly root ::1 root foo % localhost rows in set (0.00 sec)

16 MySQL Access Privilege System Wrong access SHOW GRANTS [FOR Grant tables mysql.db mysql.tables_priv mysql.columns_priv mysql.procs_priv mysql.proxies_priv SELECT USER(), CURRENT_USER()

17 MySQL Access Privilege System Wrong access mysql> show grants; Grants for GRANT ALL PRIVILEGES ON *.* TO WITH GRANT OPTION GRANT PROXY ON TO WITH GRANT OPTION rows in set (0.02 sec)

18 MySQL Access Privilege System Wrong access mysql> show grants for Grants for GRANT USAGE ON *.* TO GRANT ALL PRIVILEGES ON `test`.* TO rows in set (0.02 sec)

19 Error handling Errors vs warnings mysql> select max (f1) from t1; ERROR 1630 (42000): FUNCTION test.max does not exist. Check the 'Function Name Parsing and Resolution' section in the Reference Manual mysql> select * from t1 where "f1"=1; Empty set, 1 warning (0.05 sec) mysql> show warnings; Level Code Message Warning 1292 Truncated incorrect DOUBLE value: 'f1' row in set (0.00 sec)

20 MySQL Access Privilege System Application (C API) Error information mysql_errno mysql_error Warnings mysql_info mysql_sqlstate mysql_warning_count

21 Error handling perror ~]$ perror 1630 MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION): FUNCTION %s does not exist. Check the 'Function Name Parsing and Resolution' section in the Reference Manual ~]$ perror 1292 MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated incorrect %.32s value: '%.128s' ~]$ perror 2 OS error code 2: No such file or directory [sveta@delly ~]$ perror 150 MySQL error code 150: Foreign key constraint is incorrectly formed

22 Error handling In stored routines GET DIAGNOSTICS GET DIAGNOSTICS rows = ROW_COUNT, conditions = NUMBER; GET DIAGNOSTICS CONDITION 1 msg = MESSAGE_TEXT; code = RETURNED_SQLSTATE,

23 What can affect query execution? You run a query, it does not return an error, but still behaves not as expected Startup options or system variables How optimizer creates query plan Storage engine used Parallel execution next section

24 System variables and startup options Scope Global Control parameters, necessary for all server processes Location of server files: datadir etc. Shared buffers More Session Control connection-specific parameters

25 System variables and startup options How to set SET [GLOBAL] var_name = NEW_VAL Command line option Configuration file In default location Specified by option --defaults-file

26 System variables and startup options Who can change Global options and few session options A user with privilege SUPER Session options Anybody There is no limits!

27 System variables and startup options When allocated Those which control behavior of whole server Once at server startup Can start with low values, then grow to specified Connection options For every connection when connection opens Operation-specific For every operation when needed

28 System variables and startup options How to control SHOW [GLOBAL] STATUS GLOBAL Since server start SESSION For operations in current session Can be reset FLUSH STATUS

29 System variables and startup options How to control mysql> show global status like 'Handler_read_rnd_next'\G *************************** 1. row *************************** Variable_name: Handler_read_rnd_next Value: 27 1 row in set (0.00 sec) mysql> show status like 'Handler_read_rnd_next'\G *************************** 1. row *************************** Variable_name: Handler_read_rnd_next Value: 7 1 row in set (0.00 sec)

30 System variables and startup options Troubleshooting best practices Record currently used variables SHOW [GLOBAL] VARIABLES Make change dynamically if possible SET [GLOBAL] var_name=new_val Test in one session first Then change global variable Change configuration file after you are happy with results

31 System variables and startup options When affecting option is not known Record currently used variables SHOW [GLOBAL] VARIABLES Start mysqld with option --no-defaults This option must be first one! Check if problem is solved Change variable values one-by-one until you find one which leads to the problem

32 Diagnostic tool: INFORMATION_SCHEMA When affecting option is not known Contain metadata information Tables Indexes Other Allows to create plugins InnoDB plugins We will discuss them later Oracle analog Data Dictionary Views

33 MySQL Optimizer Overview EXPLAIN is less powerful if compare with Oracle It is improved in version Graphic EXPLAIN in MySQL Workbench 6.0 EXPLAIN EXTENDED EXPLAIN PARTITIONS EXPLAIN FORMAT=JSON INFORMATION_SCHEMA.TRACE Status variables 'Handler_%' Default in 5.7

34 EXPLAIN in Oracle EXPLAIN PLAN SET statement_id = 'example_plan4' FOR SELECT h.order_number, l.revenue_amount, l.ordered_quantity FROM so_headers_all h, so_lines_all l WHERE h.customer_id = :b1 AND h.date_ordered > SYSDATE30 AND l.header_id = h.header_id ; Plan SELECT STATEMENT NESTED LOOPS TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL INDEX RANGE SCAN SO_HEADERS_N1 TABLE ACCESS BY INDEX ROWID SO_LINES_ALL INDEX RANGE SCAN SO_LINES_N1

35 EXPLAIN in MySQL mysql> EXPLAIN SELECT user, host FROM mysql.user\g *************************** 1. row *************************** id: 1 select_type: SIMPLE table: user type: index possible_keys: NULL key: PRIMARY key_len: 228 ref: NULL rows: 4 Extra: Using index 1 row in set (0.13 sec)

36 EXPLAIN in MySQL Variant: EXTENDED mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.user\g *************************** 1. row *************************** id: 1 select_type: SIMPLE table: user type: index possible_keys: NULL key: PRIMARY key_len: 228 ref: NULL rows: 4 filtered: mysql> SHOW WARNINGS\G ******* 1. row ******* Level: Note Code: 1003 Message: /* select#1 */ select `mysql`.`user`.`user` AS `user`,`mysql`.`user`.`h ost` AS `host` from `mysql`.`user` 1 row in set (0.00 sec) Extra: Using index 1 row in set, 1 warning (0.00 sec)

37 EXPLAIN in MySQL Variant: FORMAT=JSON... mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.user\g "used_key_parts": [ "Host", ************** 1. row ************** "User" EXPLAIN: { ], "query_block": { "key_length": "228", "select_id": 1, "rows": 4, "table": { "filtered": 100, "using_index": true "table_name": "user", } "access_type": "index", } "key": "PRIMARY", }... 1 row in set, 1 warning (0.03 sec)

38 EXPLAIN in MySQL More information MySQL EXPLAIN in Practice [HOL9232] (past) MySQL 5.7: What s New in the Parser and the Optimizer? [CON2830] (past) How to Analyze and Tune MySQL Queries for Better Performance [TUT3157] (past) Using MySQL Workbench Performance Tools [HOL9237] (past)

39 MySQL Optimizer Handler_% status variables mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> SHOW STATUS LIKE 'Handler_read_%'; Variable_name Value Handler_read_first 0 Handler_read_key 0 Handler_read_last 0 Handler_read_next 0 Handler_read_prev 0 Handler_read_rnd 0 Handler_read_rnd_next rows in set (0.00 sec)

40 MySQL Optimizer Handler_% status variables mysql> select count(*) from employees join titles using(emp_no) where title='senior Engineer'\G *************************** 1. row *************************** count(*): row in set (3.24 sec) mysql> SHOW STATUS LIKE 'Handler_read_%'; Variable_name Value Handler_read_first 1 Handler_read_key Handler_read_last 0 Handler_read_next

41 MySQL Optimizer Other tools INFORMATION_SCHEMA.TRACE join_preparation, join_optimization, join_execution considered_execution_plans, refine_plan, more Query Analyzer in MEM Graphic EXPLAIN in MySQL Workbench 6.0

42 MySQL Storage Engines Specifics Own way to handle Corruption Index statistics CHECK TABLE to check for errors They care about physical data, so all data information are on their level Options usually start from engine name myisam_*, innodb_*, custom_*

43 MySQL Storage Engines InnoDB Transactional storage engine Physical layout *frm file table definition Shared tablespace *ibd file tablespace for individual table Optional: --innodb_file_per_table Recommended and default since 5.6 Redo log files OPTIMIZE TABLE = ALTER + ANALYZE Automatic startup check

44 High concurrency issues

45 High concurrency issues Common problems Query or transaction waits a lock, held by another one Fight for system resources Resource overload Resource underload

46 Transactions and their relation to locks Lock types Levels MDL Table-level Row-level Transactions Server-level MDL locks What do they lock Read locks Engine level Table locks Row locks Block writes Write locks AUTOCOMMIT Supported Block both reads and writes

47 Locking issues Diagnostic tools SHOW [FULL] PROCESSLIST Universal tool which lists all currently running connections SHOW ENGINE INNODB STATUS INFORMATION SCHEMA PROCESSLIST InnoDB tables PERFORMANCE SCHEMA MDL locks Table locks Server-level transactions

48 Storage engine specifics Diagnostic tools Transactions and engine-level locks are done at the engine level Storage engines have own diagnostic tools Use them when hit an issue with such a lock

49 Storage engine specifics InnoDB SHOW ENGINE INNODB STATUS InnoDB monitors Pseudo-tables Turn periodical logging into error log CREATE TABLE innodb_monitor(f1 int) ENGINE=INNODB; Lock monitor Changes output format of InnoDB Monitor CREATE TABLE innodb_lock_monitor(f1 int) ENGINE=INNODB; Options innodb_status_output innodb_status_output_locks

50 Storage engine specifics InnoDB Tables in INFORMATION_SCHEMA INNODB_TRX INNODB_LOCKS INNODB_LOCK_WAITS INNODB_METRICS Options innodb_monitor_* Option innodb_print_all_deadlocks

51 Diagnostic tool: performance_schema For whole server Monitors internal operations Events Waits Mutexes Statements Stages MDL, table-level locks, transactions Memory Replication 5.7+ Similar to Oracle wait interface

52 Diagnosting locks Summary Table-level locks PROCESSLIST: Waiting for table lock P_S.TABLE_HANDLES 5.7+ Row-level locks InnoDB monitors, SHOW ENGINE INNODB STATUS Tables in INFORMATION_SCHEMA Option --innodb_print_all_deadlocks MDL locks PROCESSLIST: Waiting for metadata lock P_S.METADATA_LOCKS 5.7+

53 High availability solutions

54 MySQL Cluster Special storage engine: NDB Stores data on two or more physical machines Two or more copies General troubleshooting techniques Applicable Specific NDB storage engine techniques

55 MySQL Replication Overview Always available, but you must setup it before use Asynchronous master-slave Master Keeps all updates in separate file: binary log Slave IO thread read updates from master and stores in relay log file SQL thread executes updates

56 Replication IO thread Communication issues Access error Check slave error log SHOW SLAVE STATUS P_S.replication_connection_status Try to connect using normal MySQL client using slave credentials SHOW GRANTS Fix privileges on master Restart slave

57 Replication SQL thread Typical issues Simple master-slave Data is different on master and slave Replication event can not be applied Different errors on master and slave Slave lags far behind the master Circular replication or other writes in addition to slave SQL thread Data is different on master and slave

58 Replication SQL thread Data is different on master and slave Was the table modified besides the SQL thread? How? Can it affect content of the table in the wrong way? Are the table definitions same on master and slave? MySQL Utilities mysqlrplsync, mysqldbcompare, mysqldiff Is it possible that master events was applied in wrong order? Use mysqlbinlog to find queries which caused the issue Check master's application to find what is wrong

59 Replication SQL thread Events from master were applied in wrong order Lock issues InnoDB Triggers SET GLOBAL slave_skip_counter Synchronize tables! Different options Start slave with master's options, then check

60 Replication SQL thread Slave lags far behind the master Threads Master runs in multiple update threads Slave uses single Seconds_behind_master is growing Tune slave performance Buffers Indexes (for statement-based replication)

61 More information

62 OpenWorld sessions October, 01 Practical MySQL Optimization [CON5522] Peter Zaitsev, 3:30 PM, Moscone South What s New in MySQL 5.7 Security [CON1985] Georgi Kodinov, 4:45 PM, Moscone South MySQL Cost Model [CON3163] Olav Sandstå, 5:30 PM, Moscone South - 250

63 OpenWorld sessions October, 02 MySQL Troubleshooting with the Performance Schema [HOL9215] Sveta Smirnova, Lig Isler-turmelle 8:30 AM, Hotel Nikko - Monterey MySQL Enterprise Edition Features in Practice [HOL9216] Matt Lord, 10:00 AM, Hotel Nikko - Monterey MySQL Performance: Demystified Tuning and Best Practices [CON5097] Dimitri Kravtchuk, 10:45 AM, Moscone South - 252

64 Further reading MySQL Troubleshooting book Marc Alff's Performance Schema blog Planet MySQL

65 References MySQL User Reference Manual Knowledge Management Database Forums Bug trackers Oracle Internal Bugs database My Oracle Support

66 You will find me at

67 ?

68 Thank you!

69