Common MySQL Scalability Mistakes Ronald Bradford http://ronaldbradford.com 2011.04 EffectiveMySQL.com - Its all about Performance and Scalability EffectiveMySQL.com - Its all about Performance and Scalability 2010 - Oracle ACE Director (first in MySQL) 2009 - MySQL community member of the year Co Author of Expert PHP & MySQL Top individual blog contributor to Planet MySQL 22 years of RDBMS experience, 12 years with MySQL MySQL Inc (2006-2008), Oracle Corp (96-99) Provide independent consulting/ Available NOW EffectiveMySQL.com - Its all about Performance and Scalability AUTHOR 9 EffectiveMySQL.com - Its all about Performance and Scalability9
PROBLEM CAUSE My website seems to The default MyISAM 9 freeze or responds storage engine uses randomly? exclusive table locks for DML. EffectiveMySQL.com - Its all about Performance and Scalability 9EffectiveMySQL.com - Its all about Performance and Scalability SOLUTION EXAMPLE Optimize blocking query End user report that selects 9all performance customer, order, order lines and order history data and performs Use a transactional engine poor joins. This takes shared read with MVCC and row based locks blocking future write locks locking to address the then future reads. LOCKING issue EffectiveMySQL.com - Its all about Performance and Scalability 9EffectiveMySQL.com - Its all about Performance and Scalability
WHY MySQL is unique in that it offers MySQL PROCESSLIST 9 different mechanisms for storing Blocked have State = Locked and retrieving data, each with Blocker - Same table, larger Time strengths and weaknesses. mysql> S PROCESSLIST; +----+------+-----------+-------+---------+------+------------+--------------- Id User Host db Command Time State Info The DEFAULT is not always the +----+------+-----------+-------+---------+------+------------+--------------- 13 app1 localhost odtug Query 144 User sleep UPDATE emp... 14 app1 localhost odtug Query 116 Locked select c from emp best. 15 app1 localhost odtug Query 89 Locked select c from emp EffectiveMySQL.com - Its all about Performance and Scalability 9EffectiveMySQL.com - Its all about Performance and Scalability 8 Optimize Blocker Indexes Limit query Summary table Change storage engine EffectiveMySQL.com - Its all about Performance and Scalability 9EffectiveMySQL.com - Its all about Performance and Scalability8
PROBLEM SOLUTION Why is my database so Don t store large 8static large? objects in the database EffectiveMySQL.com - Its all about Performance and Scalability 8EffectiveMySQL.com - Its all about Performance and Scalability EXAMPLE WHY 80% of data is email content/ Maximize memory 8usage attachments for important data 60% of data is PDF documents Reduce database recovery 30% of data is uncompressed large XML objects time EffectiveMySQL.com - Its all about Performance and Scalability 8EffectiveMySQL.com - Its all about Performance and Scalability
Compress large text data Table Size per schema 8 90% saving on XML data # Schema Table Usage SELECT table_schema,table_name,engine,row_format, table_rows, avg_row_length, (data_length+index_length)/1024/1024 as total_mb, (data_length)/1024/1024 as data_mb, Store static data in files (index_length)/1024/1024 as index_mb, CURDATE() AS today FROM information_schema.tables WHERE table_schema = DATABASE() Avoids DB handling overhead ORDER BY 7 DESC; http://ronaldbradford.com/mysql-dba/ EffectiveMySQL.com - Its all about Performance and Scalability 8EffectiveMySQL.com - Its all about Performance and Scalability 7 PROBLEM I can't access my 7website? EffectiveMySQL.com - Its all about Performance and Scalability 7EffectiveMySQL.com - Its all about Performance and Scalability
TRUE STORY SOLUTION Question: Integrated monitoring 7 How do you know when your server including graphical is down or not accessible? interface, real time Answer: analysis and notification The users will let us know. EffectiveMySQL.com - Its all about Performance and Scalability 7EffectiveMySQL.com - Its all about Performance and Scalability Monitoring/Alerting Dashboard Graphical The state of NOW Historical Sampling at 1s/3s/5s Necessary e.g. 0.1% of throughput Generally missing/incomplete Useless for real-time analysis EffectiveMySQL.com - Its all about Performance and Scalability 7EffectiveMySQL.com - Its all about Performance and Scalability7
6 Instrumentation Important to business viability e.g. orders per minute page load time Seamless implementation i.e. no code changes to view real-time extensible EffectiveMySQL.com - Its all about Performance and Scalability 7EffectiveMySQL.com - Its all about Performance and Scalability6 PROBLEM SOLUTION My replication slave can't Know the weakest 6link(s) keep up? of MySQL replication and don't exceed that, or cheat. EffectiveMySQL.com - Its all about Performance and Scalability6EffectiveMySQL.com - Its all about Performance and Scalability
WHY Master If replication can't catchup, 6 DML Statement slaves are useless. Write Data/Redo Log Backup & recovery may Write Binary Log also suffer. Return OK to client EffectiveMySQL.com - Its all about Performance and Scalability6EffectiveMySQL.com - Its all about Performance and Scalability Slave Replication workarounds 6 Detect master log change Restrict queries executed Retrieve binary log entry --ignore Write relay log (IO_THREAD) Different storage engines Read relay log Apply DML (SQL_THREAD) Different index structures Write Data/redo log EffectiveMySQL.com - Its all about Performance and Scalability6EffectiveMySQL.com - Its all about Performance and Scalability
EXPERT TIP 5 Advanced workarounds RAID 0 (large number of slaves) Pre fetch thread EffectiveMySQL.com - Its all about Performance and Scalability6EffectiveMySQL.com - Its all about Performance and Scalability5 PROBLEM TRUE STORY Question: My server has crashed with 5 Have you ever performed a database recovery? a hard drive failure Answer: No, why? EffectiveMySQL.com - Its all about Performance and Scalability 5EffectiveMySQL.com - Its all about Performance and Scalability
TRUE STORY SOLUTION Consultant: Have a DR Plan 5 Do you know that your daily backups only recover the data up to that time, e.g. 1am. You Documented know you have lost all your sales and changes Tested since then. Customer: Timed No, I didn t know that. Verified - End to End EffectiveMySQL.com - Its all about Performance and Scalability 5EffectiveMySQL.com - Its all about Performance and Scalability QUIZ 1. Do you have MySQL backups in place? Do you pass the MySQL backup/ 5 2. Do you backup ALL your MySQL data? 3. Do you have consistent MySQL backups? recovery quiz? 4. Do you have backups that include both static snapshot and point in time transactions? 5. Do you review your backup logs EVERY SINGLE day or have tested backup monitoring in place? 6. Do you perform a test recovery of your static backup? http://rb42.com/mysql-backup-quiz 7. Do you perform a test recovery to point in time? 8. Do you time your backup and recovery process and review over time? 9. Do you have off-site copies of your backups? 10. Do you backup your primary binary logs? http://rb42.com/mysql-backup-quiz EffectiveMySQL.com - Its all about Performance and Scalability 5EffectiveMySQL.com - Its all about Performance and Scalability
4 EffectiveMySQL.com - Its all about Performance and EffectiveMySQL.com Scalability4 SOLUTION PROBLEM Why is my database 4 executing 1,200 qps for 50 users? - Its all about Performance and Scalability CAUSE Determine what queries Excessive SQL statements 4 are running and why they Duplicate are running? Redundant Cachable Row at a time (RAT) EffectiveMySQL.com - Its all about Performance and Scalability 4EffectiveMySQL.com - Its all about Performance and Scalability
WHY EXAMPLE RAT SELECT * FROM activities_theme WHERE Reducing SQL load both 4theme_parent_id=0 SELECT * FROM activities_theme WHERE theme_parent_id=1 SELECT * FROM activities_theme WHERE theme_parent_id=2 SELECT * FROM activities_theme WHERE theme_parent_id=11 improves performance now SELECT * FROM activities_theme WHERE theme_parent_id=16 and provides greater SELECT * FROM activities_theme CAT WHERE theme_parent_id in (0,1,2,11,16) capacity as you scale http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/ EffectiveMySQL.com - Its all about Performance and Scalability 4EffectiveMySQL.com - Its all about Performance and Scalability EXAMPLE EXAMPLE Unnecessary 5 Query SELECT * FROM `artist` SELECT pages_id, pages_livestats_code, pages_title, Unnecessary 4 5 Query SELECT * FROM `artist` pages_parent, pages_exhibid, pages_theme, 5 Query SELECT * FROM `artist` Duplicate pages_accession_num 5 Query SELECT * FROM `artist` FROM pages WHERE pages_id = 0 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` WHERE (ArtistID = 196 ) Only 2 queries 5 Query SELECT * FROM `artist` WHERE (ArtistID = 2188 ) 5 Query SELECT * FROM `artist` necessary or 1 CAT 5 Query SELECT * FROM `artist` 5 Query SELECT * FROM `artist` 5 minutes 6000 executions 0 is out of bounds http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/ EffectiveMySQL.com - Its all about Performance and Scalability 4EffectiveMySQL.com - Its all about Performance and Scalability http://ronaldbradford.com/blog/optimizing-sql-performance-the-art-of-elimination-2010-07-08/
Capture & Analyze MySQL Binary Log (Archive 4Redo) mysqlbinlog DML is easy mk-query-digest SELECT is harder One Liner http://ronaldbradford.com/blog/mysql-dml-stats-per-table-2009-09-09/ http://www.slideshare.net/ronaldbradford/capturing-analyzing-and-optimizing-mysql EffectiveMySQL.com - Its all about Performance and Scalability 4EffectiveMySQL.com - Its all about Performance and Scalability 55463 update sessions Top 2 queries = 57% 25574 insert into sessions Process List 4 12820 update items 11636 insert into item_categories 7532 update users 5168 delete from item_categories 4076 update extended_item_infos General Log 3701 insert into sphinx_new_items 3701 insert into mini_items 2190 update sweet_bars 1922 update chat_users tcpdump 1662 update item_shipping_infos 1265 update search_terms 1260 insert into images 81k of 141k 931 delete from item_shipping_infos Application 825 update booths 713 update booth_stats 574 update topics 540 update offers EffectiveMySQL.com - Its all about Performance and Scalability 4EffectiveMySQL.com - Its all about Performance and Scalability
EXPERT TIP Capturing, Analyzing and Optimizing /* Comment your queries 4*/ your SQL The more products you have, the http://www.slideshare.net/ronaldbradford/capturing-analyzing-andoptimizing-mysql more developers you have, the more time you spend in code identification before you can even determine a resolution EffectiveMySQL.com - Its all about Performance and Scalability 4EffectiveMySQL.com - Its all about Performance and Scalability 3 PROBLEM The database is slow. My webpage takes five seconds to load. EffectiveMySQL.com - Its all about Performance and EffectiveMySQL.com - Its all about Performance and Scalability Scalability3 3
SOLUTION EXAMPLE Evaluate the time taken in Client example showed 3a webpage taking 5 seconds to load. The html the database and all component was taking only 400 ms. stages in between Any MySQL performance improvement will only tune 8% of the total time. EffectiveMySQL.com - Its all about Performance and Scalability 3EffectiveMySQL.com - Its all about Performance and Scalability WHY Performance is important Firebug - 3 http://getfirebug.com/ to end user Httpwatch - http://httpwatch.com/ Page speed - http://code.google.com/speed/page-speed/ Performance is perception YSlow - http://developer.yahoo.com/yslow/ wget/curl Application code instrumentation EffectiveMySQL.com - Its all about Performance and Scalability 3EffectiveMySQL.com - Its all about Performance and Scalability
EXPERT TIP 2 http://www.stevesouders.com/ http://developer.yahoo.com/ performance/rules.html EffectiveMySQL.com - Its all about Performance and Scalability 3EffectiveMySQL.com - Its all about Performance and Scalability2 PROBLEM SOLUTION I want to add new H/W. Develop a seamless 2 How do I change my integration that requires no application to support this? code changes, no downtime and very little additional physical resources. EffectiveMySQL.com - Its all about Performance and Scalability 2EffectiveMySQL.com - Its all about Performance and Scalability
Integrated monitoring and Seamless automated server 2 instrumentation deployment Deployed from Day 1 Version Control Build & Release Runtime config management Automated discovery EffectiveMySQL.com - Its all about Performance and Scalability 2EffectiveMySQL.com - Its all about Performance and Scalability API Different levels of data 2availability One code path for business Read & Write functionality Read Only Implied business documentation No Access Enforced data exchange standard Cached Testability EffectiveMySQL.com - Its all about Performance and Scalability 2EffectiveMySQL.com - Its all about Performance and Scalability
REFERENCE Different principles for scalability Successful MySQL Scalability 2 1. Integrated monitoring & instrumentation Read Scalability 2. Seamless automated server deployment Write Scalability 3. Disaster is inevitable 4. Application Programming Interface Caching 5. Support different levels of data availability 6. Support different scalability principles http://ronaldbradford.com/blog/successful-mysql-scalability-presentation-2010-09-17/ EffectiveMySQL.com - Its all about Performance and Scalability 2EffectiveMySQL.com - Its all about Performance and Scalability 1 PROBLEM My website is slow? EffectiveMySQL.com - Its all about Performance and EffectiveMySQL.com - Its all about Performance and Scalability Scalability1 1
SOLUTION Seek professional advice. Hire for example Ronald Bradford. 22+ years of system architecture, database design and performance tuning. Employment as Consultant for Oracle Corporation (96-99) Employment as Senior Consultant for MySQL Inc (06-08) R R EffectiveMySQL.com - Its all about Performance and Scalability 1EffectiveMySQL.com - Its all about Performance and Scalability RECAP RECAP Monitoring. Before, during and after NOW. 3 levels of real time data Raccess. Read/Write, Read and no access You may not be able to predict the future but you can preempt the future. 3 aspects of scalability. Read, Write and Caching Choose the best product and features for you needs. Operate below 90% capacity. That 10% is your insurance. The best SQL statement is the one you never have to execute. EffectiveMySQL.com - Its all about Performance REffectiveMySQL.com and Scalability - Its all about Performance and Scalability
CONTACT http://ronaldbradford.com me@ronaldbradford.com EffectiveMySQL.com - Its all about Performance and Scalability CEffectiveMySQL.com - Its all about Performance and Scalability