Migrating To MySQL The Live Database Upgrade Guide

Transcription

1 Migrating To MySQL The Live Database Upgrade Guide October 4, 2016 Krzysztof Książek Severalnines krzysztof@severalnines.com 1

2 Agenda! Why upgrading to MySQL 5.7?! Preparing an upgrade - changes between MySQL 5.6 and 5.7! Information_schema! SQL modes! Authentication model! Testing for upgrade! Designing test environment! Building test environment! Executing tests! Upgrade process! Tools which may help you with upgrade process 2

3 Why upgrading to MySQL 5.7? 3

4 Why upgrading to MySQL 5.7?! Better performance! New features! Multi source replication! Access to security and bugfix releases! 5.5 ended its Premier support, Extended will end in 2018! Intra-schema parallel replication! Online buffer pool resize!!!! Improved optimizer cost model 4

5 Preparing an upgrade - reviewing changes 5

6 Preparing an upgrade - reviewing changes! Information schema changes! global_status, session_status, global_variables and session_variables moved to the performance_schema! You can restore the original behavior by setting: SET GLOBAL show_compatibility_56=1! This may affect different monitoring solutions 6

7 Preparing an upgrade - reviewing changes! SQL mode changes! STRICT_TRANS_TABLES mode is used by default! No more inserting of invalid data like zeroed date or skipping column in INSERT when column doesn t have explicit DEFAULT value! This may cause issues if application relies on previous (definitely not so good) defaults 7

8 Preparing an upgrade - reviewing changes! Authentication model changes! Pre-4.1 passwords have been removed! Password expiration has been added! By default it s not affecting 5.6 users ( >=5.7.11) but this default may change in the future ( You should manually set the desired expiration policy for every user, after you upgraded to MySQL 5.7 8

9 Preparing an upgrade - reviewing changes! Redo and undo log format changed - use innodb_fast_shutdown=0 when you stop 5.6 for upgrade! YEAR(2) format has been removed! User-defined locks system has changed - multiple locks are now possible! derived_merge optimizer switch may trigger SQL incompatibility! For full list of changes look at: 9

10 Testing for upgrade 10

11 Testing for upgrade 11

12 Testing for upgrade! First step - design your test environment! It should be as close to the production as possible! Weaker hardware will lead to incorrect results! Queries may be slower on weaker CPU and I/O subsystem! or you will not catch a real regression - thinking it s the hardware s fault 12

13 Testing for upgrade! Dataset should be as close to the real data as possible! Scrubbing dataset or cleaning confidential data may impact query execution plans - if you don t have to, don t clean your data! You should be using real-world queries to test the performance! Ideally, use your application for testing the logic and making sure all processes work as expected. 13

14 Testing for upgrade! Simplest way of building a test environment:! create a slave from production master! and then cut off replication! Once we have 5.6 node with current dataset, we need to create another 5.6 node with a copy of this data! Finally, we want to upgrade one of them to MySQL 5.7! It s nice as we can test the upgrade process while building test hosts 14

15 Testing for upgrade! We have hosts but we need to collect data for tests! tcpdump or just slow query log would do! both have some impact on the overall performance! tcpdump can be executed outside of the database, to minimize the impact! You want to collect all the queries - if you have some processes scheduled to run at a particular time, collect those queries too when they happen 15

16 Testing for upgrade! Once you have data, you can run some preliminary regression testing using pt-upgrade! Run twice, first time using the save-results flag - it ll save your time! Generate report, investigate any reported issues! pt-upgrade compares performance and result set! Make sure you execute tests locally on the hosts, network latency may cause some impact! Make sure both nodes are in the same state - cold or warmed up in exactly the same way 16

17 Testing for upgrade! pt-upgrade is not enough to cover all issues - yes, it ll check performance and some obvious issues but you need to run more detailed tests! If you have a set of acceptance tests you use to test new releases (you should have them), you can use those tests to check if upgrade affected core functions of your app! Test application logic, make sure all operations work correctly, check all of the processes and APIs you may have in your app 17

18 Testing for upgrade! Once you have completed your tests, there's one more thing to check! Rebuild your 5.7 node! provision it again! run mysql_upgrade, as you did earlier! set it up as a slave! Allow it to work as a slave for couple of days - it ll confirm replication works fine between old and new versions 18

19 Upgrade process 19

20 Upgrade process! Start with a single slave and upgrade it:! Disable innodb_fast_shutdown! Shut down MySQL! Restart MySQL! Make sure replication is working and if needed, fix it! Upgrade binaries to MySQL 5.7! Start MySQL! Run mysql_upgrade 20

21 Upgrade process! Before upgrade - make sure you took the node out of rotation in a proxy layer or in your application! After upgrade - if possible, put some limited traffic on the new host! Observe how it behaves under real-world traffic! CPU! I/O! Memory utilization 21

22 Upgrade process! Step by step increase the load to a normal level! If everything is ok, proceed with upgrade on remaining slaves! Once all of the slaves are upgraded, pick one of them as a new master! Set it in a master-master replication with current, 5.6 master 22

23 Upgrade process! Perform a switchover to the new master! If the old master keeps replicating, that s great - upgrade it in a regular way! If the old master replication breaks (i.e. incompatibilities in binlog format between old and new MySQL versions), rebuild it:! Wipe the data! Perform binary upgrade to MySQL 5.7! Provision the data from slave and set up the replication 23

24 Tools which may help you with upgrade process 24

25 Tools which may help you with upgrade process! tcpdump! tcpdump -s x -nn -q -tttt -i any port 3306 > mysql.tcp.txt! Helps you to collect queries exchanged between application hosts and database server! Minimizes the impact caused by data collection (when executed outside of database node) 25

26 Tools which may help you with upgrade process! pt-upgrade, Percona Playback! pt-upgrade is designed to check for performance and basic result set regressions! Points you to queries which are not performing on the new version or they return different data! Percona Playback is designed for load testing - replays slow log, trying to simulate real-world workload 26

27 Tools which may help you with upgrade process! MaxScale and ProxySQL! Database-aware proxies help you through the upgrade process! They route the traffic off the node when it goes down! Help you with a warm-up period - use lower weight for new hosts to allow them to warm up properly! ProxySQL can handle graceful switchover when used with external tool (for example, mysqlrpladmin, MHA or ClusterControl) 27

28 Tools which may help you with upgrade process! ProxySQL can be used to mirror traffic to another hostgroup via query rules! mirror_hostgroup column in mysql_query_rules! Performance-wise, drop can be from ok-ish to serious! Yet, workload depending, this could be a feasible option! Especially couple of releases later, when existing bugs will be fixed 28

29 Tools which may help you with upgrade process 29

30 Tools which may help you with upgrade process! mysqlrpladmin, MHA! Those tools can perform a switchover for you! No need to make the changes in manual and error prone way - just run a script! Those tools can be integrated with ProxySQL for graceful switchovers 30

31 Tools which may help you with upgrade process! ClusterControl! Helps with provisioning of new nodes! Perform a switchover for you! Gives you clear insight into replication status! Gives you clear insight into all database operations 31

32 Thank You! Q&A Contact: 32