Twitter, Inc.

Transcription

1 No More Forkin - Migrating to MySQL Community Version

2 Background Migration Process Migration Challenges Post-Deployment Challenges Future Plans Q&A

3 Most Twitter Followers (109m) US (102m) (77.7m) (71.9m) Sports (72m) Tech (45.7m)

4 Answers to you questions No - These clusters do not contain tweets No - I don t have access to tweets and user profiles Yes - These clusters are used by internal applications and some external-facing applications / services.

5 Background History of Rationale for Twitter Fork List of Custom Changes Reasons for Abandoning Fork

6 History of October MySQL 5.0 Community Version Released March First Tweet from Jack Dorsey November MySQL 5.1 Community Version Released December MySQL 5.5 Released February MySQL Community Version Released October MySQL 5.7 Community Version Released September Start of Twitter Fork - MySQL Twitter Version March MySQL Twitter Version Released June Start of MySQL 5.6 Twitter Version Build March MySQL Twitter Version Released Q Twitter Evaluation of MySQL 5.7 Community Version January Last Release of MySQL Twitter Version (5.6.35)

7 Rationale for Twitter Fork General Operability by the DBA Team Agility to Change Observability Memory Management Predictability Performance Auditability Source: display/mysql/home

8 List of Custom Changes User name length increased from 16 to 32 (Twitter MySQL ) innodb_deadlock_check (Twitter MySQL ) Improve memory allocation, especially under NUMA (Twitter MySQL ) Add partitioning functions for YMD and YMDH (Twitter MySQL )

9 Reasons for Abandoning Our Branch Cost Leverage Community Support and Agility MySQL 5.7 New Features Automatic Password Expiration JSON Data Type Support Multi-Source Replication And others

10 #MigrationProcess MySQL Migration Methodology (High-Level) Migration Methodology (Server Upgrade) Deployment Schedule and Timeline

11 MySQL Sandboxes / Staging Clusters Multi-Tenant Clusters Single-Tenant Clusters Revenue Clusters Gizzard Clusters

12 Migration Methodology In-place Upgrade vs. Logical Upgrade (Uninstall / Reinstall / Restore)

13 Migration Methodology (High-Level) Create MySQL 5.7 bridge from a standby slave using mysql_twtr_to_community - dump source (standby), scan dump for incompatibilities, load to destination (bridge) Perform checksum between the bridge and standby (pt-tablechecksum) Upgrade 1 slave within the cluster and restore from the MySQL 5.7 bridge After having 1 slave in MySQL 5.7 for a while and no issues encountered, upgrade remaining slaves Upgrade standby Promote standby to master (point of no return) Upgrade old master to MySQL 5.7

14 Migration Methodology (Server Upgrade) Block host from receiving traffic and unmonitor Monitor performance of remaining slaves, check for increase in total queries and slow queries Uninstall old version of MySQL Install MySQL 5.7 Restore databases from another MySQL 5.7 slave

15 Deployment Timeline/Schedule 2016 Q4 - MySQL 5.7 Migration Gap Analysis / Strategy Development May Developed Migration Scripts June Build Staging/Sandbox Environments (24) October Upgrade Multi-Tenant Clusters (10) November/December Upgrade Single-Tenant Clusters (41) February 2018 to March Upgrade Revenue Clusters (After SB)

16 #MigrationChallenges Logistic Challenges Technical Challenges

17 Migration Logistic Challenges Multi-tenant cluster coordination Unable to find cluster owners Holidays / Moratoriums / Year-End Vacations (Black Friday, Christmas, New Year, Year-End) happy-new-year-from-the-minions

18 Migration Technical Challenges pt-show-grants - password not included anymore in IDENTIFIED BY pt-table-checksum results issues Missing user grants Twitter-specific system variables Slave capacity Cluster too active (bridge unable to catch-up) Partition by timestamp GTID issues

19 Migration Technical Challenges pt-show-grants MySQL 5.6 GRANT USAGE *.* TO <User> IDENTIFIED BY PASSWORD <Password> MySQL 5.7 GRANT USAGE *.* TO <User> Solution: Created our own script to extract user grants that returns the same format both for MySQL 5.6 and MySQL 5.7

20 Migration Technical Challenges pt-table-checksum results issues Issue: Error checksumming table <Table Name>: Possible infinite loop detected! Solution: Compared row count between standby and bridge for all tables in all databases.

21 Migration Technical Challenges Missing User Grants Solution: Extract user grants from master and apply to upgraded slaves.

22 Migration Technical Challenges Twitter-Specific System Variables Issue: Unable to install MySQL 5.7 due to unknown configuration setting. Solution: Remove offending system variable from the puppet attribute of the host (example: innodb_deadlock_check)

23 Migration Technical Challenges Slave Capacity Bridge MySQL 5.7 Standby MySQL 5.6 Master MySQL 5.6 Slave MySQL 5.6 Slave MySQL 5.6 Slave MySQL 5.6 Solution: Add more slaves (temporarily / permanently) to replace the slave to be upgraded.

24 Migration Technical Challenges Cluster too active Issue: Bridge unable to catch-up after restoring from slave due to the master receiving too much activity. Solution: Turn durability off

25 Migration Technical Challenges Partition by timestamp Issue: Constant, random or timezone-dependent expressions in (sub)partitioning function are not allowed. Cause: In MySQL 5.7, only unix_timestamp function is allowed to partition a table by RANGE based on the value of a timestamp column. Solution: Change data type of partition column to datetime

26 Migration Technical Challenges GTID Issues Error 'When is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation.

27 GTID Issue Read-Only = 1 Bridge West Coast Data Center Read-Only = 1 Standby pt-heartbeat (super user) Read-Only = 0 Master Slave Slave Slave

28 GTID Issue Connect bridge to master? Standby West Coast Data Center Read-Only = 1 Read-Only = 0 Read-Only = 1 pt-heartbeat (super user) Master Bridge MySQL 5.7 Slave Slave Slave

29 GTID Issue Make the standby read-write? West Coast Data Center Read-Only = 1 Bridge Read-Only = 0 Standby pt-heartbeat (super user) Read-Only = 0 Master Slave Slave Slave

30 GTID Issue - Solution Standby West Coast Data Center Read-Only = 1 Read-Only = 0 Read-Only = 0 pt-heartbeat (super user) Master pt-heartbeat (super user) Bridge MySQL 5.6 Slave Slave Slave Bridge MySQL 5.7 Read-Only = 1

31 Post-Deployment Challenges Compatibility Issues Application Performance Post-Migration Realizations Recommendations

32 Compatibility Issues New Key Words and Reserved Words LOAD DATA LOCAL INFILE Incorrect usage of UNION and LIMIT

33 Compatibility Issues New Key Words and Reserved Words Account, Channel, Encryption, Filter, Instance SELECT * FROM virtual Solution: Put table names within quotes: SELECT * FROM `virtual` Rename table to avoid the use of a reserved word

34 Compatibility Issues LOAD DATA LOCAL INFILE LOAD DATA LOCAL INFILE <File Name> INTO TABLE <Table Name> FIELDS TERMINATED BY, ENCLOSED BY (<Fields List>); Error: Invalid utf8 character string Cause: Change of default value of innodb_strict_mode changed from OFF to ON. Solution #1: Add CHARACTER SET latin1 LOAD DATA LOCAL INFILE <File Name> INTO TABLE <Table Name> CHARACTER SET latin1 FIELDS TERMINATED BY, ENCLOSED BY (<Fields List>); Solution #2: Convert input file to utf8 using iconv $ iconv from-code=iso to-code=utf-8 [Input File] > [Output File]

35 Compatibility Issues Incorrect usage of UNION and LIMIT SELECT SQL_CALC_FOUND_ROWS * FROM (SELECT <Columns> FROM <table> WHERE <WHERE Clause> ORDER BY <ORDER BY Clause>) AS sorted LIMIT 1 OFFSET 700 UNION ALL SELECT FOUND_ROWS(), 'Total Count', <Empty Columns> (SELECT SQL_CALC_FOUND_ROWS * FROM (SELECT <Columns> FROM <table> WHERE <WHERE Clause> ORDER BY <ORDER BY Clause>) AS sorted LIMIT 1 OFFSET 700) UNION ALL (SELECT FOUND_ROWS(), 'Total Count', <Empty Columns>) Output: ID Type Date Status My Tweets Unknown 1233 Total Rows Output: ID Type Date Status My Tweets Unknown 6 Total Rows

36 Compatibility Issues Incorrect usage of UNION and LIMIT Solution: Query re-write SELECT * FROM (SELECT <Columns> FROM <table> WHERE <WHERE Clause> ORDER BY <ORDER BY Clause> LIMIT 1 OFFSET 700) AS sorted UNION ALL SELECT COUNT(*), <Empty Columns> FROM <table> WHERE <WHERE Clause>

37 Application Performance Slow Analytics Queries with Derived Tables SELECT <Columns>, SUM(<Column>) FROM (SELECT <Columns> FROM <Table1> LEFT JOIN <Table2> ON <Condition1> LEFT JOIN <Table3> ON <Condition2> LEFT JOIN (SELECT <Columns> FROM <Table4>) ON <Condition3>) WHERE <WHERE Clause> GROUP BY <GROUP BY Clause> MySQL 5.6 : 3-4 Minutes MySQL 5.7 : Minutes Solution: set session optimizer_switch= derived_merge=off ; Execution Time: 50 seconds

38 Post-Migration Realizations Unused Clusters Grouped Clusters Use of LOAD DATA LOCAL INFILE Load Balancer Parameter Set-up Slave Load Threshold

39 Future Plans (Next 12 Months) MySQL HA MySQL 8.0 Vitess

40 #Recommendations Test, test, test Work with customers early in the process Inform customers of the progress/schedule Timing is everything (from scheduling to standby promotion to moratorium) Transparency is key Explain Plan is your friend Request for help as necessary

41 #Recommendations Test, test, test Work with customers early in the process Inform customers of the progress/schedule Timing is everything (from scheduling to standby promotion to moratorium) Transparency is key Explain Plan is your friend Request for help as necessary

42 - Q&A

43 Time for Twitter Swags What month and year was Jack Dorsey s first tweet? What was Jack Dorsey s first tweet? What was MySQL Twitter version s last release version? Who has the most number of Twitter followers? What 3 Percona toolkit tools / utilities were mentioned during the presentation?

44 We re Hiring #jointheflock