Heterogeneous Replication Between MySQL and MongoDB

Transcription

1 Heterogeneous Replication Between MySQL and MongoDB USING TUNGSTEN REPLICATOR 09/26/2017 Percona Live Europe 2017

2 HELLO! My name is Pep Pla

3 INTRODUCTION Native Replication Changes are logged on the origin Logs are transferred from origin to destination Changes are applied into the destination Limitations Restricted to same technology both in origin and destination The Pythian Group Inc.,

4 INTRODUCTION Tungsten Replication Changes are logged on the origin (native) Changes are translated to tungsten format (thl) Thl files are transferred from origin to destination Thl files are translated back to (some sort) of native format and Changes are applied into the destination Advantages Not restricted to the same technology both in origin and destination The Pythian Group Inc.,

5 TUNGSTEN ARCHITECTURE The Pythian Group Inc.,

6 ORIGINS OF HRBMAMUTR Legacy Architectures Big monolithic databases Few technologies used One-size-fits-all philosophy Contemporary Architectures Multiple databases Tens of technologies in use Specialization The Pythian Group Inc.,

7 ORIGINS OF HRBMAMUTR Transfer of information between different systems and different technologies is required: Batch transfer (ETL) Messaging Application level Replication Homogeneous Heterogeneous The Pythian Group Inc.,

8 ORIGINS OF HRBMAMUTR Hackaton : European Open Database Camp 2011 Robert Hodges Flavio Peroco Implemented a new applier in a few hours Requires some additional information from the binlogs Apply thl logs into mongodb Few changes recently The Pythian Group Inc.,

9 MySQL vs. MongoDB Collection MySQL Database Table Row MongoDB Database Collection Document Structure replication MongoDB and MySQL databases are the same But Collections are not Tables and Documents are not Rows The Pythian Group Inc.,

10 MySQL vs. MongoDB Tables vs. Collections Normalization DDL Tables should honor 1NF Collections should not comply 1NF Different sets of commands Collections are created implicitly Foreign keys The Pythian Group Inc.,

11 MySQL vs. MongoDB Row vs. Document Row has a fixed format Changes in format change existing data Document is a JSON structure Each document has its own structure The Pythian Group Inc.,

12 MySQL vs. MongoDB MySQL Product Sizes ProductId ProductDescription SizeId SizeDescription ProductSizes ProductId MongoDB Product ProductId ProductDescription SizeA SizeDescriptionA ProductSizeQuantityA SizeB SizeDescriptionB SizeId ProductSizeQuantity The Pythian Group Inc.,

13 MySQL vs. MongoDB Additional differences Transactions Isolation levels Triggers Foreign keys... The Pythian Group Inc.,

14 Create a lab using Vagrant One Vagrantfile, two yml files for provision, two tungsten cfg files MySQL node: Percona-Server-5.7 MongoDB node: MongoDB 3.4 Vagrant with VirtualBox provider Ansible local provisioning (no need to install ansible in the host) CentOS 7 Tested with MacOS and Windows Vagrant plugin vagrant-vbguest required The Pythian Group Inc.,

15 Create a lab using Vagrant Most recent version of Tungsten Replicator available in github Vagrantfile, tungsten.ini and ansible yml files will be available in github (not yet) or to plasanz@pythian.com Disclaimer Not intended for production Security features are disabled Trivial passwords Mongod open to the network The Pythian Group Inc.,

16 What you can do with HRBMAMUTR Mirror tables to collections DML are replicated to MongoDB Empty tables are not replicated Not a big deal The Pythian Group Inc.,

17 What you can t do with HRBMAMUTR DDL are not replicated Collections are created implicitly When you perform a DML against one table, this is replicated and the collection is created if needed PK indexes are created Do not trust, in some circumstances indexes are not created No other indexes are created Renaming a table DOES NOT rename the collection Removing a table DOES NOT remove the collection The Pythian Group Inc.,

18 What you can t do with HRBMAMUTR Grants are not replicated But inserts/updates/deletes in any database are replicated as long as they are in the binlog Beware of mysqldump The Pythian Group Inc.,

19 Indexes MySQL and MongoDB should be used for different purposes Different indexes should be created? Implicit indexes (from MySQL pk) are not unique Primary key MongoDB has _id You can have a column in MySQL _id and it is going to be used Must be unique (will NOT fail) Can use filters, as long as your primary key is not composed The Pythian Group Inc.,

20 Implementation strategies Usually you already have a MySQL database MySQL replication strategy Backup Restore Configure and start replication This can t be done, as you can t backup from MySQL and restore to MongoDB Recreate all the tables you need to replicate Do not replicate the whole database The Pythian Group Inc.,

21 Online schema changes Using MySQL commands Be careful and think it twice No recommended Percona or Github tools Recommended No downtime in MySQL You need to stop replication to MongoDB for a short period of time The Pythian Group Inc.,

22 Online schema changes No automatic cutover Remember, DDL are not replicated Stop MongoDB applier trepctl offline Perform the cutover trepctl status before the cutover to retrieve current thl file Find the cutover position in the THL logs Using thl list -file command The Pythian Group Inc.,

23 Online schema changes Start MongoDB applier until that position trepctl online -until-seqno <sequence number> Perform the cutover in MongoDB Rename collections Restart MongoDB applier trepctl online The Pythian Group Inc.,

24 Online schema changes Check replication filters before the change! Both pt-osc and gh-ost create tables that should be replicated Dummy online schema changes can be used for migration! The Pythian Group Inc.,

25 Lets recap Tables are not collections Rows are not documents Choose carefully what you are going to replicate This can be done using filters in the destination Create indexes in MongoDB Use online schema change tools to migrate Use our lab and plan your new architecture and the migration process The Pythian Group Inc.,

26 How could we do it better? Denormalizing and document embedding

27 What is the problem? MySQL Product Sizes ProductId ProductDescription SizeId SizeDescription ProductSizes ProductId MongoDB Product ProductId ProductDescription SizeA SizeDescriptionA ProductSizeQuantityA SizeB SizeDescriptionB SizeId ProductSizeQuantity The Pythian Group Inc.,

28 What is the problem? Tables are not collections Rows are not documents MongoDB is a different language We re trying to speak MySQL language using MongoDB words We need to create MongoDB documents from multiple MySQL rows Implement the relational thing into one document This is not possible with current implementation The Pythian Group Inc.,

29 How to do it? Fix the problem at MongoDB applier Retrieve missing information from MySQL and create the document Problems Latency and changes in data can lead to corrupt documents MySQL connection from MongoDB host The Pythian Group Inc.,

30 How to do it? Fix the problem at MongoDB applier Retrieve missing information from MongoDB and create the document Problems Latency and changes in data can lead to corrupt documents New documents/initial table creation The Pythian Group Inc.,

31 How to do it? Fix the problem at MySQL replicator Retrieve missing information from MySQL and create the document Problems We are near the database but there is still latency and: Latency and changes in data can lead to corrupt documents The Pythian Group Inc.,

32 How to do it? Fix the problem at MySQL Retrieve the changes within the transaction and compose the document Using triggers Use a blackhole table Operation field Destination collection JSON field with Document Problems JSON is not supported by tungsten The Pythian Group Inc.,

33 One small change for MySQL and a giant change for MongoDB Changing one row in MySQL can lead to millions of changes in MongoDB documents Change the description of one size requires only one update in the related table But requires to change all MongoDB documents The Pythian Group Inc.,

34 THANK YOU! The Pythian Group Inc.,

35 ABOUT PYTHIAN Pythian s 400+ IT professionals help companies adopt and manage disruptive technologies to better compete The Pythian Group Inc.,