Spread the Database Love with Heterogeneous Replication. MC Brown, VP, Products

Transcription

1 Spread the Database Love with Heterogeneous Replication MC Brown, VP, Products

2 Heterogeneous Replication is NOT Exporting and Importing Data One Time Exports Moving to a different database platform ETL

3 Heterogeneous Replication IS Live, constant, low-latency movement of data For analytics For migration For upgrades For Caching Data/format matching Effective target reproduction

4 Know Your Databases

5 Not all Databases are Created Equal Transactional over non transactional Object Reference Rows Columns Documents Free text Unstructured

6 Is that a record, a field, a row a column? Row of data? Collection of related tables? What does it look like as a document? What does a document look like as a row? Databases, tables, collections, objects, buckets

7 Related Tables or Document?

8 Mapping DB Compatibility RDBMS RDBMS Vendor specific only Columnar Store Vendor specific only Document Database Field mappings only Freetext/ unstructured data store Application specific Columnar Store Vendor specific only Vendor specific only Field mappings only Application specific Document Database Field mappings only Field mappings only Vendor specific only Application specific Freetext/ unstructured data Application specific Application specific Application specific Application specific Vendor specific - i.e. unique data types Field mappings - how we map the data App Specific - how the data is used

9 Know Your Data

10 Hetero Replication Challenges Effective data replication low latency Automatic mapping Data typing Indexing and native use nothing lost or removed

11 Challenges: Data Typing Data types are not supported everywhere For some, the type does not matter Even if the type does matter, the format, precision, structure might be different Numbers, Dates, Strings, Compound Data Types all cause problems

12 Extraction/Apply Rates Data extraction rates vary Data apply rates Different solution handle data loading at different rtes Rows-based extraction/bulk apply Bulk extraction/row apply Non-destructive

13 Whats the solution?

14 Replicator Needs Native, neutral format Ability to change, reformat, restructure information Standalone nature Two-way Handle impedance problem

15 Guess What? Tungsten Replicator does this High Performance Flexible storage interchange format Built-in filtering Operates standalone Stop and restart Transactionally consistent Open Source

16 Applying Data

17 Native is Best, Batch an Alternative Native: Applying to JDBC Adapt JDBC Applier to construct statement Or apply a record to target using API Batch Use CSV for data interchange Call scripts to import

18 How Batch Apply Works Replicator Transactions from master Service ora2vr COPY to stage tables Staging Staging Tables Staging Tables Tables SELECT to base tables Base Base Tables Base Tables Tables Merge Script CSV CSV Files CSV Files Files (or) COPY directly to base tables

19 How Batch Apply Works Works on one table at a time Five functions in JavaScript Apply - During transaction Prepare - Run when going online Begin - Start of transaction Commit - After transaction Release - When going offline

20 During a transaction Copy, import, load the CSV Have access to column, key and transaction information Merge the data Done Delete and Insert, or Delete, Update and Insert

21 Case Study: Cassandra/CQL Load table data: COPY staging_tablename (optype,seqno,uniqno,id,message) from FILENAME Delete: delete from sample where id in (#{deleteidlist}) Insert: insert into sample ("+collist+") values ("+substlist+")

22 Filters

23 Filter Execution Stage Extract Filter Apply Stage Extract Filter Apply Binlog tcp/ip Slave Replicators MySQL Master In-Memory Queue Transaction History Log

24 Filter Operation Always get one transaction at a time Metadata Data blocks SQL or ROW Info Always returns the transaction Transaction must be processed inline

25 JS Filters prepare() - called when going online filter() - does the work release() - called when going offline Access to: Connection to DB Full Java class environment Bunch of utility functions

26 Data Structure ReplDBMSEvent DBMSData StatementData DBMSData StatementData DBMSData RowChangeData OneRowChange OneRowChange... StatementData ReplDBMSEvent DBMSData RowChangeData OneRowChange OneRowChange...

27 Get/Set Values for(j = 0; j < rowchanges.size(); j++) { onerowchange = rowchanges.get(j); columns = onerowchange.getcolumnspec(); columnvalues = onerowchange.getcolumnvalues(); for (c = 0; c < columns.size(); c++) { columnspec = columns.get(c); type = columnspec.gettype(); if (type == TypesDATE type == TypesTIMESTAMP) { for (row = 0; row < columnvalues.size(); row++) { values = columnvalues.get(row); value = values.get(c); } } } } if (value.getvalue() == 0) { value.setvaluenull() }

28 What you can do in a filter Anything

29 Case Study: Building a Kafka Applier

30 Kafka? Message queue/bus Huge flexible Very practical High performance Not a database Full publish/subscribe model

31 Message Format for Data Embedded JSON CSV Row Message topic? Encoded binary fields Schema/table/primary key?

32 Impedance What happens with multi-row transactions? What happens when a multi-row transaction is not applied? Should we split data into chunks?

33 What we do already Sources MySQL Oracle Targets MySQL Oracle RedShift Vertical Hadoop Text SQLite RabbitMQ S3 MongoDB

34 What are we adding? Sources REST API Input MongoDB Couchbase CouchDB PostgreSQL Targets Cassandra Amazon Athena Couchbase CouchDB ElasticSearch Flume Kafka Native JDBC to Hadoop PostgreSQL

35 Where Next github.com/continuent/tungsten-replicator mcb.guru