Tungsten Replicator for Kafka, Elasticsearch, Cassandra

Transcription

1 Tungsten Replicator for Kafka, Elasticsearch, Cassandra

2 Topics In todays session Replicator Basics Filtering and Glue Kafka and Options Elasticsearch and Options Cassandra Future Direction 2

3 Asynchronous replication decouples transaction processing on master and slave DBMS nodes MySQL/Oracle MySQL/Oracle DBMS-specific Logging (i.e. Redo or Binary) DBMS Logs Option 1: Local Install Extractor reads directly from the logs, even when the DBMS service is down. This is the default. Option 2: Remote Extractor gets log data via MySQL Replication Slave protocols (which requires the DBMS service to be online) or the Redo Reader feature. This is how we handle RDS and Oracle extraction tasks. Extractor Options 2 1 Master Replicator: Extractor THL Download transactions via network THL = Events + Metadata Apply using JDBC Slave Replicator: Applier THL 39

4 Parallel apply maximizes DBMS I/O bandwidth when updating replicas Slave Replicator Pipeline Stage Stage Stage remote-to-thl thl-to-q q-to-dbms Extract Filter Apply Extract Filter Apply Extract Filter Apply Extract Filter Apply Master replicator THL (Events+ metadata) Parallel Queue Extract Filter Apply Slave 30

5 Why Kafka Kafka is a high performance message bus NOT a database Great for distributing messages and firing/triggering operations on content Log aggregation Activity/security tracking Metrics Auditing Data ingestion for Hadoop

6 Mass Data Collection with Kafka Kafka Tungsten Replicator Kafka Kafka

7 Multiple Target Distribution Database Tungsten Replicator Kafka Image Process Database Kafka Database Kafka Metrics

8 How Kafka Replication Works MySQL/Oracle DBMS-specific Logging (i.e. Redo or Binary) DBMS Logs Kafka Applier (Native) Master Replicator: Extractor Download transactions via network Slave Replicator: Applier Zookeeper THL THL = Events + Metadata THL

9 What Tungsten Replicator Does to Apply into Kafka Takes an incoming row and converts it to a message Message consists of metadata: Schema name, table name Sequence number Commit timestamp Operation Type Embedded Message Content

10 Message Structure Schema Table Row Row Row Row Row Row Topic: Schema_Table MsgID: Schema Table PKey Row MsgID: Schema Table PKey Row

11 Sample Message { } "_meta_committime" : " :27:18.0", "_meta_source_schema" : "sbtest", "_meta_seqno" : "10130", "_meta_source_table" : "sbtest", "_meta_optype" : "INSERT", "record" : { "c" : "Base Msg", "k" : "100", "id" : "255759", "pad" : "Some other submsg" }

12 Customizable Elements Whether acknowledgements are required from Kafka How much distribution/replication is required before sending the message Format of the message key Whether to embed schema and table name Whether the commit timestamp should be embedded

13 Demo

14 Elasticsearch Immediately replicate data into Elasticsearch for searching Contains the core text and content of the records Provides the original information to track back to the record Content structure against the schema (index type) and tablename (index) Document ID based on the pkey and other information which is configurable

15 How Elasticsearch Replication Works Redo Logging DBMS Logs Generated PLOG Redo Reader Elasticsearch Applier (REST API) Master Replicator: Extractor Download transactions via network Slave Replicator: Applier THL THL = Events + Metadata THL

16 Sample Entry { "_id" : "99999", "_type" : "mg", "found" : true, "_version" : 2, "_index" : "msg", "_source" : { "msg" : "Hello ElasticSearch", "id" : "99999" } }

17 Replicating into Cassandra Replicating into Cassandra

18 Demo

19 Cassandra Great for fast online and CRM style deployments Highly fault tolerant and scalable Has some data and formatting changes Currently needs our DDL translation tool (soon built-in) Quasi table/doccument style

20 How Cassandra Replication Works base base Master Replicator Slave Replicator JS Cassandra Ruby Connector merge staging CSV 46

21 Demo

22 Future Direction for these appliers and related technology Full transaction support for Kafka Support for Amazon Elasticsearch Kafka Extraction Parsing contents of Kafka message queues Database updates Large scale distribution of database changes Filtering and re-submission

23 General Tungsten Replicator Functionality Expanding the standard filter technology Data translation (dates, numbers, hex) Basic lookup/combination to aid ETL style deployments Data munging/obfuscation (PII, credit cards) for analytics More appliers InfluxDB SQL Server PostgreSQL Hadoop JDBC MemSQL Amazon (Aurora, Elasticsearch) CouchDB/Base THL Compression/Encryption

24 Next Steps If you are interested in knowing more about Tungsten Replicator and would like to try it out for yourself, please contact our sales team who will be able to take you through the details and setup a POC sales@continuent.com Read the documentation at Subscribe to our Tungsten University YouTube channel! 14

25 For more information, contact us: MC Brown VP Products