Through O Shaped Glasses

Transcription

1 Through O Shaped Glasses Introducing Kafka to the Oracle DBA Mike Donovan CTO Dbvisit Software

2 Mike Donovan Chief Technology Officer, Dbvisit Software Multi-platform DBA, (Oracle, MSSQL..) Conference speaker: OOW, RMOUG, dbtech Showcase, Collaborate NZOUG member Technical Writer and Editor Kafka enthusiast Say that I am an oracle ACE Professional not-knower of things 2

3 Why am I interested in Kafka?

4 BEFORE: Many Ad Hoc Pipelines

5 Stream Data Platform with Kafka Distributed Fault Tolerant Stream Processing Data Integration Message Store

6 Agenda What is Kafka Looking at this new technology as an Oracle DBA Why should an Oracle professional care? How do I get started with Kafka

7 What s all the fuss about?

8 The New World of data Data centralization Real time delivery Integration Stream data processing New data end points/stores

9 What is Kafka? An open-source publish-subscribe messaging implemented as a distributed commit log A scalable, fault tolerant, distributed system where messages are kept in topics that are partitioned and replicated across multiple nodes. Developed at LinkedIn ~2010 Confluent and the OS project

10 What is Kafka? Data is written to Kafka in the form of key-value pair messages (can have null) Each message belongs to a topic Messages as a continuous flow (stream) of events Producers (writers) decoupled from Consumers (readers) A delivery channel/platform (if you like) crossing systems (data Integration) TOPICS (Kafka) (~)= TABLES (ORACLE)

11 Kafka - components data Schema Registry Zookeeper REST Proxy Kafka What about KSQL and Kafka Streams? Kafka Connect

12 Kafka basic operations demo 1. Download the Confluent platform 2. Run the CLI (scripts alternative) CLI = SQL Plus? (or svrmgr) 3. Push data into Kafka topic (bundled Producer) 4. Read some data out of a Kafka topic (bundled Consumer)

13 Kafka why would you use it? 3 propositions: Messaging system Data streaming platform Data storage Messaging Website Activity Tracking Metrics Log Aggregation Stream Processing Event Sourcing Commit Log

14 Apples and Oranges? Kafka and Oracle Messaging system - transmission channel - integration priority Data streaming (always on) platform in line transformations - push Data storage (topics) X Data delivery end point (periodic/batch) - materialised views? - logical replication? - pull? Data store (source of truth tables)

15

16 Oracle Database tables: State active record model Oracle Database Source Persistent Store - retains current known STATE. ID Name Salary 1 Chris Jim Bob 500 select * from employees where ID = 2;

17 Event Streaming Source ID Name Salary 1 Chris Jim 350 Insert into stage_emp values (2, 300, 350, Machine_2, QA, U, 2016-May-12 14:22:03) Target 3 Bob 500 Update emp set salary = 350 Where id = 2 ID Name Old Salary New Salary Machine ID User TRANS_ TYPE 2 Jim Machine_2 QA U Commit Timestamp 2016-May-12 14:22:03 INSERT ALL ROWS mode

18 What s missing with state alone? Train No Start Location End Location Passengers Engineer Status TRANS_ TYPE Commit Timestamp 1 London Cardiff 100 Smythe Good I 2016-Nov-12 14:22:00 Map of Europe Show SQL Statement 1 Cardiff Cardiff 0 Smythe Good U 1 Cardiff Edinburgh 312 Johnson Good U 2016-Nov-12 17:24: Nov-12 18:00:09 1 Edinburgh Edinburgh 0 Rest U 2016-Nov-13 04:02:33

19 Table/Stream Duality Changes Time

20 Table/Stream Duality Log Compaction KSQL demo

21

22 The Online Redo Log Files The redo log stores a continuous chain of chronological order of every change vector applied to the database. This will be the bare minimum of information required to reconstruct, or redo, all the work that has been done. If a datafile (or the whole database) is damaged or destroyed, these change vectors can be applied to datafile backups to redo the work, bringing them forward in time until the moment that the damage occurred. P89 OCA exam guide

23 The Redo Log! CHANGE #3 TYP:0 CLS:1 AFN:4 DBA:0x b OBJ:27521 SCN:0x ab0ab9 SEQ:2 OP:11.2 ENC:0 RBL:0 KTB Redo op: 0x01 ver: 0x01 compat bit: 4 (post-11) padding: 1 op: F xid: 0x c uba: 0x00c31f4a.043f.15 KDO Op code: IRP row dependencies Disabled xtype: XA flags: 0x bdba: 0x b hdba: 0x a itli: 1 ispac: 0 maxfr: 4858 tabn: 0 slot: 0(0x0) size/delt: 24 fb: --H-FL-- lb: 0x1 cc: 6 null: col 0: [ 2] c1 07 col 1: [ 5] col 2: [ 2] c1 16 col 3: [ 2] c1 02 col 4: [ 2] col 5: [ 2] c1 07 insert into HR.EMPLOYEES values (6,'Perry',21,1,'IT,6);

24 An old methodology: Event Sourcing Event Sourcing ensures that all changes to application state are stored as a sequence of events... The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself. Martin Fowler: Don't save the current state of objects Instead write the events that lead to the current state An APPEND-ONLY log

25 An old methodology: Event Sourcing Martin Kleppmann - Designing Data Intensive Applications EVENT Details Meta-data

26 Event Sourcing Benefits Fowler suggests: Complete Rebuild - rehydrate secondary systems Temporal Queries Event Replay - forward and reverse

27 Event Streaming Capture all changes in the database and record these as events Every change becomes an insert, even a delete and update become an insert Adds additional information (metadata) about these changes such as who, where, what, when Turning the database inside out (turn the redo log into a normal log) See the full lifecycle of the data, now possible in real time!

28 The Online Redo Log Files I created a topic (in Confluent 4.0) called connect-dbmessage Boils down to a file on disk here (where is this determined?) /tmp/confluent.sz1gda5f/kafka/data/connect-dbmessage-0 We can run a strings command on it: AND we can also dump it using some Kafka tools (need to show this)...

29 Oracle Change Data delivered to Kafka metadata INSERT... into SCOTT.TEST9

30 Kafka - a log writer/reader Partition 0 Partition 1 Partition 2 Old Organized by topics Sub-categorization by partitions (log files on disk) Replicated between nodes for redundancy New

31 Indexes, Offsets and Data files Kafka - a log writer/reader [oracle@dbvrep01 REP-TX.META-0]$ ll total rw-r--r-- 1 oracle oinstall Jun 15 18: index -rw-r--r-- 1 oracle oinstall Jun 15 18: log -rw-r--r-- 1 oracle oinstall Jun 15 18: timeindex DUMP LOG SEGMENTS COMMAND: kafka-run-class kafka.tools.dumplogsegments --print-data-log --files /tmp/kafka-logs/rep-tx.meta-0/ log

32 Topic vs Table Creation Create a topic bin/kafka-topics create zookeeper localhost: topic TOPIC_NAME --replication-factor 1 --partitions 1

33 Kafka Connect - export/import tool Datapump anyone? Cassandra Elasticsearch Google BigQuery Hbase HDFS JDBC Kudu MongoDB Postgres S3 SAP HANA Solr Vertica

34 SMTs and KStreams Create a topic bin/kafka-topics create zookeeper localhost: topic TOPIC_NAME --replication-factor 1 --partitions 1

35 Get started with Kafka Kafka and Kafka Connect Download the Confluent Platform (bundled connectors) Check out the available community connectors Try running it in Docker

36 About Dbvisit Software Real-time Oracle Database Streaming software solutions In the Cloud Hybrid On-Premise New Zealand-based, US office, Asia Sales office, EU office (Prague) Unique offering: disaster recovery solutions for Oracle Standard Edition Logical replication for moving data where ever and whenever you wish Flexible licensing, cost effective pricing models available Exceptional growth, customers Peerless customer support

37 Thank