Microservices with Kafka Ecosystem. Guido Schmutz

Microservices with Kafka Ecosystem Guido Schmutz @gschmutz doag2017

Guido Schmutz Working at Trivadis for more than 20 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz 2

Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems. 3

With over 600 specialists and IT experts in your region. COPENHAGEN HAMBURG 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants DÜSSELDORF Research and development budget: CHF 5.0 million FRANKFURT Financially self-supporting and sustainably profitable BASEL FREIBURG STUTTGART BRUGG ZURICH MUNICH VIENNA Experience from more than 1,900 projects per year at over 800 customers GENEVA BERN LAUSANNE 4

Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. Why Kafka for Event-Driven Microservices? 5. Asynchronous Microservices in Enterprise Architecture 6. References 5

Where do we come from? 6

Traditional Approach Customer Fat Client App Data Storage Customer UI GUI Customer BO Data Shop Backend Application Shop Rich UI Search Facade Order DAO Shared Database Shop UI GUI UI Logic Customer DAO sync request/response Order Facade Business Product DAO Data 7

SOA Approach Shop Web App Business Activity Service Business Entity Service Data Storage SOAP SOAP Shop UI GUI UI Logic Custer BAS Customer BES SOAP Customer Database Shop UI App SOAP Search BAS Payment BES Order DAO SOAP Shop UI GUI UI Logic SOAP Order BAS Product Customer BES DAO SOAP Order and Product DB Service Product DAO Order BES Data 8

Virtualized SOA Approach Service Virtualization Layer Shop Web App Business Activity Service Business Entity Service Data Storage Shop UI GUI UI Logic SOAP SOAP Service Bus Custer BAS Customer BES SOAP Customer Database Shop UI App SOAP Payment BES Order DAO SOAP Shop UI GUI UI Logic Search BAS SOAP Product Customer BES DAO SOAP Order and Product DB 9 Order BAS Order BES

What are Microservices? 10

Key Principles of Microservices Tightly Scoped Tightly Encapsulated behind concrete interfaces Responsible for managing their own data Highly cohesive Highly decoupled Independently deployable, self-contained and autonomous 11

Microservice Approach Customer Microservice REST Customer API Customer Logic Customer Shop Web App Order Microservice REST Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST Product API Product Logic Product Stock Microservice REST Stock API Stock Logic Stock 12

Microservice Approach with API Gateway Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice REST Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST Product API Product Logic Product Stock Microservice REST Stock API Stock Logic Stock 13

Synchronous World of Request-Response leads to tight, point-to-point couplings Service 1 Service 2 Service 3 Service 4 API Logic API Logic API Logic API Logic State State State State problem in lower end of chain have a ripple effect on the other service crash of service overloaded service / slow response time change of interface Service 5 API Logic Service 6 API Logic Service 7 API Logic State State State 14

Why not Event-Driven? 15

3 mechanisms through which services interact Request-Driven Event Driven Command Order IPad order(ipad) Service Event Consume IPad OrderedEvent Event Broker Query Retrieve my Orders) getallorders(myself) Logic State Event Publish OrderValidatedEvent 16

3 mechanisms through which services interact Commands represents an action request for some operation to be performed in another service Something that will change the state of the system Commands expect a response. Events represent both a Fact and a Trigger Something that has happened, expressed as a notification Queries represent a request to look something up Importantly, queries are side effect free; they leave the state of the system unchanged 17

Request-Driven Communication Highest Coupling Shop UI Application Order Microservice UI Logic Call PlaceOrderAPI API Logic State Call UpdateStockAPI Stock Microservice Maybe Call Reorder API API Logic State 18

Decoupling through Events Lowest Coupling 1) Raise OrderRequested Event Order Microservice 2) Listen to OrderRequested Event Event Broker 4) Listen to OrderConfirmed Event Stock Microservice API Logic State 3) Raise OrderConfirmed Event 5) Maybe Raise Reorder Event API Logic State sync request/response async, event pub/sub async request/response 19

Event-Driven (Async) Microservice Approach Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice Shop UI GUI UI Logic REST Order API Order Logic Order Event Broker Product Microservice REST Product API Product Logic Product sync request/response async, event pub/sub async request/response Stock Microservice REST Stock API Stock Logic Stock 20

Process & Virtualized SOA Approach Sync & Async Service Virtualization Layer Shop Web App Orchestration Business Activity Service Business Entity Service Data Storage Shop UI UI Logic GUI Service Bus Custer BAS Customer BES Customer Database Shop UI App Payment BES Order DAO Shop UI GUI UI Logic Search BAS Product Customer BES DAO Order and Product DB 21 Order BAS Order BES

Command Query Responsibility Segregation (CQRS) Optimize different nonfunctional requirements for read and write behavior App Service Data Storage interface to the service is split between commands that trigger changes in state queries that provide read access to the state of resources UI Command Service Write Data Store UI Logic used to support services with higher performance and capacity requirements for reading data than for writing data Projection Service number of instances supporting queries can be scaled up independently of instance supporting commands Query Service Read Data Store (Materialized Views) 22

Event Sourcing persists the state of a business entity as a sequence of state-changing events App Service Data Storage Whenever state of business entity changes, a new event is appended to the list of events UI Event Service Since saving an event is a single operation, it is inherently atomic UI Logic Replayer The application reconstructs an entity s current state by replaying the events Other App Event Store Publisher 23

Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern App Service Data Storage materializing views from the stored events UI Command Service Event Store Optionally Commands can be stored in event store and transformed into events by the command handler UI Logic Command Handler Projection Service Query Service Read Data Store (Materialized Views) 24

Using Event Sourcing with Microservices Event sourcing enables building a forward-compatible application architecture the ability to add more applications in the future that need to process the same event but create a different materialized view. Microservice Event Subscribe REST Command Handler Event Store Neha Narkhede, Confluent Blog API Event Handler(s) Projection Handler(s) Query Logic State 25

How many Event Stores do we need? OR Microservice Event Store Microservice REST REST API Logic State Event API Logic State Microservice Event Store Microservice Event Store REST REST API Logic State Event API Logic State Event Microservice Event Store Microservice REST REST API Logic State Event API Logic State 26

Have one source of truth Avoid double write! Would need distributed transactions Write Event first then consume it from same micro service eat your own dog food Microservice Microservice REST API Logic State REST API Publisher Consumer State Event Store Event Store Event Event 27

Publish Event through Database Customer Fat Client App Data Store Integration Customer UI Customer BO CDC Customer CDC Connector Event Store Microservice Event Logic State 28

Why Kafka for Event-Driven Microservices? 29

Apache Kafka - Unix Analogy KSQL Kafka Connect API Kafka Streams API Kafka Connect API $ cat < in.txt grep "kafka" tr a-z A-Z > out.txt Kafka Core (Cluster) Adapted from: Confluent 30

Kafka High Level Architecture The who is who Producers write data to brokers. Consumers read data from brokers. All this is distributed. The data Data is stored in topics. Topics are split into partitions, which are replicated. Zookeeper Ensemble Producer Producer Producer Kafka Cluster Broker 1 Broker 2 Broker 3 Consumer Consumer Consumer 31

Leverage the Log at the heart of Apache Kafka sits a distributed log collection of messages, appended sequentially to a file log-structured character makes Kafka well suited to performing the role of an Event Store in Event Sourcing service seeks to the position of the last message it read, then scans sequentially, reading messages in order Reads are a single seek & scan Writes are append only Event Hub 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 32

Scale-Out Architecture Kafka Cluster Broker 1 Consumer Group 1 Consumer 1 topic consists of many partitions Producer 1 Consumer 2 producer load load-balanced over all partitions Producer 2 Broker 2 Consumer 3 consumer can consume with as many threads as there are partitions Producer 3 Broker 3 Consumer Group 2 Consumer 4 33

Strong Ordering Guarantees most business systems need strong ordering guarantees messages that require relative ordering need to be sent to the same partition supply same key for all messages that require a relative order Producer 1 Key-1 Key-2 Key-3 Key-4 Key-5 Broker 1 Broker 2 Key-1 Key-3 Consumer 1 Consumer 2 Consumer 3 To maintain global ordering use a single partition topic Key-6 Broker 3 34

Durable and Highly Available Messaging Broker 1 Broker 1 Producer 1 Broker 2 Broker 2 Consumer 1 Consumer 1 Producer 1 Consumer 2 Consumer 2 Broker 3 Broker 3 35

Durable and Highly Available Messaging (II) Broker 1 Broker 1 Producer 1 Broker 2 Broker 2 Consumer 1 Consumer 1 Producer 1 Broker 3 Consumer 2 Broker 3 Consumer 2 36

Replay-ability Logs never forget by keeping events in a log, we have a version control system for our data if you were to deploy a faulty program, the system might become corrupted, but it would always be recoverable sequence of events provides an audit point, so that you can examine exactly what happened rewind and reply events, once service is back and bug is fixed Rewind Service Event Hub Replay Logic State 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 37

Hold Data for Long-Term Data Retention 1. Never Broker 1 2. Time based (TTL) log.retention.{ms minutes hours} 3. Size based log.retention.bytes Producer 1 Broker 2 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 \ --create --topic customers \ --replication-factor 1 \ --partitions 1 \ --config cleanup.policy=compact Broker 3 38

Keep Topics in Compacted Form 0 1 2 3 4 5 6 7 8 9 10 11 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value Compaction Offset Key Value 3 4 6 8 9 10 K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 K1 K2 K3 K4 K5 K6 V4 V10 V5 V7 V9 V11 V3 V6 V8 V1 V2 39

Topic Viewed as Event Stream or State Stream (Change Log) Event Stream State Stream (Change Log Stream) 2017-10-02T20:18:46 2017-10-02T20:18:55 2017-10-02T20:18:59 2017-10-02T20:19:01 2017-10-02T20:19:02 2017-10-02T20:19:23 11,Normal,41.87,-87.67 11,Normal,40.38,-89.17 21,Normal,42.23,-91.78 21,Normal,41.71,-91.32 11,Normal,38.65,-90.2 21,Normal41.71,-91.32 11 2017-10-02T20:18:46,11,Normal,41.87,-87.67 11 2017-10-02T20:18:55,11,Normal,40.38,-89.17 21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78 21 2017-10-02T20:19:01,21,Normal,41.71,-91.32 11 2017-10-02T20:19:02,11,Normal,38.65,-90.2 21 2017-10-02T20:19:23,21,Normal41.71,-91.32 40

Keep Topics both in Original and Compacted Form OR Producer TX Kafka Broker A B C B A E K A E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K B E A C K Consume and Produce 41

Change Data Capture (CDC) Legacy Microservice RDBMS cdc-source trucking_ driver Driver Topic elasticsearch -sink NoSQL 42

Enrich Stream with Static Data with Kafka Streams Movement Topic Consume and Produce Join Movement & Driver Topic Driver Topic Driver Handler Driver Table Driver Local State 43

Building Embedded, Materialized View with Kafka Streams Movement & Driver Topic Join Consume and Produce Window Aggregation Engine Metric Topic Join Store Result Store Result State Join State 44

Building Embedded, Queryable Materialized View with Kafka Streams Movement & Driver Topic Join Consume and Produce Window Aggregation API Application Engine Metric Topic Join Store Result Store Result State Join State 45

Asynchronous Microservices in Enterprise Architecture 46

Asynchronous Microservices in Enterprise Architecture Billing & Ordering CRM / Profile Marketing Campaigns File Import / SQL Import Raw Refined Hadoop Clusterd Hadoop Cluster Big Data Cluster Storage Storage Results Parallel Processing SQL Enterprise Data Warehouse BI Tools Location Mobile Apps Event Hub Event Stream Stream Analytics Cluster Search Search / Explore Social Click stream Weather Data Call Center Stream Processor State Microservice Cluster API Service Online & Mobile Apps 47 Sensor Data Event Stream Microservice State API

References 48

References Microservices Blog Series, Ben Stopford, Confluent: https://www.confluent.io/blog/tag/microservices Apache Kafka for Microservices: A Confluent Online Talk Series: https://www.confluent.io/landing-page/microservices-online-talk-series/ Turning the database inside-out with Apache Samza, Martin Kleppmann, Con https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ Event sourcing, CQRS, stream processing and Apache Kafka: What s the connection?, Neha Narkhede, Confluent: https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Immutability Changes Everything, Pat Helland, Salesforce: http://cidrdb.org/cidr2015/papers/cidr15_paper16.pdf Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: https://www.youtube.com/watch?v=b1-gs0oetyc 49

Technology on its own won't help you. You need to know how to use it properly. 50

Trivadis @ DOAG 2017 #opencompany Booth: 3rd Floor next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit 51