Microservices with Kafka Ecosystem. Guido Schmutz

Transcription

1 Microservices with Kafka Ecosystem Guido doag2017

2 Guido Schmutz Working at Trivadis for more than 20 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: Slideshare: Twitter: gschmutz 2

3 Our company. Trivadis is a market leader in IT consulting, system integration, solution engineering and the provision of IT services focusing on and technologies in Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields: O P E R A T I O N Trivadis Services takes over the interacting operation of your IT systems. 3

4 With over 600 specialists and IT experts in your region. COPENHAGEN HAMBURG 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants DÜSSELDORF Research and development budget: CHF 5.0 million FRANKFURT Financially self-supporting and sustainably profitable BASEL FREIBURG STUTTGART BRUGG ZURICH MUNICH VIENNA Experience from more than 1,900 projects per year at over 800 customers GENEVA BERN LAUSANNE 4

5 Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. Why Kafka for Event-Driven Microservices? 5. Asynchronous Microservices in Enterprise Architecture 6. References 5

6 Where do we come from? 6

7 Traditional Approach Customer Fat Client App Data Storage Customer UI GUI Customer BO Data Shop Backend Application Shop Rich UI Search Facade Order DAO Shared Database Shop UI GUI UI Logic Customer DAO sync request/response Order Facade Business Product DAO Data 7

8 SOA Approach Shop Web App Business Activity Service Business Entity Service Data Storage SOAP SOAP Shop UI GUI UI Logic Custer BAS Customer BES SOAP Customer Database Shop UI App SOAP Search BAS Payment BES Order DAO SOAP Shop UI GUI UI Logic SOAP Order BAS Product Customer BES DAO SOAP Order and Product DB Service Product DAO Order BES Data 8

9 Virtualized SOA Approach Service Virtualization Layer Shop Web App Business Activity Service Business Entity Service Data Storage Shop UI GUI UI Logic SOAP SOAP Service Bus Custer BAS Customer BES SOAP Customer Database Shop UI App SOAP Payment BES Order DAO SOAP Shop UI GUI UI Logic Search BAS SOAP Product Customer BES DAO SOAP Order and Product DB 9 Order BAS Order BES

10 What are Microservices? 10

11 Key Principles of Microservices Tightly Scoped Tightly Encapsulated behind concrete interfaces Responsible for managing their own data Highly cohesive Highly decoupled Independently deployable, self-contained and autonomous 11

12 Microservice Approach Customer Microservice REST Customer API Customer Logic Customer Shop Web App Order Microservice REST Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST Product API Product Logic Product Stock Microservice REST Stock API Stock Logic Stock 12

13 Microservice Approach with API Gateway Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice REST Shop UI GUI UI Logic Order API Order Logic Order Product Microservice REST Product API Product Logic Product Stock Microservice REST Stock API Stock Logic Stock 13

14 Synchronous World of Request-Response leads to tight, point-to-point couplings Service 1 Service 2 Service 3 Service 4 API Logic API Logic API Logic API Logic State State State State problem in lower end of chain have a ripple effect on the other service crash of service overloaded service / slow response time change of interface Service 5 API Logic Service 6 API Logic Service 7 API Logic State State State 14

15 Why not Event-Driven? 15

16 3 mechanisms through which services interact Request-Driven Event Driven Command Order IPad order(ipad) Service Event Consume IPad OrderedEvent Event Broker Query Retrieve my Orders) getallorders(myself) Logic State Event Publish OrderValidatedEvent 16

17 3 mechanisms through which services interact Commands represents an action request for some operation to be performed in another service Something that will change the state of the system Commands expect a response. Events represent both a Fact and a Trigger Something that has happened, expressed as a notification Queries represent a request to look something up Importantly, queries are side effect free; they leave the state of the system unchanged 17

18 Request-Driven Communication Highest Coupling Shop UI Application Order Microservice UI Logic Call PlaceOrderAPI API Logic State Call UpdateStockAPI Stock Microservice Maybe Call Reorder API API Logic State 18

19 Decoupling through Events Lowest Coupling 1) Raise OrderRequested Event Order Microservice 2) Listen to OrderRequested Event Event Broker 4) Listen to OrderConfirmed Event Stock Microservice API Logic State 3) Raise OrderConfirmed Event 5) Maybe Raise Reorder Event API Logic State sync request/response async, event pub/sub async request/response 19

20 Event-Driven (Async) Microservice Approach Customer Microservice REST Customer API Customer Logic Customer Shop Web App API Gateway Order Microservice Shop UI GUI UI Logic REST Order API Order Logic Order Event Broker Product Microservice REST Product API Product Logic Product sync request/response async, event pub/sub async request/response Stock Microservice REST Stock API Stock Logic Stock 20

21 Process & Virtualized SOA Approach Sync & Async Service Virtualization Layer Shop Web App Orchestration Business Activity Service Business Entity Service Data Storage Shop UI UI Logic GUI Service Bus Custer BAS Customer BES Customer Database Shop UI App Payment BES Order DAO Shop UI GUI UI Logic Search BAS Product Customer BES DAO Order and Product DB 21 Order BAS Order BES

22 Command Query Responsibility Segregation (CQRS) Optimize different nonfunctional requirements for read and write behavior App Service Data Storage interface to the service is split between commands that trigger changes in state queries that provide read access to the state of resources UI Command Service Write Data Store UI Logic used to support services with higher performance and capacity requirements for reading data than for writing data Projection Service number of instances supporting queries can be scaled up independently of instance supporting commands Query Service Read Data Store (Materialized Views) 22

23 Event Sourcing persists the state of a business entity as a sequence of state-changing events App Service Data Storage Whenever state of business entity changes, a new event is appended to the list of events UI Event Service Since saving an event is a single operation, it is inherently atomic UI Logic Replayer The application reconstructs an entity s current state by replaying the events Other App Event Store Publisher 23

24 Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern App Service Data Storage materializing views from the stored events UI Command Service Event Store Optionally Commands can be stored in event store and transformed into events by the command handler UI Logic Command Handler Projection Service Query Service Read Data Store (Materialized Views) 24

25 Using Event Sourcing with Microservices Event sourcing enables building a forward-compatible application architecture the ability to add more applications in the future that need to process the same event but create a different materialized view. Microservice Event Subscribe REST Command Handler Event Store Neha Narkhede, Confluent Blog API Event Handler(s) Projection Handler(s) Query Logic State 25

26 How many Event Stores do we need? OR Microservice Event Store Microservice REST REST API Logic State Event API Logic State Microservice Event Store Microservice Event Store REST REST API Logic State Event API Logic State Event Microservice Event Store Microservice REST REST API Logic State Event API Logic State 26

27 Have one source of truth Avoid double write! Would need distributed transactions Write Event first then consume it from same micro service eat your own dog food Microservice Microservice REST API Logic State REST API Publisher Consumer State Event Store Event Store Event Event 27

28 Publish Event through Database Customer Fat Client App Data Store Integration Customer UI Customer BO CDC Customer CDC Connector Event Store Microservice Event Logic State 28

29 Why Kafka for Event-Driven Microservices? 29

30 Apache Kafka - Unix Analogy KSQL Kafka Connect API Kafka Streams API Kafka Connect API $ cat < in.txt grep "kafka" tr a-z A-Z > out.txt Kafka Core (Cluster) Adapted from: Confluent 30

31 Kafka High Level Architecture The who is who Producers write data to brokers. Consumers read data from brokers. All this is distributed. The data Data is stored in topics. Topics are split into partitions, which are replicated. Zookeeper Ensemble Producer Producer Producer Kafka Cluster Broker 1 Broker 2 Broker 3 Consumer Consumer Consumer 31

32 Leverage the Log at the heart of Apache Kafka sits a distributed log collection of messages, appended sequentially to a file log-structured character makes Kafka well suited to performing the role of an Event Store in Event Sourcing service seeks to the position of the last message it read, then scans sequentially, reading messages in order Reads are a single seek & scan Writes are append only Event Hub

33 Scale-Out Architecture Kafka Cluster Broker 1 Consumer Group 1 Consumer 1 topic consists of many partitions Producer 1 Consumer 2 producer load load-balanced over all partitions Producer 2 Broker 2 Consumer 3 consumer can consume with as many threads as there are partitions Producer 3 Broker 3 Consumer Group 2 Consumer 4 33

34 Strong Ordering Guarantees most business systems need strong ordering guarantees messages that require relative ordering need to be sent to the same partition supply same key for all messages that require a relative order Producer 1 Key-1 Key-2 Key-3 Key-4 Key-5 Broker 1 Broker 2 Key-1 Key-3 Consumer 1 Consumer 2 Consumer 3 To maintain global ordering use a single partition topic Key-6 Broker 3 34

35 Durable and Highly Available Messaging Broker 1 Broker 1 Producer 1 Broker 2 Broker 2 Consumer 1 Consumer 1 Producer 1 Consumer 2 Consumer 2 Broker 3 Broker 3 35

36 Durable and Highly Available Messaging (II) Broker 1 Broker 1 Producer 1 Broker 2 Broker 2 Consumer 1 Consumer 1 Producer 1 Broker 3 Consumer 2 Broker 3 Consumer 2 36

37 Replay-ability Logs never forget by keeping events in a log, we have a version control system for our data if you were to deploy a faulty program, the system might become corrupted, but it would always be recoverable sequence of events provides an audit point, so that you can examine exactly what happened rewind and reply events, once service is back and bug is fixed Rewind Service Event Hub Replay Logic State

38 Hold Data for Long-Term Data Retention 1. Never Broker 1 2. Time based (TTL) log.retention.{ms minutes hours} 3. Size based log.retention.bytes Producer 1 Broker 2 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 \ --create --topic customers \ --replication-factor 1 \ --partitions 1 \ --config cleanup.policy=compact Broker 3 38

39 Keep Topics in Compacted Form K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value Compaction Offset Key Value K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 K1 K2 K3 K4 K5 K6 V4 V10 V5 V7 V9 V11 V3 V6 V8 V1 V2 39

40 Topic Viewed as Event Stream or State Stream (Change Log) Event Stream State Stream (Change Log Stream) T20:18: T20:18: T20:18: T20:19: T20:19: T20:19:23 11,Normal,41.87, ,Normal,40.38, ,Normal,42.23, ,Normal,41.71, ,Normal,38.65, ,Normal41.71, T20:18:46,11,Normal,41.87, T20:18:55,11,Normal,40.38, T20:18:59, 21,Normal,42.23, T20:19:01,21,Normal,41.71, T20:19:02,11,Normal,38.65, T20:19:23,21,Normal41.71,

41 Keep Topics both in Original and Compacted Form OR Producer TX Kafka Broker A B C B A E K A E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K B E A C K Consume and Produce 41

42 Change Data Capture (CDC) Legacy Microservice RDBMS cdc-source trucking_ driver Driver Topic elasticsearch -sink NoSQL 42

43 Enrich Stream with Static Data with Kafka Streams Movement Topic Consume and Produce Join Movement & Driver Topic Driver Topic Driver Handler Driver Table Driver Local State 43

44 Building Embedded, Materialized View with Kafka Streams Movement & Driver Topic Join Consume and Produce Window Aggregation Engine Metric Topic Join Store Result Store Result State Join State 44

45 Building Embedded, Queryable Materialized View with Kafka Streams Movement & Driver Topic Join Consume and Produce Window Aggregation API Application Engine Metric Topic Join Store Result Store Result State Join State 45

46 Asynchronous Microservices in Enterprise Architecture 46

47 Asynchronous Microservices in Enterprise Architecture Billing & Ordering CRM / Profile Marketing Campaigns File Import / SQL Import Raw Refined Hadoop Clusterd Hadoop Cluster Big Data Cluster Storage Storage Results Parallel Processing SQL Enterprise Data Warehouse BI Tools Location Mobile Apps Event Hub Event Stream Stream Analytics Cluster Search Search / Explore Social Click stream Weather Data Call Center Stream Processor State Microservice Cluster API Service Online & Mobile Apps 47 Sensor Data Event Stream Microservice State API

48 References 48

49 References Microservices Blog Series, Ben Stopford, Confluent: Apache Kafka for Microservices: A Confluent Online Talk Series: Turning the database inside-out with Apache Samza, Martin Kleppmann, Con Event sourcing, CQRS, stream processing and Apache Kafka: What s the connection?, Neha Narkhede, Confluent: Immutability Changes Everything, Pat Helland, Salesforce: Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: 49

50 Technology on its own won't help you. You need to know how to use it properly. 50

51 DOAG 2017 #opencompany Booth: 3rd Floor next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit 51