Complex event flows in distributed systems

Transcription

1 Complex event flows in distributed With thoughts

2 3 common hypotheses I check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines are painful

3 Bernd Ruecker Co-founder and Developer Advocate of Camunda Berlin,

4

5 Simplified example: dash button Photo by 0xF2, available under Creative Commons BY-ND 2.0 license.

6 Three steps

7 Who is involved? Some bounded contexts Checkout Inventory Shipment

8 (Micro-)services Checkout Inventory Shipment

9 Autonomous (micro-)services Checkout Dedicated Application Processes Dedicated infrastructure Inventory Shipment Dedicated Development Teams

10 Events decrease coupling

11 Example Checkout The button blinks if we can ship within 24 hours Inventory Shipment

12 Request/response: temporal coupling Checkout The button blinks if we can ship within 24 hours Request Response Inventory Shipment

13 Temporal decoupling with events and read models Read Model Checkout The button blinks if we can ship within 24 hours Good Fetched Inventory Good Stored Shipment *Events are facts about what happened (in the past)

14 Events can decrease coupling* *e.g. decentral data-management, read models, extract cross-cutting aspects

15 Peer-to-peer event chains Order placed Checkout received Inventory Goods fetched Shipment Goods shipped

16 Peer-to-peer event chains Order placed Checkout received Inventory Goods fetched Shipment Goods shipped

17 The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

18 The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

19 The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

20 Peer-to-peer event chains Order placed Fetch the goods before the payment Checkout received Inventory Goods fetched Shipment Goods shipped

21 Peer-to-peer event chains Order placed Fetch the goods before the payment Checkout received Inventory Goods fetched Shipment Goods shipped Customers can pay via invoice

22 Peer-to-peer event chains Order placed Fetch the goods before the payment Checkout received Inventory Goods fetched Shipment Goods shipped

23 Peer-to-peer event chains Order placed Fetch the goods before the payment Checkout received Inventory Goods fetched Shipment Goods shipped

24 Photo by born1945, available under Creative Commons BY 2.0 license.

25 Extract the end-to-end responsibility Order placed Checkout Retrieve payment Order received Inventory Shipment *Commands have an intent about what needs to happen in the future

26 Commands help to avoid (complex) peer-to-peer event chains

27 Orchestration needs to be avoided

28 Smart ESB-like middleware Order Checkout Order placed received Inventory Shipment Good fetched Good shipped

29 Dumb pipes Checkout Order Martin Fowler Inventory Shipment Smart endpoints and dumb pipes

30 Danger of god services? Checkout Order Sam Newmann Inventory Shipment A few smart god services tell anemic CRUD services what to do

31 Danger of god services? Checkout Order Sam Newmann Inventory Shipment A few smart god services tell anemic CRUD services what to do

32 A god service is only created by bad API design!

33 Example Order Retrieve

34 Example Order Retrieve Credit Card

35 Example Order Retrieve Credit Card Rejected

36 Example Order Retrieve Rejected Rejected Credit Card If the credit card was rejected, the customer can provide new details Client of dumb endpoints easily become a god services.

37 Who is responsible to deal with problems? Order Retrieve Credit Card Rejected received failed If the credit card was rejected, the customer can provide new details

38 Long-running execution Order Retrieve Credit Card Smart endpoints are potentially long-running Rejected received failed If the credit card was rejected, the customer can provide new details Clients of smart endpoints remains lean.

39 Handling State Persist thing (Entity, Document, Actor, ) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Scheduling, Versioning, operating, visibility, scalability,

40 Workflow engines are painful Complex, proprietary, heavyweight, central, developer adverse,

41 Avoid the wrong tools! Low-code is great! (You can get rid of your developers!) Death by properties panel Complex, proprietary, heavyweight, central, developer adverse,

42 Workflow engines, state machines It is relevant in modern architectures

43 Workflow engines, state machines CADENCE Silicon valley has recognized

44

45 Workflow engines, state machines CADENCE

46 Workflow engines, state machines CADENCE also at scale

47 Zeebe.io

48 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); } engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //.startevent().servicetask("step1").camundaclass(sysoutdelegate.class).servicetask("step2").camundaclass(sysoutdelegate.class) What do I mean by.endevent().done() ).deploy(); leightweight? engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); }

49 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //.startevent().servicetask("step1").camundaclass(sysoutdelegate.class).servicetask("step2").camundaclass(sysoutdelegate.class).endevent().done() ).deploy(); Build engine in one line of code (using inmemory H2) engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); } }

50 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow").startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent().done() ).deploy(); Define flow e.g. in Java DSL engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); } }

51 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow").startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent().done() ).deploy(); Define flow e.g. in Java DSL engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); } }

52 BPMN Business Process Model and Notation ISO Standard

53 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow").startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent().done() ).deploy(); We can attach code engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); } }

54 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow").startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent().done() ).deploy(); engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); } } that is called when workflow instances pass through

55 public static void main(string[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine(); engine.getrepositoryservice().createdeployment() //.addmodelinstance("flow.bpmn", Bpmn.createExecutableProcess("flow").startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent().done() ).deploy(); engine.getruntimeservice().startprocessinstancebykey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(delegateexecution execution) throws Exception { System.out.println("Hello " + execution.getvariable("city")); } } Start instances

56 Now you have a state machine!

57 Easy to handle time

58 @martinschimak

59 Synchronous communication Order REST Credit Card

60 Synchronous communication Order REST Credit Card

61 Synchronous communication Order REST Credit Card

62 Synchronous communication Order REST Credit Card Stateful Retry

63 Distributed systems

64 It is impossible to differentiate certain failure scenarios. Client Service Provider Independant of communication style!

65 Distributed systems introduce complexity you have to tackle! REST Credit Card

66 Distributed systems introduce complexity you have to tackle! REST Credit Card

67 Distributed systems

68 Distributed transactions using compensation * Compensation

69 Relaxed consistency Temporarily inconsistent state But eventually consistent No Isolation (as in ACID)

70

71 Use cases for workflow automation Business Distributed Transactions Business processes automation IT Communication in distributed systems Orchestration <= milliseconds seconds minutes, weeks, always short running short running, but potentially long running long running

72 Biz Dev Ops improve communication improve communication

73 Biz Dev Ops improve communication improve communication Leverage state machine & workflow engine Living documentation Visibility in testing

74 Visual HTML reports for test cases

75 Biz Dev Ops improve communication improve communication Understand and discuss business processes Evaluate optimizations in-sync with implementation Leverage state machine & workflow engine Living documentation Visibility in testing

76 Living documentation for long-running behaviour

77 Biz Dev Ops improve communication improve communication Understand and discuss business processes Evaluate optimizations in-sync with implementation Leverage state machine & workflow engine Living documentation Visibility in testing

78 Biz Dev Ops improve communication improve communication Understand and discuss business processes Evaluate optimizations in-sync with implementation Leverage state machine & workflow engine Living documentation Visibility in testing Operate with visibility and context

79 Proper Operations Visibility + Context

80 Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on. Jimmy Floyd, 24 Hour Fitnesse

81 Workflows live inside service boundaries

82 Manigfold architecture options

83 Manigfold architecture options

84 Manigfold architecture options

85 Manigfold architecture options

86 Manigfold architecture options

87 Lightweight workflow engines are great* don t DIY *e.g. enabling potentially long-running services, solving hard developer problems, can run decentralized

88 Sales-Order & Order-Fulfillment via Camunda for every order worldwide (Q2 2017: 22,2 Mio)

89 Code example & live demo Checkout Order Inventory Shipping Monitor Human Tasks H2 H2

90 # Events decrease coupling: sometimes read-models, but no complex peer-to-peer event chains! # Orchestration needs to be avoided: sometimes no ESB, smart endpoints/dumb pipes, important capabilities need a home # Workflow engines are painful: some of them lightweight engines are easy to use and can run decentralized, they solve hard developer problems, don t DIY

91 Thank you!

92 Contact: Slides: Blog: application-development/ 3-common-pitfalls-of-microservicesintegrationand-how-to-avoid-them.html With thoughts