O'Reilly SA NYC 2018: Complex event flows in distributed systems
2 likes997 views
Slides from my talk at O'Reilly Software Architecture Conference in New York City on 28th of February 2018. Code from the demo is available: http://github.com/flowing/flowing-retail
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„leightweight?“](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-58-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-59-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-60-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-61-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-63-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-64-320.jpg)
![public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances](https://image.slidesharecdn.com/complexeventflowsindistributedsystems-180228222020/85/O-Reilly-SA-NYC-2018-Complex-event-flows-in-distributed-systems-65-320.jpg)
More Related Content
What's hot (20)
Similar to O'Reilly SA NYC 2018: Complex event flows in distributed systems (20)
More from Bernd Ruecker (20)
Recently uploaded (20)
O'Reilly SA NYC 2018: Complex event flows in distributed systems
- 1. Complex event flows in distributed systems @berndruecker With thoughts from http://flowing.io @berndruecker | @martinschimak
- 2. 3 common hypotheses I check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines are painful
- 3. Berlin, Germany [email protected] @berndruecker Bernd Ruecker Co-founder and Developer Advocate of Camunda
- 5. Simplified example: dash button Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/
- 6. Three steps…
- 7. Who is involved? Some bounded contexts… Checkout Payment Inventory Shipment
- 9. Autonomous (micro-)services Checkout Payment Inventory Shipment Dedicated Application Processes Dedicated infrastructure Dedicated Development Teams
- 11. Example Checkout Payment Inventory Shipment The button blinks if we can ship within 24 hours
- 12. Request/response: temporal coupling Checkout Payment Inventory Shipment Request Response The button blinks if we can ship within 24 hours
- 13. Temporal decoupling with events and read models Checkout Payment Inventory Shipment Good Stored Read Model Good Fetched The button blinks if we can ship within 24 hours *Events are facts about what happened (in the past)
- 14. Events can decrease coupling* *e.g. decentral data-management, read models, extract cross-cutting aspects
- 17. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
- 18. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
- 19. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
- 20. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment
- 21. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped
- 22. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Customers can pay via invoice Goods fetched Order placed Payment received Goods shipped …
- 23. Photo by born1945, available under Creative Commons BY 2.0 license.
- 24. Order Extract the end-to-end responsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Order placed Payment received Retrieve payment
- 25. Commands help to avoid (complex) peer-to-peer event chains
- 26. Orchestration needs to be avoided
- 28. Dumb pipes Checkout Payment Inventory Shipment Order Smart endpoints and dumb pipes Martin Fowler
- 29. Danger of god services? Checkout Order A few smart god services tell anemic CRUD services what to do Sam Newmann Payment Inventory Shipment
- 30. Danger of god services? Checkout Payment Inventory Shipment Order A few smart god services tell anemic CRUD services what to do Sam Newmann
- 31. A god service is only created by bad API design!
- 35. Example Order Payment Client of dumb endpoints easily become a god services. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Rejected
- 36. Payment failed Who is responsible to deal with problems? Order Payment If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received
- 37. Payment failed Who is responsible to deal with problems? Order Payment Clients of smart endpoints remains lean. If the credit card was rejected, the customer can provide new details Credit Card Retrieve Payment Rejected Payment received Smart endpoints are potentially long-running
- 38. Photo by Tookapic, available under Creative Commons CC0 1.0 license.
- 39. „There was an error while sending your boarding pass“
- 43. Check-in Barcode Generator Web-UI Me Output Mgmt Current situation – the bad part
- 44. Check-in Barcode Generator Web-UI Me Output Mgmt Current situation – the bad part Stateful Retry
- 46. We are having some technical difficulties and cannot present you your boarding pass right away. But we do actively retry ourselves, so lean back, relax and we will send it on time.
- 48. Check-in Barcode Generator Web-UI Me Output Mgmt Possible situation – much better! Stateful Retry
- 49. Check-in Barcode Generator Web-UI Me Output Mgmt Stateful Retry Possible situation – much better! The failure never leaves this scope!
- 50. Persist thing (Entity, Actor, …) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Scheduling, Versioning, operating, visibility, scalability, … Handling State
- 51. Workflow engines are painful Complex, proprietary, heavyweight, central, developer adverse, …
- 52. Avoid the wrong tools! Death by properties panel Low-code is great! (You can get rid of your developers!) Complex, proprietary, heavyweight, central, developer adverse, …
- 53. Workflow engines, state machines It is relevant in modern architectures
- 54. CADENCE Silicon valley has recognized Workflow engines, state machines
- 56. CADENCE also at scale Workflow engines, state machines
- 57. CADENCE for todays demo Workflow engines, state machines
- 58. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } What do I mean by „leightweight?“
- 59. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Build engine in one line of code (using in- memory H2)
- 60. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL
- 61. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Define flow e.g. in Java DSL
- 62. BPMN Business Process Model and Notation ISO Standard
- 63. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } We can attach code…
- 64. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } …that is called when workflow instances pass through
- 65. public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } } Start instances
- 66. Payment Now you have a state machine!
- 67. Payment Easy to handle time
- 71. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client
- 72. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
- 73. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
- 75. Distributed transaction with compensations*
- 76. Living documentation for long-running behaviour
- 77. Visual HTML reports for test cases
- 79. Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on. Jimmy Floyd, 24 Hour Fitnesse „
- 80. Workflows live inside service boundaries
- 81. Explicit flows help separate domain and infrastructure Infrastructure Aggregates, Domain Events, Domain Services, etc … + the flow Workflow Engine Payment Application Domain
- 87. Lightweight workflow engines are great – don‘t DIY* *e.g. enabling potentially long-running services, solving hard developer problems, can run decentralized
- 89. Zalando Sales-Order and Order-Fulfillment via Camunda for every order worldwide Orders Q2-2017: 22,2 Mio. Sales 2016: 3,6 Mrd. EUR Growth 2016: 23%
- 90. Code example & live demo InventoryPaymentOrder ShippingCheckout Monitor https://github.com/flowing/flowing-retail/ Human Tasks H2 H2
- 91. # Events decrease coupling: sometimes read-models, but no complex peer-to-peer event chains! # Orchestration needs to be avoided: sometimes no ESB, smart endpoints/dumb pipes, important capabilities need a home # Workflow engines are painful: some of them lightweight engines are easy to use and can run decentralized, they solve hard developer problems, don‘t DIY
- 92. Thank you! Meet me at Meet the experts Now!
- 93. [email protected] @berndruecker https://bernd-ruecker.com https://blog.bernd-ruecker.com https://github.com/flowing https://www.infoq.com/articles/ events-workflow-automation With thoughts from http://flowing.io @berndruecker | @martinschimak Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html