The resurgence of event driven architecture
1 like568 views
The document discusses the resurgence of event-driven architecture (EDA) and its significance in building independent, decoupled microservices that leverage asynchronous communication for rapid innovation. It outlines essential patterns and principles of EDA, such as event sourcing and command query responsibility segregation (CQRS), emphasizing scalable, resilient system designs. Furthermore, it highlights the evolving landscape of application and integration development and the importance of effective API and event management in agile integration contexts.
The resurgence of event driven architecture
- 1. The Resurgence of Event Driven Architecture Kim Clark Integration Architect IBM Cloud / 28th May 2020 / © 2020 IBM Corporation Webinar: http://ibm.biz/eda-resurgence Slides: http://ibm.biz/eda-resurgence-slides
- 2. Evolution to agile integration: Summary view Webinar http://ibm.biz/agile-integration-webinar Webpage http://ibm.biz/agile-integration IBM Redbook http://ibm.biz/agile-integration-redbook API Management API ManagementIntegration Centralized ESB Socialized APIs fine-grained deployment Decentralized ownership & events API & Event Management
- 3. Creating truly independent digital applications requires asynchronous communication as well as APIs 3IBM Cloud / Agileintegration/ Month 10, 2019 / © 2019 IBM Corporation Microservice Application SoR SoR SoRSoR API consumption EventStream consumption µService µServiceµService µService µService µService µServiceµService µService Agility Innovate rapidly without affecting other components APIs Are simplest to use, but create a real-time dependency on the underlying system of record Truly independent, decoupled microservice components enable To provide those benefits they need to be independent of the systems of record Scalability Scale only what you need, and only when you need to Resilience Fail fast, return fast, without affecting other components Event streams Enable microservices to build decoupled views of the data and respond to real time events API and Event management
- 4. What does it mean to have truly a cloud-native/microservice component? Fine-grained components • Self-contained well decoupled components of an “appropriate” size (e.g. microservices architecture) Strong decoupling • Clear ownership boundaries, formalized API and event interfaces, component grouping. Independent persistence. Minimal state • Components are clones, and/or can re-assign/re- balance work. No caller affinity, no two phase commits Immutable deployment • Image based deployment and updates, • No live configuration. • Rollback by rolling forward to previous image. Infrastructure as code • Declarative infrastructure configuration (GitOps) at the level of individual components. CI/CD • Pipeline automation (build, test, deploy, verify) for creation and maintenance. Agile methods • Short, fixed length iteration cycles. Intrinsic business collaboration/feedback DevOps • Close collaboration between development and operations with rapid feedback resolution Lightweight runtimes • Fast start up/shut down. File- system based install, deployment, and configuration. Elastic, agnostic platform • Platform agnostically providing deployment, scaling, HA, security, monitoring... Observability • Platform neutral logging, tracing to streams enabling retrospective inferred analysis of internal state. IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 5. Comparing messaging and events Stream History Scalable Subscription Fine grained messaging Assured Deliveryü Messaging Focused on message exchange and transactions Events (e.g. Kafka) Focused on streaming of events Supported connectivity Topics and Subscriptions Critical data exchange Event driven Event streaming Strengths IBM Cloud / © 2020 IBM Corporation
- 6. Event driven architecture patterns that build on one another Event stream data distribution: Event distribution with history Event stream projections: Consumer specific data views Event sourcing: Using an event log as a data master CQRS: Command Query Responsibility Segregation Event processing: Evaluating events over time Saga: Combining multiple actions together Supporting/related patterns: change data capture, event connectors (source/sink), function as a service… https://ibm-cloud-architecture.github.io/refarch-edaIBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 7. Event Log Event streams for data distribution External website application Partner website application Mobile application Subscribe and consume Event stream Back end datastore … 4 5 6 7 … Subscribe and consume Subscribe and consume Publish (change events) IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 8. Event Log (Topic A) Event processing Event Event stream Event Propagator … 42 43 44 45 46 Publish … 4 5 6 7 8 … 172 173 174 175 Event Log (Topic B) … 4 5 6 7 Event processor … 44 456 7 8 46 Event pattern matched Subscribe and consume Publish 4 5 Event processing could involve: • Filtering • Redaction • Routing • Event pattern matching • Real-time analytics • Machine learning model* (training and/or using) * Note that event processing can be re-run over the event stream history in order to gain new insights as the learning improves IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 9. Event Log Event stream projections Local read-optimized datastore External website application Partner website application Mobile application Local datastore Local datastore Local datastore Subscribe and consume Direct invocation Event stream Back end datastore … 4 5 6 7 … Subscribe and consume Subscribe and consume Publish (change events) IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 10. Event sourcing Local “event sourced” read-optimized datastore Read implementation A Read implementation B Local datastore Local datastore Subscribe and consume Direct invocation Event streamSubscribe and consume “Event Sourced” implies the use of an event log as the primary datastore Application/Context boundary Event Log … 4 5 6 7 … Command implementation COMMAND QUERY Exposed API , and CQRS IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 11. Event sourcing/CQRS within a bounded context vs (inter) domain events CQRS and Event Sourcing patterns are typically scoped to within a bounded context. From outside the context, the event sourced nature of the implementation should be hidden. Bounded Context Event Log COMMAND QUERY Bounded Context Only selected events (and for that matter APIs) should be exposed across bounded contexts. Their data models should be decoupled from the internal model. Internal events (inter) domain events API & Event Management IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 12. CQRS at different scopes These two diagrams represent the same logical pattern, but have radically different implementations Logical component boundary Event Log Event Processing Command Processing API Management API In an enterprise scope, the same pattern often exists but is broken up across multiple different sub-systems, integration specialists would recognize these as enterprise integration patterns. Is this still CQRS? Search & Read Event Log Read Datastore Event Consumptio n Command Processing APIM API Datastore Events Command Store APIM API Command Forwarding API In an application scope, application developers would consider this pattern part of their application design (and perhaps call it CQRS). APIM Search & Read Read Datastore Consumer specific projection (event-fed read- optimized view) Consumer specific store-forward pattern to improve availability and performance of updates Existing back end system exposed as managed APIs. Fire-forget updates Enabled to emit data change events Internal events Cross boundary events Cross boundary API Event Processing IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 13. asynchronous(messaging,events,…) trad. on premisescloud platform cloud platformcloud platform private cloudSaaS platform iPaaS/iSaaS SaaS platform cloud platform cloud platform The asynchronous backplane (messaging, events) The asynchronous backplane provides reliable message/event storage and a distribution network that can traverse application and cloud boundaries robustly. 13
- 14. Example of complementary use of messaging and events Event Log Event Log Replication Cloud On-Premise Data Warehouse / Lake App App App App App SaaS SaaS SaaS App App App App Messaging Store Messaging Device Device Device App Messaging Store Messaging Device Device Device App Store Messaging Device Device Device App Messaging Event Log Integration Engine Messaging Provider Event Log API Management App Application App App App App Microservice application Messaging network
- 15. Saga Pattern: Orchestration vs Choreography Orchestrated interactions (orchestration) Pros First class notion of “process” Modelled process visible in implementation Monitoring/reporting at process level Process version migration options Clearer modelling of exception paths Cons Higher CPU/network due to persistence Solution split between layers Chained interactions (choreography) Pros Higher performance due to minimal persistence Can be implemented entirely in the integration layer Cons No notion of “process”, only the individual steps Process “model” not visible in implementation Custom work for effective monitoring/reporting Exception paths harder to understand Complex to migrate to new processes Queued events holding stateOrchestration state persistence https://microservices.io/patterns/data/saga.html IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 16. Microservice component Microservice component Microservice component Microservice component Microservice component Microservice component Events within the application boundary MongoDB Redis SaaS SaaS Blockchain replica ML model enterprise applications Internet Of Things SaaS Application Boundary Event broker IBM Cloud / 28th May 2020 / © 2020 IBM Corporation API Management
- 17. Microservice component Microservice component Microservice component Microservice component Microservice component Microservice component Events within and beyond the application boundary MongoDB Redis SaaS SaaS Blockchain replica ML model enterprise applications Internet Of Things SaaS Application Boundary API Management Event broker Event Management? Event broker IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 18. The maturing space of event stream management Easily found, quickly consumed • Described • Catalogued and discoverable • Policy controlled access Completely De-coupled • Self-service consumption • No technology ties between provider and consumers • No operational ties Consumed reliably • Consumption options suitable for many application types • Replay history, durable subs, horizontal scale Delivered in time to be useful • Latency measured in seconds not minutes • Delivered rather than polled IBM Integration / © 2020 IBM Corporation 18 Event Distribution Event Stream Management
- 19. On premises A target high level architecture for enterprise integration in 2020 Systems of Record Systems of Engagement Mobile Partners API Economy IoT SaaS SaaS SaaS SaaS IaaS PaaS PaaS PaaS SaaS Cloud platform • SaaS = Software as a service • PaaS = Platform as a service • IaaS = Infrastructure as a service Polyglot (e.g. microservice application) • Language runtimes • Specialist runtimes (e.g. integration) Exposure points • APIs • Events Key features • Technology layering has been removed. • Grouping is based on business domains/functions • Applications are self-contained, including their integration needs. • Events have become more a first class citizen rather than just a transport • Well defined application boundaries, breached only by governed APIs and Events • Cloud platform type (IaaS/PaaS etc.) is hidden when looking from the outside of an application. • Architecture balances isolation (decoupling), and interaction (defined interfaces). Caveats • Represents an idealized target integration architecture. Most organizations do not have this level of clarity in the separations of their applications. • Describes how the things within the enterprise communicate with one another. It does not show the enterprise in the context of its broader ecosystem (e.g. with partners). IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
- 20. IBM Cloud / © 2020 IBM Corporation This webinar Resurgence of event driven architecture ibm.biz/eda-resurgence http://ibm.biz/eda-resurgence-slides Learn more Agile integration http://ibm.biz/agile-integration-webinar ibm.com/cloud/integration/agile-integration Convergence of application and integration development ibm.biz/agile-integration-convergence ibm.biz/agile-integration-convergence-pdf
- 21. Thank you! IBM Cloud / 28th May 2020 / © 2020 IBM Corporation