Microservices with Kafka Ecosystem
3 likes1,273 views
The document discusses the principles and architecture of microservices, emphasizing the importance of event-driven communication and the use of Kafka as an event broker. It outlines key concepts such as service autonomy, asynchronous communication, and patterns like CQRS and event sourcing. The content also highlights the benefits of Kafka for building scalable and reliable microservices ecosystems.
Microservices with Kafka Ecosystem
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Microservices with Kafka Ecosystem Guido Schmutz Frankfurt, 12.12.2017 @gschmutz guidoschmutz.wordpress.com
- 2. Guido Schmutz Working at Trivadis for more than 21 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: [email protected] Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Microservices with Kafka Ecosystem2
- 3. COPENHAGEN MUNICH LAUSANNE BERN ZURICH BRUGG GENEVA HAMBURG DÜSSELDORF FRANKFURT STUTTGART FREIBURG BASEL VIENNA With over 600 specialists and IT experts in your region. 14 Trivadis branches and more than 600 employees 200 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 million Financially self-supporting and sustainably profitable Experience from more than 1,900 projects per year at over 800 customers Microservices with Kafka Ecosystem4
- 4. Agenda 1. Where do we come from? 2. What are Microservices? 3. Why not Event Driven? 4. Why Kafka for Event-Driven Microservices? 5. Asynchronous Microservices in Enterprise Architecture 6. Summary 7. References Microservices with Kafka Ecosystem5
- 5. Microservices with Kafka Ecosystem6 Where do we come from?
- 6. Shop Rich UI Shop Backend Application Traditional Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database Microservices with Kafka Ecosystem7 sync request/response
- 7. Shop UI App Business Activity Service SOA Approach Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic Data GUI Service Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP Microservices with Kafka Ecosystem8
- 8. Shop UI App Business Activity Service Virtualized SOA Approach Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP 9
- 9. Microservices with Kafka Ecosystem10 What are Microservices?
- 10. Key Principles of Microservices Tightly Scoped Tightly Encapsulated behind concrete interfaces Responsible for managing their own data Highly cohesive Highly decoupled Independently deployable, self-contained and autonomous Microservices with Kafka Ecosystem11
- 11. Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Microservices with Kafka Ecosystem12
- 12. Microservice Approach with API Gateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Microservices with Kafka Ecosystem13
- 13. Synchronous World of Request-Response leads to tight, point-to-point couplings Microservices with Kafka Ecosystem14 problem in lower end of chain have a ripple effect on the other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State
- 14. Microservices with Kafka Ecosystem15 Why not Event-Driven?
- 15. 3 mechanisms through which services interact Request-Driven Event Driven Service Logic State Event Consume “IPad OrderedEvent” Command ”Order IPad” order(iPad) : boolean Event Publish “OrderValidatedEvent” Query ”Retrieve my Orders) getAllOrders(myself) Event Broker Microservices with Kafka Ecosystem16
- 16. Request-Driven Communication – Highest Coupling Shop UI Application UI Logic Order Microservice Logic State { } API Stock Microservice State { } API Logic Call PlaceOrderAPI Call UpdateStockAPI Maybe Call Reorder API Microservices with Kafka Ecosystem18
- 17. Decoupling through Events – Lowest Coupling Event Broker Microservices with Kafka Ecosystem19 Order Microservice Logic State { } API Stock Microservice State { } API Logic3) Raise OrderConfirmed Event 4) Listen to OrderConfirmed Event 5) Maybe Raise Reorder Event 2) Listen to OrderRequested Event 1) Raise OrderRequested Event sync request/response async request/response async, event pub/sub
- 18. Event-Driven (Async) Microservice Approach Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Microservices with Kafka Ecosystem20 Event Broker sync request/response async request/response async, event pub/sub
- 19. Shop UI App Business Activity Service Process & Virtualized SOA Approach – Sync & Async Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus Orchestration Microservices with Kafka Ecosystem21
- 20. Command Query Responsibility Segregation (CQRS) Optimize different nonfunctional requirements for read and write behavior interface to the service is split between • commands that trigger changes in state • queries that provide read access to the state of resources support services with higher performance and capacity requirements for reading data than for writing data scare reading capacity independently of writing Data Storage Write Data Store Read Data Store (Materialized Views) Service Command Service Query Service App UI Projection Service UI Logic Microservices with Kafka Ecosystem22
- 21. Event Sourcing persists the state of a business entity as a sequence of state-changing events Whenever state of business entity changes, a new event is appended to the list of events Saving an event is a single operation and is inherently atomic The application reconstructs an entity’s current state by replaying the events Data Storage Event Store Service Event Service Publisher App UI UI Logic Replayer Other App Microservices with Kafka Ecosystem23
- 22. Event Sourcing & CQRS Event sourcing is commonly combined with the CQRS pattern materializing views from the stored events Optionally Commands can be stored in event store and transformed into events by the command handler Data Storage Event Store Service Command Service App UI UI Logic Query Service Read Data Store (Materialized Views) Projection Service Command Handler Microservices with Kafka Ecosystem24
- 23. Using Event Sourcing with Microservices Microservices with Kafka Ecosystem25 “Event sourcing enables building a forward-compatible application architecture — the ability to add more applications in the future that need to process the same event but create a different materialized view.” Neha Narkhede, Confluent Blog Microservice State Command Handler { } API REST Event Store Projection Handler(s) Query Logic Event Handler(s) Event Subscribe
- 24. How many Event Stores do we need ? Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST Microservice { } API StateLogic REST Event Store Event Microservice { } API StateLogic REST OR Microservices with Kafka Ecosystem26
- 25. Have only one „source of truth“ Avoid double write! • Would need distributed transactions Write Event first then consume it from same micro service • “eat your own dog food” Microservice { } API StateLogic REST Event Store Event Microservice { } API StateConsumer REST Event Store Event Publisher Microservices with Kafka Ecosystem27
- 26. Data Store Publish Events through Database Customer Event Store Event Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Microservices with Kafka Ecosystem28
- 27. Why Kafka for Event-Driven Microservices? Microservices with Kafka Ecosystem29
- 28. What is apache Apache Kafka? The who is who • Producers write data to brokers. • Consumers read data from brokers. • All this is distributed. The data • Data is stored in topics. • Topics are split into partitions, which are replicated. Kafka Cluster Consumer Consumer Consumer Producer Producer Producer Broker 1 Broker 2 Broker 3 Zookeeper Ensemble Microservices with Kafka Ecosystem31
- 29. Apache Kafka – scalable message processing and more! Microservices with Kafka Ecosystem32 Source Connector trucking_ driver Kafka Broker Sink Connector Stream Processing
- 30. Leverage the Log at the heart of Apache Kafka sits a distributed log collection of messages, appended sequentially to a file service ‘seeks’ to the position of the last message it read, then scans sequentially, reading messages in order log-structured character makes Kafka well suited to performing the role of an Event Store in Event Sourcing Event Hub 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 Reads are a single seek & scan Writes are append only Microservices with Kafka Ecosystem33
- 31. Scale-Out Architecture Microservices with Kafka Ecosystem34 topic consists of many partitions producer load load-balanced over all partitions consumer can consume with as many threads as there are partitions Producer 1 Consumer 1 Broker 1 Producer 2 Producer 3 Broker 2 Broker 3 Consumer 2 Consumer 3 Consumer 4 Consumer Group 1 Consumer Group 2 Kafka Cluster
- 32. Strong Ordering Guarantees most business systems need strong ordering guarantees messages that require relative ordering need to be sent to the same partition supply same key for all messages that require a relative order To maintain global ordering use a single partition topic Producer 1 Consumer 1 Broker 1 Broker 2 Broker 3 Consumer 2 Consumer 3 Key-1 Key-2 Key-3 Key-4 Key-5 Key-6 Key-3 Key-1 Microservices with Kafka Ecosystem35
- 33. Replay-ability – Logs never forget by keeping events in a log, we have a version control system for our data if you were to deploy a faulty program, the system might become corrupted, but it would always be recoverable sequence of events provides an audit point, so that you can examine exactly what happened rewind and reply events, once service is back and bug is fixed Event Hub 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 Replay Rewind Service Logic State Microservices with Kafka Ecosystem38
- 34. Hold Data for Long-Term – Data Retention Producer 1 Broker 1 Broker 2 Broker 3 1. Never 2. Time based (TTL) log.retention.{ms | minutes | hours} 3. Size based log.retention.bytes 4. Log compaction based (entries with same key are removed): kafka-topics.sh --zookeeper zk:2181 --create --topic customers --replication-factor 1 --partitions 1 --config cleanup.policy=compact Microservices with Kafka Ecosystem39
- 35. Keep Topics in Compacted Form 0 1 2 3 4 5 6 7 8 9 10 11 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 Offset Key Value 3 4 6 8 9 10 K1 K3 K4 K5 K2 K6 V4 V5 V7 V9 V10 V11 Offset Key Value Compaction Microservices with Kafka Ecosystem40 V1 V2 V3 V4 V5 V6 V7 V8 V9V10 V11 K1 K3 K4 K5K2 K6
- 36. Topic Viewed as Event Stream or State Stream (Change Log) Event Stream State Stream (Change Log Stream) 2017-10-02T20:18:46 11,Normal,41.87,-87.67 2017-10-02T20:18:55 11,Normal,40.38,-89.17 2017-10-02T20:18:59 21,Normal,42.23,-91.78 2017-10-02T20:19:01 21,Normal,41.71,-91.32 2017-10-02T20:19:02 11,Normal,38.65,-90.2 2017-10-02T20:19:23 21,Normal41.71,-91.32 11 2017-10-02T20:18:46,11,Normal,41.87,-87.67 11 2017-10-02T20:18:55,11,Normal,40.38,-89.17 21 2017-10-02T20:18:59, 21,Normal,42.23,-91.78 21 2017-10-02T20:19:01,21,Normal,41.71,-91.32 11 2017-10-02T20:19:02,11,Normal,38.65,-90.2 21 2017-10-02T20:19:23,21,Normal41.71,-91.32 Microservices with Kafka Ecosystem41
- 37. Keep Topics both in Original and Compacted Form Microservices with Kafka Ecosystem42 Kafka Broker A B C B A E K A E A C K B E A C K Producer Kafka Broker A B C B A E K A E A C K B E A C K Producer Consume and Produce OR TX
- 38. Legacy Microservice Change Data Capture (CDC) Microservices with Kafka Ecosystem43 RDBMS cdc-source trucking_ driver Driver Topic elasticsearch -sink NoSQL
- 39. Enrich Stream with Static Data with Kafka Streams Microservices with Kafka Ecosystem44 Movement Topic Consume and Produce Driver Table Driver Topic Driver Handler Join Driver Local State Movement & Driver Topic
- 40. Building Embedded, Materialized View with Kafka Streams Microservices with Kafka Ecosystem45 Movement & Driver Topic Consume and Produce Join Store Engine Metric Topic Join Join State Window Aggregation Result Store Result State
- 41. Building Embedded, Queryable Materialized View with Kafka Streams Microservices with Kafka Ecosystem46 Movement & Driver Topic Consume and Produce Join Store Engine Metric Topic Join Join State Window Aggregation Result Store Result State API Application
- 42. Asynchronous Microservices in Enterprise Architecture Microservices with Kafka Ecosystem47
- 43. Hadoop Clusterd Hadoop Cluster Big Data Cluster Asynchronous Microservices in Enterprise Architecture Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps SQL Search BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data Event Hub Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Analytics Cluster Stream Processor State { } API Event Stream Event Stream Service 48
- 44. Summary Microservices with Kafka Ecosystem49
- 45. Summary Microservices with Kafka Ecosystem50 • service autonomy is key in a Microservices Architecture! • not all communication need to be synchronous => separate into • commands • events • queries • Kafka is well suited as an event broker / event store • brings many more interesting features beyond just “message passing”
- 46. References Microservices with Kafka Ecosystem51
- 47. References Microservices with Kafka Ecosystem52 Microservices Blog Series, Ben Stopford, Confluent: • https://www.confluent.io/blog/tag/microservices Apache Kafka for Microservices: A Confluent Online Talk Series: • https://www.confluent.io/landing-page/microservices-online-talk-series/ Turning the database inside-out with Apache Samza, Martin Kleppmann, Con • https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Immutability Changes Everything, Pat Helland, Salesforce: • http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: • https://www.youtube.com/watch?v=B1-gS0oEtYc
- 48. Technology on its own won't help you. You need to know how to use it properly. Microservices with Kafka Ecosystem53