Managing Data in Microservices
Download as PPTX, PDF15 likes18,908 views
The document discusses the prerequisites and techniques for managing data in microservices, emphasizing the need for domain and organizational maturity. It outlines various approaches for microservice persistence, shared data management, event handling, and transaction workflows, highlighting the shift from monolithic databases to distributed systems. The author advocates for clear ownership of data within microservices, effective decoupling from shared databases, and using events as fundamental building blocks in service interfaces.
Managing Data in Microservices
- 1. Managing Data in Microservices Randy Shoup VP Engineering, Stitch Fix @randyshoup linkedin.com/in/randyshoup
- 2. First Law of Distributed Object Design: Don’t distribute your objects! -- Martin Fowler
- 3. Prerequisites for Microservices • Domain Maturity: Know enough about your domain to decompose it cleanly • Organizational Maturity: Small teams with well-defined areas of responsibility • Process Maturity: Automated Testing and Continuous Delivery • Operational Maturity: Monitoring, alerting, oncall
- 4. Microservices • Single-purpose • Simple, well-defined interface • Modular and independent A C D E B
- 5. Microservices are nothing more than SOA done properly. -- me
- 6. Microservices • Single-purpose • Simple, well-defined interface • Modular and independent • Isolated persistence (!) A C D E B
- 7. Microservice Persistence • Approach 1: Operate your own data store o Store to your own instance(s) of {Postgres, MySQL, etc.}, owned and operated by the service team • Approach 2: Use a persistence service o Store to your own table(s) in {Dynamo, RDS, Spanner, etc.}, operated as a service by another team or by a third-party provider o Isolated from all other users of the service • Only external access to data store is through published service interface
- 8. Extracting Microservices • Problem: Monolithic shared DB • Clients • Shipments • Items • Styles, SKUs • Warehouses • etc. stitchfix.com Styling app Warehouse app Merch app CS app Logistics app Payments service Profile service
- 9. Extracting Microservices • Decouple applications / services from shared DB • Clients • Shipments • Items • Styles, SKUs • Warehouses • etc. stitchfix.com Styling app Warehouse app Merch app CS app Logistics app Payments service Profile service
- 10. Extracting Microservices • Decouple applications / services from shared DB Styling app Warehouse app core_item core_sku core_client
- 11. Extracting Microservices • Step 1: Create a service Styling app Warehouse app core_item core_sku core_client client-service
- 12. Extracting Microservices • Step 2: Applications use the service Styling app Warehouse app core_item core_sku core_client client-service
- 13. Extracting Microservices • Step 3: Move data to private database Styling app Warehouse app core_item core_sku client-service core_client
- 14. Extracting Microservices • Step 4: Rinse and Repeat Styling app Warehouse app core_sku client-service core_client item-service core_item
- 15. Extracting Microservices • Step 4: Rinse and Repeat Styling app Warehouse app client-service core_client item-service core_item style-service core_sku
- 16. Extracting Microservices • Step 4: Rinse and Repeat Styling app Warehouse app client-service core_client item-service core_item style-service core_sku
- 17. Microservice Techniques: Shared Data • Problem o Monolithic database makes it easy to leverage shared data o Where does shared data go in a microservices world? • Principle: Single System of Record o Every piece of data is owned by a single service o That service represents its canonical system of record o Every other copy of that data is a read-only, non-authoritative cache
- 18. Microservice Techniques: Shared Data • Approach 1: Synchronous Lookup o Customer service owns customer data o Order service calls customer service in real time order-service customer-service
- 19. Microservice Techniques: Shared Data • Approach 2: Async event + local cache o Customer service owns customer data o Customer service sends customer-updated event when customer changes o Order service caches current customer information order-servicecustomer-service
- 20. Microservice Techniques: Shared Data • Approach 3: Shared metadata library o Read-only metadata, basically immutable o E.g., size schemas, colors, fabrics, US States, etc. receiving-serviceitem-service style-service
- 21. Events as First-Class Construct • “A significant change in state” o Statement that some interesting thing occurred o 0 | 1 | N consumers subscribe to the event, typically asynchronously • Fourth fundamental building block o Presentation interface / interaction o Application stateless business logic o Persistence database o State changes events • Events represent how the real world works o Finance, software development process, any “workflow”
- 22. Microservices and Events • Events are a first-class part of the interface • A service interface includes o Synchronous request-response (REST, gRPC, etc) o Events the service produces o Events the service consumes o Bulk reads and writes (ETL) • The interface includes any mechanism for getting data in or out of the service (!)
- 23. Microservice Techniques: Joins • Problem o Monolithic database makes joins very easy o Splitting the data into separate services makes joins very hard
- 24. Microservice Techniques: Joins • Approach 1: Service that “Materializes the View” o Listen to events from item-service, events from order-service o Maintain denormalized join of items and orders together in local storage • Best for high cardinality A and high cardinality B (M:N join) Items Orders item-feedback-service item-service order-service
- 25. Microservice Techniques: Joins • Many common systems do this o Most NoSQL approaches o “Materialized view” in database systems o Search engines o Analytic systems o Log aggregators
- 26. Microservice Techniques: Joins • Approach 2: Join in Application / Client o Get a single customer from customer-service o Query matching orders for that customer from order-service • Best for single A, multiple Bs (1:N join) A B order-history-page customer-service order-service
- 27. Microservice Techniques: Joins • Many common systems do this o Web application “mashup”
- 28. Microservice Techniques: Workflows and Sagas • Problem o Monolithic database makes transactions across multiple entities easy o Splitting data across services makes transactions very hard
- 29. Microservice Techniques: Workflows and Sagas • Transaction Saga o Model the transaction as a state machine of atomic events • Reimplement as a workflow • Roll back by applying compensating operations in reverse A B C D A B C D
- 30. Microservice Techniques: Workflows and Sagas • Many common systems do this o Payment processing flow o Most approval workflows
- 31. Microservice Techniques: Workflows and Sagas • Ideal use for Functions as a Service (“Serverless”) o Very lightweight logic o Stateless o Triggered by an event A B C D A B C D ƛ ƛ ƛ ƛ ƛ ƛ ƛ ƛ