![Sensitivity: Secret
An event archive is a collection of events that
have been published on an event bus.
Resources:
VoucherArchive:
Type: AWS::Events::Archive
Properties:
ArchiveName: OrdersArchive
Description: Orders to be resubmitted
EventPattern:
<YourEventFilter>
RetentionDays: 10
SourceArn: 'arn:aws:..:event-bus/loyalty'
{
"detail": {
"metadata": {
"domain": [
"LEGO-LOYALTY"
],
"service": [
“order-submission"
],
"category": [
"task-status"
],
"type": [
“order"
],
"status": [
"retry"
]
}
}
}](https://image.slidesharecdn.com/eventbridgedublin-230607215623-907beed5/85/Advanced-Event-Driven-Patterns-AWS-Community-Day-Dublin-64-320.jpg)
![Sensitivity: Secret
HandlingReplayedEvents
StartReplay - API
EventStartTime: timestamp
EventEndTime: timestamp
ReplayName: “OrdersReplay_20220330201522”
{
"detail":{
"metadata":{
"domain":"LEGO-LOYALTY",
"service":“order-submission",
"category":"task-status",
"type":“order",
"status":"retry"
},
"data":{
"request_id":"AbLhmQ",
"reference":"P6IQd",
"order":"M101"
}
},
"replay-name": “OrdersReplay_20220330201522”
}
Replayed Event
Replay API
{
"detail": {
"metadata": {
"domain": [
"LEGO-LOYALTY"
],
"service": [
“order-submission"
],
"category": [
"task-status"
],
"type": [
“order"
],
"status": [
"retry"
]
}
},
"replay-name": [{
“prefix": “OrdersReplay”
}],
}
Replay Event
Filter Pattern](https://image.slidesharecdn.com/eventbridgedublin-230607215623-907beed5/85/Advanced-Event-Driven-Patterns-AWS-Community-Day-Dublin-67-320.jpg)
![Sensitivity: Secret
"Dispatch Voucher": {
"Type": "Task",
"Resource": "arn:aws:states:::events:putEvents.waitForTaskToken",
"HeartbeatSeconds": 6000,
"Parameters": {
"Entries": [
{
"Detail": {
"metadata": {
"domain": "LEGO-LOYALTY",
"service": "service-loyalty-orders",
"category": “internal",
"type": “request",
"status": "processed"
},
"data": {
"loyalty_request_id.$": "$$.Task.Token",
"loyalty_reference.$": "$.loyalty_reference",
"merchant_reference.$": "$.merchant_reference",
"loyalty_order_reference.$": "$.loyalty_order_reference",
"vouchers": [
{
"voucher_code.$": "$.voucher_code"
}
]
}
},
"DetailType": "event",
"EventBusName": "the-custom-event-bus-arn",
"Source": "service-loyalty-order-process"](https://image.slidesharecdn.com/eventbridgedublin-230607215623-907beed5/85/Advanced-Event-Driven-Patterns-AWS-Community-Day-Dublin-81-320.jpg)
Advanced Event-Driven Patterns - AWS Community Day Dublin
- 1. Sensitivity: Secret AdvancedEvent-DrivenPatterns with AmazonEventBridge Sheen Brisals The LEGO Group sheenbrisals
- 2. Sensitivity: Secret Sheen Brisals Engineer. Architect. Writer. Speaker. Leader. AWS Serverless Hero @ The LEGO Group Ex Oracle, Hewlett-Packard, Omron, TATA sheenbrisals sbrisals.medium.com sheen-brisals
- 4. Sensitivity: Secret Amazon S3 CDN/WAF AWS WAF AWS Fargate Amazon ElastiCache Amazon API Gateway AWS Lambda AWS Lambda Amazon Kinesis Data Firehose AWS Lambda Amazon Event bridge Amazon SQS Amazon Dynamo DB Amazon SNS Amazon CloudWatch Thirdparty payments ecommerce etc AWS Step Functions LEGO.com high level architecture Serverless FE & BFF Amazon ECS on Fargate
- 5. Sensitivity: Secret Event-driven architecture The importance of events The need for Amazon EventBridge Event-driven patterns Resources Amazon EventBridge Custom event bus Default event bus Partner event bus Event producers (sources) Event consumers (targets) Partner apps Custom apps AWS services Event bus Filtering & routing rules
- 7. Sensitivity: Secret Event-Driven Architecture (EDA) is an architectural concept that uses events to communicate between decoupled microservices asynchronously.
- 10. Sensitivity: Secret Four main elements of EDA publisher subscriber event carrier event event
- 12. Sensitivity: Secret An event is something that has already happened.
- 13. Sensitivity: Secret { "metadata": { "version": "1.0", "domain": "ecommerce", "subdomain": "orders", "service": "service-payments", "type": “data" }, "data": { "customer_id": "730e-4dfb-9166", "order_number": "123-987-456", "payment_reference": "cc-visa-9076-cv3s5s", "payment_type": "creditcard", "amount": 35.99, "currency": "GBP", "paid_at": "2023-01-30T09:12:27Z" } } "data": { "customer_id": "730e-4dfb-9166", "order_number": "123-987-456", "payment_reference": "cc-visa-9076-cv3s5s", "payment_type": "creditcard", "amount": 35.99, "currency": "GBP", "paid_at": "2023-01-30T09:12:27Z" }
- 14. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events
- 15. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events { "metadata": { "version": "1.0", "domain": "ecommerce", "subdomain": "orders", "service": "service-payments", "type": “data" }, "data": { "customer_id": "730e-4dfb-9166", "order_number": "123-987-456", "payment_reference": "cc-visa-9076-cv3s5s", "payment_type": "creditcard", "amount": 35.99, "currency": "GBP", "paid_at": "2023-01-30T09:12:27Z" } }
- 16. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events { "metadata": { "version": "1.0", "domain": "holiday-travel", "subdomain": "flight-booking", "service": "service-status-checker" }, "data": { "system": "anytime-payment-provider", "current_status": "down", "previous_status": "up", "status_count": "2", "checked_at": "2023-03-20T02:15:00Z" } }
- 17. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events { "metadata": { "created_at": "2023-01-30T09:12:27Z", "trace_id": "skdj834sd3-j3ns-cmass23", "domain": "rewards", "service": "third-party-CRM", "status": "retry" }, "data": { "customer_id": "730e-4dfb-9166", "reward_code": "DXT876LSA536MBS", "order_numer": "123-987-456", "info": { "activity": "reward_update", "code": 500, "cause": "Internal Server Error" }
- 18. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events { "detail": { "SaleToPOIRequest": { "EventNotification": { "EventDetails": "newstate", "EventToNotify": "Shutdown", "TimeStamp": "2019-08-07..." }, "MessageHeader": { "SaleID": "POSSystemID12345", "ProtocolVersion": "3.0", "MessageType": "Notification", "POIID": "V400m-324688179", "MessageClass": "Event", "MessageCategory": "Event", "DeviceID": "1517998561" } } } } { “event": “$.detail.SaleToPOIRequest.EventToNotify“, “device": “$.detail.MessageHeader.DeviceID“ } { “eventName": <event>, “deviceId": <device> } original event transformation logic transformed event
- 19. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events }, "responseElements": { "x-amz-request-id": "EXAMPLE123456789", "x-amz-id-2": "EXAMPLE123/5678abcdefghi/mnopqrsEFGH" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "testConfigRule", "bucket": { "name": "festival-promotions", "ownerIdentity": { "principalId": "EXAMPLE" }, "arn": "arn:aws:s3:::festival-promotions" }, "object": { "key": "discount_codes.csv", "size": 1024, "eTag": "0123456789a6789abcdef",
- 20. Sensitivity: Secret Event Categories • Domain events • Operational events • Internal / local events • Transformed events • AWS events custom events
- 21. Sensitivity: Secret Event Categories Event Types • Domain events • Operational events • Internal / local events • Transformed events • AWS events • Command • Query • Request • Response • Data • Status • Task
- 23. Sensitivity: Secret • Event producer is agnostic of the consumers • Producer adds its origin identity Domain, service, function • Producer conforms to a contract & structure Think as API • Packs just-the-required data Least data privilege • Producer adds a unique event identifier Think traceability
- 25. • Consumer deals with event duplicates Remember idempotency? • Stores events before processing Storage-first pattern • Ordering of events not guaranteed Well, the world is asynchronous! • Don’t modify & relay events Respect authenticity
- 27. Sensitivity: Secret Prove yourself that serverless works! Convince your colleagues that serverless is fun! Satisfy your stakeholders that serverless is good!
- 28. Sensitivity: Secret Tangled event-drivenarchitecture Lambda pinball architecture Lambda pinball architectures characteristically lose sight of important domain logic in the tangled web of lambdas, buckets and queues as requests bounce around increasingly complex graphs of cloud services. Thoughtworks
- 29. Sensitivity: Secret Tangled event-drivenarchitecture Lambda pinball architecture Ball of Serverless Mud (BoSM)
- 33. Sensitivity: Secret Amazon EventBridge is a serverless event bus service that makes it easy to connect your applications with data from a variety of sources
- 34. Sensitivity: Secret Amazon EventBridge Custom event bus Default event bus Partner event bus Event producers (sources) Event consumers (targets) Partner apps Custom apps AWS services Event bus Filtering & routing rules
- 35. Sensitivity: Secret CORE Features COOL Features • Custom event bus • Event filtering • Event routing • Multiple targets • Event transformation • AWS services integration • Scheduler • Partner event source • Schema registry • Schema discovery • Event archive & replay • API destinations • Pipes • Sandbox
- 37. Sensitivity: Secret There are only patterns, patterns on top of patterns, patterns that affect other patterns. Patterns hidden by patterns. Patterns within patterns. Chuck Palahniuk
- 39. Sensitivity: Secret Usecase–Checkouteventprocessing Login Checkout Payment Order Data sync Shipping
- 42. Sensitivity: Secret Hub-and-spokeeventbus Customer login Login Shipping Send order to SAP Data sync Customer, VIP, wishlist sync Checkout Submit order Payment Authorize payment Commerce platform Order Process order Order and customer updates Event relay Customer login Invoke every minute Events Order complete Customer login Payment authorized Order submit Order complete EventBridge FIFO queue
- 43. Sensitivity: Secret Choreography This Photo by Unknown Author is licensed under CC BY-SA-NC Event Bus Publisher Subscriber Publisher Publisher Publisher Subscriber Subscriber Subscriber • No conductor • Pub-sub model • Service decoupling • Asynchronous • Idempotency • Traceability Use choreography to coordinate different microservices
- 46. Sensitivity: Secret Transformedeventflow Vehicle entry Billboard feeder Vehicle entered Car park status Vehicle exit Vehicle exited
- 47. Sensitivity: Secret { "detail": { "metadata": { “domain": “LEGOLAND-WINDSOR", “service": “vehicle-entry", “carpark": “Upper Level", "type": “CAR", "status": “ENTERED" }, "data": { “registration": “XY12ABC", “entryTime": “1576826331“, “spaces”: 125 } } } } Transformedeventflow Vehicle entry event { “location": “$.detail.metadata.carpark“, “available": “$.detail.data.spaces“, } Data paths • Limits event data exposure • Customize target delivery data “<location> has <available> spaces!!" Data template Upper Level has 125 spaces!!
- 48. Sensitivity: Secret Staticeventflow Checkout Monitor Order placed Order prompt
- 49. Sensitivity: Secret { "detail": { "metadata": { “domain": “LEGO-SHOP", “service": “service_checkout", "type": "ORDER", "status": “SUBMITTED" }, "data": { "orderNumber": "T123456789", "customerId": "bf3703467718“, “locale”: “en-GB”, “value”: 59.99 } } } } Checkout Monitor Order placed Order prompt Extensive order data { “prompt": “order" } Static constant data • Original event is not exposed • Ideal for hiding sensitive data • Controls data flow to 3rd parties
- 51. Sensitivity: Secret Loyalty registration Joins loyalty program Retail customer Retail POS device Customer accounts Loyalty Service Customer Service Async registration Registration process Loyalty number Loyalty number UseCase:Customerregistration Enroll
- 52. Sensitivity: Secret Retail POS app Enrollment API Auth Auth Handler Customer API Invoke Enrollment Registration X DLQ DLQ handler TraditionalSyncTroubles
- 53. Sensitivity: Secret Enrollment event Retail POS app Loyalty event bus Enrollment API Auth Auth Handler Target rule Customer API Invoke Enrollment Choreography Registration Event-driven
- 54. Sensitivity: Secret Enrollment event Retail POS app Loyalty event bus Enrollment API Auth Auth Handler Target rule Customer API Invoke Enrollment Choreography Registration Destination Event-drivenwithAPIDestination
- 56. Sensitivity: Secret API Destination target API connection API endpoint Basic auth OAuth auth API key auth Custom Partner Endpoint URL HTTP method Invocation rate Username Password Key name Value Auth endpoint HTTP method Client ID Client secret ***** APIDestinationComponents
- 57. Sensitivity: Secret • Secrets Manager integration • Rate limit support • Built-in retry • Functionless • Partner targets • Payload size • Fire and forget • No API response handling • 5 seconds timeout • Observability challenges APIDestination–BenefitsvsBeAwareOf
- 59. Sensitivity: Secret Circuit breaker Service consumer Closed circuit Service Service consumer Open circuit Service Service consumer Service status Service Failed requests • Service status check • Open, half-open, closed • Essential in distributed services • Failure replay considerations
- 60. Sensitivity: Secret Use case: Guaranteed data update Order processingflow waits for the responses
- 61. Sensitivity: Secret Pattern: Circuit-breaker with archive-replay Order Processing Vendor Mediator Event bus SaaS Platform Orders Cache SUCCESS – All good ERROR – Data RETRY – Timeout, 5XX
- 63. Sensitivity: Secret Vendor Mediator Event bus Order status events Order status consumer rule Order retry archive rule Event archive success error retry retry Events archive An event archive is a collection of events that have been published on an event bus.
- 64. Sensitivity: Secret An event archive is a collection of events that have been published on an event bus. Resources: VoucherArchive: Type: AWS::Events::Archive Properties: ArchiveName: OrdersArchive Description: Orders to be resubmitted EventPattern: <YourEventFilter> RetentionDays: 10 SourceArn: 'arn:aws:..:event-bus/loyalty' { "detail": { "metadata": { "domain": [ "LEGO-LOYALTY" ], "service": [ “order-submission" ], "category": [ "task-status" ], "type": [ “order" ], "status": [ "retry" ] } } }
- 65. Sensitivity: Secret Vendor Mediator Event bus Order status events Order status consumer rule Order retry archive rule Event archive success error retry retry SaaS Platform Replay trigger Status Scheduler API retry events retry events Events replay
- 66. Sensitivity: Secret EventsReplay StartReplay - API EventStartTime: timestamp EventEndTime: timestamp ReplayName: “OrdersReplay_20220330201522” { "detail":{ "metadata":{ "domain":"LEGO-LOYALTY", "service":“order-submission", "category":"task-status", "type":“order", "status":"retry" }, "data":{ "request_id":"AbLhmQ", "reference":"P6IQd", "order":"M101" } }, "replay-name": “OrdersReplay_20220330201522” } Replayed Event Replay API
- 67. Sensitivity: Secret HandlingReplayedEvents StartReplay - API EventStartTime: timestamp EventEndTime: timestamp ReplayName: “OrdersReplay_20220330201522” { "detail":{ "metadata":{ "domain":"LEGO-LOYALTY", "service":“order-submission", "category":"task-status", "type":“order", "status":"retry" }, "data":{ "request_id":"AbLhmQ", "reference":"P6IQd", "order":"M101" } }, "replay-name": “OrdersReplay_20220330201522” } Replayed Event Replay API { "detail": { "metadata": { "domain": [ "LEGO-LOYALTY" ], "service": [ “order-submission" ], "category": [ "task-status" ], "type": [ “order" ], "status": [ "retry" ] } }, "replay-name": [{ “prefix": “OrdersReplay” }], } Replay Event Filter Pattern
- 68. Sensitivity: Secret • Flexible retention period • Multiple archives • Replay start & end time • Replay target rule • Event separation • Replay event exodus • Order not maintained • Take care of idempotency • No archive visibility • Replay delay Archive-Replay–BenefitsvsBeAwareOf
- 69. Sensitivity: Secret Pattern:Distributedorchestration Usecase:Cross-serviceorchestrationwithtasktoken
- 70. Sensitivity: Secret Choreography This Photo by Unknown Author is licensed under CC BY-SA-NC Event Bus Publisher Subscriber Publisher Publisher Publisher Subscriber Subscriber Subscriber • No conductor • Pub-sub model • Service decoupling • Asynchronous • Idempotency • Traceability Use choreography to coordinate different microservices
- 72. Sensitivity: Secret Orchestration • Orchestrator • Workflow • Long running • Coordination of tasks • Low code • Pause & resume Data Process Process Process
- 73. Sensitivity: Secret Orchestration • Orchestrator • Workflow • Long running • Coordination of tasks • Low code • Pause & resume Data Process Process Process “Cross-service orchestration. Good or bad?” “Logic: Inside a function or in a state machine?” Use orchestration inside microservices Use distributed orchestration across microservices
- 74. Sensitivity: Secret Use case: Cross-service task coordination Data Process Process Long running process API Function Microservice A Microservice B Microservice C
- 75. Sensitivity: Secret Conversationalevent-drivenpattern A B Event bus Request Request Response Response
- 76. Sensitivity: Secret Publicclient Loyalty Service SaaS CRM Data Feeder Loyalty Sessions Event Bus Event Source Orders Service Status Admin Rewards Members SaaS Survey LEGO Data Lake NPS Provider CRM Mediator Fraud Engine Events Ingestion LEGO Event Broker Order Management
- 78. Sensitivity: Secret Pattern: Distributed orchestration Microservice A Microservice X • Keep services decoupled • Request - response • Receive completion note
- 79. Sensitivity: Secret Pattern: Distributed orchestration Microservice A Microservice X
- 81. Sensitivity: Secret "Dispatch Voucher": { "Type": "Task", "Resource": "arn:aws:states:::events:putEvents.waitForTaskToken", "HeartbeatSeconds": 6000, "Parameters": { "Entries": [ { "Detail": { "metadata": { "domain": "LEGO-LOYALTY", "service": "service-loyalty-orders", "category": “internal", "type": “request", "status": "processed" }, "data": { "loyalty_request_id.$": "$$.Task.Token", "loyalty_reference.$": "$.loyalty_reference", "merchant_reference.$": "$.merchant_reference", "loyalty_order_reference.$": "$.loyalty_order_reference", "vouchers": [ { "voucher_code.$": "$.voucher_code" } ] } }, "DetailType": "event", "EventBusName": "the-custom-event-bus-arn", "Source": "service-loyalty-order-process"
- 86. Sensitivity: Secret Publicclient Loyalty Service Platform SaaS CRM Data Feeder Loyalty Sessions Event Bus Event Source Orders Service Status Admin Rewards Members SaaS Survey LEGO Data Lake NPS Provider CRM Mediator Fraud Engine Events Ingestion
- 87. Sensitivity: Secret Publicclient Loyalty Service Platform SaaS CRM Data Feeder Loyalty Sessions Event Bus Event Source Orders Service Status Admin Rewards Members SaaS Survey LEGO Data Lake NPS Provider CRM Mediator Fraud Engine Events Ingestion
- 88. Sensitivity: Secret Publicclient Loyalty Service Platform SaaS CRM Data Feeder Loyalty Sessions Event Bus Event Source Orders Service Status Admin Rewards Members SaaS Survey LEGO Data Lake NPS Provider CRM Mediator Fraud Engine Events Ingestion Domain events • Businesssignificant • Outward & inwardflowing • Restricted access Operational/internalevents • Internal visibility • Messengers& Catalysts • Available within the boundary
- 89. Sensitivity: Secret Publicclient Loyalty Service Platform SaaS CRM Data Feeder Loyalty Sessions Event Bus Event Source Orders Service Status Admin Rewards Members SaaS Survey LEGO Data Lake NPS Provider CRM Mediator Fraud Engine Webhook Domain events • Businesssignificant • Outward & inwardflowing • Restricted access Operational/internalevents • Internal visibility • Messengers& Catalysts • Available within the boundary Gatekeeperservicebus Internal Event Bus Gate Keeper Bus External subscribers
- 90. Sensitivity: Secret Publicclient Loyalty Service Platform SaaS CRM Data Feeder Loyalty Sessions Event Source Orders Service Status Admin Rewards Members SaaS Survey LEGO Data Lake NPS Provider CRM Mediator Fraud Engine Events Ingestion Domain events • Businesssignificant • Outward & inwardflowing • Restricted access Operational/internalevents • Internal visibility • Messengers& Catalysts • Available within the boundary Gatekeeper Bus Internal Event Bus External Targets
- 92. Sensitivity: Secret • Complex • Automate pub / sub onboarding Singleenterprisebus Multipleenterprisebuses Singlebusperdomain Multiplebusesperdomain • Domain specific • Better event flow control • Cross-account event sharing • Internal vs external bus • Better control & visibility • Cross-bus, cross-account sharing • Compact & controlled option • Better visibility & governance • Cross-service sharing
- 94. Sensitivity: Secret • Event streamer • Event size: 1 MB • Event batching • Event store: 24hr – 1yr • Sequence number • Limited targets • On-demand & provisioned o Event choreographer o Event size: 256 KB o No event batching o Event archive o No sequencing o Many targets o On-demand scaling Kinesis EventBridge Services-scale refined event ingestion and routing Cloud-scale raw event ingestion and processing