![Advanced Filters
Works on the data portion of the event
Supports ten levels of nesting
Supports one or more values
Up to 10 advanced filters can be combined in
OR
Numbers
NumberLessThan
NumberLessThanOrEquals
NumberGreaterThan
NumberGreaterThanOrEquals
NumberIn–thevaluefordata.keyequalsavalueintheset[0,2.08,3.14]
NumberNotIn–thevaluefordata.keyisnotintheset[1,11,112,1124]
Strings
StringContains–thevaluefordata.keycontains“the”
StringIn–thevaluefordata.keyequalsavalueintheset[“small”,“brown”,“fox”]
StringNotIn–thevaluefordata.keyequalsavalueintheset[“jumped”,“over”,“the”]
StringBeginsWith–thevaluefordata.keybeginswith“lazy”
StringEndsWith–thevaluefordata.keyendswith“dog”
Boolean
BoolEquals](https://image.slidesharecdn.com/aguidethroughtheazuremessagingservices-updateconference-191115160447/85/A-guide-through-the-Azure-Messaging-services-Update-Conference-30-320.jpg)
A guide through the Azure Messaging services - Update Conference
- 1. A guide through the Azure Messaging services Eldert Grootenboer
- 3. Messaging?
- 4. Messaging in Azure Event Hubs Streaming data Service Bus Enterprise messaging Event Grid Cross cloud reactive eventing Storage Queues Simple task queues Relay Secure sockets without changes to your network
- 5. Why so many choices?
- 6. Command Report Measurement Job Query Handover Transfer Update Assignment Request Notification Trace
- 8. Events Independent Report State Change Actionable Time Ordered Context Partitioned Report Condition Analyzable Discrete Series
- 10. Let’s have a look at the services
- 11. Simple Queues
- 12. Storage queues are task queues Coordinate work across compute (VMs, Functions, etc.) Low cost Pay as you go Easy to use Messages are generally homogeneous Unlimited number of queues Unlimited number of concurrent connections Maximum queue size 500TB Maximum message size 64KB Few features
- 13. Few features No headers/metadata No pub/sub No Тime To Live No scheduling No dead-letter queue Peeking no more than 32 messages only No de-duplication 7 days retention period No transactional guarantees
- 14. A simple queue Sender sends message to queue Queue ACKs receipt Receiver connects to queue & retrieves message Receiver ACKs complete (or other action)
- 16. Enterprise Messaging as a Service Queues & Topics Reliable asynchronous communication Rich features for temporal control Message headers Routing and filtering Convoys & Sessions (related messages with state)
- 17. Features of Service Bus Scheduled delivery Time To Live ForwardTo Defer Sessions Batching Ordering Auto-delete on idle OnMessage Duplicate detection Actions Transactions Poison message handling
- 18. Features of Service Bus Premium Dedicated resources Geo-DR Availability Zones Virtual Network Service Endpoints
- 19. Topics & Subscriptions Sender only knows about Topic Receivers only know about Subscriptions Filters and Actions exist on Subscriptions Topic Subscription 1 Subscription 2 type=order type=quote Filter Filter “type” = “order” “type” = “quote”
- 20. Where does Service Bus fit in? Decoupling and providing durability with transactional consistency Queue
- 21. Push based publish-subscribe event routing
- 22. What is an event? Something happened Notification Triggers a reaction Based on facts Can be series based or discrete
- 24. Concepts Events: what happened Event Publishers: where it took place Topics: where publishers send events Event Subscriptions: how you receive events Event Handlers: the app or service reacting to the event Events
- 25. Trigger a functionto run CognitiveAPI when a file is added to storage Use a function to run a compliance checkon each newly created SQL database Tag newly provisioned VMs with AzureAutomation and add to metadata store Use custom“drive start” and “driveend” events to log vehicle performancemetrics What is Event Grid For?
- 26. "topic": "/subscriptions/{id}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{storageacc}", "subject": "/blobServices/default/containers/monitored/blobs/blob.json", "eventType": "Microsoft.Storage.BlobCreated", "eventTime": "2017-09-20T01:12:35.8066557Z", "id":"9ac0de1a-001e-00da-23ad-311d4a06bb2f", "data":{ "api":"PutBlob", "clientRequestId": "9ac0de1a-9da0-11e7-9a3b-51cc40a2ca27", "requestId": "9ac0de1a-001e-00da-23ad-311d4a000000", "eTag":"0x9AC0FC4ADA5997D", "contentType": "application/json", "contentLength": 268, "blobType": "BlockBlob", "url":"https://{storageacc}.blob.core.windows.net/monitored/blob.json", "sequencer": "00000000000019200000000000F12B37", "storageDiagnostics": { "batchId": "9ac0de1a-e1a1-44c5-a0a1-a66b2a191fc4“ } What does an Event look like?
- 27. Publishing custom events Custom Topics Topic: <Name of Topic> Subject: <whatever you choose> EventType: <whatever you choose> mytopic.westus2-1.eventgrid.azure.net sub1 sub2 mytopic Topic:mytopic.westus2-1.eventgrid.azure.net Subject: /products/beverages EventType: skuCreated Topic:mytopic.westus2-1.eventgrid.azure.net Subject: /products/snacks EventType: skuUpdated Publisher
- 28. The challenge with Custom Topics Having many topics forces you to do routing This is what Grid was made for There is a better way! sub1 sub2 mytopic2 sub1 sub2 mytopic3 sub1 sub2 mytopic4 sub1 sub2 mytopic1 Publisher
- 29. Event Domains All events to single endpoint Sender sets topic Subscribers are authenticated via AAD (cross org support) Subscribers identity not required to be Azure Entire service can be white labeled
- 30. Advanced Filters Works on the data portion of the event Supports ten levels of nesting Supports one or more values Up to 10 advanced filters can be combined in OR Numbers NumberLessThan NumberLessThanOrEquals NumberGreaterThan NumberGreaterThanOrEquals NumberIn–thevaluefordata.keyequalsavalueintheset[0,2.08,3.14] NumberNotIn–thevaluefordata.keyisnotintheset[1,11,112,1124] Strings StringContains–thevaluefordata.keycontains“the” StringIn–thevaluefordata.keyequalsavalueintheset[“small”,“brown”,“fox”] StringNotIn–thevaluefordata.keyequalsavalueintheset[“jumped”,“over”,“the”] StringBeginsWith–thevaluefordata.keybeginswith“lazy” StringEndsWith–thevaluefordata.keyendswith“dog” Boolean BoolEquals
- 31. Engineered for scale and reliability Retry intervals 10 seconds 30 seconds 1 minute 5 minutes 10 minutes 30 minutes 1 hour Once an hour up to 24 hours Defaults: 30 delivery attempts / 24 hours Dead-lettering Requires Storage account + container Dead-lettered events stored as blobs
- 32. Storage Queues Flexible Interoperability Multi-Cloud Solutions IoT Hub Blob Storage Resource Groups Azure Subscriptions Event Hubs Custom Topics Storage (GPv2) Container Registry Service Bus Azure Event Grid + CNCF CloudEvents Azure Functions Logic Apps Azure Automation WebHooks Event Hubs
- 34. Records a stream Recording moves forward only You can play the tape over and over again Can play from anywhere on the tape FastFwd or Rewind then Play Multiple channels with different data Event streaming
- 35. Event Hubs conceptual architecture Kafka HTTP AMQP
- 36. Where does data streaming fit in? Cloud gateways (web APIs) Field gateways Applications LegacyIOT (customprotocols) Devices IP-capabledevices (Windows/Linux) Low-powerdevices (RTOS) Search and query Cortana Analytics PowerBI Dashboards Devices to take action
- 37. Event Hubs Capture: slow data stream Policy based push to own storage Avro format Does not impact throughput Better batching and archival scenarios Offloads batch processing from your real-time stream (no pressure drop)
- 38. How does Capture work?
- 39. How are Event Hubs and Kafka similar? Both use partitioned consumer Partitions are independent Both use a client side cursor concept Both can scale to very high workloads Conceptually they are nearly the same
- 40. Azure Event Hubs for Apache Kafka™ Kafka 1.0 compatible endpoint backed by Event Hubs Use your existing Kafka application & tools with Event Hubs Only need to change your connection string All defaults supported
- 42. Cloud relayed communication Hybrid Connections & Relay Securely connect to on premises data and services from anywhere Load balance multiple sites behind a single endpoint Protect internal services by projecting a cloud endpoint Does not require any network changes
- 43. Where does Relay fit in? When you have scenarios that are too complex or costly for VPN When VNET integration is not possible When you have to keep your data where it is
- 44. How does it work? Service registers cloud endpoint Client connects to cloud endpoint Network magic happens
- 45. Relay as a gateway Can front multiple service instances Round robin controlled in Relay service Sticky sessions ServiceInstances
- 46. Almost done
- 47. Serverless Streaming Data Enterprise Key Takeaways Segment Simple Queuing Eventing PubSub Streaming Enterprise Messaging Cloud Socket Product Storage Queues Event Grid Event Hubs Service Bus Relay & HC What do you care about • Communication within an app • Individual message • Queue (polling) semantics • Easy to use • Pay as you go • Communication between apps / orgs • Individual message • Push semantics • Filtering and routing • Pay as you go • Fan out • Many messages in a Stream (think in MBs) • Ease of use and operation • Low cost • Fan in • Strict ordering • Works with other tools • Instantaneous consistency • Strict ordering • JMS • Non-repudiation & Security • Geo-Replication & Availability • Rich features (de-dupe, scheduling, etc.) • Raw TCP socket that can traverse firewalls • Load balanced live traffic What you care less about • Ordering of messaging • Instantaneous consistency • Ordering of messaging • Instantaneous consistency • Individual message semantics • Server-side cursor • Only Once • Cost • Simplicity • Buffered messaging • Queue semantics Focus point • Single partition lightweight queue • Reliable fan out push at massive scale • High scale distributed log • Highly indexed full featured message broker • High density TCP tunneling Hybrid
Editor's Notes
- #4: Exchange of messages between systems NOT SMS, WhatsApp etc.
- #8: Commands are an instruction, telling a system to "do something". Approve the invoice. Reject the policy. Turn the light off. Hop on one foot. Commands can be validated, approved/rejected, processed, replied. Events on the other hand reflect an immutable fact. Invoice approved. Policy rejected. Light turned off. Ankle broken. We can't approve/reject events in the past, even if we would love to turn back the clock on certain recent elections. https://jimmybogard.com/message-naming-conventions/
- #11: What the service is What is the key scenario How we address that challenge
- #12: Rename from Azure Storage Queue to Azure Simple Queue or Elastic Queue
- #13: - A reference to MSMQ (when on-premises solutions roamed the world) - Mention it's HTTP-based implementation - Tasks queues coordinate work across compute (VMs, Functions, etc.) - The messages in a queue are generally homogeneous - Low cost - Pay as you Go billing model - Easy to use - Have as many queues as you like (or connections) - Maximum queue size is insane while message size is small - mention claim check that's required to be implemented - Very few features – next slide
- #32: Event Grid adds a small randomization to all retry intervals. After one hour, event delivery is retried once an hour. By default, Event Grid expires all events that aren't delivered within 24 hours. You can customize the retry policy when creating an event subscription. You provide the maximum number of delivery attempts (default is 30) and the event time-to-live (default is 1440 minutes). [30=7 in the first hour + 23 with one-per-hour]
- #33: CloudEvents simplifies interoperability by providing a common event schema for publishing, and consuming cloud based events. This schema allows for uniform tooling, standard ways of routing & handling events, and universal ways of deserializing the outer event schema. With a common schema, you can more easily integrate work across platforms.
- #44: Kill heart. Data sovereignty and accessing on-prem data securely from remote locations. Law firms with client and case notes in their office that needs to be accessed by attorneys remotely. Same with medical data by doctors and nurses. Mom and pop shop could use this as a cheaper alternative to azure vpn.