Building IoT and Big Data Solutions on Azure
Download as PPTX, PDF2 likes1,081 views
This document discusses building IoT and big data solutions on Microsoft Azure. It provides an overview of common data types and challenges in integrating diverse data sources. It then describes several Azure services that can be used to ingest, process, analyze and visualize IoT and other large, diverse datasets. These services include IoT Hub, Event Hubs, Stream Analytics, HDInsight, Data Factory, DocumentDB and others. Examples and demos are provided for how to use these services to build end-to-end IoT and big data solutions on Azure.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Building IoT and Big Data Solutions on Azure (20)
![[「RDB技術者のためのNoSQLガイド」出版記念セミナー] Azure DocumentDB](https://cdn.slidesharecdn.com/ss_thumbnails/20160317azuredocumentdb-160329053855-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Ido Flatow (14)
Recently uploaded (20)
Building IoT and Big Data Solutions on Azure
- 1. Building IoT and Big Data Solutions on Azure Ido Flatow Senior Architect, Sela Group Microsoft MVP & RD @idoflatow Level: Intermediate
- 2. Modern Data – The Big Picture IoT User Data Media Files Documents Machine Data Log Files
- 3. IoT 2010
- 4. IoT 2016
- 5. What We Need • An integrated data solution that will be: – Able to process events from external sources – Able to walk data through different pipelines – Fast and responsive – Big-Data ready
- 6. In Other Words Consume BI Dashboards Applications Process ETL Aggregations Computation Analysis Querying Persist Hadoop SQL NoSQL Ingest Structured Data Un-Structured Data
- 7. Microsoft Azure Services for IoT and BigData Devices Device Connectivity Storage Analytics Presentation & Action Event Hubs SQL Database Machine Learning App Service IoT Hub Table/Blob Storage Stream Analytics Power BI External Data Sources DocumentDB HDInsight Notification Hubs External Data Sources Data Factory Mobile Apps BizTalk Services { }
- 8. Microsoft Azure Services for IoT and BigData Devices Device Connectivity Storage Analytics Presentation & Action Event Hubs SQL Database Machine Learning App Service IoT Hub Table/Blob Storage Stream Analytics Power BI External Data Sources DocumentDB HDInsight Notification Hubs External Data Sources Data Factory Mobile Apps BizTalk Services { }
- 9. Field Gateway Device Connectivity & Management Analytics & Operationalized Insights IoT & Data Processing PatternsDevices RTOS,Linux,Windows,Android,iOS Protocol Adaptation Batch Analytics & Visualizations Azure HDInsight, AzureML, Power BI, Azure Data Factory Hot Path Analytics Azure Stream Analytics, Azure HDInsight Storm Hot Path Business Logic Service Fabric & Actor Framework Cloud Gateway Event Hubs & IoT Hub Field Gateway Protocol Adaptation
- 10. Field Gateway Device Connectivity & Management IoT with Event Hubs Devices RTOS,Linux,Windows,Android,iOS Cloud Gateway Event Hubs Field Gateway Protocol Adaptation
- 11. IoT Hub Cloud Gateway Endpoints device Event processing (hot and cold path) Device provisioning and management Your IoT Hub Device id C2D queue endpoint D2C send endpoint Device … Device … Device … D2C receive endpoint C2D send endpoint Msg feedback and monitoring endpoint Device identity managementIoT Hub management Device business logic, Connectivity monitoring Field GW / Cloud GW
- 12. IoT Hub Features • Connection – Bidirectional communication – Reliable & secure channel – Per-device authentication – Multiplexing • Features – Device to cloud telemetry – Cloud to device commands and notifications (with TTL & feedback) – File uploads/downloads – Monitoring devices (connection, activity, ...) – Multi protocols (AMQP, HTTP) IoT Protocol Gateway (MQTT)
- 13. IOT HUB Demo
- 14. Azure Stream Analytics • Automatic recovery • Monitoring and alerting • Scale on demand • Managed Cloud Service • Each unit handles 1MB/s • Can scale up to 1GB/s • SQL like language • temporal windowing semantics • support for reference data
- 15. Tumbling Windows • How many vehicles enter each toll booth every 5 minutes? SELECT TollId, COUNT(*) FROM EntryStream GROUP BY TollId, TumblingWindow(minute,5)
- 17. What is Azure Data Factory? Azure Data Factory is a managed service to produce trusted information from data stored in the cloud and on-premises. Easily create, orchestrate and schedule highly-available, fault tolerant work flows to move and transform your data at scale.
- 18. Evolving Approaches to Analytics ETL Tool (SSIS, etc) EDW (SQL Svr, Teradata, etc) Extract Original Data Load Transformed Data Transform BI Tools Ingest Original Data Scale-out Storage & Compute (HDFS, Blob Storage, etc) Transform & Load Data Marts Data Lake(s) Dashboards Apps Streaming data
- 19. Data Factory Concepts Call Log Files Customer Table Customer Churn Table Customer Call Details Customers Likely to Churn
- 21. DocumentDB and Azure Data Services fully managed, scalable, queryable, schema free JSON document database service for modern applications transactional processing rich query managed as a service elastic scale internet accessible http/rest schema-free data model arbitrary data formats
- 22. Data size Access Updates Structure Integrity Scaling Hadoop vs. Relational DB
- 23. Hadoop Ecosystem and OSS vs. Azure IoT and BigData Services Azure Services OSS Solutions Event Hubs Kafka IoT Hub Kafka + Mosquitto (MQTT broker) Stream Analytics Storm HDInsight Hadoop Map Reduce Map Reduce Hive Hive Spark Spark HBase HBase Azure ML Mahout Data Factory Pig DocumentDB MongoDB / Couchbase
- 25. Azure IoT Hub vs Event Hub Area IoT Hub Event Hubs Communication patterns Device-to-Cloud Cloud-to-Device event ingress (device-to-cloud) Device protocol support AMQP, AMQP over WebSockets, HTTP, and MQTT AMQP, AMQP over WebSockets, and HTTP Security Per-device identity and revocable access control Event Hubs-wide shared access policies, with limited revocation support Operations monitoring Rich set of device identity management Only aggregate metrics File upload File notification endpoint for workflow integration Manually request files from devices Scale Millions of simultaneously connected devices Limited number of simultaneous connections-- up to 5,000 AMQP connections Device SDKs C, Node.js, Java, .NET, Python Java, .NET C and Node.js in preview
- 27. Resources • Azure IoT Dev center – http://www.azure.com/iotdev • Azure Services – https://azure.microsoft.com/en-us/services/event-hubs – http://azure.microsoft.com/en-us/services/iot-hub – https://azure.microsoft.com/en-us/services/stream-analytics – https://azure.microsoft.com/en-us/services/data-factory – https://azure.microsoft.com/en-us/services/documentdb – https://azure.microsoft.com/en-us/services/hdinsight • Microsoft and IoT – https://www.microsoft.com/en-us/cloud-platform/internet-of-things – https://blogs.microsoft.com/iot/ • My Info – @IdoFlatow // [email protected] // http://www.idoflatow.net/downloads
Editor's Notes
- #8: Key goal of slide: IoT as you know is a hot area these days and there are a number of players that claim to be active in this space…. And they tend to focus on specific elements you see in this diagram. Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions. Customers are adopting these services and are successfully deploying their solutions today (reference Rockwell, ThyssenKrupp) Talk track [Short Version for Sam’s Leadership Session]: As we think about Azure IoT services, Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions Ranging from devices that produce data, to connecting them to the cloud storage, and driving analytics to gain valuable business insights that allows enterprises to take actions Talk track [Long Version Chris’ Breakout Session]: As we think about Azure IoT services, there are a collection of capabilities involved. First there are Producers. These can be basic sensors, small form factor devices, traditional computer systems, or even complex assets made up of a number of data sources. Next we have the Connect Devices capabilities on the ingress level within and around Azure. The primary destination is Service Bus & Event Hubs, but this relies on client agent technology either at the edge device level or within a field or cloud gateway. We also have capabilities for other external data sources o provide data As data is ingressed to Azure, there are various Storage options there can be a number of destinations engaged. Traditional database technology, table or blob, or even more complex destinations like Document DB are possible. External or third party technologies can also be used. This is where the flexibility and agility of a platform shows its strength, This is where analysts like Gartner are forming opinions about just how robust our platform can be. As this data is processed in Azure, there are a number of capabilities that can be utilized. Machine Learning, HD Insight, Stream Analytics are examples of tools that can analytics the data in various ways. Finally the concept of Take Actions uses Azure services. Data may populate a LOB portal, be pushed to apps, or presented in analytics and productivity tools. These are all ways that the data gets out of these architecture points to allow organizations to use analysis to change / transform their business. Through all of these areas, there is the possibility of utilizing existing investments either within your Azure environment, or elsewhere.
- #9: Key goal of slide: IoT as you know is a hot area these days and there are a number of players that claim to be active in this space…. And they tend to focus on specific elements you see in this diagram. Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions. Customers are adopting these services and are successfully deploying their solutions today (reference Rockwell, ThyssenKrupp) Talk track [Short Version for Sam’s Leadership Session]: As we think about Azure IoT services, Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions Ranging from devices that produce data, to connecting them to the cloud storage, and driving analytics to gain valuable business insights that allows enterprises to take actions Talk track [Long Version Chris’ Breakout Session]: As we think about Azure IoT services, there are a collection of capabilities involved. First there are Producers. These can be basic sensors, small form factor devices, traditional computer systems, or even complex assets made up of a number of data sources. Next we have the Connect Devices capabilities on the ingress level within and around Azure. The primary destination is Service Bus & Event Hubs, but this relies on client agent technology either at the edge device level or within a field or cloud gateway. We also have capabilities for other external data sources o provide data As data is ingressed to Azure, there are various Storage options there can be a number of destinations engaged. Traditional database technology, table or blob, or even more complex destinations like Document DB are possible. External or third party technologies can also be used. This is where the flexibility and agility of a platform shows its strength, This is where analysts like Gartner are forming opinions about just how robust our platform can be. As this data is processed in Azure, there are a number of capabilities that can be utilized. Machine Learning, HD Insight, Stream Analytics are examples of tools that can analytics the data in various ways. Finally the concept of Take Actions uses Azure services. Data may populate a LOB portal, be pushed to apps, or presented in analytics and productivity tools. These are all ways that the data gets out of these architecture points to allow organizations to use analysis to change / transform their business. Through all of these areas, there is the possibility of utilizing existing investments either within your Azure environment, or elsewhere.