Big Data on Azure Tutorial

Editor's Notes

• #4: All kinds of data being generated   Stored on-premises and in the cloud – but vast majority in hybrid   Reason over all this data without requiring to move data   They want a choice of platform and languages, privacy and security   <Transition> Microsoft’s offerng
• #20: https://aka.ms/sg4qzc https://customers.microsoft.com/doclink/pros
• #42: Objective: This slide describes the architecture of how Apache Spark is different, allowing it to offer better performance for data sharing. Table Source: https://gist.github.com/jboner/2841832 Talking points: Spark provides primitives for in-memory cluster computing. A Spark job can load and cache data into memory and query it repeatedly, much more quickly than disk-based systems. Spark integrates into the Scala programming language to let you manipulate distributed data sets like local collections. No need to structure everything as map and reduce operations. Data sharing between operations is faster, since data is in-memory. Hadoop shares data through HDFS, an expensive option. It also maintains three replicas. Spark stores data in-memory without any replication.
• #43: Objective: This slide explains the two types of operations that RDDs support: transformation and actions. Talking points: Transformations create a new data set from an existing data set. Transformations do not compute their results right away. They are only computed when an action requires a result to be returned to the driver program. Does not apply to persistent RDDs. Examples include: map, filter, sample, union, and more. Actions return a value to the driver program after running a computation on the data set. Examples include: reduce, collect, count, first, foreach, and more.
• #45: Objective: This slide shows an example of how transformations and actions are enabled to search through error messages. Talking points: Cache errors – Implementing this action will collect all the errors present Count all errors – Implementing this action counts all the errors in the data Count errors mentioning MySQL – When implementing this code, MySQL errors are counted Fetch the MySQL errors as an array of strings – When implementing this code, MySQL errors are extracted as an array of strings
• #67: Event Detection in Realtime FINANCIAL ENGINES CONNECTED CAR – SENSORS FIRE Data Landing for Learning Use cases Connected Car Insurance companies for Connected Driving
• #70: What are the three Big components that You need to stand up when you ASK: Who knows what Lambda architecture is Who has helped implement one? Walk through VERTICALS Ingest Prep + Analyze Serve Consume Horizontals Drive by speed – realtime vs Batch
• #71: What are the three Big components that You need to stand up when you ASK: Who knows what Lambda architecture is Who has helped implement one? Walk through VERTICALS Ingest Prep + Analyze Serve Consume Horizontals Drive by speed – realtime vs Batch
• #74: Let’s Walk through an example of this
• #77: We will demo this soon
• #80: We will demo this soon
• #81: TODO – add logos for Bing Ads, Office365, Delve Analytics
• #87: How to monitor all of our resources across subscriptions with single pane of glass? How to Analyze Hadoop Logs & Metrics easily? How to setup alerting?