Unlocking insights in streaming data

1 © Hortonworks Inc. 2011 – 2017 All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Future of Data Boston

2 © Hortonworks Inc. 2011 – 2017 All Rights Reserved
Agenda
 Networking, Food and drink
 Announcements
 Main Presentation
– Unlocking Insights in Streaming Data with Open Source
 Question and Answer
 Networking and Wrap up

Announcements
 Thanks to our sponsors
– Hortonworks
– Pivotal Labs
 What topics would you like to hear about?

About Carolyn Duby
 Big Data Solutions Architect
 High performance data intensive systems
 Data science
 ScB ScM Computer Science, Brown University
 LinkedIn: https://www.linkedin.com/in/carolynduby/
 Twitter: @carolynduby Github: carolynduby
 Hortonworks
– Innovation through data
– Enterprise ready, 100% open source, modern data platforms
– Engineering, Technical Support, Professional Services, Training

Streaming Analytics
 Streaming data is valuable
– Make decisions in real time
– Gain new understanding of business
– Detect/resolve/predict/warn of conditions
– Recommend at the right moment

Not very easy
 Streams flows in at variable rates
– High and low points
– Data streams arrive with different latency
– Often can’t control input rate
 Lots of different choices of libraries
– Storm, Spark Streaming, Samza, Flink…. Oh my!
 Complex time series analytics
– Windowing
– Joining streams

Components
 Schema registry
– AVRO schemas for streaming data
 Model registry
– Register machine learning models
 Streaming Analytics Manager
– Build and monitor streaming applications
 Superset
– Visualize streaming time series data
 Druid
– Store and aggregate time series data
 Streaming Substrate – Kafka, Storm, HDFS

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reference Architecture: Real-time
Streaming Analytics

Trucking company w/ large fleet of international trucks
A truck generates millions of events for a given route;
an event could be:
 'Normal' events: starting / stopping of the vehicle
 ‘Violation’ events: speeding, excessive acceleration and
breaking, unsafe tail distance
 ‘Speed’ Events: The speed of a driver that comes in every
minute.
Company uses an application that monitors truck
locations and violations from the truck/driver in real-
time
Route?
Truck?
Driver?
Analysts query a broad
history to understand if
today’s violations are
part of a larger problem
with specific routes,
trucks, or drivers

Event Broker Cluster
Sensor Sources
Truck Sensors
Truck Sensors
Truck Sensors
Truck Sensors
Real-time Analytics Architecture with HDF
Flow Management
Clusters
Ingress
Gateway
Niﬁ
Site to Site
Protocol
Egress
Gateway
Cloud Instance in
Different Geo
Locations
China Cloud Instance
(IBM)
Germany Cloud Instance
(Azure)
US Cloud Instance
(Amazon)
Stream Analytics Cluster
Ingest
Streams
Generate
Insights
Real-Time Apps
Real-time
Apps &
Exploration Platform
Centralized Schema
Repository

Enterprise Services:
Schema Registry

Problem Statement:
 No centralized store to manage schemas for event data. Schema has to be hardcoded, passed with
data or inferred. Producers and Consumers cannot evolve at different rates
Solution:
 Introducing new component in HDF platform called: Hortonworks Schema Registry
 A shared repository of schemas that allows applications to flexibly interact with each other - in order to
save or retrieve schemas for the data they need to access
Why does this matter?
 Meets governance and operations requirements with a centralized registry to manage event schemas.
 Provides a reusable schema and avoids attaching a schema to every payload, allows consumers and
producers to evolve at different rates
#1: Schema Registry: The Problem & Solution Defined

How Schema Registry Work with the Rest of the Platform
 NiFi Processors for Schema Registry
– Schema aware flow management (record readers/writers)
– Live schema reference / automated schema conversion
 Streaming Analytics Manager processors for Schema Registry
– For example: Lookup a schema of a Kafka Topic
– Context/schema aware user experience eases time-to-market of building stream apps
 Atlas integration with Schema Registry (In a future release )
– Just like Atlas pulls schema info from Hive MetaStore, Atlas can now capture schema, format and
semantic metadata from events in HDF via the registry.

Schema Registry Demo

Stream Processing

Why SAM?
Make it a delightful experience to build streaming analytics applications.
Provide the same experience for streaming analytics that developers
have today with Apache NiFi/MiniFi.

Stream Processing – Introducing Streaming Analytics Manager (SAM)
Streaming Analytics Manager
 Design, develop, deploy and manage streaming analytics app with drag-and-drop ease
– Build streaming analytics applications that do event correlation, context enrichment , complex
pattern matching, analytical aggregations and creation of alerts/notifications when insights are
discovered.
– Supports multiple streaming substrates/engine (e.g: Storm, Spark Streaming, etc.)
– Extensibility is a first class citizen (add custom sinks, processors, spouts, etc..)

SAM’s 3 Modules for 3 Different Personas in the Enterprise

Stream Ops Module for IT Operations
 Service Pool Abstraction
 Create and manage different environments in which
individual streaming applications will be built
 Environments consists of services such as HDFS, Kafka,
Storm from different service pools
 Save time and reduce operational overhead with same
drag and drop paradigm as the stream build module
 SAM takes away the complexity of deploying secure
streaming analytics on kerberized cluster

Stream Builder Module for App Developers
 Builder components, shown on the canvas
palette, are the building blocks used by the app
developer to build streaming apps.
 Drag and drop to build a working streaming
application without writing a single line of code.
 4 Types of Components: Sources, Processors,
Sinks and Custom

SAM is All about Doing Real-Time Analytics on the Stream
Real-Time
Prescriptive
Analytics
Real-Time Analytics
Real-Time
Predictive
Analytics
Real-Time
Descriptive
Analytics
What should we do
right now?
What could happen
now/soon?
What is happening
right now?

Real-Time Prescriptive Analytics
 Question: What should we do right
now?
 Context: It is rainy, the driver is
been on the road for 12 hours and
he has 30 high speeding alerts over
a 3 minute window in the last 2
hours.
 Answer: Dispatch a radio call to the
Driver to slow down

Real-Time Predictive Analytics
 Question: No violation events but what might happen that I need to be worried about?
 My data science team has a model that can predict that based on
– Weather
– Roads
– Driver HR info like driver certification status, wagePlan
– Driver timesheet info like hours, and miles logged over the last week

Real-Time Predictive Analytics
Use SAM’s enrich/custom processors to enrich the event
with the features required for the model2
Enrich with Features
Use SAM’s projection/custom processors to
transform/normalize the streaming event and the
features required for the model
3
Transform/Normalize
Use SAM’s PMML processor to score the model for each
stream event with its required features4
Score Model
Use SAM’s rule and notification processors to alert,
notify and take action using the results of the model5
Alert / Notify / Action
Export the Spark Mllib model and import into the HDF’s
Model Registry
1 Model
Registry

Real-Time Prescriptive Analytics for Business Analysts
 A tool to create time-
series and real-time
analytics dashboards,
charts and graphs
 30+ visualization
charts out of the box
with customization
capability
 Druid is the Analytics
Engine that powers
the Stream Insight
Module.

Druid

What Is Druid?
Druid is a distributed, real-time, column-oriented datastore
designed to quickly ingest and index large amounts of data
and make it available for real-time query.
Features:
• Streaming Data Ingestion
• Sub-Second Queries
• Merge Historical and Real-Time Data
• Approximate Computation
TECHNICAL PREVIEW: 2.6.2
GA: 2.6.3

Cool stuff you can do with Druid (and pretty much nothing else)
 Real Time Ingest and Query At Scale
– Scale: 100m+ events per second with highly
concurrent queries.
– Stream data from Kafka to Druid and query it as it
arrives.
 Use cases:
– Real-time bidding / market making.
– Realtime analytics on clickstream data.
– IoT monitoring applications.
– Real-time dashboards and KPI tracking.
 Learn how PayPal hypercharged self-service
analytics:
– https://www.slideshare.net/anilmadan902/paypal-
business-intelligence-and-real-time-analytics

Cool stuff you can do with Druid (and pretty much nothing else)
 Datasketches: Fast, multi-dimensional approximate set intersections at high scale.
– Question: Is a CPM of $7.00 a good deal for my iOS app, popular among middle-aged in the US?
– Decision Support: How many iOS users, in the US, age range 30-45 visited in the last week?
 Use cases:
– Targeted advertising / offer management.
– Personalized recommendations.
– Anything aimed at a segment or an individual.
 Learn how Nielsen Marketing Cloud takes Micro Targeting to the next level with Druid:
– https://www.slideshare.net/ItaiYaffe/using-druid-for-interactive-count-distinct-queries-at-scale
Age Range
Country
Mobile
Platforms
Small Intersection:
Not worth it!

Streaming Analytics Manager Demo

Extensibility:
SAM Software Development Kit

Extensibility with SAM SDK
 Custom Processor - allows users to write their own business logic

 Multi-lang support (upcoming)

 UDAFs - compute aggregates within a window Built in functions
 STDDEV
 STDDEVP
 VARIANCE
 VARIANCEP
 MEAN
 MIN
 MAX
 SUM
 COUNT
 UPPER
 LOWER
 INITCAP
 SUBSTRING
 CHAR_LENGTH
 CONCAT

 UDFs - does simple transformations Built in functions
 STDDEV
 STDDEVP
 VARIANCE
 VARIANCEP
 MEAN
 MIN
 MAX
 SUM
 COUNT
 UPPER
 LOWER
 INITCAP
 SUBSTRING
 CHAR_LENGTH
 CONCAT

 Notifier - sends notifications such as Email, SMS or more complex ones that can invoke
external APIs
Built in notifiers
 Email
 More in future…

Questions?

Learn More about Streaming Analytics Manager (SAM)
 Download the tutorial – https://hortonworks.com/tutorial
– Real-Time Event Processing in NIFI, SAM, Schema Registry and Superset
 Blogs – https://hortonworks.com/blog
– Hortonworks Thoughts on Building A Successful Streaming Analytics Platform
 Hortonworks Community – https://community.hortonworks.com
 Github - https://github.com/hortonworks/streamline

Unlocking insights in streaming data

More Related Content

What's hot (20)

Similar to Unlocking insights in streaming data (20)

Recently uploaded (20)

Unlocking insights in streaming data

Editor's Notes