Designing data pipelines for analytics and machine learning in industrial settings

© 2018 MapR TechnologiesMapR Confidential 1
Building Data Pipelines for AI-
enhanced Industrial Automation
Ian Downard idownard@mapr.com
Will Ochandarena wochandarena@mapr.com

About Us
Ian Downard Will Ochandarena
• Current: Product Management @ MapR
– Focus: IoT, Cloud, Streaming
• Past: Product Manager @ Cisco
• Schools:
– Engineering @ Rensselaer
Polytechnic Institute
– Business @ Santa Clara
University
• Current: Software Engineer @ MapR
– Focus: developer enablement
• Past: Software Engineer @ US Navy,
Rockwell Automation
• Schools:
– Post-grad Engineering @ Missouri
University of Science and
Technology

Agenda
1. Trends in Industrial IoT & AI
2. A Practitioner’s Guide to IoT Data Pipelines
3. Demo: HVAC Monitoring and Predictive Maintenance

Industrial IoT Trends

• Manufacturer’s adoption of ML/AI will increase 38% in the next five years.
Source: Digital Factories 2020: Shaping the future of manufacturing (48 pp., PDF, no opt-in) PriceWaterhouseCoopers
• ML will reduce supply chain forecasting errors by 50% and lost sales by 65%.
Source: Smartening up with Artificial Intelligence (AI) - What’s in it for Germany and its Industrial Sector? (52 pp., PDF, no opt-in) McKinsey &
Company.
• Manufacturers are improving semiconductor yields up to 30%
Source: Smartening up with Artificial Intelligence (AI) - What’s in it for Germany and its Industrial Sector? (52 pp., PDF, no opt-in) McKinsey &
Company.
Why You Should Care:
AI technologies are yielding real business results for Manufacturers
(Source: https://www.forbes.com/sites/louiscolumbus/2018/03/11/10-ways-machine-learning-is-revolutionizing-manufacturing-in-2018/#3532c8ec23ac)

Data Science is Leaving the Playground
Data
Analysis
Data
Science
Operationalization
of AI
Backward
Looking
Forward
Looking
Forward
Looking
Long-term
Decision
Making
Short-term
Decision
Making
In-the-moment
Decision
Making

Successful AI requires robust data pipelines
An AI
use-case
Models
running in
production
E.g. Predicting
remaining useful life
(RUL) of an
equipment
Requires
Feature(s) extraction,
model dev, algo
selection, supervised
& unsupervised
learning
Requires
Partition the
deployment logic
between equipment,
sensors, cloud, and
on-prem
Ability to tap into ALL
data silos
Build a system of
record from real-time
and batch data
Establish data
security, governance,
stewardship
A flexible, extensible
data framework
Requires
Iterative process, piloting with
a smaller data sample, explore
classification, regression,
toolkit selection and more.
Fundamental tenets of a
robust data platform
Gartner estimated that 85% percent of big data projects fail due to challenges in integrating with
existing business processes and applications, … , and security and governance challenges.
www.techrepublic.com/article/85-of-big-data-projects-fail-but-your-developers-can-help-yours-succeed/

Data-Driven Trends – From Yesterday to Today to Tomorrow
Manual assembly
Inadequate visibility into the
supply chain
Un-connected equipment.
Manual prognosis.
Manual business processes
for selling goods & services
Semi-automated assembly
Robots-based fully automated
assembly
Partially connected via Supply Chain
Management Software (SCMS)
IoT-ized equipment
E-commerce & e-retailers
enabled via the Internet
Automated supply ordering
based on depletion monitoring
Predictive maintenance and
proactive health monitoring
Quicker and seamless selling using
bots, voice-services
Drones for delivery. Autonomous
delivery vehicles.
Handover to the “delivery guys”
Connected delivery vehicles with
accurate delivery alerts to consumers
Factories with
monolithic apps
Connected factories
with connected apps
Smart factories with data-
driven apps
YESTERDAY TODAY TOMORROW

Rise of Edge Computing
Need for real-time
collection drives
move to pub/sub
data protocols
Modern vision
algorithms spur
rise of video as
the uber-sensor
Moore’s law putting
unprecedented
compute and storage
capacity @ the edge
Remote locations
have unreliable, low-
bandwidth network
connectivity

• “Act locally, learn globally”
• Small footprint at the edge for local data
services, processing and core connectivity
• Collect, process, classify data @ edge
• Pub-sub streaming of summary data &
metadata from edge to core
• Core persists streams as stream-of-record
for centralized use cases & learning
• Raw data held at edge, available for recall
from core for period of time
Architectural Trend #1 - Edge-first Processing
StreamT
o
pi
c
Stream
T
o
pi
c
Stream
T
o
pi
c
Edge
Edge
Edge
Cloud Core
On-prem Core
Core
External
Application

Architectural Trend #2 - Stream of Record
Sensor Readings
& Events
Persisted Stream
of Record
Materialized
Views
Apps
• Development Agility - New apps access new and historical data in one place.
• Compliance - Persistent, tamper-proof record of historical activity.
• Architecture Simplicity - Less data copies and pipelines.
Yield Mgmt
Remaining
Useful Life
Exploration
temperatures
positions
events

A Practitioner’s Guide to
IoT Data Pipelines

© 2018 MapR Technologies 13
Roadmap to building AI into Factory IoT
Build Data Pipelines
Run AI experiments
Analyze Data
Instrument Machinery
Automate AI utilities
Monitoring
Data Exploration
Applied Machine Learning

Data Acquisition
CONTROL NETWORK
CORPORATE NETWORK
Run AI experiments
Analyze Data
PLC PLC
Gateway to data.
MQTT, REST…

Data Pipelines
Ingest Persist Analyze
Data Flow
IDEs, notebooks,
online AI platforms
Files, Tables, Streams

Data Pipelines
Run AI experiments
Analyze Data

Monitoring and Data Exploration
Run AI experiments
Analyze Data

Data Exploration, Feature Engineering, and AI
IDEs, notebooks
platforms
Programming
Libraries
Data
Run AI experiments
Analyze Data
Files, Tables, Streams

What AI techniques are viable?
• Linear Regression
– predict a value (e.g. remaining useful life)
• Logistic Regression
– predict a probability (e.g. chance that failure will occur within 30
seconds)
• Anomaly Detection
– Alert when time-series prediction != actual
Run AI experiments
Analyze Data

Sensor Data
Device ID time x y z
1 8:00:00 .431 .123 .145
1 8:00:01 .735 .112 .672
1 8:00:02 .932 .141 .431
1 8:00:03 .988 .241 .625

Device ID time x y z _operator _weekend
1 11:59:58 .431 .123 .145 Joe False
1 11:59:59 .735 .112 .672 Joe False
1 12:00:00 .932 .141 .431 Moe True
1 12:00:01 .988 .241 .625 Moe True
Derived Features
Deriving new properties that correlate to failures requires flexible
schema data storage.

Sensor Data
Device
ID
time x y z Remaining
Life
30s to
failure
1 8:00:00 .431 .123 .145
1 8:00:01 .735 .112 .672
1 8:00:02 .932 .141 .431
1 8:00:03 .988 .241 .625
Lagging Features

Device
ID
time x y z Remaining
Life
30s to
failure
1 8:00:00 .431 .123 .145 3 true
1 8:00:01 .735 .112 .672 2 true
1 8:00:02 .932 .141 .431 1 true
1 8:00:03 --- --- --- 0 true
Lagging Features
When a failure happens…
…then lagging features get labeled.

NoSQL DB
Spark DB Connectors
Operate on
data in Spark
without data
movement.
r
DB pushdown =
fast filtering and
sorting.
Objective Solution
“Database pushdown” means you can update
feature stores without data movement.

Feature table size
Number of Lagging variable
records to update.
Alert sent to Grafana
Listening to stream
topic for failure events.

Architecting for Fast Data
• Continuous time signals require
high speed sampling.
• Full resolution is required.
– Aggregation hides important things.
– High fidelity makes machine
learning more effective.
• Streams and compute must scale.

Case Study: Vibration analysis
• Vibrations give the first clue that a machine is failing
• Vibration sensors measure physical displacement
• Capturing a 10kHz vibration requires > 20k samples / second
Detecting vibration anomalies requires continuously
processing high speed data streams.

(1 record / sec)
>20k samples/sec
Anomaly
notifications
Feature
Store
Spark can consume high speed streams and
persist derived signals to new streams or tables.

Anomaly
notifications
Feature
Store
Spark can filter streams by device id
and parallel compute transformations.
As long as the ingest stream can keep
up, this will scale. What if it doesn’t?

Anomaly
notifications
Feature
Store
Affinity of producers to topics
can significantly improve
throughput in MapR Streams
and Kafka.
Spark consumers can
subscribe to multiple
topics, so just subscribe
to all and scale away!

Machine Learning in Production
A traditional starting point:
Run AI experiments
Analyze Data
Model
Inference
Request
Remaining
Useful Life
Inferences
Streams ensure requests
and responses are saved,
replicated, and re-playable.

Run AI experiments
Analyze Data
Models
Telemetry
Remaining
Useful Life
Model Diffs
Inferences
Rendezvous
Models must be monitored, too!

Run AI experiments
Analyze Data
Automate AI utilities http://bit.ly/ml-logistics

IoT Data Pipelines in Action

Predictive
Maintenance
Real-Time
Monitoring
Running AI experiments
Data Exploration and
Feature Engineering
Building Data Pipelines
Predictive Maintenance Demonstration
https://github.com/mapr-demos/predictive-maintenance
Streams, JSON DB

Data Pipeline for HVAC Monitoring
Ingest
Stream
Stream-to-DB
consumer
Interactive monitoring
in Grafana
Kafka API REST APISensor data Time-series
storage in
OpenTSDB

Data Pipeline for HVAC Analytics and Predictive
Maintenance
Ingest
Stream
Feature
Engineering
in Spark
Feature
storage in
NoSQL DB
SQL analytics in Drill
Anomaly detection,
failure prediction,
and data science
tools.
Kafka API OJAI API
Failure events
Sensor data

Designing data pipelines for analytics and machine learning in industrial settings

More Related Content

What's hot (20)

Similar to Designing data pipelines for analytics and machine learning in industrial settings (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Designing data pipelines for analytics and machine learning in industrial settings