Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python

1 © Hortonworks Inc. 2011–2018. All rights reserved.
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Open Source Computer Vision with TensorFlow, MiniFi,
Apache NiFi, OpenCV, Apache Tika and Python
Timothy Spann, Solutions Engineer
Hortonworks @PaaSDev
Vision Thing

Disclaimer
• This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be
developed.
• Technical feasibility, market demand, user feedback, and the Apache Software
Foundation community development process can all effect timing and final delivery.
• This document’s description of these features and technology directions does not
represent a contractual commitment, promise or obligation from Hortonworks to deliver
these features in any generally available product.
• Product features and technology directions are subject to change, and must not be
included in contracts, purchase orders, or sales agreements of any kind.
• Since this document contains an outline of general product development plans,
customers should not rely upon it when making a purchase decision.

What?
• For processing images from IoT devices like Raspberry Pis and NVidia Jetson TX1 and
more that are equipped with attached cameras or external USB webcams, we use
Python to interface via OpenCV and PiCamera. From there we run image processing at
the edge on these IoT device using OpenCV and TensorFlow to determine attributes and
image analytics. Apache MiniFi coordinates running these Python scripts and decides
when and what to send from that analysis and the image to a remote Apache NiFi server
for additional processing.
• At the Apache NiFi cluster, in the cluster it routes the images to one processing path and
the JSON encoded metadata to another flow. The JSON data (with its schema referenced
from a central Schema Registry) is routed using Record Processing and SQL. This data in
enriched and augmented before conversion to Apache ORC to be stored in HDFS with
Apache Hive tables.

Agenda
• OpenCV
• TensorFlow
• Apache Tika
• Apache NiFi and MiniFi
• Demos
• Questions

So Why Am I Ingesting Images From Edge Devices?
Use Cases
Object Recognition
• WebCam Security
• Anomaly Detection
• Logging
• Metadata about images
• Customer Analysis
Image Classification Motion Estimation
• Movement tracking
• Security
• Occupied Room
Active Archive
• Store all images
• Training datasets
• Joining With Other Data
• Cameras Everywhere

Architecture

Ingestion
Simple Event Processing
Engine
Stream Processing
Destination
Data Bus
Build
Predictive Model
From Historical Data
Deploy
Predictive Model
For Real-time Insights
Perishable Insights
Historical Insights
Open Computer Vision Flow

Open Source Image Analytical Components
Streaming Analytics
Manager
Image Ingest
Distributed queue
Buffering
Process decoupling
Routing and Pre-Processing
Orchestration
Queueing
Simple Event Processing
Image Capture
Image Processing

Streaming Analytics
Manager
Part of MiniFi C++ Agent
Detect metadata and data
Extract metadata and data
Content Analysis
Deep Learning Framework
Open Source Image Analytical Components
Enabling Record Processing
Schema Management

Aggregate all data from sensors, geo-location devices, machines and social
feeds
Collect: Bring Together
Mediate point-to-point and bi-directional data flows, delivering data
reliably to Apache HBase, Apache Hive, Slack and Email.
Conduct: Mediate the Data Flow
Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather,
location, and TensorFlow.
Curate: Gain Insights

{
"imagefilename" : "/opt/demo/images/2018-04-
17_1127.jpg",
"yaw" : 100.0,
"host" : "sensehatmovidius",
"top3" : "n06874185 traffic light, traffic signal,
stoplight",
"top5" : "n03773504 missile",
"humidity" : 31.2,
"uuid" : "uuid_json_20180417152727.json",
"ipaddress" : 192.168.1.104,
"top2" : "n04286575 spotlight, spot",
"top3pct" : "6.199999898672104",
"top2pct" : "10.199999809265137",
"cputemp2" : 56.92,
"z" : 1.0,
"diskfree" : "4152.5 MB",
"top1pct" : "13.79999965429306",
"currenttime" : "2018-04-17 15:27:37",
"label2" : "n04592741 wing",
"pitch" : 360.0,
"pressure" : 1026.2,
"roll" : 1.0,
"label1" : "n04286575 spotlight, spot",
"top5pct" : "4.30000014603138",
"label4" : "n06874185 traffic light, traffic signal, stoplight",
"y" : 0.0,
"label3" : "n04009552 projector",
"cputemp" : 58,
"top1" : "n02930766 cab, hack, taxi, taxicab",
"top4pct" : "5.000000074505806",
"tempf" : 75.81,
"memory" : 56.5,
"top4" : "n03345487 fire engine, fire truck",
"starttime" : "2018-04-17 15:27:25",
"runtime" : "12",
"label5" : "n09229709 bubble",
"temp" : 35.45,
"x" : 0.0
}
Example Data

OpenCV

What Is OpenCV?
• OpenCV is a an open source computer vision library
• Nearly 20 years old
• Started by Intel
• Current Version 3.4.1
• C++, Python and Java Interfaces
• Runs on Windows, Linux, Mac, BSD, iOS, and Android.
• Can be built from source with Make
• Runs on Raspberry PIs
https://en.wikipedia.org/wiki/OpenCV

What Can I Do with OpenCV?
• Facial Recognition
• Image Capture From Cameras
• Object Identification
• Motion Tracking
• Pixel Manipulation
• Image Properties
• Image Data Manipulation
• Image Processing including filtering, color conversion and histograms
• Image Labelling
https://github.com/jdye64/nifi-opencv
https://www.learnopencv.com/
https://docs.opencv.org/3.4.0/d9/df8/tutorial_root.html

What Does OpenCV Python Code Look Like?
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://github.com/aruizga7/Self-Driving-Car-in-DSX/tree/master/1.%20Line%20Lane%20Detection
import cv2
cap = cv2.VideoCapture(0)
ret, frame = cap.read()
filename = ‘images/ilovedataworkssummit.jpg’
cv2.imwrite(filename, frame)
img = cv2.cvtColor(cv2.imread(filename),cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224))
cv2.rectangle(image, (x, y), (x + w, y + h), (255, 255, 0), 2)

TensorFlow

Apache NiFi Integration with TensorFlow Options
• TensorFlow (C++, Python, Java)
via ExecuteStreamCommand
• TensorFlow NiFi Java Custom Processor
• TensorFlow Running on Edge Nodes (MiniFi)

python classify_image.py --image_file /opt/demo/dronedata/Bebop2_20160920083655-0400.jpg
solar dish, solar collector, solar furnace (score = 0.98316)
window screen (score = 0.00196)
manhole cover (score = 0.00070)
radiator (score = 0.00041)
doormat, welcome mat (score = 0.00041)
bazel-bin/tensorflow/examples/label_image/label_image --
image=/opt/demo/dronedata/Bebop2_20160920083655-0400.jpg
tensorflow/examples/label_image/main.cc:204] solar dish (577): 0.983162I
tensorflow/examples/label_image/main.cc:204] window screen (912): 0.00196204I
tensorflow/examples/label_image/main.cc:204] manhole cover (763): 0.000704005I
tensorflow/examples/label_image/main.cc:204] radiator (571): 0.000408321I
tensorflow/examples/label_image/main.cc:204] doormat (972): 0.000406186
TensorFlow via Python or C++ Binary (Java Library Is New!)

TensorFlow Python ExecuteStreamCommand NiFi
https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html

Run TensorFlow on YARN 3.0
https://community.hortonworks.com/articles/83872/data-lake-30-containerization-erasure-coding-gpu-p.html

Why TensorFlow? Also Apache MXNet, PyTorch and DL4J.
• Google
• Multiple platform
support
• Hadoop integration
• Spark integration
• Keras
• Large Community
• Python and Java APIs
• GPU Support
• Mobile Support
• Inception v3
• Clustering
• Fully functional demos
• Open Source
• Apache Licensed
• Large Model Library
• Buzz
• Extensive Documentation
• Raspberry Pi Support

TensorFlow Java Processor in NiFi
https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html
https://github.com/tspannhw/nifi-tensorflow-processor

TensorFlow Running on Edge Nodes (MiniFi)

Apache Tika with Apache NiFi
• Detection
• Parsing
• Output Formats including Text and HTML
• Translation
• Language Identification
• Apache NiFi - Bundled ExtractMediaMetadata Processor
• Apache NiFi - Extract the content metadata from flowfiles
https://tika.apache.org/1.18/gettingstarted.html
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-media-nar/1.6.0/org.apache.nifi.processors.media.ExtractMediaMetadata/

HORTONWORKS DATA FLOW
NIFI
1.2.0HDF 3.0
Jul 2017
1.0.0
HDF 2.0
Mar 2016
1.1.0
NiFiRegistry
Ranger
0.7.0
0.5.0
0.6.0
Ambari
2.5.1
2.4.0
2.4.2
Kafka
0.10.1.0
0.9.0
0.10.0
Zookeeper
3.4.6
3.4.6
3.4.6
Storm
1.1.0
1.0.1
1.0.2SAM
0.5.0
SchemaRegistry
0.3.0
HDF 2.1
Aug 2016
Ongoing Innovation in Apache
HDF 1.0
Dec 2014
0.3.0
0.6.1
HDF 1.2
Oct 2015
MiNiFiC++andJava0.2.0
Ongoing Innovation in OpenSource
1.0.0
0.0.1
0.10.0
HDF 3.1.1
Jan 2018 1.5.0 0.1.0 0.7.02.6.11.0.0 3.4.61.1.10.6.0 0.5.00.4.0
SECURITYSTREAMING & INTEGRATION OPERATIONS
Hortonworks Data Flow 3.1.1
https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_release-notes/content/ch_hdf_relnotes.html

Apache Tika Supported File Formats
• HTML, XML
• Microsoft Word, Excel, PowerPoint, Outlook
• OpenOffice
• RSS
• RTF
• Zip, Tar, 7zip, Gzip, RAR
• PDF
• MP3, WAV, MIDI
• MP4, FLV
• TIFF, JPEG, PNG, BMP, GIF
• And more!

https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html
https://community.hortonworks.com/articles/81694/extracttext-nifi-custom-processor-powered-by-apach.html
https://community.hortonworks.com/articles/76924/data-processing-pipeline-parsing-pdfs-and-identify.html
https://github.com/tspannhw/nifi-extracttext-processor
https://community.hortonworks.com/content/kbentry/177370/extracting-html-from-pdf-excel-and-word-
documents.html

HDF Data-In-Motion Platform – with HDF 3.1

Why Apache NiFi?
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a fifty sources
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control

Apache MiNiFi
• NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide
bidirectional data transfer"

Edge Intelligence with Apache MiNiFi
 Guaranteed delivery
 Data buffering
‒ Backpressure
‒ Pressure release
 Prioritized queuing
 Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
 Data provenance
 Recovery / recording a rolling log
of fine-grained history
 Designed for extension
Different from Apache NiFi
 Design and Deploy
 Warm re-deploys
Key Features

Custom Apache NiFi Processors for Open Source Computer Vision

TensorFlow with MiniFi
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html

https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html
Image Analytics

NiFi and Kafka Are Complementary
NiFi
Provide dataflow solution
• Centralized management, from edge to core
• Great traceability, event level data provenance
starting when data is born
• Interactive command and control – real time
operational visibility
• Dataflow management, including prioritization,
back pressure, and edge intelligence
• Visual representation of global dataflow
Kafka
Provide durable stream store
• Low latency
• Distributed data durability
• Decentralized management of producers &
consumers
+

Integrated Provisioning and Security
Kafka 1.0 Support
To enhance data governance and lineage, users can
now manage access control policies using resource or
tag-based security in Ranger for Kafka 1.0 clusters.
Users can now install, configure, manage, upgrade,
monitor, and secure Kafka 1.0 clusters with Ambari.
New processors in NiFi and Streaming Analytics
Manager support Kafka 1.0 features including message
headers and transactions.

What Is Apache Kafka?
• Distributed streaming platform that allows
publishing and subscribing to streams of
records
• Streams of records are organized into
categories called topics
• Topics can be partitioned and/or replicated
• Records consist of a key, value, and
timestamp
http://kafka.apache.org/intro
Kafka
Cluster
producer
producer
producer
consumer
consumer
consumer
APACHE KAFKA

https://community.hortonworks.com/articles/177349/big-data-devops-apache-nifi-
hwx-schema-registry-sc.html

Completion of Schema Lifecycle:
Merged Schema from Dev Branch to Master

Schema Registry Support for Different “States”:
Enable, Disable, Archive

SAM and Schema Registry Integration
• Streaming Apps Require a Schema
• Unlike NiFi, SAM requires a schema to build
streaming analytics applications.
• Every SAM builder component requires a schema to
function.
• SAM’s primary mechanism for connecting to a stream
of data is Kafka, but Kafka does not have a schema.
• This is where HDF’s Schema Registry component
becomes incredibly valuable.
• SAM’s Kafka Source Component integrated
with Schema Registry
• When you configure a Kafka source and supply
kafka topic, SAM calls the Schema Registry.
• Using the Kafka topic as the key, SAM will retrieve
the schema.
• This schema is then displayed on the tile component,
and is passed to downstream components.

Streaming Analytics Manager

Contact
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://www.meetup.com/futureofdata-princeton/
https://twitter.com/PaaSDev
https://dzone.com/refcardz/introduction-to-tensorflow

Hortonworks Community Connection
Read access for everyone, join to participate and be recognized
• Full Q&A Platform (like StackOverflow)
• Knowledge Base Articles
• Code Samples and Repositories

Thank you

Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python

More Related Content

What's hot (20)

Similar to Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python

Editor's Notes