Apache NiFi Crash Course Intro

Apache NiFi Crash
Course Intro
Joe Percivall - @JPercivall
Hadoop Summit – Melbourne
1 Sept 2016

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About Me
• Software Engineer at Hortonworks
• Apache NiFi committer and PMC member
• Github: github.com/JPercivall

Agenda
What is dataflow and what are the challenges?
Apache NiFi
Architecture
Live Demo
Community

Agenda
Apache NiFi
Architecture
Live Demo
Community

Let’s Connect A to B
Producers A.K.A Things
Anything
AND
Everything
Internet!
Consumers
• User
• Storage
• System
• …More Things

Moving data effectively is hard
Standards: http://xkcd.com/927/

Why is moving data effectively hard?
 Standards
 Formats
 “Exactly Once” Delivery
 Protocols
 Veracity of Information
 Validity of Information
 Ensuring Security
 Overcoming Security
 Compliance
 Schemas
 Consumers Change
 Credential Management
 “That [person|team|group]”
 Network
 “Exactly Once” Delivery

Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs
Let’s consider the needs of a courier service
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/

Great! I am collecting all this data! Let’s use it!
Finding our needles in the haystack
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center
Kafka
Core Data Center at HQ
Server Cluster
Others
Storm / Spark /
Flink / Apex
Kafka
Storm / Spark / Flink / Apex
On Delivery Routes
Trucks Deliverers

Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs
Oh, that courier service is global

Agenda
Apache NiFi
Architecture
Live Demo
Community

• Web-based User Interface for creating, monitoring,
& controlling data flows
• Directed graphs of data routing and transformation
• Highly configurable - modify data flow at runtime,
dynamically prioritize data
• Easily extensible through development of custom
components
• Data Provenance tracks data through entire system
[1] https://nifi.apache.org/
Dataflow
Apache NiFi

Apache NiFi
Key Features
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Recovery/recording
a rolling log of fine-
grained history
• Visual command and
control
• Flow templates
• Pluggable/multi-role
security*
• Designed for extension
• Clustering

Revisit: Courier service from the perspective of NiFi
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Server Cluster
Trucks Deliverers
NiFi NiFi NiFi NiFi NiFi NiFi
On Delivery Routes

Courier service from the perspective of NiFi & MiNiFi
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Server Cluster
Trucks Deliverers
Client
Libraries
Client
Libraries
MiNiFi
MiNiFi
NiFi NiFi NiFi NiFi NiFi NiFi
Client
Libraries
On Delivery Routes

Apache NiFi MiNiFi
Key Features
• Data buffering
- Backpressure
- Pressure release
- Loss tolerance
• Data provenance
grained history
• Design and Deploy
• Warm re-deploys

Visual Command and Control
vs.
Design and Deploy

Apache NiFi Managed Dataflow
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

NiFi is based on Flow Based Programming (FBP)
FBP Term NiFi Term Description
Information
Packet
FlowFile Each object moving through the system.
Black Box FlowFile
Processor
Performs the work, doing some combination of data routing, transformation,
or mediation between systems.
Bounded
Buffer
Connection The linkage between processors, acting as queues and allowing various
processes to interact at differing rates.
Scheduler Flow
Controller
Maintains the knowledge of how processes are connected, and manages the
threads and allocations thereof which all processes use.
Subnet Process
Group
A set of processes and their connections, which can receive and send data via
ports. A process group allows creation of entirely new component simply by
composition of its components.

FlowFiles & Data Agnosticism
 NiFi is data agnostic!
 But, NiFi was designed understanding that users
can care about specifics and provides tooling
to interact with specific formats, protocols, etc.
ISO 8601 - http://xkcd.com/1179/
Robustness principle
Be conservative in what you do,
be liberal in what you accept from others“

FlowFiles are like HTTP data
HTTP Data FlowFile
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g
Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT
ETag: "45b6-834-49130cc1182c0"
Accept-Ranges: bytes
Content-Length: 13
Connection: close
Content-Type: text/html
Hello world!
Standard FlowFile Attributes
Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'fileSize’ Value: '23609'
FlowFile Attribute Map Content
Key: 'filename’ Value: '15650246997242'
Key: 'path’ Value: './’
Binary Content *
Header
Content

Agenda
Apache NiFi
Architecture
Live Demo
Community

OS/Host
JVM
Flow Controller
Web Server
Processor 1 Extension N
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
OS/Host
JVM
Flow Controller
Web Server
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
NiFi Architecture 0.x line OS/Host
JVM
NiFi Cluster Manger – Request Replicator
Web Server
Master
NiFi Cluster
Manager (NCM)
OS/Host
JVM
Flow Controller
Web Server
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
Slaves
NiFi Nodes

NiFi Architecture 1.x line

NiFi Architecture – Repositories - Pass by reference
FlowFile Content Provenance
F1 C1 C1 P1 F1
Excerpt of demo flow… What’s happening inside the repositories…
BEFORE
AFTER
F2 C1 C1 P3 F2 – Clone (F1)
F1 C1 P2 F1 – Route
P1 F1 – Create

NiFi Architecture – Repositories – Copy on Write
FlowFile Content Provenance
F1 C1 C1 P1 F1 - CREATE
Excerpt of demo flow… What’s happening inside the repositories…
BEFORE
AFTER
F1 C1
F1.1 C2 C2 (encrypted)
C1 (plaintext)
P2 F1.1 - MODIFY
P1 F1 - CREATE

NiFi vs MiNiFi Java Processes
NiFi Framework
Components
MiNiFi
NiFi Framework
User Interface
Components
NiFi

NiFi Java Processes
Bootstrap
NiFi
UI
bootstrap.conf
nifi.properties
flow.xml.gzreads &
modifies
reads
reads
starts
NiFi MiNiFi

MiNiFi Java Processes
MiNiFi
Bootstrap
Configuration
Change Notifier(s)
bootstrap.conf
nifi.properties
flow.xml.gz
reads
reads
starts
config.ymltransforms
reads
into
NiFi MiNiFi

Simple Config.yml
Tail a rolling file -> Site to Site

Config Change Notifiers
 Two implementations
– RestChangeNotifier
• Http(s)
– FileChangeNotifier
 Configured in bootstrap.conf

Change notifier update
MiNiFi
Bootstrap
Configuration
Change Notifiers
1. Initial state
–Both running

MiNiFi
Bootstrap
Configuration
Change Notifiers
user creates new configuration
2. User sends update through
notifier
–HTTP(S) post request
–Change watched file

MiNiFi
Bootstrap
Configuration
Change Notifiers
3. Bootstrap validation
–Basic validation
–Rest notifier will respond
accordingly
–Results logged
validate new configuration

MiNiFi
Bootstrap
Configuration
Change Notifiers
config.yml
saves new
4. Bootstrap saves and
transforms
–Copy old config.yml to a
swap file
nifi.properties
flow.xml.gz
transforms into

MiNiFi
Bootstrap
Configuration
Change Notifiers
nifi.properties
flow.xml.gz
attempt restart
config.yml
saves new
reads
transforms into
5. Bootstrap attempts restart
–MiNiFi reads in the new
nifi.properties and
flow.xml.gz

6. Success or Fail
–Successful restart continue
processing
–Failure, rollback to old
config
–Existing Data is mapped or
orphaned
MiNiFi
Bootstrap
Configuration
Change Notifiers
nifi.properties
flow.xml.gz
attempt restart
config.yml
saves new
reads
transforms into

Agenda
Apache NiFi
Architecture
Demo
Community

Matured at NSA 2006-2014
Brief history of the Apache NiFi Community
• Contributors from Government and several commercial industries
• Releases on a 6-8 week schedule
Code developed
at NSA
2006
Today
Achieved TLP
status in just
7 months
July 2015
Code available
open source
ASL v2
November 2014

MiNiFi Prospective Plans - Centralized Command and Control
 Design at a centralized place, deploy on the edge
– Flow deployment
– NAR deployment
– Agent deployment
 Version control of flows
 Agent status monitoring
 Bi-directional command and control
Centralized management console with a UI

Hortonworks Dataflow
Ingestion
Simple Event Processing
Engine
Stream Processing
DestinationData Bus

Questions?

Learn more and join us!
Apache NiFi site
http://nifi.apache.org
Subproject MiNiFi site
http://nifi.apache.org/minifi/
Subscribe to and collaborate at
dev@nifi.apache.org
users@nifi.apache.org
Submit Ideas or Issues
https://issues.apache.org/jira/browse/NIFI
Follow us on Twitter
@apachenifi

Thank you!

Apache NiFi Crash Course Intro

Recommended

More Related Content

What's hot (20)

Similar to Apache NiFi Crash Course Intro (20)

More from DataWorks Summit/Hadoop Summit (20)

Recently uploaded (20)

Apache NiFi Crash Course Intro

Editor's Notes