The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
4 likes2,834 views
Apache NiFi MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Abstract: Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love. Expected prior knowledge / intended audience: developers and data flow managers should have a passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi. Takeaways: Attendees will learn about opportunities to bring their data flow and capture closer to the "edge" -- sources of data like IoT devices, vehicles, machinery, etc. They will understand the possibilities to prioritize, filter, secure, and manipulate this data earlier in the data lifecycle to enhance their data visibility and performance. Speaker: Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
![© Hortonworks Inc. 2011–2019. All rights reserved;8
• Standards
• Formats
• Protocols
• Veracity
• Validity
• Schemas
• Partitioning/
Bundling
Data
Dataflow Challenges In 3 Categories
Infrastructure
• “Exactly Once”
Delivery
• Ensuring
Security
• Overcoming
Security
• Credential
Management
• Network
People
• Compliance
• “That [person|
team|group]”
• Consumers
Change
• Requirements
Change
• “Exactly Once”
Delivery](https://image.slidesharecdn.com/thefirstmile-190212141454/85/The-First-Mile-Edge-and-IoT-Data-Collection-with-Apache-NiFi-and-MiNiFi-8-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi (20)
Ad
More from DataWorks Summit (20)
Ad
Recently uploaded (20)
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
- 1. © Hortonworks Inc. 2011–2019. All rights reserved;1 The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi Andy LoPresto | @yolopey Sr. Member of Technical Staff at Hortonworks, Apache NiFi PMC & Committer 06 February 2019 Dataworks Summit Melbourne
- 2. © Hortonworks Inc. 2011–2019. All rights reserved;2 Acknowledgement of Country I acknowledge the Traditional Owners of the land on which we are meeting. I pay my respects to their Elders, past and present, and the Aboriginal Elders of other communities who may be here today.
- 3. © Hortonworks Inc. 2011–2019. All rights reserved;3 Gauging Audience Familiarity With NiFi “What’s a NeeFee?” No experience with dataflow No experience with NiFi “I can pick this up pretty quickly” Some experience with dataflow Some experience with NiFi “I refactored the Ambari integration endpoint to allow for mutual authentication TLS during my coffee break” Forgotten more about NiFi than most of us will ever know
- 4. © Hortonworks Inc. 2011–2019. All rights reserved;4 Agenda • What is dataflow and what are the challenges? • Apache NiFi • IoT Challenges • Apache MiNiFi • Exploration • Community • All slides provided online, so no need to transcribe
- 5. © Hortonworks Inc. 2011–2019. All rights reserved;5 What is dataflow?
- 6. © Hortonworks Inc. 2011–2019. All rights reserved;6 What is dataflow? • Moving some content from A to B • Content could be any bytes • Logs • HTTP • XML • CSV • Images • Video • Telemetry Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 7. © Hortonworks Inc. 2011–2019. All rights reserved;7 Moving data effectively is hard “Data Pipeline” https://xkcd.com/2054/
- 8. © Hortonworks Inc. 2011–2019. All rights reserved;8 • Standards • Formats • Protocols • Veracity • Validity • Schemas • Partitioning/ Bundling Data Dataflow Challenges In 3 Categories Infrastructure • “Exactly Once” Delivery • Ensuring Security • Overcoming Security • Credential Management • Network People • Compliance • “That [person| team|group]” • Consumers Change • Requirements Change • “Exactly Once” Delivery
- 9. © Hortonworks Inc. 2011–2019. All rights reserved;9 Raise your hand if you want to maintain Python scripts for the rest of your life Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs
- 10. © Hortonworks Inc. 2011–2019. All rights reserved;10 What is Apache NiFi?
- 11. © Hortonworks Inc. 2011–2019. All rights reserved;11 • Guaranteed delivery • Data buffering • Backpressure • Pressure release • Prioritized queuing • Flow specific QoS • Latency vs. throughput • Loss tolerance Key Features Apache NiFi • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine-grained history • Visual command and control • Flow templates • Pluggable, multi-tenant security • Designed for extension • Clustering
- 12. © Hortonworks Inc. 2011–2019. All rights reserved;12 Flowfiles Are Like HTTP Data HTTP Data FlowFile HTTP/1.1 200 OK Date: Sun, 10 Oct 2010 23:26:07 GMT Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT ETag: "45b6-834-49130cc1182c0" Accept-Ranges: bytes Content-Length: 13 Connection: close Content-Type: text/html Hello world! Standard FlowFile Attributes Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'fileSize’ Value: '23609' FlowFile Attribute Map Content Key: 'filename’ Value: '15650246997242' Key: 'path’ Value: './’ Binary Content * Header Content
- 13. © Hortonworks Inc. 2011–2019. All rights reserved;13 User Interface Less of this…… more of this
- 14. © Hortonworks Inc. 2011–2019. All rights reserved;14 Deeper Ecosystem Integration: 286+ Processors, 61 Controller Services Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute All Apache project logos are trademarks of the ASF and the respective projects. Fetch HTTP Syslog Email HTML Image HL7 FTP UDP XML SFTP AMQP WebSocket Parse Records Convert Records
- 15. © Hortonworks Inc. 2011–2019. All rights reserved;15 What are the IoT challenges?
- 16. © Hortonworks Inc. 2011–2019. All rights reserved;16 IoT Challenges • Limited computing capability • Limited power/network • Restricted software library/platform availability • No UI • Physically inaccessible • Not frequently updated • Competing standards/protocols • Scalability • Privacy & Security @_lennart
- 17. © Hortonworks Inc. 2011–2019. All rights reserved;17 • When the Mirai attack has its own Wikipedia page, that’s not good • Hackers stole high-roller database from casino via aquarium thermometer connected to internet (04/2018) Recent Examples
- 18. © Hortonworks Inc. 2011–2019. All rights reserved;18 • Runs on JVM • Provides UI for flow design & monitoring • Security built-in • TLS, authentication/authorization, encrypted data • Handles practically any format/protocol NiFi Solves Everything*
- 19. © Hortonworks Inc. 2011–2019. All rights reserved;19 • NiFi supports AMQP, MQTT, UDP, TCP, HTTP(S), CEF, JMS, (S)FTP, AWSIoT • With a little pruning, NiFi can run on a Raspberry Pi NiFi for IoT
- 20. © Hortonworks Inc. 2011–2019. All rights reserved;20 • NiFi is designed to “own the box” • NiFi 0.7.x started up in about 10-15 minutes on RP3 (593 MB) • NiFi 1.x started up in about 30 minutes on RP3 (760 MB) • 33 new processors • Rewrite for multi tenant authorization • Complete UI overhaul So Why Do We Need A Different Solution?
- 21. © Hortonworks Inc. 2011–2019. All rights reserved;21 Enter Apache MiNiFi
- 22. © Hortonworks Inc. 2011–2019. All rights reserved;22 • Get the key parts of NiFi close to where data begins and provide bidirectional communication • NiFi lives in the data center — give it an enterprise server or a cluster of them • MiNiFi lives as close to where data is born and is a guest on that device or system • IoT • Connected car • Legacy hardware Apache NiFi Subproject: MiNiFi
- 23. © Hortonworks Inc. 2011–2019. All rights reserved;23 • NiFi is big • 1.8.0 release is 1.2 GB compressed • Can be modified to run in restricted environments, but requires manual surgery • Provides UI, provenance query, etc. • Runs on dedicated machines/clusters — “owns the box” • MiNiFi lives at the edge • No UI • 0.5.0 Java release is 67 MB, C++ release is 6.1 MB (0.2.0 fits on a floppy disk) • “Good guest” Why build MiNiFi?
- 24. © Hortonworks Inc. 2011–2019. All rights reserved;24 • MiNiFi Java (v0.5.0) • Modified version of NiFi • No UI • YAML configuration • Reduced processor count • 63+ by default, more available with additional NARs • MiNiFi C++ (v0.5.0) • Written from scratch • 33 processors by default • Bi-directional site-to-site & provenance data Flavors of MiNiFi
- 25. © Hortonworks Inc. 2011–2019. All rights reserved;25 NiFi vs MiNiFi Java Processes NiFi Framework Components MiNiFi NiFi Framework User Interface Components NiFi
- 26. © Hortonworks Inc. 2011–2019. All rights reserved;26 • NiFi • Design flows • Aggregate data from many sources • Perform routing/analysis/SEP • MiNiFi • Receive flows • Collect data • Send for processing How Does MiNiFi Interact With NiFi?
- 27. © Hortonworks Inc. 2011–2019. All rights reserved;27 • We’ve been imagining EDGE to CORE as a bi-directional linear system • Let’s expand that to the real world Let’s Add Dimensionality
- 28. © Hortonworks Inc. 2011–2019. All rights reserved;28 • Data tagging/provenance • Governance from edge (geopolitical restrictions) • Security (encryption, certificate-based authentication) • Low latency (immediate reactions & decision-making) What does MiNiFi provide? Connected Car Reference Platform Box Tuner + DSRC CardConnectivity Card
- 29. © Hortonworks Inc. 2011–2019. All rights reserved;29 MiNiFi on a Connected Car Comprehension Collection Processing / Synthesis Parse <> Listen <> CAN Bus Gateway MCU MCU MCU Ethernet / Ethernet AVB Local Interconnect Network Yet to be established protocol Listen Ethernet Listen LINListen CAN Parse CAN Parse Ethernet Parse LIN Route Transmit Execute PrioritizeFilter
- 30. © Hortonworks Inc. 2011–2019. All rights reserved;30 MiNiFi on a Connected Car
- 31. © Hortonworks Inc. 2011–2019. All rights reserved;31 • Site-to-Site • NiFi protocol • Two implementations • Raw socket • HTTP(S) • Secured with mutual authentication TLS • HTTP(S), (S)FTP, JMS, Syslog, File, Email, Process MiNiFi Exfil
- 32. © Hortonworks Inc. 2011–2019. All rights reserved;32 Edge Data Exploration
- 33. © Hortonworks Inc. 2011–2019. All rights reserved;33 • IoT Device generating log messages • Need to encrypt data on device • Need to prioritize some data for unreliable network connectivity • Transmit data to central node • Decrypt data and analyze • Make determinations and modify live flow Scenario
- 34. © Hortonworks Inc. 2011–2019. All rights reserved;34 • Simulate the log generation • Schedule is customizable • Script can write to dynamic location NiFi As Test Harness/Environment
- 35. © Hortonworks Inc. 2011–2019. All rights reserved;35 • Tails a log file • Logs the raw contents (can be multiple lines in time window) • Splits into individual lines • Filters the content • Using parity of the timestamp • Prioritizes • Encrypts using AES/GCM • Exfils to remote NiFi Build the MiNiFi Flow
- 36. © Hortonworks Inc. 2011–2019. All rights reserved;36 • Save as template from NiFi • Run $ ./bin/config.sh transform template.xml config.yml • MiNiFi flow ready to run* *Still need to set up TLS & encrypted properties Export from NiFi to MiNiFi
- 37. © Hortonworks Inc. 2011–2019. All rights reserved;37 • NiFi TLS Toolkit makes certificates & keystores simple (and secure) • Copy encrypted property value from flow.xml.gz to config.yml (flow repo) Setting Up Crypto
- 38. © Hortonworks Inc. 2011–2019. All rights reserved;38 • All data transmitted over TLS is encrypted • On NiFi, automatically decrypted • Attributes visible • Content still encrypted because of EncryptContent processor • Can serve as secure route for follow-on systems If We Really Have TLS, Why Encrypt?
- 39. © Hortonworks Inc. 2011–2019. All rights reserved;39 • Receive the data over S2S • Log the incoming messages • Decrypt content • Log again Process Data In NiFi
- 40. © Hortonworks Inc. 2011–2019. All rights reserved;40 Does It Work?
- 41. © Hortonworks Inc. 2011–2019. All rights reserved;41 • Increase the write frequency • Check that newer records (within tail window) with higher priority arrive first Prioritization?
- 42. © Hortonworks Inc. 2011–2019. All rights reserved;42 • Window Aggregator • If >60% odd in window, switch prioritization • Encrypt with different keys for different tags & send to different follow-on systems • Spotty network? Tell MiNiFi to cache low priority and send in batches • MiNiFi rollover & pruning of monitored log • Exfil MiNiFi provenance data to NiFi Next Steps
- 43. © Hortonworks Inc. 2011–2019. All rights reserved;43 Community
- 44. © Hortonworks Inc. 2011–2019. All rights reserved;44 Community Example • Jeremy Dyer • Alexa + MiNiFi + Dyer 2.0 http://www.opensourcedad.com/apache/minifi-cpp/2016/12/18/poop-scale.html
- 45. © Hortonworks Inc. 2011–2019. All rights reserved;45 What’s Next?
- 46. © Hortonworks Inc. 2011–2019. All rights reserved;46 • NiFi 1.8.0 — 26 Oct 2018 (212+ Jiras) • Jetty, DB improvements • Auto load-balancing queues • TLS Toolkit w/ external CA • Record processor improvements • MiNiFi C++ 0.5.0 — 6 June 2018 • MiNiFi Java 0.5.0 — 7 July 2018 • NiFi Registry 0.3.0 — 25 Sept 2018 Introducing Apache NiFi Registry New Announcements
- 47. © Hortonworks Inc. 2011–2019. All rights reserved;47 • Previously, flows were exported via XML templates • Didn’t contain sensitive values • Couldn’t be updated in-place • No tracking system • NiFi Registry brings asset management as first-class citizen to NiFi • Flows can be versioned • Flows can be promoted between environments Introducing Apache NiFi Registry 0.3.0 NiFi Registry for Dataflows
- 48. © Hortonworks Inc. 2011–2019. All rights reserved;48 Community Health
- 49. © Hortonworks Inc. 2011–2019. All rights reserved;49 Apache NiFi site https://nifi.apache.org Subproject MiNiFi site https://nifi.apache.org/minifi/ Subscribe to and collaborate at [email protected] [email protected] Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi Learn more and join us
- 50. © Hortonworks Inc. 2011–2019. All rights reserved;50 More NiFi Today Title Time Room The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi 1100 - 1140 Room 103 Apache NiFi Crash Course 1400 - 1600 Room 109 Dataflow Management From Edge to Core with Apache NiFi 1650 - 1730 Room 112 Using Spark Streaming and NiFi for the Next Generation of ETL in the Enterprise 1650 - 1730 Room 103
- 51. © Hortonworks Inc. 2011–2019. All rights reserved;51 Thank you [email protected] | [email protected] | @yolopey github.com/alopresto/slides