Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
2 likes1,372 views
The document discusses intelligent data collection at the edge using Apache NiFi and Minifi, including challenges in dataflow, especially for IoT devices. It highlights key features of Apache NiFi, its role in managing data flow, and the necessity for Minifi to operate in limited environments. The presentation emphasizes the need for effective data management tools to address various challenges in data movement, storage, and security.
![© Hortonworks Inc. 2011 – 2017. All Rights Reserved7
Why is moving data effectively hard?
⬢ Standards
⬢ Formats
⬢ “Exactly Once” Delivery
⬢ Protocols
⬢ Veracity of Information
⬢ Validity of Information
⬢ Ensuring Security
⬢ Overcoming Security
⬢ Compliance
⬢ Schemas
⬢ Consumers Change
⬢ Credential Management
⬢ “That [person|team|group]”
⬢ Network*
⬢ “Exactly Once” Delivery](https://image.slidesharecdn.com/dwssydney2017intelligentlycollectingdataattheedge-pdf-170927171349/85/Intelligently-Collecting-Data-at-the-Edge-Intro-to-Apache-MiNiFi-7-320.jpg)
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
- 5. © Hortonworks Inc. 2011 – 2017. All Rights Reserved5 Let’s Connect A to B Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 7. © Hortonworks Inc. 2011 – 2017. All Rights Reserved7 Why is moving data effectively hard? ⬢ Standards ⬢ Formats ⬢ “Exactly Once” Delivery ⬢ Protocols ⬢ Veracity of Information ⬢ Validity of Information ⬢ Ensuring Security ⬢ Overcoming Security ⬢ Compliance ⬢ Schemas ⬢ Consumers Change ⬢ Credential Management ⬢ “That [person|team|group]” ⬢ Network* ⬢ “Exactly Once” Delivery
- 10. © Hortonworks Inc. 2011 – 2017. All Rights Reserved10 Apache NiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine-grained history • Visual command and control • Flow templates • Pluggable, multi-tenant security • Designed for extension • Clustering
- 11. © Hortonworks Inc. 2011 – 2017. All Rights Reserved11 FlowFiles are like HTTP data HTTP Data FlowFile HTTP/1.1 200 OK Date: Sun, 10 Oct 2010 23:26:07 GMT Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT ETag: "45b6-834-49130cc1182c0" Accept-Ranges: bytes Content-Length: 13 Connection: close Content-Type: text/html Hello world! Standard FlowFile Attributes Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'fileSize’ Value: '23609' FlowFile Attribute Map Content Key: 'filename’ Value: '15650246997242' Key: 'path’ Value: './’ Binary Content * Header Content
- 12. © Hortonworks Inc. 2011 – 2017. All Rights Reserved12 User Interface Less of this… … more of this
- 15. © Hortonworks Inc. 2011 – 2017. All Rights Reserved15 Data Provenance ▪ Constrained ▪ High-latency ▪ Localized context ▪ Hybrid – cloud/on-premises ▪ Low-latency ▪ Global context Origin – attribution Replay – recovery Evolution of topologies Long retention Types of Lineage • Event • Configuration
- 17. © Hortonworks Inc. 2011 – 2017. All Rights Reserved17 IoT Challenges ⬢ Limited computing capability ⬢ Limited power/network ⬢ Restricted software library/platform availability ⬢ No UI ⬢ Physically inaccessible ⬢ Not frequently updated ⬢ Competing standards/protocols ⬢ Scalability ⬢ Privacy & Security
- 19. © Hortonworks Inc. 2011 – 2017. All Rights Reserved19 NiFi Solves Everything* ⬢ Runs on JVM ⬢ Provides UI for flow design & monitoring ⬢ Security built-in ⬢ TLS, authn/authz, encrypted data ⬢ Handles practically any format/protocol
- 21. © Hortonworks Inc. 2011 – 2017. All Rights Reserved21 So Why Do We Need A Different Solution? ⬢ NiFi is designed to “own the box” ⬢ NiFi 0.7.x started up in about 10-15 minutes on RP3 (593 MB) ⬢ NiFi 1.x started up in about 30 minutes on RP3 (760 MB) ⬢ 33 new processors ⬢ Rewrite for multi tenant authorization ⬢ Complete UI overhaul
- 23. © Hortonworks Inc. 2011 – 2017. All Rights Reserved23 Apache NiFi Subproject: MiNiFi ⬢ Get the key parts of NiFi close to where data begins and provide bidirectional communication ⬢ NiFi lives in the data center — give it an enterprise server or a cluster of them ⬢ MiNiFi lives as close to where data is born and is a guest on that device or system ⬢ IoT ⬢ Connected car ⬢ Legacy hardware
- 24. © Hortonworks Inc. 2011 – 2017. All Rights Reserved24 Why build MiNiFi? ⬢ NiFi is big ⬢ 1.3.0 release is 933 MB compressed ⬢ Can be modified to run in restricted environments, but requires manual surgery ⬢ Provides UI, provenance query, etc. ⬢ Runs on dedicated machines/clusters — “owns the box” ⬢ MiNiFi lives at the edge ⬢ No UI ⬢ 0.2.0 Java binary is 59 MB, C++ binary is 1193 KB ⬢ “Good guest”
- 25. © Hortonworks Inc. 2011 – 2017. All Rights Reserved25 How Does MiNiFi Interact With NiFi? ⬢ NiFi ⬢ Design flows ⬢ Aggregate data from many sources ⬢ Perform routing/analysis/ SEP ⬢ MiNiFi ⬢ Receive flows ⬢ Collect data ⬢ Send for processing
- 27. © Hortonworks Inc. 2011 – 2017. All Rights Reserved27 Flavors of MiNiFi ⬢ MiNiFi Java (v0.2.0) ⬢ Modified version of NiFi ⬢ No UI ⬢ YAML configuration ⬢ Reduced processor count ⬢ 110 by default, more available with additional NARs ⬢ MiNiFi C++ (v0.2.0) ⬢ Written from scratch ⬢ 10 processors by default ⬢ Bi-directional site-to-site & provenance data
- 29. © Hortonworks Inc. 2011 – 2017. All Rights Reserved29 What does MiNiFi provide? ⬢ Data tagging/provenance ⬢ Governance from edge (geopolitical restrictions) ⬢ Security (encryption, certificate-based authentication) ⬢ Low latency (immediate reactions & decision-making) Connected Car Reference Platform Box Tuner + DSRC CardConnectivity Card
- 30. © Hortonworks Inc. 2011 – 2017. All Rights Reserved30 MiNiFi Exfil ⬢ Site-to-Site ⬢ NiFi protocol ⬢ Two implementations ⬢ Raw socket ⬢ HTTP(S) (Java only) ⬢ Secured with mutual authentication TLS ⬢ HTTP(S), (S)FTP, JMS, Syslog, File, Email, Process (Java only)
- 32. © Hortonworks Inc. 2011 – 2017. All Rights Reserved32 Scenario ⬢ IoT Device generating log messages ⬢ Need to encrypt data on device ⬢ Need to prioritize some data for unreliable network connectivity ⬢ Transmit data to central node ⬢ Decrypt data and analyze ⬢ Make determinations and modify live flow
- 33. © Hortonworks Inc. 2011 – 2017. All Rights Reserved33 NiFi As Test Harness/Environment ⬢ Simulate the log generation ⬢ Schedule is customizable ⬢ Script can write to dynamic location
- 34. © Hortonworks Inc. 2011 – 2017. All Rights Reserved34 Build the MiNiFi Flow ⬢ Tails a log file ⬢ Logs the raw contents (can be multiple lines in time window) ⬢ Splits into individual lines ⬢ Filters the content ⬢ Using parity of the timestamp ⬢ Prioritizes ⬢ Encrypts using AES/GCM ⬢ Exfils to remote NiFi
- 35. © Hortonworks Inc. 2011 – 2017. All Rights Reserved35 Export from NiFi to MiNiFi ⬢ Save as template from NiFi ⬢ Run $ ./bin/config.sh transform template.xml config.yml ⬢ MiNiFi flow ready to run* *Still need to set up TLS & encrypted properties
- 37. © Hortonworks Inc. 2011 – 2017. All Rights Reserved37 If We Really Have TLS, Why Encrypt? ⬢ All data transmitted over TLS is encrypted ⬢ On NiFi, automatically decrypted ⬢ Attributes visible ⬢ Content still encrypted because of EncryptContent processor ⬢ Can serve as secure route for follow-on systems
- 38. © Hortonworks Inc. 2011 – 2017. All Rights Reserved38 Process Data In NiFi ⬢ Receive the data over S2S ⬢ Log the incoming messages ⬢ Decrypt content ⬢ Log again
- 41. © Hortonworks Inc. 2011 – 2017. All Rights Reserved41 Next Steps ⬢ Window Aggregator ⬢ If >60% odd in window, switch prioritization ⬢ Encrypt with different keys for different tags & send to different follow-on systems ⬢ Spotty network? Tell MiNiFi to cache low priority and send in batches ⬢ MiNiFi rollover & pruning of monitored log ⬢ Exfil MiNiFi provenance data to NiFi
- 44. © Hortonworks Inc. 2011 – 2017. All Rights Reserved44 Why NiFi & MiNiFi? ⬢ Moving data is multifaceted in its challenges and these are present in different contexts at varying scopes – Inter vs intra, domestically, internationally ⬢ Provide common tooling and extensions that are needed but be flexible for extension – Leverage existing libraries and expansive Java ecosystem for functionality – Allow organizations to integrate with their existing infrastructure ⬢ Empower folks managing your infrastructure to make changes and reason about issues that are occurring – Data Provenance to show context and data’s journey – User Interface/Experience a key component
- 46. © Hortonworks Inc. 2011 – 2017. All Rights Reserved46 Learn more and join us Apache NiFi site https://nifi.apache.org Subproject MiNiFi site https://nifi.apache.org/minifi/ Subscribe to and collaborate at [email protected] [email protected] Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi