Intelligently collecting data at the edge—intro to Apache MiNiFi
3 likes2,082 views
The document discusses the challenges and solutions of dataflow management using Apache NiFi and MiNiFi, focusing on data collection at the edge, particularly for IoT applications. It highlights the capabilities of NiFi for data processing and how MiNiFi can operate in resource-constrained environments close to data sources. Additionally, it introduces the Apache NiFi registry for versioning dataflows and outlines community contributions and upcoming features.
![© Hortonworks Inc. 2011–2018. All rights reserved6
• Standards
• Formats
• Protocols
• Veracity
• Validity
• Schemas
• Partitioning/
Bundling
Data
Dataflow Challenges In 3 Categories
Infrastructure
• “Exactly Once”
Delivery
• Ensuring
Security
• Overcoming
Security
• Credential
Management
• Network
People
• Compliance
• “That [person|
team|group]”
• Consumers
Change
• Requirements
Change
• “Exactly Once”
Delivery](https://image.slidesharecdn.com/intelligentlycollectingdataattheedge-180516182501/85/Intelligently-collecting-data-at-the-edge-intro-to-Apache-MiNiFi-6-320.jpg)
Intelligently collecting data at the edge—intro to Apache MiNiFi
- 3. © Hortonworks Inc. 2011–2018. All rights reserved3 Agenda • What is dataflow and what are the challenges? • Apache NiFi • IoT Challenges • Apache MiNiFi • Exploration • Community • All slides provided online, so no need to transcribe
- 5. © Hortonworks Inc. 2011–2018. All rights reserved5 What is dataflow? • Moving some content from A to B • Content could be any bytes • Logs • HTTP • XML • CSV • Images • Video • Telemetry Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 6. © Hortonworks Inc. 2011–2018. All rights reserved6 • Standards • Formats • Protocols • Veracity • Validity • Schemas • Partitioning/ Bundling Data Dataflow Challenges In 3 Categories Infrastructure • “Exactly Once” Delivery • Ensuring Security • Overcoming Security • Credential Management • Network People • Compliance • “That [person| team|group]” • Consumers Change • Requirements Change • “Exactly Once” Delivery
- 9. © Hortonworks Inc. 2011–2018. All rights reserved9 • Guaranteed delivery • Data buffering • Backpressure • Pressure release • Prioritized queuing • Flow specific QoS • Latency vs. throughput • Loss tolerance Key Features Apache NiFi • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine-grained history • Visual command and control • Flow templates • Pluggable, multi-tenant security • Designed for extension • Clustering
- 10. © Hortonworks Inc. 2011–2018. All rights reserved10 Flowfiles Are Like HTTP Data HTTP Data FlowFile HTTP/1.1 200 OK Date: Sun, 10 Oct 2010 23:26:07 GMT Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT ETag: "45b6-834-49130cc1182c0" Accept-Ranges: bytes Content-Length: 13 Connection: close Content-Type: text/html Hello world! Standard FlowFile Attributes Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'fileSize’ Value: '23609' FlowFile Attribute Map Content Key: 'filename’ Value: '15650246997242' Key: 'path’ Value: './’ Binary Content * Header Content
- 15. © Hortonworks Inc. 2011–2018. All rights reserved14 IoT Challenges • Limited computing capability • Limited power/network • Restricted software library/platform availability • No UI • Physically inaccessible • Not frequently updated • Competing standards/protocols • Scalability • Privacy & Security
- 17. © Hortonworks Inc. 2011–2018. All rights reserved16 • Runs on JVM • Provides UI for flow design & monitoring • Security built-in • TLS, authentication/authorization, encrypted data • Handles practically any format/protocol NiFi Solves Everything*
- 19. © Hortonworks Inc. 2011–2018. All rights reserved18 • NiFi is designed to “own the box” • NiFi 0.7.x started up in about 10-15 minutes on RP3 (593 MB) • NiFi 1.x started up in about 30 minutes on RP3 (760 MB) • 33 new processors • Rewrite for multi tenant authorization • Complete UI overhaul So Why Do We Need A Different Solution?
- 21. © Hortonworks Inc. 2011–2018. All rights reserved20 • Get the key parts of NiFi close to where data begins and provide bidirectional communication • NiFi lives in the data center — give it an enterprise server or a cluster of them • MiNiFi lives as close to where data is born and is a guest on that device or system • IoT • Connected car • Legacy hardware Apache NiFi Subproject: MiNiFi
- 22. © Hortonworks Inc. 2011–2018. All rights reserved21 • NiFi is big • 1.6.0 release is 1.2 GB compressed • Can be modified to run in restricted environments, but requires manual surgery • Provides UI, provenance query, etc. • Runs on dedicated machines/clusters — “owns the box” • MiNiFi lives at the edge • No UI • 0.4.0 Java binary is 65 MB, C++ binary is 4.5 MB (0.2.0 fits on a floppy disk) • “Good guest” Why build MiNiFi?
- 23. © Hortonworks Inc. 2011–2018. All rights reserved22 • MiNiFi Java (v0.4.0) • Modified version of NiFi • No UI • YAML configuration • Reduced processor count • 110+ by default, more available with additional NARs • MiNiFi C++ (v0.4.0) • Written from scratch • 28 processors by default • Bi-directional site-to-site & provenance data Flavors of MiNiFi
- 25. © Hortonworks Inc. 2011–2018. All rights reserved24 • NiFi • Design flows • Aggregate data from many sources • Perform routing/analysis/SEP • MiNiFi • Receive flows • Collect data • Send for processing How Does MiNiFi Interact With NiFi?
- 27. © Hortonworks Inc. 2011–2018. All rights reserved26 • Data tagging/provenance • Governance from edge (geopolitical restrictions) • Security (encryption, certificate-based authentication) • Low latency (immediate reactions & decision-making) What does MiNiFi provide? Connected Car Reference Platform Box Tuner + DSRC CardConnectivity Card
- 28. © Hortonworks Inc. 2011–2018. All rights reserved27 MiNiFi on a Connected Car Comprehension Collection CAN Bus Gateway MCU MCU MCU Ethernet / Ethernet AVB Local Interconnect Network Yet to be established protocol Listen Ethernet Listen LINListen CAN Listen <> Parse CAN Parse Ethernet Parse LIN Parse <> Processing / Synthesis Route Transmit Execute PrioritizeFilter
- 30. © Hortonworks Inc. 2011–2018. All rights reserved29 • Site-to-Site • NiFi protocol • Two implementations • Raw socket • HTTP(S) (Java only) • Secured with mutual authentication TLS • HTTP(S), (S)FTP, JMS, Syslog, File, Email, Process (Java only) MiNiFi Exfil
- 32. © Hortonworks Inc. 2011–2018. All rights reserved31 • IoT Device generating log messages • Need to encrypt data on device • Need to prioritize some data for unreliable network connectivity • Transmit data to central node • Decrypt data and analyze • Make determinations and modify live flow Scenario
- 33. © Hortonworks Inc. 2011–2018. All rights reserved32 • Simulate the log generation • Schedule is customizable • Script can write to dynamic location NiFi As Test Harness/Environment
- 34. © Hortonworks Inc. 2011–2018. All rights reserved33 • Tails a log file • Logs the raw contents (can be multiple lines in time window) • Splits into individual lines • Filters the content • Using parity of the timestamp • Prioritizes • Encrypts using AES/GCM • Exfils to remote NiFi Build the MiNiFi Flow
- 35. © Hortonworks Inc. 2011–2018. All rights reserved34 • Save as template from NiFi • Run $ ./bin/config.sh transform template.xml config.yml • MiNiFi flow ready to run* *Still need to set up TLS & encrypted properties Export from NiFi to MiNiFi
- 37. © Hortonworks Inc. 2011–2018. All rights reserved36 • All data transmitted over TLS is encrypted • On NiFi, automatically decrypted • Attributes visible • Content still encrypted because of EncryptContent processor • Can serve as secure route for follow-on systems If We Really Have TLS, Why Encrypt?
- 38. © Hortonworks Inc. 2011–2018. All rights reserved37 • Receive the data over S2S • Log the incoming messages • Decrypt content • Log again Process Data In NiFi
- 41. © Hortonworks Inc. 2011–2018. All rights reserved40 • Window Aggregator • If >60% odd in window, switch prioritization • Encrypt with different keys for different tags & send to different follow-on systems • Spotty network? Tell MiNiFi to cache low priority and send in batches • MiNiFi rollover & pruning of monitored log • Exfil MiNiFi provenance data to NiFi Next Steps
- 46. © Hortonworks Inc. 2011–2018. All rights reserved44 • NiFi 1.6.0 — 08 April 2018 • MongoDB, InfluxDB, Druid, HBase components • Granular @Restricted components • MiNiFi C++ 0.4.0 — 27 January 2018 • MiNiFi Java 0.4.0 — 22 January 2018 • NiFi Registry 0.1.0 — 1 January 2018 Introducing Apache NiFi Registry New Announcements
- 47. © Hortonworks Inc. 2011–2018. All rights reserved45 • Previously, flows were exported via XML templates • Didn’t contain sensitive values • Couldn’t be updated in-place • No tracking system • NiFi Registry brings asset management as first-class citizen to NiFi • Flows can be versioned • Flows can be promoted between environments Introducing Apache NiFi Registry 0.1.0 NiFi Registry for Dataflows Learn more at Forget Duplicating Local Changes: Apache NiFi and the Flow Development Lifecycle (FDLC) Thursday 19/4 @ 1600, Room II
- 48. © Hortonworks Inc. 2011–2018. All rights reserved46 • Centralized C2 for classes of agents • Tag devices, platforms, capabilities • Distribute flows to edge via multiple mechanisms Command & Control for MiNiFi MiNiFi C2 Server IN DEVELOPMENT
- 50. © Hortonworks Inc. 2011–2018. All rights reserved48 Apache NiFi site https://nifi.apache.org Subproject MiNiFi site https://nifi.apache.org/minifi/ Subscribe to and collaborate at [email protected] [email protected] Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi Learn more and join us
- 51. © Hortonworks Inc. 2011–2018. All rights reserved49 More NiFi This Week… Title Room Time Speaker(s) Apache NiFi Crash Course Hall I - D 1115 - 1345 Andy LoPresto, Tim Spann IoT with Apache MXNet and Apache NiFi and MiNiFi Hall I - C 1150 - 1230 Tim Spann Best practices and lessons learnt from Running Apache NiFi at Renault Europe 1650 - 1730 Adel Gacem, Abdelkrim Hadjidj From an experiment to a real production environment Room V 1650 - 1730 Jeroen Wolffensperger, Martijn Groen IoT, Streaming, and Dataflow Birds of a Feather Room I 1740 - 1855 George Vetticaden, Davor Bonaci, Andy LoPresto, Stephan Ewen Intelligently Collecting Data at the Edge — Intro to Apache MiNiFi Room II 1100 - 1140 Andy LoPresto The Power of Intelligent Flows: Realtime IoT Botnet Classification with Apache NiFi Hall I - C 1400 - 1440 Andy LoPresto Forget Duplicating Local Changes: Apache NiFi and the Flow Development Lifecycle (FDLC) Room II 1600 - 1640 Andy LoPresto