Unconference Round Table Notes

Real-Time Streaming
Processing Round Table
Timothy Spann
Principal Developer Advocate

FLiPN-FLaNK Stack
Tim Spann
@PaasDev // Blog: www.datainmotion.dev
Principal Developer Advocate. Princeton Future of Data Meetup.
https://github.com/tspannhw/EverythingApacheNiFi
https://medium.com/@tspann
Apache NiFi x Apache Kafka x Apache Flink

© 2023 Cloudera, Inc. All rights reserved. 4
Round Table Discussion
• Current trends of real-time stream processing in 2023
• Current challenges of real-time stream processing and proposed
solutions
• Performance / Benchmarking real-time stream processing
• The future of real-time stream processing

Current trends of real-time stream processing in 2023
• Lightweight serverless
• Hazelcast
• SQL
• Flink
• Kafka or Pulsar as Messaging Hub
• Java 17+
• Managed Clusters, Containers and Environments
• Real-Time Analytics
• Fast Storage Options

Current challenges of real-time stream processing and proposed
solutions
• Deployment, Automation and Scaling
• Choosing right project/sizing for use case
– Simple Event Processing vs Complex Event Processing
– Leveraging existing applications
• Developer Skills
• Self management and monitoring
• Cost issues -> autoscaling, optimizing, performance, hybrid
deployment

Performance / Benchmarking real-time stream processing
• Kafka/Pulsar: https://openmessaging.cloud/docs/benchmarks/
• NiFi: https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
• Flink: https://github.com/ververica/flink-sql-benchmark
• Hazelcast:
https://hazelcast.com/press-release/hazelcast-demonstrates-cloud-efficiency-r
eal-time-stream-processing-of-one-billion-events-per-second/

The future of real-time stream processing
• WASM (Web Assembly)
• Petabyte, 5000 Node Clusters, Smart Hyper Scaling
• Multi-language support (Python, Rust, Kotlin, Golang, Carbon, JVM)
• Machine Learning, Deep Learning, AI and Advanced Math
• Low Code Development like Apache NiFi, DataFlow Designer, SQL
• Dynamic Hybrid Deployment
• Citizen Stream Engineer
• Edge Streaming and Hybrid Edge Streaming
• Java 20, 21; Java Loom Virtual Threading
• Ultra low latency, trillions of events per second, massive RAM/network

Round Table Resources
• https://dzone.com/articles/real-time-stream-processing-with-hazelcast-and-
streamnative
• https://ﬂipstackweekly.com/
• https://www.datainmotion.dev/
• https://www.ﬂankstack.dev/
• https://github.com/tspannhw
• https://medium.com/@tspann
• https://medium.com/@tspann/predictions-for-streaming-in-2023-ad4d7395
d714
• https://www.apachecon.com/acna2022/slides/04_Spann_Tim_Citizen_Strea
ming_Engineer.pdf

https://github.com/tspannhw https://www.datainmotion.dev/

11
CONNECTED DEVICES ARE EVERYWHERE
EDGE
DATA
CENTER
Capture data from all
these sources and
scale with a data
streaming platform in a
hybrid architecture
Miles driven Wear-out
tires
Wearing of
the doors
Engine wear
Rising temperature
Data capacity Compute speed

12
TODAY’S NEEDS FOR DATA STREAMING
Gain Competitive Advantage
“Many leading enterprises realize that real-time analytics — the analytics of the present — is an
incredible competitive advantage because they can act now to serve ﬁckle customers, ﬁx
operational problems, power internet-of-things (IoT) apps, and respond decisively to competitors.”
Forrester
Supply chain
impacts
manufacturing
Predict
customer
buying pattern
Utilities
prevent power
outage
Telecoms deliver
continuous
QoS
Reduce
cyber
threats

Source
NiFi Kafka
Flink
SQL
Real Time
Dashboard
Flink Stream SQL
Streaming
SQL
Source
Source
Data
Collection
Central
Cache
Real Time
ETL
Real-time Analytic
Real Time
OLAP

FLaNK Stack
MiNiFi
Agent
https://flankstack.dev/

17
Buffer Analyze Connect
Analytics Apps
Apache Kafka
gateway-west-
raw-sensors
gateway-central-
raw-sensors
C++ agent
US-Central Plants
C++ agent
US-East Plants
Data-at-Rest
opDB
CDP
S3 Azure
Streaming
Analytics Powered
by FLINK
OLAP Analytics /
Time series with
Kudu
Ingest Gateway
Powered by Kafka
US-West Plants
C++ agent
Model Scoring
Powered by CML
Data-at-Rest
Cloud Storage
Data Lake
Operational Stores
Data Collection
at the Edge
Collect
Apache NiFi & MiNiFi
Distribute
Apache NiFi
Data Flow Apps
Powered by NIFI
A DATA-IN-MOTION REFERENCE ARCHITECTURE

19
CLOUDERA DATAFLOW DATA-IN-MOTION PLATFORM

20
Connect to any data source anywhere, process, and deliver to any
destination
CLOUDERA DATAFLOW
Solve the First Mile Data Collection Problem
Process
Route
Filter
Enrich
Transform
Distribute
Connectors
Any
destination
Deliver
Ingest
Active
Passive
Connectors
Gateway
Endpoint
Connect & Pull
Send
Data born in
the cloud
Data born
outside the
cloud

CLOUDERA FLOW MANAGEMENT - POWERED BY APACHE NiFi
Ingest and manage data from edge-to-cloud using a no-code interface
● #1 data ingestion/movement engine
● Strong community
● Product maturity over 11 years
● Deploy on-premises or in the cloud
● Over 400+ pre-built processors
● Built-in data provenance
● Guaranteed delivery
● Throttling and Back pressure

22
Cloudera Flow Management
Ingest and manage data from edge-to-cloud using a no-code interface
ACQUIRE PROCESS DELIVER
• Over 300 pre-built processors
• Easy to build your own processors
• Parse, enrich & apply schema
• Filter, Split, Merge & Route
• Throttle & Backpressure
• Guaranteed delivery
• Full data provenance
• Eco-system integration
Advanced tooling to industrialize ﬂow development
(Flow Development Life Cycle)
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG
HASH
MERGE
EXTRACT
DUPLICATE
SPLIT
ROUTE TEXT
ROUTE CONTENT
ROUTE CONTEXT
CONTROL RATE
DISTRIBUTE LOAD
GEOENRICH
SCAN
REPLACE
TRANSLATE
CONVERT
ENCRYPT
TALL
EVALUATE
EXECUTE

23
Cloudera Edge Management
Manage, control, and monitor the edge for all your streaming and IoT initiatives
• Small footprint agent with MiNiFi
• Java and C++ agents
• Rich edge processors (edge collection &
processing)
• End to end lineage and security
• Central Command and Control
• Design and deploy to thousands of agents
• Edge Applications lifecycle management
• Multitenancy with Agent classes
• Native integration with other CDF services
+
A lightweight edge agent that implements the core
features of Apache NiFi, focusing on data collection
and processing at the edge
Edge Flow Manager
Flow
Authorship
Flow
Deployment
Flow
Monitoring

24
Comprehensive Streams Messaging
Extend streams messaging services for Schema Mgmt, Replication & Monitoring
Schema Registry
Kafka Schema Governance
Streams Replication Manager
Kafka Replication Service powered by
MirrorMaker2
Streams Messaging Manager
Management & Monitoring Service
for all of your Kafka clusters

25
Next Generation Stream Processing & Analytics
Low latency stateful stream processing
● Advanced features - late arriving data,
checkpointing, event time processing,
Exactly Once Processing
● We support Apache Flink along with other
stream processing engines like Kafka
Streams and Spark Structured Streaming.
Real-Time
Insights
Event
Processing
Low
Latency

26
KEY DIFFERENTIATORS
Comprehensive streaming platform – Only vendor to offer a open and comprehensive streaming
platform for real-time data ingestion and processing to produce prescriptive and predictive analytics
Stream to Cloud – Extend the same on-premises streaming capabilities to the cloud with full support
for multi-cloud and hybrid cloud models
400+ pre-built processors – Only product to offer such comprehensive connectivity to a wide range of
data sources from edge to cloud
Enterprise-Grade Security & Governance – Deploy your streaming applications with conﬁdence and
trust with Cloudera SDX offering uniﬁed security and governance across the entire platform
Democratize access to real-time data – Enable data analysts and other personas to quickly build
streaming applications with just SQL

Development & Runtime of DataFlow Functions
Step1. Develop functions on
local workstation or in CDP
Public Cloud using no-code,
UI designer
Step 2. Run functions on
serverless compute
services in AWS, Azure &
GCP
AWS Lambda Azure Functions Google Cloud Functions

DataFlow Functions Use Cases
Trigger Based, Batch, Scheduled and Microservice Use Cases
Serverless Trigger-Based
File Processing Pipeline
Develop & run data processing pipelines when
files are created or updated in any of the cloud
object stores
Example: When a photo is uploaded to object
storage, a data flow is triggered which runs image
resizing code and delivers resized image to
different locations.
Serverless Workflows /
Orchestration
Chain different low-code functions to build
complex workflows
Example: Automate the handling of support
tickets in a call center or orchestrate data
movement across different cloud services.
Serverless
Scheduled Tasks
Develop and run scheduled tasks without any
code on pre-defined timed intervals
Example: Offload an external database running
on-premises into the cloud once a day every
morning at 4:00 a.m.
Serverless
Microservices
Build and deploy serverless independent modules
that power your applications microservices
architecture
Example: Event-driven functions for easy
communication between thousands of decoupled
services that power a ride-sharing application.
Serverless
Web APIs
Easily build endpoints for your web applications
with HTTP APIs without any code using DFF and
any of the cloud providers' function triggers
Example: Build high performant, scalable web
applications across multiple data centers.
Serverless
Customized Triggers
With the DFF State feature, build flows to create
customized triggers allowing access to
on-premises or external services
Example: Near real time offloading of files from a
remote SFTP server.

29
Flow Catalog
• Central repository for flow
definitions
• Import existing NiFi flows
• Manage flow definitions
• Initiate flow deployments

30
ReadyFlows
• Cloudera provided flow
definitions
• Cover most common data flow
use cases
• Can be deployed and adjusted
as needed
• Made available through docs
during Tech Preview

31
Deployment
Wizard
• Turns flow definitions into flow
deployments
• Guides users through providing
required configuration
• Pick from pre-defined NiFi
node sizes
• Define KPIs for the deployment
Start Deployment Wizard Provide Parameters
Configure Sizing & Scaling Define KPIs

32
Key Performance
Indicators
• Visibility into flow deployments
• Track high level flow
performance
• Track in-depth NiFi component
metrics
• Defined in Deployment Wizard
• Monitoring & Alerts in
Deployment Details
KPI Definition in Deployment Wizard KPI Monitoring

33
Dashboard
• Central Monitoring View
• Monitors flow deployments
across CDP environments
• Monitors flow deployment
health & performance
• Drill into flow deployment to
monitor system metrics and
deployment events

34
DATA FLOW
DESIGN FOR
EVERYONE
• Cloud-native data ﬂow
development
• Developers get their own
sandbox
• Start developing ﬂows without
installing NiFi
• Redesigned visual canvas
• Optimized interaction patterns
• Integration into CDF-PC Catalog
for versioning

Unconference Round Table Notes

Recommended

More Related Content

Similar to Unconference Round Table Notes (20)

More from Timothy Spann (20)

Recently uploaded (20)

Unconference Round Table Notes