Apache Kafka Streams + Machine Learning / Deep Learning
64 likes35,745 views
This document discusses applying machine learning models to real-time stream processing using Apache Kafka. It covers building analytic models from historical data, applying those models to real-time streams without redevelopment, and techniques for online training of models. Live demos are presented using open source tools like Kafka Streams, Kafka Connect, and H2O to apply machine learning to streaming use cases like flight delay prediction. The key takeaway is that streaming platforms can leverage pre-built machine learning models to power real-time analytics and actions.
Apache Kafka Streams + Machine Learning / Deep Learning
- 1. 1Confidential Apache Kafka + Machine Learning Analytic Models Applied to Real Time Stream Processing Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.kai-waehner.de
- 2. 2Apache Kafka and Machine Learning Agenda 1) Machine Learning in the Real World 2) Building an Analytic Model 3) Applying an Analytic Model in Real Time 4) Online Training of Models
- 3. 3Apache Kafka and Machine Learning Agenda 1) Machine Learning in the Real World 2) Building an Analytic Model 3) Applying an Analytic Model in Real Time 4) Online Training of Models
- 4. 4Apache Kafka and Machine Learning Machine Learning ... allows computers to find hidden insights without being explicitly programmed where to look.
- 5. 5Apache Kafka and Machine Learning Real World Examples of Machine Learning Spam Detection Search Results + Product Recommendation Picture Detection (Friends, Locations, Products) Your Company The Next Disruption: Google Beats Go Champion
- 6. 6Apache Kafka and Machine Learning Leverage Machine Learning to Analyze and Act on Critical Business Moments Seconds Minutes Hours Price Optimization Predictive Maintenance Fraud Detection Cross Selling Transportation Rerouting Customer Service Inventory Management Windows of Opportunity
- 7. 7Apache Kafka and Machine Learning How to realize these use cases?
- 8. 8Apache Kafka and Machine Learning Big Data Analytics Volume (terabytes, petabytes) Variety (social networks, blog posts, logs, sensors, etc.) Velocity („real time“) Value
- 9. 9Apache Kafka and Machine Learning Big Data Analytics for Actionable Insights From Insight to Action (continuously closed loop)
- 10. 10Apache Kafka and Machine Learning Streaming Platform Big Data Analytics Database IoT Device Streaming Producer ….. DWH Data Integration C O N N E C T C O N N E C T Data Lake Model Building Batch Real Time Stream Processing REST Interface IoT Device Mobile App Streaming Consumer C O N N E C T C O N N E C T BI Tool Messaging Web Application Model Schema Registry / Governance 1) Data Producer 2) Analytics Platform 3) Streaming Platform 4) Data Consumer
- 11. 11Apache Kafka and Machine Learning Agenda 1) Machine Learning in the Real World 2) Building an Analytic Model 3) Applying an Analytic Model in Real Time 4) Online Training of Models
- 12. 12Apache Kafka and Machine Learning Streaming Platform Big Data Analytics Database IoT Device Streaming Producer ….. DWH Data Integration C O N N E C T C O N N E C T Data Lake Model Building Batch Real Time Stream Processing REST Interface IoT Device Mobile App Streaming Consumer C O N N E C T C O N N E C T BI Tool Messaging Web Application Model Schema Registry / Governance 1) Data Producer 2) Analytics Platform 3) Streaming Platform 4) Data Consumer
- 13. 13Apache Kafka and Machine Learning Hidden Technical Debt in Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf Writing source code is not the time-consuming task! !
- 14. 14Apache Kafka and Machine Learning Analytical Pipeline 1. Data Access 2. Data Preparation 3. Exploratory Data Analysis 4. Model Building 5. Model Execution 6. Model Validation 7. Deployment
- 15. 15Apache Kafka and Machine Learning Data Access Find insights to create added business value by correlating various data sources!
- 16. 16Apache Kafka and Machine Learning Data Preparation http://www.slideshare.net/odsc/feature-engineering Data Preparation
- 17. 17Apache Kafka and Machine Learning Exploratory Data Analysis © Copyright 2000-2017 TIBCO Software Inc. • Scripting • Visual Analytics • Machine Learning
- 18. 18Apache Kafka and Machine Learning Model Building A model is a simplification of the truth that helps you with decision making.
- 19. 19Apache Kafka and Machine Learning Model Execution (Coding) Apply Model to New Data
- 20. 20Apache Kafka and Machine Learning Model Execution (Tooling) Apply Model to New Data
- 21. 21Apache Kafka and Machine Learning Model Validation https://genome.tugraz.at/proclassify/help/pages/XV.html Cross-Validation Procedure
- 22. 22Apache Kafka and Machine Learning Frameworks and Tooling?
- 23. 23Apache Kafka and Machine Learning Languages, Frameworks and Tools Many more …. Portable Format for Analytics (PFA)
- 24. 24Apache Kafka and Machine Learning Live Demos with Open Source Technologies Development of Analytic Models with R, TensorFlow, Apache Spark, H2O.ai, RapidMiner
- 25. 25Apache Kafka and Machine Learning Live Demo Use Case: Customer Churn Prediction Machine Learning Algorithm: Generalized Linear Model (GLM) using Logistic Regression Technology: Open Source R
- 26. 26Apache Kafka and Machine Learning Live Demo Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Gradient Boosted Machines (GBM) using Decision Trees Technology: H2O.ai
- 27. 27Apache Kafka and Machine Learning Live Demo Use Case: Predictive Maintenance (Anomaly Detection in Telco Networks) Deep Learning Algorithm: Artificial Neural Networks (ANN) using Autoencoders Technology: TensorFlow + Python API
- 28. 28Apache Kafka and Machine Learning Live Demo Use Case: Classification (Prediction of Titanic Survivors) Deep Learning Algorithm: Recurrent Neural Networks (RNN) Technology: RapidMiner
- 29. 29Apache Kafka and Machine Learning Agenda 1) Machine Learning in the Real World 2) Building an Analytic Model 3) Applying an Analytic Model in Real Time 4) Online Training of Models
- 30. 30Apache Kafka and Machine Learning Analytical Pipeline 1. Data Access 2. Data Preparation 3. Exploratory Data Analysis 4. Model Building 5. Model Execution 6. Model Validation 7. Deployment
- 31. 31Apache Kafka and Machine Learning Streaming Platform Big Data Analytics Database IoT Device Streaming Producer ….. DWH Data Integration C O N N E C T C O N N E C T Data Lake Model Building Batch Real Time Stream Processing REST Interface IoT Device Mobile App Streaming Consumer C O N N E C T C O N N E C T BI Tool Messaging Web Application Model Schema Registry / Governance 1) Data Producer 2) Analytics Platform 3) Streaming Platform 4) Data Consumer
- 32. 32Apache Kafka and Machine Learning Definition of Stream Processsing Data at Rest Data in Motion
- 33. 33Apache Kafka and Machine Learning Key Concepts
- 34. 34Apache Kafka and Machine Learning Key Concepts
- 35. 35Apache Kafka and Machine Learning Key Concepts
- 36. 36Apache Kafka and Machine Learning Stream Processing Use Cases • Real Time Applications • Stateful Streaming Analytics • Stateless “Real Time ETL”
- 37. 37Apache Kafka and Machine Learning Event Processing Windows Various Options for Windowing (Fixed, Sliding, Session, …)
- 38. 38Apache Kafka and Machine Learning How to apply analytic models to real time processing without redevelopment?
- 39. 39Apache Kafka and Machine Learning Application of Analytic Models to Real Time without Redevelopment Stream Processing H20.ai R Python Spark ML MATLAB SAS PMML
- 40. 40Apache Kafka and Machine Learning Streaming Analytics - Processing Pipeline APIs Adapters / Channels Integration Messaging Stream Ingest Transformation Aggregation Enrichment Filtering Stream Preprocessing Process Management Analytics (Real Time) Applications & APIs Analytics / DW Reporting Stream Outcomes • Contextual Rules • Windowing • Patterns • Analytics • Machine Learning • … Stream Analytics Index / SearchNormalization Applying an Analytic Model is just a piece of the puzzle!
- 41. 41Apache Kafka and Machine Learning Frameworks and Tooling?
- 42. 42Apache Kafka and Machine Learning Frameworks and Products OPEN SOURCE CLOSED SOURCE PRODUCT FRAMEWORK Azure Microsoft Stream Analytics
- 43. 43Apache Kafka and Machine Learning When to use Kafka Streams for Stream Processing?
- 44. 44Apache Kafka and Machine Learning When to use Kafka Streams for Stream Processing? No need for a Big Data cluster Deploy in your existing infrastructure Kafka manages scalability / fail-over Focus on development of business logic in your department
- 45. 45Apache Kafka and Machine Learning Kafka Streams Map, filter, aggregate, apply analytic model, „any business logic“ Input Stream (Kafka Topic) Kafka Cluster Output Stream (Kafka Topic) Kafka Cluster Stream Processing Microservice (Kafka Streams) Deployed anywhere: Docker, Kubernetes, Mesos, Java App, …
- 46. 46Apache Kafka and Machine Learning A complete streaming microservices, ready for production at large-scale Word Count App configuration Define processing (here: WordCount) Start processing
- 47. 47Apache Kafka and Machine Learning Confluent Platform: the Free, Open-Source Streaming Platform Open Source ExternalCommercial Confluent Platform Monitoring Analytics Custom Apps Transformations Real-time Applications … CRM Data Warehouse Database Hadoop Data Integration … Control Center Auto-data Balancing Multi-Data Center Replication 24/7 Support Supported Connectors Clients Schema Registry REST Proxy Apache Kafka Kafka Connect Kafka Streams Kafka Core Database Changes Log Events loT Data Web Events …
- 48. 48Apache Kafka and Machine Learning Streaming Platform Big Data Analytics Database IoT Device Streaming Producer ….. DWH Data Integration C O N N E C T C O N N E C T Data Lake Model Building Batch Real Time Stream Processing REST Interface IoT Device Mobile App Streaming Consumer C O N N E C T C O N N E C T BI Tool Messaging Web Application Model Schema Registry / Governance 1) Data Producer 2) Analytics Platform 3) Streaming Platform 4) Data Consumer
- 49. 49Apache Kafka and Machine Learning STREAMING PLATFORM BIG DATAANALYTICS Oracle DB CoaP IoT Kafka Java Client ….. HP Vertica Data Integration F L U M E H2O.ai, Spark, TensorFlow Batch Real Time Confluent REST Proxy MQTT IoT iPhone App Kafka Go Client C K O A N F N K E A C T H I V E Grafana Kafka Java EE Web App Hadoop C K O A N F N K E A C T Confluent Schema Registry Kafka Streams H2O.ai Mesos Kafka Streams TensorFlow Kubernetes Avro Avro 1) Data Producer 2) Analytics Platform 3) Streaming Platform 4) Data Consumer
- 50. 50Apache Kafka and Machine Learning Live Demos with Open Source Technologies Development of Analytic Models with Apache Kafka Messaging, Kafka Streams, Kafka Connect, Confluent Schema Registry
- 51. 51Apache Kafka and Machine Learning Live Demo Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Any! (in our example, H2O.ai GBM) Streaming Platform: Apache Kafka Core, Kafka Connect, Kafka Streams, Confluent Schema Registry
- 52. 52Apache Kafka and Machine Learning H2O.ai Model + Kafka Streams Filter Map 1) Create H2O ML model 2) Configure Kafka Streams Application 3) Apply H2O ML model to Streaming Data 4) Start Kafka Streams App
- 53. 53Apache Kafka and Machine Learning End-to-End Stream Monitoring and Alerting Confluent Control Center Data Stream Monitoring and Alerting Multi-cluster monitoring and management Kafka Connect Configuration • Message delivery? • Delays? • Where got it stuck? • Lost messages? • Broker issues? • Performance? http://docs.confluent.io/3.2.0/control-center/docs/monitoring.html
- 54. 54Apache Kafka and Machine Learning Agenda 1) Machine Learning in the Real World 2) Building an Analytic Model 3) Applying an Analytic Model in Real Time 4) Online Training of Models
- 55. 55Apache Kafka and Machine Learning Let’s improve the analytic model continuously…
- 56. 56Apache Kafka and Machine Learning Analytical Pipeline 1. Data Access 2. Data Preparation 3. Exploratory Data Analysis 4. Model Building 5. Model Execution 6. Model Validation 7. Deployment Online Training Continuously train and improve the model with every new event
- 57. 57Apache Kafka and Machine Learning Online Model Training of Analytic Models How to improve models? 1.Manual Update 2.Automated Batch 3.Real Time
- 58. 58Apache Kafka and Machine Learning STREAMING PLATFORM BIG DATAANALYTICS F L U M E H2O.ai, Spark, TensorFlow H I V E Kafka Hadoop Confluent Schema Registry Kafka Streams H2O.ai Mesos Kafka Streams TensorFlow Kubernetes Avro Avro 1) Get new Input Event via Kafka Topic 2) Improve Model in Big Data Cluster 3) Update deployed Model via Kafka Topic 4) Leverage Improved Model for new Events
- 59. 59Apache Kafka and Machine Learning Caveats for Online Model Training • Processes and infrastructure not ready • Validation needed before production • Slows down the system • Only a few ML implementations supported • Many use cases do not need it
- 60. 60Apache Kafka and Machine Learning Key Take-Aways Ø Insights are hidden in Historical Data on Big Data Platforms Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models Ø Streaming Platform uses these Models (without Redevelopment) to take Action in Real Time
- 61. 61Apache Kafka and Machine Learning Kai Waehner Technology Evangelist [email protected] @KaiWaehner www.kai-waehner.de LinkedIn Questions? Feedback? Please contact me!