Machine Learning on Streaming Data using Kafka, Beam, and TensorFlow (Mikhail Chrestkha, Google Cloud; Stephane Maarek, DataCumulus) Kafka Summit NYC 2019
3 likes2,085 views
The document discusses integrating machine learning with Apache Kafka, Apache Beam, and TensorFlow, highlighting their architecture and a use case for credit card fraud detection. It emphasizes the advantages of using managed services for data processing, model training, and deployment. Key takeaways suggest a shift from batch processing to real-time streaming, with applications spanning various industries.
Machine Learning on Streaming Data using Kafka, Beam, and TensorFlow (Mikhail Chrestkha, Google Cloud; Stephane Maarek, DataCumulus) Kafka Summit NYC 2019
- 1. Machine Learning on Streaming Data with Apache Kafka, Apache Beam, & TensorFlow
- 2. About Us Mikhail Chrestkha Machine Learning Specialist Google Cloud linkedin.com/in/mchrestkha Stéphane Maarek CEO & Kafka Instructor DataCumulus linkedin.com/in/stephanemaarek Big Thanks to: Julianne Cuneo Big Data Specialist, Google Cloud Kai Waehner Technology Evangelist, Confluent
- 3. Agenda 1. Motivation 2. Architecture 3. Use Case Walk-Through w/ Demo 4. Summary
- 4. 1 Motivation
- 5. Technology Landscape Smart Analytics Streaming InfoWorld’s 2019 Technology of the Year Award Winners: ● Apache Beam ● Apache Kafka ● Elastic Stack ● DataStax Enterprise ● Firebase ● Horovod ● H2O Driverless AI ● Keras ● Kubernetes ● LLVM ● .Net Core ● PyTorch ● Redis ● TensorFlow ● Visual Studio Code ● XGBoost Cloud ? https://www.globenewswire.com/news-release/2019/01/30/1707685/0/en/InfoWorld-Announces-2019-Technology-of-the-Year-Award-Winners.html
- 6. Data Ingestion Data Analysis & Transformation Trainer Model Evaluation & Validation Serving Notebook Orchestration ML Framework ML Platform
- 7. OSS Managed Service Apache Kafka Event streaming platform Confluent Cloud Monitoring, Replication, Data Balancing Apache Beam Data processing pipelines Unified batch & streaming Dataflow Automated resource management of workers TensorFlow Robust foundation for machine and deep learning Cloud Machine Learning Engine ● Training: Distributed training infrastructure that supports CPUs, GPUs, and TPUs ● Serving: Host models for batch & online prediction
- 9. Reference Kafka ML Architecture ● Data pipelines are simplified ● Building analytic modules is decoupled from servicing them ● Usage of real time or batch as needed ● Analytic models can be deployed in a performant, scalable and mission-critical environment Kai Waehner Technology Evangelist, Confluent https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/
- 10. Confluent Cloud Managed by Confluent Analytics, ML training & deployment path ML serving path Data warehouse BigQuery ML Training Cloud ML Engine Topic 1 Raw transaction Topic 2 Predictions Kafka Cluster Processing Cloud Dataflow Leverage managed services to simplify & focus on code not infrastructure Producer Consumer Consumer ML Notebook KSQL SQL Submit ML Training jobs ML Serving Cloud ML Engine ML notebook development / experimentation Deploy ML model Automate w/ AirFlow Dataflow Template
- 11. 3 Use Case Walk-Through
- 12. Kaggle Case Study Fraud Detection of Credit Card Transactions ● Collect transaction data ● Analyze historical data ● Train model on historic sample ● Evaluate model based on precision & recall ● Predict fraud on new streaming data 492 Fraud (0.172%) 284,807 transactions ● Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015 ● Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon ● Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE ○ Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi) ● Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-Aël; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier ● Carcillo, Fabrizio; Le Borgne, Yann-Aël; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing https://opendatacommons.org/licenses/dbcl/1.0/
- 13. DEMO 1 - 5 min Sending our credit card data Confluent Cloud, Creating a Topic, Python Script, Security
- 14. Kafka to BigQuery Dataflow Template Java Code KafkaIO.<String, String>read() BigQueryIO.writeTableRows() Create a template for easy re-usability by an analyst Images from https://beam.apache.org/documentation/pipelines/design-your-pipeline/ redacted
- 15. Explore data & train ML model from ksql import KSQLAPI redacted %%bigquery redacted gcloud ml-engine jobs submit training redacted Query directly from topic Query petabytes of data Submit ML training job
- 16. DEMO 2 - 5 min Dataflow template & job Jupyter: KSQL, BQML, TensorFlow CMLE job
- 17. Send Predictions back to Kafka Java Code Cloud Machine Learning Engine Request Response Hosted ML Model Image from https://beam.apache.org/documentation/pipelines/design-your-pipeline/ Publish models KafkaIO.<String, String>read() KafkaIO.<String, String>write() Train Model
- 18. DEMO 3 - 5 min (1) Deploy model as an end point (2) Prediction sent to Kafka topic to be consumed (3) Track models & monitor predictions in CMLE UI
- 19. Futuristic Architecture: Pure Kafka-based ML Resilient, highly available, sync & async Confluent Cloud Managed by Confluent Topic 1 Raw transaction Topic 2 Predictions Producer Consumer Consumer Kafka Streams ML Synchronous Application training serving Interactive Query API gRPC or REST API Internal Topic ML Model (compacted?) Model state
- 20. 4 Summary
- 21. Summary ● Kafka + Beam + TensorFlow = Great foundation for future ○ Batch today → streaming tomorrow ○ Small data → big data tomorrow ○ Shallow learning today → deep learning tomorrow ● Make data & ML easier for yourself by using managed services ● Build for many other use cases: ○ Predictive maintenance ○ Logistics routing ○ Image search & recommendations in e-commerce Smart AnalyticsStreaming Cloud
- 22. Talk to Google Cloud K1Booth Learn More Blog: Enabling connected transformation with Apache Kafka and TensorFlow on Google Cloud Platform bit.ly/2CHERol KafkaIO on Beam bit.ly/2YwL3Jc KafkaToBigQuery Dataflow Template Example bit.ly/2HQqVN0 Contact us linkedin.com/in/mchrestkha linkedin.com/in/stephanemaarek
- 23. Confluent Cloud Managed by Confluent Analytics, ML training & deployment path ML serving path Data warehouse BigQuery ML Training Cloud ML Engine Topic 1 Raw transaction Topic 2 Predictions Kafka Cluster Processing Cloud Dataflow Questions Producer Consumer Consumer Cloud ML Notebook KSQL SQL Submit ML Training jobs ML Serving Cloud ML Engine ML notebook development / experimentation Deploy ML model Automate w/ AirFlow Dataflow Template