Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFlow (Kai Waehner, Confluent) Kafka Summit SF 2019
0 likes4,810 views
Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly "at the edge".
![15Apache Kafka and Machine Learning – Kai Waehner
Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)](https://image.slidesharecdn.com/kaiwaehner-191005041926/85/Event-Driven-Model-Serving-Stream-Processing-vs-RPC-with-Kafka-and-TensorFlow-Kai-Waehner-Confluent-Kafka-Summit-SF-2019-13-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFlow (Kai Waehner, Confluent) Kafka Summit SF 2019 (20)
Ad
More from confluent (20)
Ad
Recently uploaded (20)
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFlow (Kai Waehner, Confluent) Kafka Summit SF 2019
- 1. 1Apache Kafka and Machine Learning – Kai Waehner Event-Driven Stream Processing and Model Deployment with Kafka and TensorFlow Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. 2Apache Kafka and Machine Learning – Kai Waehner Key Takeaways • Model deployment can be completely separated from model training (process and technology) • Two alternatives for model deployment: Model server vs. embedded into application • Model embedding into Kafka applications is simple and creates a robust and performant environment
- 3. 3Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 4. 4Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 5. 6Apache Kafka and Machine Learning – Kai Waehner Machine Learning (ML) ...allows computers to find hidden insights without being explicitly programmed where to look. Machine Learning • Decision Trees • Naïve Bayes • Clustering • Neural Networks • Etc. Deep Learning • CNN • RNN • Autoencoder • Etc.
- 6. 7Apache Kafka and Machine Learning – Kai Waehner The First Analytic Models How to deploy the models in production? …real-time processing? …at scale? …24/7 zero downtime?
- 7. 8Apache Kafka and Machine Learning – Kai Waehner Hidden Technical Debt in Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
- 8. 10Apache Kafka and Machine Learning – Kai Waehner Scalable, Technology-Agnostic Machine Learning Infrastructures https://www.infoq.com/presentations/netflix-ml-meson https://eng.uber.com/michelangelo https://www.infoq.com/presentations/paypal-data-service-fraud What is this thing used everywhere?
- 9. 11Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 10. 12Apache Kafka and Machine Learning – Kai Waehner Apache Kafka - A Distributed Commit Log Producers Kafka Cluster Consumers
- 11. 13Apache Kafka and Machine Learning – Kai Waehner Kafka Streams Your app sinksource KafkaConnect KafkaConnect Kafka Cluster Apache Kafka includes Kafka Connect and Kafka Streams
- 12. 14Apache Kafka and Machine Learning – Kai Waehner Apache Kafka at Scale at Tech Giants > 4.5 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
- 13. 15Apache Kafka and Machine Learning – Kai Waehner Confluent - Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
- 14. 16Apache Kafka and Machine Learning – Kai Waehner Apache Kafka’s Open Source Ecosystem as Infrastructure for ML
- 15. 17Apache Kafka and Machine Learning – Kai Waehner Apache Kafka’s Open Ecosystem as Infrastructure for ML Kafka Streams / KSQL Kafka Connect Rest Proxy Schema Registry Go/.NET /Python Kafka Producer KSQL Kafka Streams Focus of this talk
- 16. 19Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 17. 20Apache Kafka and Machine Learning – Kai Waehner TensorFlow TensorFlow is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains. https://www.tensorflow.org/
- 18. 21Apache Kafka and Machine Learning – Kai Waehner TensorFlow Ecosystem + large community + integration with most 3rd party ML tools + support by all major cloud providers Focus of this talk
- 19. 23Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 20. 24Apache Kafka and Machine Learning – Kai Waehner Model Deployment with a Model Server Open source vs. commercial Self-managed vs hosted Features (Versioning, A/B Testing, …) Communication (RPC via HTTP or gRPC) Serving
- 21. 25Apache Kafka and Machine Learning – Kai Waehner TensorFlow Serving • Serve multiple models or multiple versions simultaneously • gRPC and HTTP endpoints • Model versioning without changing any client code • Canarying and A/B testing • Optimized for minimal latency to inference time • Scheduler grouping individual inference requests into batches for joint execution • Supports many servables • Tensorflow models • Embeddings, • Vocabularies, • Feature transformations, • Non-Tensorflow-based models https://github.com/tensorflow/serving
- 22. 26Apache Kafka and Machine Learning – Kai Waehner Stream Processing with External Model and RPC Streams Input Event Prediction Request Response Model Serving TensorFlow Serving gRPC / HTTP Application
- 23. 27Apache Kafka and Machine Learning – Kai Waehner TensorFlow + Kafka Streams Filter Map 2) Configure Kafka Streams Application 3) RPC to TensorFlow Serving (and catch Exceptions) 4) Start Kafka Streams App 1) Import Kafka and TensorFlow Serving API Model Server Request Response
- 24. 28Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 25. 29Apache Kafka and Machine Learning – Kai Waehner Stream processing with embedded model Application Input Event Prediction
- 26. 30Apache Kafka and Machine Learning – Kai Waehner Stream processing with embedded model Streams Input Event Prediction Stream Processing Model doPrediction() return value
- 27. 31Apache Kafka and Machine Learning – Kai Waehner Stream processing with embedded model Input Event Prediction Kafka Client REST Client Model doPrediction() return value
- 28. 32Apache Kafka and Machine Learning – Kai Waehner TensorFlow + Kafka Streams Filter Map 2) Load TensorFlow Model 3) Configure Kafka Streams Application 4) Apply TensorFlow Model to Streaming Data 5) Start Kafka Streams App 1) Import Kafka and TensorFlow API
- 29. 33Apache Kafka and Machine Learning – Kai Waehner What Models should be directly embedded into an App? Examples • Python Model à Slow • H2O Java Class (e.g. Decision Tree) à Fast • TensorFlow protobuf (small Neural Network, few Mb or less) à Quick load • TensorFlow protobuf (large Neural Network, 100Mb and more) à Much memory • Standards-based (e.g. XML / JSON, based on PMML, ONNX ) • Includes other steps like pre-processsing • Often organizational challenges and technical limitations / constraints • Model Performance à Faster is better • Model Binary Format à Best case: Java Bytecode • Model Size à Less MB and Memory is better • Model Server Features à Out-of-the-box vs. build-yourself vs. not needed
- 30. 34Apache Kafka and Machine Learning – Kai Waehner Rebuilding the Features of a Model Server is not that hard… Application Input Event Prediction - Start new version of application (e.g. K8s Pod) - Send and consume model or weights via Kafka topic - Load new version via API (e.g. Java TensorFlow API) - Service Mesh (Envoy / Linkerd / Istio etc.) for A/B Testing, Green / Blue Deployments / Dark Launch, … Update Model? Versioning? A/B Testing? Canary?
- 31. 35Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 32. 36Apache Kafka and Machine Learning – Kai Waehner Stream Processing with Model Server vs. Embedded Model Why use a model server and RPC? • Simple integration with existing technologies and organizational processes • Easier to understand if you come from non- streaming world • Later migration to real streaming is also possible • Model management built-in for different models, versioning and A/B testing • Monitoring built-in Why embed model into streaming app? • Better latency as remote call instead of local inference • Offline inference (devices, edge processing, etc.) • No coupling of the availability, scalability, and latency/throughput of your Kafka Streams application with the SLAs of the RPC interface • No side-effects (e.g., in case of failure), all covered by Kafka processing (e.g., exactly once) Application Input Event Prediction
- 33. 37Apache Kafka and Machine Learning – Kai Waehner Separation of Model Training and Model Inference Local Predictions Model Training in Cloud Model Deployment On Premise Analytic Model
- 34. 38Apache Kafka and Machine Learning – Kai Waehner Where to Deploy the Model? • Public Cloud • VM / Container • Serverless • On Premise • Traditional Application • Cloud-Native • Edge • Data Center • Machine • Device VM
- 35. 39Apache Kafka and Machine Learning – Kai Waehner Traditional Application Deployment VM Application Input Event Prediction
- 36. 40Apache Kafka and Machine Learning – Kai Waehner Traditional Application Deployment – A Kafka Streams App 1) Java Code (Kafka Streams App + Unit Test) 2) Maven Build 3) Application Deployment of JAR File https://github.com/kaiwaehner/kafka-streams-machine-learning-examples Streams
- 37. 41Apache Kafka and Machine Learning – Kai Waehner Cloud-Native Deployment leveraging Kubernetes
- 38. 42Apache Kafka and Machine Learning – Kai Waehner Kubernetes Cluster K8 NodeK8 NodeK8 Node Replicator Pod C3 Pod SR Pod K8 NodeConfluent Operator Kafka Pod ZK Pod Persistent Volumes (AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.) External Access Load Balancers Configurations ConfigMaps Kafka Application Pod Cloud-Native Deployment – Embedded Model Kafka Streams App (TensorFlow Model Embedded)
- 39. 43Apache Kafka and Machine Learning – Kai Waehner Kubernetes Cluster K8 NodeK8 NodeK8 Node Replicator Pod C3 Pod SR Pod K8 NodeConfluent Operator Kafka Pod ZK Pod Persistent Volumes (AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.) External Access Load Balancers Configurations ConfigMaps Kafka Application Pod Cloud-Native Deployment – Model Inference via Sidecar Pattern and RPC Kafka Streams App TensorFlow Serving (Sidecar)
- 40. 44Apache Kafka and Machine Learning – Kai Waehner Model Deployment at the Edge Questions to discuss • Edge Data Center vs. Edge Device / Machine • Kafka Cluster vs. 1 Kafka Broker vs. Kafka Client at the Edge • Powerful Client (KSQL, Java) vs. Lightweight Client (C, JavaScript) • Embedded vs. RPC Model Inference • Local vs. Remote Training • Legal and Compliance Implications What is “the Edge”?
- 41. 45Apache Kafka and Machine Learning – Kai Waehner Example: Edge Computing in Telco Business • StarlingX (https://www.starlingx.io/) - open source “cloud infrastructure software stack for the edge used by the most demanding applications in industrial IOT, telecom, video delivery and other ultra-low latency use cases” • Wind River Titanium Cloud is „the industry’s highest-performing cloud infrastructure for the network edge“ based on StarlingX • Definition of “Edge Computing”: Ultra low latency with < 100ms end-to-end communication Let’s always define what you mean with “Edge Computing”
- 42. 46Apache Kafka and Machine Learning – Kai Waehner Agenda • Challenges of Machine Learning • Apache Kafka Ecosystem for Machine Learning • TensorFlow Ecosystem • Stream Processing with Model Server and RPC • Stream Processing with Embedded Models • Deployment Options and Trade-Offs • A Cutting Edge Cloud-Native ML Streaming Infrastructure
- 43. Data Lake Batch Analytics Event Streaming Platform Batch Integration Real Time Pre- processing Machine Sensors Streaming Platform Other Components Real Time Processing (6b) All Data (7) Potential Defect (3) Read Data Optimization / Analytics (5) Deploy Optimization Model (8b) Alert Person (e.g. Mobile App) (2) Preprocess Data (6a) Consume machine data Model Standard- based Integration (8a) Stop Machine (1) Ingest Data Real Time Edge Computing Model Lite Real Time App Model Server RPC PLC Proprietary- based Integration Standard Interface Proprietary Interface (9) Manual user-based analytics and reporting to find insights and improve real time process (4) Train Model
- 44. 48Apache Kafka and Machine Learning – Kai Waehner Time Model BModel A Producer Distributed Commit Log Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (https://github.com/tensorflow/io). (no additional data storage like S3 or HDFS required!) Streaming Ingestion and Model Training with TensorFlow IO
- 45. 49Apache Kafka and Machine Learning – Kai Waehner Model Deployment with Apache Kafka, KSQL and TensorFlow “CREATE STREAM AnomalyDetection AS SELECT sensor_id, detectAnomaly(sensor_values) FROM machine_engine;“ User Defined Function (UDF)
- 46. Spark Notebooks (Jupyter) Kafka Cluster Kafka Connect KSQL Machine Sensors Kafka Ecosystem Other Components Real Time Kafka Streams Application (Java / Scala) (6b) All Data (7) Potential Defect (3) Read Data TensorFlow I/O TensorFlow (5) Deploy Model (2) Preprocess Data (6a) Consume machine data TensorFlow MQTT File HTTP (8a) Stop Machine (1) Ingest Data Real Time Edge Computing (C / librdkafka) TensorFlow Lite Real Time Kafka App TensorFlow Serving HTTP / gRPC (4) Train Model PLC Beckhoff S7 Modbus OPC-UA PLC4X Connector Kafka Connect Standard Interface Proprietary Interface (8b) Alert Person (e.g. Mobile App) (9) Manual user-based analytics and reporting to find insights and improve real time process
- 47. 54Apache Kafka and Machine Learning – Kai Waehner Machine Learning + Apache Kafka à Examples @ Github https://github.com/kaiwaehner
- 48. 55Apache Kafka and Machine Learning – Kai Waehner Key Takeaways • Model deployment can be completely separated from model training (process and technology) • Two alternatives for model deployment: Model server vs. embedded into application • Model embedding into Kafka applications is simple and creates a robust and performant environment
- 49. 56Apache Kafka and Machine Learning – Kai Waehner Kai Waehner Technology Evangelist [email protected] @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!