Maximize the Business Value of Machine Learning and Data Science with Kafka (Chirag Dadia & Tom Szumowski, Nuuly) Kafka Summit 2020

Sep 10, 20200 likes1,371 views

Today, many companies that have lots of data are still struggling to derive value from machine learning (ML) and data science investments. Why? Accessing the data may be difficult. Or maybe it’s poorly labeled. Or vital context is missing. Or there are questions around data integrity. Or standing up an ML service can be cumbersome and complex. At Nuuly, we offer an innovative clothing rental subscription model and are continually evolving our ML solutions to gain insight into the behaviors of our unique customer base as well as provide personalized services. In this session, I’ll share how we used event streaming with Apache Kafka® and Confluent Cloud to address many of the challenges that may be keeping your organization from maximizing the business value of machine learning and data science. First, you’ll see how we ensure that every customer interaction and its business context is collected. Next, I’ll explain how we can replay entire interaction histories using Kafka as a transport layer as well as a persistence layer and a business application processing layer. Order management, inventory management, logistics, subscription management – all of it integrates with Kafka as the common backbone. These data streams enable Nuuly to rapidly prototype and deploy dynamic ML models to support various domains, including pricing, recommendations, product similarity, and warehouse optimization. Join us and learn how Kafka can help improve machine learning and data science initiatives that may not be delivered to their full potential.

Maximize the Business Value of
Machine Learning and Data
Science with Kafka
Kafka Summit 2020
Nuuly, Urban Outfitters
Chirag Dadia, Director of Engineering
Tom Szumowski, Sr. Data Scientist
Alan Rosenwinkel, Sr. Data Science Manager

Agenda
● Overview of Nuuly
● Customer Experience & Inventory Management
● Warehouse Management
● General Architecture & Kafka
● Data Science at Nuuly
● Q&A

What is Nuuly?
A women’s subscription clothing rental service

How does it work?
Kafka as the central nervous system

Kafka’s Role at Nuuly
Kafka stream processors orchestrate all the business workflows

How does Kafka help the Data Science (DS) workflow?
Extraction: Flexibly connect generic or custom consumers to hydrate data stores
Access: Safe access to any data stream without new apps
Kafka helps Data Scientists access production data for offline analysis

How does Kafka help the Data Science (DS) workflow?
Integration: Low-latency streaming data for real-time apps
Deployment: Push artifacts into production apps
Kafka helps bridge the gap between DS artifacts and operational data feeds

How do these workflows tie into the Nuuly business?
Online
Optimization
Order
Fulfillment
Personalization
● Recommendations
● Finding similar products
● Optimizing garment pick-
paths in warehouse
● Enabling data-flows for
real-time user feedback
assessment
Offline Processing Batch Topic Processing
Real-Time Feedback
Streams

Personalization: Recommendations
Objective: Simplify user’s box-selection journey through
recommendations
Data Science Solution: Custom Collaborative Filtering
Business Value:
● Promote purchases
● Retain customers
Technical Challenges:
● Resources: GPU-heavy on train, memory-heavy on serve
● Scope: Used across various site components, with
varying configurations
● Scale: several thousands users and products
● Latency: Response time within 100 ms

Integrating Personalization with Kafka
1. Real-time event
streams that land in a
queryable database
2. User recommendations
computed offline 3. Resulting artifact feeds a real-
time Kafka State Store, used for
trays, browse pages, etc.
User/Item
Interactions
Offline
Nightly
Low-latency
rankings
fetch
Rankings +
Context Metadata
Offline workflow that fuels a low-latency, real-time Kafka State Store

Order Fulfillment: Technical Challenges
Objective: Minimize walking path in warehouse
Business Value: Reduce operational costs, meet shipping SLAs
Data Science Solution: Route Planning algorithms
Technical Challenges:
● Resources: Very heavy CPU/memory required online
● Reliability: Interacting with critical data paths
● Timeliness: Must always have batches of N orders ready to
disperse
● Optimality: Works best with lots of orders (not streaming)
How do we safely connect in our DS app while ensuring consistency in orders?

Connecting into Core Business Flow with Kafka
Periodic cronjob that processes all (non-tombstoned)
messages from first offset in batch

Online Optimization
Objective:
● Adaptively update backend algorithms based
on renting and purchasing feedback
Business Value:
● Promotion optimization
● Inventory/assortment optimization
Other Operational Benefits:
● Reduce experimentation tracking overhead
● Reduce incremental deployment updates
Data Science Solution
● Online Contextual Multi-Armed Bandits
Technical Challenges:
● Delayed feedback, up to a month after rented
● Multiple data sources to capture {state, action,
reward}
Source: https://slivkins.com/work/bandits-svc
Product Value Promotion Rented, Purchased, Rated, etc.
weeks later ...

Developing Closed-Loop Feedback with Kafka
1. Derive “context”
(feature extraction)
2. Select action,
triggered by event
3. Optimize:
Incrementally
update model
Leveraging real-time producers &
consumers to enable closed-loop
feedback

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...HostedbyConfluent

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...confluent

Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.

Tackling Kafka, with a Small Team ( Jaren Glover, Robinhood) Kafka Summit SF ...confluent

This is a story about what happens when a distributed system becomes a big part of a small team's infrastructure. This distributed system was Kafka and the team size was one engineer. I will discuss my failures along with my journey of deploying Kafka at scale with very little prior distributed systems experience. In this presentation, we will discuss how unique insights in the following organization culture, engineering and metrics created tailwinds and headwinds. This presentation will be a tactical approach to conquering a complex system with an understaffed team while your business is growing fast. I will discuss how the use case and resilience requirements for our Kafka cluster change as the user base grew from 100K users to over 6 million.

Death of the dumb pipes: Using Apache Kafka® for Integration projectsHostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...HostedbyConfluent

This document summarizes Activision Data's transition from a batch data pipeline to a real-time streaming data pipeline using Apache Kafka and Kafka Streams. Some key points: - The new pipeline ingests, processes, and stores game telemetry data from over 200k messages per second and over 5PB of data across 9 years of games. - Kafka Streams is used to transform the raw streaming data through multiple microservices with low 10-second end-to-end latency, compared to 6-24 hours previously. - Kafka Connect integrates the streaming data with data stores like AWS S3, Cassandra, and Elasticsearch. - The new pipeline provides real-time and historical access to structured

Apache kafka-a distributed streaming platformconfluent

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...HostedbyConfluent

Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better. The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka. We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices. Some potential takeaways for the discerning audience: 1. Opportunities in a platform approach to Event Driven Architecture in the enterprise 2. Adopting a product mindset around Data & Event Streams 3. Seeking harmony with allied enterprise applications

Leveraging Mainframe Data for Modern Analyticsconfluent

The document provides an overview of leveraging mainframe data for modern analytics using Attunity Replicate and Confluent streaming platform powered by Apache Kafka. It discusses the history of mainframes and data migration, how Attunity enables real-time data migration from mainframes, the Confluent streaming platform for building applications using data streams, and how Attunity and Confluent can be combined to modernize analytics using mainframe data streams. Use cases discussed include query offloading and cross-system customer data integration.

How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

Stream Processing Live Traffic Data with Kafka StreamsTom Van den Bulck

In this workshop we will set up a streaming framework which will process realtime data of traffic sensors installed within the Belgian road system. Starting with the intake of the data, you will learn best practices and the recommended approach to split the information into events in a way that won't come back to haunt you. With some basic stream operations (count, filter, ... ) you will get to know the data and experience how easy it is to get things done with Spring Boot & Spring Cloud Stream. But since simple data processing is not enough to fulfill all your streaming needs, we will also let you experience the power of windows. After this workshop, tumbling, sliding and session windows hold no more mysteries and you will be a true streaming wizard.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

This document discusses streaming data between Confluent Cloud and MongoDB Atlas. It provides an overview of MongoDB Atlas and its fully managed database capabilities in the cloud. It then demonstrates how to stream data from a Python generator application to MongoDB Atlas using Confluent Cloud and its connectors. The presentation concludes by providing a reference architecture for connecting Confluent Platform to MongoDB.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020HostedbyConfluent

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...HostedbyConfluent

This document summarizes Jan Svoboda's presentation on building event streaming microservices with Spring Boot and Apache Kafka. Jan discusses his motivation for learning Kafka and microservices development. He outlines a typical journey from strangling monoliths to building event-driven architectures using Kafka. Jan explains how Spring and Kafka integrate well together. The remainder of the presentation demonstrates building a sample application using these techniques and dives into common architecture patterns like event sourcing, CQRS, scaling stateful services, and reverse proxy routing.

Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...confluent

This document discusses Airbnb's use of Kafka as the foundation for its highly reliable logging system. It describes the types of data Airbnb collects, including product events, database exports, service events, and derived data. Airbnb uses a simple logging pipeline where events are delivered reliably through Kafka in real time. Key components of its production logging pipeline include Jitney for standardized messaging, a central schema repository, client SDKs, producer and consumer agents, and a self-service portal. The pipeline provides continuous integration through schema authoring, deployment, implementation, processing and storage, and monitoring. It handles large volumes of data reliably with 150 brokers processing 1 million messages per second and 10 billion events collected daily with very low

Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...HostedbyConfluent

Despite great advances in Kafka's SaaS offerings it can still be challenging to create a sustainable event-driven ecosystem. Often platform engineers become de facto ‘gatekeepers’ of events & topics, yet their day job is not about data modelling or domain expertise. We've all seen the bottlenecks these unsustainable processes create. Realising the potential of event streams requires much more than infrastructure. Beyond an event-driven mindset, it requires domain experts to lead creation of well-defined discoverable events through fit-for-purpose governance. AsyncAPI is the OpenAPI for events that can form the basis of the required self-governing, self-service eventing framework. This session will introduce a self-governing framework using AsyncAPI and share how the Bank of New Zealand applied this framework to leverage a passionate Kafka community and embed event-driven thinking. You’ll leave with a tangible set of ideas to give your own events a bit more swagger using AsyncAPI.

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Surviveconfluent

1. A streaming platform like Kafka can provide the benefits of Hadoop for batch processing but in a faster, real-time way by processing data as it arrives rather than storing all data. 2. Virtual reality applications require stream processing to power features like VR mirroring and capture in real-time. Kafka's stream processing capabilities address challenges like this for VR. 3. The document discusses how AltspaceVR uses Kafka stream processing for applications like VR mirroring and capture, presence tracking, scheduled tasks, and more to power their real-time VR experiences.

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...HostedbyConfluent

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

Elastically Scaling Kafka Using Confluentconfluent

This document discusses how Confluent Platform provides elastic scaling for Apache Kafka. It offers fully managed cloud services through Confluent Cloud or self-managed software. Confluent Cloud allows users to easily scale Kafka workloads from 0 MBps to GBps without complex provisioning. It also offers pay-for-use pricing where customers only pay for the data streamed, with the ability to scale to zero. For self-managed deployments, Confluent Platform enables dynamic scaling of Kafka clusters on Kubernetes through features like tiered storage and self-balancing clusters that can rebalance partitions in seconds versus hours for other Kafka services.

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

ETL as a Platform: Pandora Plays Nicely Everywhere with Real-Time Data Pipelinesconfluent

ETL can be painful with dirty data and outdated batch processes slowing you down; there has to be a better way. In this talk we’ll discuss the benefits of introducing a streaming platform to your architecture including how it can greatly simplify complexity, speed up performance, and help your team deliver the features they need with real-time data integration. Pandora’s Lawrence Weikum will discuss what they’ve done to bring real-time data integration to the team. We’ll review their Kafka-powered data pipelines and how they make the most of Kafka’s Connect API to make it surprisingly system to keep systems in sync. Presented by: Lawrence Weikum, Senior Software Engineer, Pandora Gehrig Kunz, Technical Product Marketing Manager, Confluent

Kafka at the core of an AIOps pipeline | Sunanda Kommula, Selector.ai and Ala...HostedbyConfluent

Large networks consist of a diverse range of equipment, across private, public, hybrid clouds and partner networks. A hierarchical network has layers of infrastructure, catering to access, core, or distribution roles, managed by different organizations specialized to architect the right network hardware, software, and features for that network layer. The nature of data generated by each component can vary in type and form, including logs, events, metrics, or alarms. The diversity of data generated by a large network is beyond human scale. Apache Kafka® is a critical hub in large networks, empowering AIOps to enhance decision making, improve analysis and insights by contextualizing large volumes of operational data. Kafka solved the big problem of collecting, processing, storing and normalizing data at scale, allowing us to focus on building the AIOps pipeline. Our platform connects the dots across relevant operations data and provides operations teams with simple and powerful access to insights, from within increasingly popular collaboration environments like Slack and Microsoft teams. The pipeline must also integrate with automation solutions. This session will cover how large volumes of streaming messages can be received by parallel Kafka consumers, and turned into action by network operations teams, dramatically reducing downtime and improving performance.

Enabling Insight to Support World-Class Supercomputing (Stefan Ceballos, Oak ...confluent

The Oak Ridge Leadership Facility (OLCF) in the National Center for Computational Sciences (NCCS) division at Oak Ridge National Laboratory (ORNL) houses world-class high-performance computing (HPC) resources and has a history of operating top-ranked supercomputers on the TOP500 list, including the world's current fastest, Summit, an IBM AC922 machine with a peak of 200 petaFLOPS. With the exascale era rapidly approaching, the need for a robust and scalable big data platform for operations data is more important than ever. In the past when a new HPC resource was added to the facility, pipelines from data sources spanned multiple data sinks which oftentimes resulted in data silos, slow operational data onboarding, and non-scalable data pipelines for batch processing. Using Apache Kafka as the message bus of the division's new big data platform has allowed for easier decoupling of scalable data pipelines, faster data onboarding, and stream processing with the goal to continuously improve insight into the HPC resources and their supporting systems. This talk will focus on the NCCS division's transition to Apache Kafka over the past few years to enhance the OLCF's current capabilities and prepare for Frontier, OLCF's future exascale system; including the development and deployment of a full big data platform in a Kubernetes environment from both a technical and cultural shift perspective. This talk will also cover the mission of the OLCF, the operational data insights related to high-performance computing that the organization strives for, and several use-cases that exist in production today.

user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent

For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior. While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement. In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.

Simplify Governance of Streaming Data confluent

The document discusses the importance of data governance and schemas for streaming data platforms using Apache Kafka. It recommends using a schema registry to define schemas for Kafka topics, handle schema changes, and prevent incompatible changes. A schema registry provides a single source of truth for schemas, prevents bad data, and allows for increased agility when modifying schemas while maintaining compatibility. It benefits the entire application lifecycle from development to production.

Data Streaming with Apache Kafka & MongoDBconfluent

Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent

APAC Kafka Summit - Best Of confluent

More Related Content

What's hot (20)