Spark for Recommender Systems

May 24, 2018Download as PPTX, PDF3 likes549 views

We will cover Apache Spark's Machine Learning Library (MLlib) This presentation covers using Spark for recommender systems. MLlib is a library built on top of Spark's engine which allows us to train, test, validate and operationalize machine learning models while working with lots of data in a convenient way thanks to its robust abstractions over data sets. Find out how you can use MLlib to build product recommendation systems by employing both traditional ML techniques such as collaborative filtering, as well as more novel, deep-learning approaches which make use of Neural Networks.

Using Spark’s Machine
Learning Library to Make
Product Recommendations
Sorin Pește
Technology Solutions Professional, Data & AI
Microsoft
source: xkcd.com

A PA C H E S PA R K
A unified, distributed, open source engine for large-scale data processing
Spark Structured
Streaming
Stream processing
Spark MLlib
Machine
Learning
Spark Core Engine
Spark SQL
Interactive
Queries
Yarn Mesos
Standalone
Scheduler
Spark MLlib
Machine
Learning
Spark
Streaming
Stream processing
GraphX
Graph
Computation

S PA R K D ATA F R A M E S
A distributed collection of data that’s conceptually equivalent to a table

S P A R K M A C H I N E L E A R N I N G ( M L L I B )
 Offers a set of parallelized machine learning algorithms for ML
 Supports Model Selection (hyperparameter tuning) using Cross
Validation and Train-Validation Split.
 Supports Java, Scala or Python apps using DataFrame-based API
Enables Parallel, Distributed ML for large datasets on Spark Clusters

S P A R K M L L I B A L G O R I T H M S
Spark MLlib
Algorithms

C O L L A B O R A T I V E F I L T E R I N G

C O L L A B O R A T I V E F I L T E R I N G
User Latent Factors
Item Latent Factors

A L T E R N A T I N G L E A S T S Q U A R E S ( A L S )

ALS
https://github.com/neaorin/databricks-demos/

A L S : E X P L I C I T V S I M P L I C I T F E E D B A C K
 Explicit feedback — user rates items
 Implicit feedback — system records user activity
 Browses a product page
 Watches a movie trailer
 Plays a song
 Shares on social media
 etc
Implicit feedback is generally used in real-world implementations

A L S : H Y P E R P A R A M E T E R T U N I N G
 Hyperparameters which can be adjusted:
 rank = the number of latent factors in the model
 maxIter = the maximum number of iterations
 regParam = the regularization parameter

A L S : H Y P E R P A R A M E T E R T U N I N G

A L S : W H A T A B O U T R E A L - T I M E ?
 Near real-time computation of ALS algorithm may be unfeasible
 Streaming variant of ALS, using Stochastic Gradient Descent
https://github.com/brkyvz/streaming-matrix-factorization
• Oryx Framework (http://oryx.io ) also offers streaming ALS

B E Y O N D A L S
 ALS-learned latent factors can be useful as input for other algorithms

D E E P L E A R N I N G
 A set of machine learning techniques that use multiple layers of non-linear processing units to
learn useful data representations of input

D E E P L E A R N I N G W I T H S P A R K
 Integrations with existing DL libraries
• Microsoft CNTK (mmlspark)
• TensorFlow (TensorFlowOnSpark)
• DeepLearning4J
• Caffe (CaffeOnSpark)
• Keras (Elephas)
• mxnet
• Paddle
• and more…
 Implementations of DL on Spark
• BigDL
• DeepDist
• SparkCL
• SparkNet
• Deep Learning Pipelines (Databricks)
• and more…
Distributed Hyperparameter Tuning

D E E P L E A R N I N G F O R R E C O M M E N D E R S
• Neural Collaborative Filtering (He et al, 2017)
https://arxiv.org/abs/1708.05031
https://github.com/hexiangnan/neural_collaborative_filtering
Neural Collaborative Filtering

D E E P L E A R N I N G F O R R E C O M M E N D E R S
• Predict the next item the user will want to interact with
Recommendations as sequence prediction
[a] -> b
[a, b] -> c
[a, b, c] -> d
[0, 0, 0, a] -> b
[0, 0, a, b] -> c
[0, a, b, c] -> d

D E E P L E A R N I N G F O R R E C O M M E N D E R S
• Predict the next item the user will want to interact with
Recommendations as sequence prediction

D E E P L E A R N I N G F O R R E C O M M E N D E R S
 Session-based Recommendations with Recurrent Neural Networks
(Hidasi et al., 2015)
https://arxiv.org/abs/1511.06939
https://github.com/hidasib/GRU4Rec
Recommendations as sequence prediction

D E E P L E A R N I N G F O R R E C O M M E N D E R S

https://arxiv.org/pdf/1510.01784.pdf
Featurize product images

How to Build a Recommendation Engine on Spark was a presentation given by Joe Caserta, CEO and founder of Caserta Concepts, at @AnalyticsWeek in Boston. Boston's Data AnalyticsStreet Conference is a 2 day packed event with thought provoking keynotes, knowledge filled sessions, intense workshops, insightful panels, and real-world case studies - engaging analytics community with latest methodologies and trends. The conference encompasses largest Speaker-to-Attendee ratio for unmatched networking and learning opportunity. For more information on the services and solutions Caserta Concepts offers, visit our website at http://casertaconcepts.com/.

Developing a Movie recommendation Engine with SparkEdureka!

This document discusses building a movie recommendation engine using Apache Spark and the MLlib machine learning library. It begins by explaining what recommendation engines are and some major companies that use them, such as Amazon, YouTube, and LinkedIn. It then covers the different approaches to building recommendation engines, including collaborative filtering, content-based filtering, and hybrid methods. The document goes on to provide a high-level architecture for implementing a recommendation engine using Spark, MLlib, and data from Hadoop. It concludes with an example of code to build such an engine.

Building an Implicit Recommendation Engine with Spark with Sophie WatsonDatabricks

Many of today’s most engaging – and commercially important – applications provide personalised experiences to users. Collaborative filtering algorithms capture the commonality between users and enable applications to make personalised recommendations quickly and efficiently. The Alternating Least Squares (ALS) algorithm is still deemed the industry standard in collaborative filtering. In this talk Sophie will show you how to implement ALS using Apache Spark to build your own recommendation engine. Sophie will show that, by splitting the recommendation engine into multiple cooperating services, it is possible to reduce the system’s complexity and produce a robust collaborative filtering platform with support for continuous model training. In this presentation you will learn how to build a recommendation system for the case where recorded data is explicitly given as a rating, as well as for the case where the data is less succinct. You will walk away from this talk with the knowledge and tools needed to implement your own recommendation system using collaborative filtering and microservices.

Social Media Mining - Chapter 6 (Community Analysis)SocialMediaMining

This document discusses community analysis in social media mining. It defines social media communities as groups of users who form links and interact based on common interests. Community detection aims to discover these implicit communities through algorithms. Member-based detection examines node characteristics like degree and similarity, while group-based detection finds communities with properties like being balanced, robust, modular, dense, or hierarchical. Analyzing communities provides insight into user interactions and behaviors that are only observable at a group level.

Personalizing "The Netflix Experience" with Deep LearningAnoop Deoras

Recommendation at Netflix ScaleJustin Basilico

Netflix provides personalized recommendations at scale to over 37 million members across 40 countries. They take a multi-layered approach using offline, nearline, and online computation. In the offline layer, large datasets are processed to train machine learning models. The nearline layer incrementally refines recommendations based on member events. In the online layer, recommendations are generated and presented to members in real-time based on signals from live services and precomputed results. Netflix recommendations are powered by a massive dataset of over 30 million daily plays and sophisticated algorithms running across distributed cloud computing infrastructure.

Few shot learning/ one shot learning/ machine learningﺁﺻﻒ ﻋﻠﯽ ﻣﯿﺮ

The document discusses few-shot learning approaches. It begins with an introduction explaining that current deep learning models require large datasets but humans can learn from just a few examples. It then discusses the problem of few-shot learning, where models must perform classification, detection, or regression on novel categories represented by only a few samples. Popular approaches discussed include meta-learning methods like MAML and prototypical networks, metric learning methods like relation networks, and data augmentation methods. The document provides an overview of the goals and techniques of few-shot learning.

User behavior analyticsShankar Vedaraman

Graph Gurus Episode 6: Community DetectionTigerGraph

This document summarizes Graph Gurus Episode 6 which discusses community detection algorithms for graphs. The episode introduces community detection and explains that communities can emerge naturally in networks like human relationships or protein interactions, or can be engineered like congressional committees. It then demonstrates how to find influential providers and their communities in a healthcare referral network using community detection algorithms like label propagation and modularity optimization. These algorithms are implemented as GSQL queries in TigerGraph's graph algorithm library. Finally, the episode applies the connected components algorithm to the Zachary's Karate Club network.

Collaborative Filtering 1: User-based CFYusuke Yamamoto

Collaborative Filtering using KNNŞeyda Hatipoğlu

Collaborative filtering is a technique used by recommender systems to predict items users may like based on opinions of similar users. K-nearest neighbors (KNN) is a collaborative filtering algorithm that finds the k most similar users and bases predictions on the ratings of those neighbors. The document describes KNN collaborative filtering, including finding neighbor similarity, making predictions, and evaluating error rates on a movie recommendation system using the MovieLens dataset.

Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain

The document summarizes a presentation on recommender systems given by Xavier Amatriain. It begins with introductions to recommender systems and collaborative filtering. Traditional collaborative filtering approaches include user-based and item-based methods. User-based CF finds similar users to a target user and recommends items they liked. Item-based CF finds similar items to those a target user liked and predicts ratings. Both approaches address sparsity and scalability challenges with dimensionality reduction techniques.

Big data analytics in banking sectorAnil Rana

The document discusses big data and big data analytics in banking. It defines big data as large, complex datasets that are difficult to process and store using traditional databases. Sources of big data include social media, sensors, transportation services, online shopping, and mobile apps. Characteristics of big data include volume, velocity, and variety. Hadoop is presented as an open source framework for analyzing big data using HDFS for storage and MapReduce for processing. The benefits of big data analytics in banking include fraud detection, risk management, customer segmentation, churn analysis, and sentiment analysis to improve customer experience.

Sentimental Analysis - Naive Bayes AlgorithmKhushboo Gupta

This document provides an overview of sentiment analysis on Twitter. It discusses how sentiment analysis can be used to determine sentiment behind texts and social media updates. The document outlines the methodology used, including data collection from Twitter, preprocessing, feature extraction, and using classifiers like Naive Bayes to predict sentiment. It also discusses applications of sentiment analysis and future areas of improvement, such as using different algorithms and temporal analysis. The goal is to more accurately analyze human sentiment from social media data.

Interactive Recommender Systems with Netflix and SpotifyChris Johnson

Interactive recommender systems enable the user to steer the received recommendations in the desired direction through explicit interaction with the system. In the larger ecosystem of recommender systems used on a website, it is positioned between a lean-back recommendation experience and an active search for a specific piece of content. Besides this aspect, we will discuss several parts that are especially important for interactive recommender systems, including the following: design of the user interface and its tight integration with the algorithm in the back-end; computational efficiency of the recommender algorithm; as well as choosing the right balance between exploiting the feedback from the user as to provide relevant recommendations, and enabling the user to explore the catalog and steer the recommendations in the desired direction. In particular, we will explore the field of interactive video and music recommendations and their application at Netflix and Spotify. We outline some of the user-experiences built, and discuss the approaches followed to tackle the various aspects of interactive recommendations. We present our insights from user studies and A/B tests. The tutorial targets researchers and practitioners in the field of recommender systems, and will give the participants a unique opportunity to learn about the various aspects of interactive recommender systems in the video and music domain. The tutorial assumes familiarity with the common methods of recommender systems.

Recommender SystemsT212

CF Models for Music Recommendations At SpotifyVidhya Murali

Zero shot-learning: paper presentationJérémie Kalfon

Recommendation system Vikrant Arya

Recommendation systems provide users with information they may be interested in based on their preferences and interests. They help address the problem of information overload by retrieving desired information for the user based on their preferences or those of similar users. The two main types of recommendation systems are personalized and non-personalized systems. Common techniques used include collaborative filtering, which finds users with similar tastes, and content-based filtering, which recommends items similar to those a user has liked based on item attributes.

딥 러닝 자연어 처리 학습을 위한 PPT! (Deep Learning for Natural Language Processing)WON JOON YOO

Deep learning with KerasQuantUniversity

Interest in Deep Learning has been growing in the past few years. With advances in software and hardware technologies, Neural Networks are making a resurgence. With interest in AI based applications growing, and companies like IBM, Google, Microsoft, NVidia investing heavily in computing and software applications, it is time to understand Deep Learning better! In this workshop, we will discuss the basics of Neural Networks and discuss how Deep Learning Neural networks are different from conventional Neural Network architectures. We will review a bit of mathematics that goes into building neural networks and understand the role of GPUs in Deep Learning. We will also get an introduction to Autoencoders, Convolutional Neural Networks, Recurrent Neural Networks and understand the state-of-the-art in hardware and software architectures. Functional Demos will be presented in Keras, a popular Python package with a backend in Theano and Tensorflow.

Recommendation systemsSaurabhWani6

Collaborative Filtering - MF, NCF, NGCFPark JunPyo

This document discusses neural graph collaborative filtering. It begins with an overview of recommender systems and collaborative filtering. It then discusses latent factor models, including matrix factorization and neural collaborative filtering (NCF). A key development was neural graph collaborative filtering (NGCF), which uses graph neural networks to model high-order connectivity in the user-item graph to address sparsity issues. NGCF aims to learn better embeddings by propagating user and item information through the graph.

Introduction to Recommendation SystemsTrieu Nguyen

Sentiment analysis of Twitter DataNurendra Choudhary

Sentiment Analysis/Opinion Mining of Twitter Data on Unigram/Bigram/Unigram+Bigram Model using: 1. Machine Learning 2. Lexical Scores 3. Emoticon Scores YouTube Video: https://youtu.be/VuR16P87yPE Link to the WebPage: http://akirato.github.io/Twitter-Sentiment-Analysis-Tool Github Page: https://github.com/Akirato/Twitter-Sentiment-Analysis-Tool

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...MLconf

Hands on Explainable Recommender Systems with Knowledge Graphs @ RecSys22GiacomoBalloccu

This document provides an overview of an upcoming tutorial on explainable recommender systems with knowledge graphs. The tutorial will include two sessions - an introductory session on explainable recommendation principles and modeling approaches, and a hands-on session using Jupyter notebooks to build and evaluate recommendation models using knowledge graphs. Attendees will learn about explainable recommendation methods, loading and preprocessing interaction datasets with knowledge graphs, building recommendation models with knowledge graphs, and evaluating and generating explanations from models. The tutorial aims to help attendees understand explainable recommender systems and apply techniques using knowledge graphs.

Artificial Intelligence (AI) Interview Questions and Answers | EdurekaEdureka!

(** Machine Learning Engineer Masters Program: https://www.edureka.co/masters-progra... **) This PPT on Artificial Intelligence Interview Questions covers all the important concepts involved in the field of AI. This PPT is ideal for both beginners as well as professionals who want to learn or brush up their knowledge on AI concepts. Below are the topics covered in this tutorial: 1. Artificial Intelligence Basic Level Interview Question 2. Artificial Intelligence Intermediate Level Interview Question 3. Artificial Intelligence Scenario based Interview Question Check out the entire Machine Learning Playlist: https://bit.ly/2NG9tK4 Follow us to never miss an update in the future. YouTube: https://www.youtube.com/user/edurekaIN Instagram: https://www.instagram.com/edureka_learning/ Facebook: https://www.facebook.com/edurekaIN/ Twitter: https://twitter.com/edurekain LinkedIn: https://www.linkedin.com/company/edureka

20181003 Whirlwind tour into PysparkAndrey Vykhodtsev

This document provides an overview of Apache Spark, including: 1) A brief history of Spark and description of its core components for batch processing, streaming, SQL queries, machine learning and graph processing. 2) Benefits of Spark such as performance, developer productivity, ecosystem support, and a unified engine. 3) Explanations of what makes Spark fast, including in-memory computing and caching of remote data locally. 4) A link to learn more about Databricks, a company that provides a cloud-based analytics platform with Spark.

Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)Jason Dai

This document outlines an agenda for a talk on building deep learning applications on big data platforms using Analytics Zoo. The agenda covers motivations around trends in big data, deep learning frameworks on Apache Spark like BigDL and TensorFlowOnSpark, an introduction to Analytics Zoo and its high-level pipeline APIs, built-in models, and reference use cases. It also covers distributed training in BigDL, advanced applications, and real-world use cases of deep learning on big data at companies like JD.com and World Bank. The talk concludes with a question and answer session.

More Related Content

What's hot (20)

Graph Gurus Episode 6: Community DetectionTigerGraph

Collaborative Filtering 1: User-based CFYusuke Yamamoto

Collaborative Filtering using KNNŞeyda Hatipoğlu

Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain

Big data analytics in banking sectorAnil Rana

Sentimental Analysis - Naive Bayes AlgorithmKhushboo Gupta

Interactive Recommender Systems with Netflix and SpotifyChris Johnson

Recommender SystemsT212

CF Models for Music Recommendations At SpotifyVidhya Murali

Zero shot-learning: paper presentationJérémie Kalfon

Recommendation system Vikrant Arya

딥 러닝 자연어 처리 학습을 위한 PPT! (Deep Learning for Natural Language Processing)WON JOON YOO

Deep learning with KerasQuantUniversity

Recommendation systemsSaurabhWani6

Collaborative Filtering - MF, NCF, NGCFPark JunPyo

Introduction to Recommendation SystemsTrieu Nguyen

Sentiment analysis of Twitter DataNurendra Choudhary

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...MLconf

Hands on Explainable Recommender Systems with Knowledge Graphs @ RecSys22GiacomoBalloccu

Artificial Intelligence (AI) Interview Questions and Answers | EdurekaEdureka!

Graph Gurus Episode 6: Community DetectionTigerGraph

Collaborative Filtering 1: User-based CFYusuke Yamamoto

Collaborative Filtering using KNNŞeyda Hatipoğlu

Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain

Big data analytics in banking sectorAnil Rana

Sentimental Analysis - Naive Bayes AlgorithmKhushboo Gupta

Interactive Recommender Systems with Netflix and SpotifyChris Johnson

Recommender SystemsT212

CF Models for Music Recommendations At SpotifyVidhya Murali

Zero shot-learning: paper presentationJérémie Kalfon

Recommendation system Vikrant Arya

딥 러닝 자연어 처리 학습을 위한 PPT! (Deep Learning for Natural Language Processing)WON JOON YOO

Deep learning with KerasQuantUniversity

Recommendation systemsSaurabhWani6

Collaborative Filtering - MF, NCF, NGCFPark JunPyo

Introduction to Recommendation SystemsTrieu Nguyen

Sentiment analysis of Twitter DataNurendra Choudhary

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...MLconf

Hands on Explainable Recommender Systems with Knowledge Graphs @ RecSys22GiacomoBalloccu

Artificial Intelligence (AI) Interview Questions and Answers | EdurekaEdureka!

Similar to Spark for Recommender Systems (20)

20181003 Whirlwind tour into PysparkAndrey Vykhodtsev

Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)Jason Dai

898-Azure Databricks Technical Deck - sorinpe.pptxssuserf55974

Apache Spark is an open-source unified analytics engine for large-scale data processing. Databricks provides a managed Apache Spark service that runs on Azure. Databricks offers a unified analytics platform that is optimized for speed and ease of use. It provides frictionless, optimized Spark clusters on Azure along with big data analytics and storage services to reduce administration and control costs.

Azure 機器學習 - 使用Python, R, Spark, CNTK 深度學習 Herman Wu

The document discusses Microsoft's Cognitive Toolkit (CNTK), an open source deep learning toolkit developed by Microsoft. It provides the following key points: 1. CNTK uses computational graphs to represent machine learning models like DNNs, CNNs, RNNs in a flexible way. 2. It supports CPU and GPU training and works on Windows and Linux. 3. CNTK achieves state-of-the-art accuracy and is efficient, scaling to multi-GPU and multi-server settings.

Tiny Batches, in the wine: Shiny New Bits in Spark StreamingPaco Nathan

London Spark Meetup 2014-11-11 @Skimlinks http://www.meetup.com/Spark-London/events/217362972/ To paraphrase the immortal crooner Don Ho: "Tiny Batches, in the wine, make me happy, make me feel fine." http://youtu.be/mlCiDEXuxxA Apache Spark provides support for streaming use cases, such as real-time analytics on log files, by leveraging a model called discretized streams (D-Streams). These "micro batch" computations operated on small time intervals, generally from 500 milliseconds up. One major innovation of Spark Streaming is that it leverages a unified engine. In other words, the same business logic can be used across multiple uses cases: streaming, but also interactive, iterative, machine learning, etc. This talk will compare case studies for production deployments of Spark Streaming, emerging design patterns for integration with popular complementary OSS frameworks, plus some of the more advanced features such as approximation algorithms, and take a look at what's ahead — including the new Python support for Spark Streaming that will be in the upcoming 1.2 release. Also, let's chat a bit about the new Databricks + O'Reilly developer certification for Apache Spark…

Apache® Spark™ MLlib: From Quick Start to Scikit-LearnDatabricks

These are the slides to support the Apache® Spark™ MLlib: From Quick Start to Scikit-Learn webinar. In this webcast, Joseph Bradley from Databricks will be speaking about Apache Spark’s distributed Machine Learning Library - MLlib. We will start off with a quick primer on machine learning, Spark MLlib, and a quick overview of some Spark machine learning use cases. We will continue with multiple Spark MLlib quick start demos. Afterwards, the talk will transition toward the integration of common data science tools like Python pandas, scikit-learn, and R with MLlib

OpenML Tutorial ECMLPKDD 2015Joaquin Vanschoren

IBM Strategy for SparkMark Kerzner

This document provides an introduction and overview of Spark: - Spark is an open-source in-memory data processing engine that can handle large datasets across clusters of computers using an API in Scala, Python, or R. - IBM is heavily committed to Spark, contributing the most code and fixing the most issues reported by other organizations to continually improve the full analytics stack. - An example is presented on using Spark to predict hospital readmissions from diabetes patient data, obtaining AUC scores comparable to other published models.

Distributed Deep Learning + others for Spark MeetupVijay Srinivas Agneeswaran, Ph.D

Building and deploying LLM applications with Apache AirflowKaxil Naik

Behind the growing interest in Generate AI and LLM-based enterprise applications lies an expanded set of requirements for data integrations and ML orchestration. Enterprises want to use proprietary data to power LLM-based applications that create new business value, but they face challenges in moving beyond experimentation. The pipelines that power these models need to run reliably at scale, bringing together data from many sources and reacting continuously to changing conditions. This talk focuses on the design patterns for using Apache Airflow to support LLM applications created using private enterprise data. We’ll go through a real-world example of what this looks like, as well as a proposal to improve Airflow and to add additional Airflow Providers to make it easier to interact with LLMs such as the ones from OpenAI (such as GPT4) and the ones on HuggingFace, while working with both structured and unstructured data. In short, this shows how these Airflow patterns enable reliable, traceable, and scalable LLM applications within the enterprise. https://airflowsummit.org/sessions/2023/keynote-llm/

eScience Cluster Arch. OverviewFrancesco Bongiovanni

Dr. Francesco Bongiovanni has expertise in scalable distributed systems and algorithms, cloud computing, applied formal methods, and distributed optimizations. He has a B.Sc. in Computer Systems, M.Sc. in Software Engineering of Distributed Systems, and Ph.D. in Computer Science. He has worked at INRIA and Verimag Laboratory. This presentation provides an overview of big data frameworks and tools including HDFS, Mesos, Spark, Spark Streaming, Spark SQL, GraphX, MLLib, Chapel, ZooKeeper, and SparkR that can be run on the eScience cluster for processing large datasets in a scalable, fault-tolerant manner. Examples demonstrate performing operations like averaging 1 billion elements

Enabling exploratory data science with Spark and RDatabricks

R is a favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large datasets with R is challenging, especially when data scientists use R with frameworks or tools written in other languages. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show how SparkR solves these problems to enable a much smoother experience. In this talk we will present an overview of the SparkR architecture, including how data and control is transferred between R and JVM. This knowledge will help data scientists make better decisions when using SparkR. We will demo and explain some of the existing and supported use cases with real large datasets inside a notebook environment. The demonstration will emphasize how Spark clusters, R and interactive notebook environments, such as Jupyter or Databricks, facilitate exploratory analysis of large data.

Leveraging NLP and Deep Learning for Document Recommendations in the CloudDatabricks

Efficient recommender systems are critical for the success of many industries, such as job recommendation, news recommendation, ecommerce, etc. This talk will illustrate how to build an efficient document recommender system by leveraging Natural Language Processing(NLP) and Deep Neural Networks (DNNs). The end-to-end flow of the document recommender system is build on AWS at scale, using Analytics Zoo for Spark and BigDL. The system first processes text rich documents into embeddings by incorporating Global Vectors (GloVe), then trains a K-means model using native Spark APIs to cluster users into several groups. The system further trains a recommender model for each group, and gives an ensemble prediction for each test record. By adopting the end-to-end pipeline of Analytics Zoo solution, we saw about 10% improvement of mean reciprocal ranking and 6% of precision respectively compared to the search recommendations for a job recommendation study. Speaker: Guoqiong Song

Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...Big Data Spain

The document discusses using a graph database to store and query graph data stored in a Hadoop data lake more efficiently. It describes the limitations of the typical approach of using Spark/GraphFrames on HDFS for graph queries. A graph database allows for faster ad hoc graph queries by leveraging graph traversals. The document proposes using a multi-model database that combines a document store, graph database, and key-value store with a common query language. It suggests this approach could run on a DC/OS cluster for easy deployment and management of resources. Examples show importing data into ArangoDB and running graph queries.

Fishing Graphs in a Hadoop Data Lake DataWorks Summit/Hadoop Summit

Hadoop clusters can store nearly everything in a cheap and blazingly fast way to your data lake. Answering questions and gaining insights out of this ever growing stream becomes the decisive part for many businesses. Increasingly data has a natural structure as a graph, with vertices linked by edges, and many questions arising about the data involve graph traversals or other complex queries, for which one does not have an a priori given bound on the length of paths. Spark with GraphX is great for answering relatively simple graph questions which are worth starting a Spark job for, because they essentially involve the whole graph. But does it make sense to start one for every ad-hoc query or is it suitable for complex real-time queries? In this talk I will introduce an alternative solution that adds those features to an existing Hadoop/Spark setup and enables real-time insights. I will address the following topics: * Challenges in gaining deeper insights from large amounts of graph data * Benefits and limitations of graph analysis with Spark * Introduction to ArangoDB SmartGraphs * Deployment of Hadoop, Spark and ArangoDB using DC/OS * Performing complex queries on billions of nodes and vertices leveraging ArangoDB SmartGraphs (Live Demo)

Graph Analytics in SparkPaco Nathan

Fishing Graphs in a Hadoop Data LakeArangoDB Database

Dev Ops TrainingSpark Summit

The document provides an agenda for a DevOps advanced class on Spark being held in June 2015. The class will cover topics such as RDD fundamentals, Spark runtime architecture, memory and persistence, Spark SQL, PySpark, and Spark Streaming. It will include labs on DevOps 101 and 102. The instructor has over 5 years of experience providing Big Data consulting and training, including over 100 classes taught.

Composable Parallel Processing in Apache Spark and WeldDatabricks

The main reason people are productive writing software is composability -- engineers can take libraries and functions written by other developers and easily combine them into a program. However, composability has taken a back seat in early parallel processing APIs. For example, composing MapReduce jobs required writing the output of every job to a file, which is both slow and error-prone. Apache Spark helped simplify cluster programming largely because it enabled efficient composition of parallel functions, leading to a large standard library and high-level APIs in various languages. In this talk, I'll explain how composability has evolved in Spark's newer APIs, and also present a new research project I'm leading at Stanford called Weld to enable much more efficient composition of software on emerging parallel hardware (multicores, GPUs, etc). Speaker: Matei Zaharia

Overview of Apache Flink: Next-Gen Big Data Analytics FrameworkSlim Baltagi

These are the slides of my talk on June 30, 2015 at the first event of the Chicago Apache Flink meetup. Although most of the current buzz is about Apache Spark, the talk shows how Apache Flink offers the only hybrid open source (Real-Time Streaming + Batch) distributed data processing engine supporting many use cases: Real-Time stream processing, machine learning at scale, graph analytics and batch processing. In these slides, you will find answers to the following questions: What is Apache Flink stack and how it fits into the Big Data ecosystem? How Apache Flink integrates with Apache Hadoop and other open source tools for data input and output as well as deployment? What is the architecture of Apache Flink? What are the different execution modes of Apache Flink? Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark? Who is using Apache Flink? Where to learn more about Apache Flink?

20181003 Whirlwind tour into PysparkAndrey Vykhodtsev

Build Deep Learning Applications for Big Data Platforms (CVPR 2018 tutorial)Jason Dai

898-Azure Databricks Technical Deck - sorinpe.pptxssuserf55974

Azure 機器學習 - 使用Python, R, Spark, CNTK 深度學習 Herman Wu

Tiny Batches, in the wine: Shiny New Bits in Spark StreamingPaco Nathan

Apache® Spark™ MLlib: From Quick Start to Scikit-LearnDatabricks

OpenML Tutorial ECMLPKDD 2015Joaquin Vanschoren

IBM Strategy for SparkMark Kerzner

Distributed Deep Learning + others for Spark MeetupVijay Srinivas Agneeswaran, Ph.D

Building and deploying LLM applications with Apache AirflowKaxil Naik

eScience Cluster Arch. OverviewFrancesco Bongiovanni

Enabling exploratory data science with Spark and RDatabricks

Leveraging NLP and Deep Learning for Document Recommendations in the CloudDatabricks

Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...Big Data Spain

Fishing Graphs in a Hadoop Data Lake DataWorks Summit/Hadoop Summit

Graph Analytics in SparkPaco Nathan

Fishing Graphs in a Hadoop Data LakeArangoDB Database

Dev Ops TrainingSpark Summit

Composable Parallel Processing in Apache Spark and WeldDatabricks

Overview of Apache Flink: Next-Gen Big Data Analytics FrameworkSlim Baltagi

More from Sorin Peste (8)

Microsoft Automated ML ServiceSorin Peste

The Microsoft Automated ML Service automates the machine learning process by using techniques like probabilistic matrix factorization to automatically train and deploy models for a given dataset with minimal coding. It handles tasks like feature engineering, model training, evaluation, and selection to build accurate machine learning models. The service is part of Azure Machine Learning and can be accessed through Azure Notebooks with sample projects provided to demonstrate its capabilities for automated machine learning.

Using Deep Learning (Computer Vision) to Search for Oil and GasSorin Peste

Several areas of Earth with large accumulations of oil and gas also have huge deposits of salt below the surface. Salt bodies are known for their propensity to form nice oil traps. However, knowing where large salt deposits are precisely is very difficult. Professional seismic imaging still requires expert human interpretation of salt bodies. This leads to very subjective, highly variable renderings. More alarmingly, it leads to potentially dangerous situations for oil and gas company drillers. That's why the oil & gas industry is now employing AI-based approaches to automatically identify subsurface salt bodies. This presentation showcases how Deep Learning is used to scan underground seismic images, looking for potentially resource-rich areas. Python code included and publicly available at: https://github.com/neaorin/kaggle-tgs-challenge

Introduction to Reinforcement LearningSorin Peste

Reinforcement Learning is learning what to do – what action to take in a specific situation – in order to maximize some type of reward. It’s one of the most promising areas of Machine Learning today. It plays an important part in some very high-profile success stories of AI, such as mastering Go, learning to play computer games, autonomous driving, autonomous stock trading, and more. In this talk we’ll introduce the main theoretical and practical aspects of Reinforcement Learning, discuss its very distinctive set of challenges, and explore what the future looks like for self-training machines.

SQL Server 2017 Machine Learning ServicesSorin Peste

Neural networks and boosted decision trees are popular machine learning algorithms supported in SQL Server. Python is the primary scripting language used for machine learning tasks in SQL Server. Python has a vast user community and is widely used among data scientists and developers. Common machine learning tasks performed include classification, regression, anomaly detection, and sentiment analysis.

Build an Intelligent Bot (Node.js)Sorin Peste

Bots are all the rage these days. But if you need one that's capable of understanding your users' natural language, so it can figure out what they WANT and not just what they say, then you're going to need some pretty strong AI behind it. Come to this session to find out how you can use Node.js and Microsoft Bot Framework together with Language Understanding Intelligent Service (luis.ai) to build an intelligent bot for Skype, Slack, Facebook, SMS and more.

Automate your UI testing for Android and iOS apps with the Xamarin Test CloudSorin Peste

You've done a nice job building your new mobile app - but are your users going to stick around enough to actually use it? Research shows that two thirds of users will quickly abandon your app if it doesn't work well for them. But with hundreds of iOS and Android devices out there, can you reasonably expect to test your app on all of them? Enter Xamarin Test Cloud: a service to help you automate mobile testing on actual, physical devices, improve your app's quality, and give your users that great first impression - all without leaving your own desk.

Build an Intelligent BotSorin Peste

SQL Server on Linux - march 2017Sorin Peste

SQL Server on Linux will provide the SQL Server database engine running natively on Linux. It allows customers choice in deploying SQL Server on the platform of their choice, including Linux, Windows, and containers. The public preview of SQL Server on Linux is available now, with the general availability target for 2017. It brings the full power of SQL Server to Linux, including features like In-Memory OLTP, Always Encrypted, and PolyBase.

Microsoft Automated ML ServiceSorin Peste

Using Deep Learning (Computer Vision) to Search for Oil and GasSorin Peste

Introduction to Reinforcement LearningSorin Peste

SQL Server 2017 Machine Learning ServicesSorin Peste

Build an Intelligent Bot (Node.js)Sorin Peste

Automate your UI testing for Android and iOS apps with the Xamarin Test CloudSorin Peste

Build an Intelligent BotSorin Peste

SQL Server on Linux - march 2017Sorin Peste

Recently uploaded (20)

Ch3MCT24.pptx measure of central tendencyayeleasefa2

How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345

I was devastated when I realized that I had fallen victim to an online fraud, losing a significant amount of money in the process. After countless hours of searching for a solution, I came across iCode cybertech. From the moment I reached out to their team, I felt a sense of hope that I can recommend iCode Cybertech enough for anyone who has faced similar challenges. Their commitment to helping clients and their exceptional service truly set them apart. Thank you, iCode cybertech, for turning my situation around! [email protected]

VKS-Python-FIe Handling text CSV Binary.pptxVinod Srivastava

Medical Dataset including visualizationsvishrut8750588758

Digilocker under workingProcess Flow.pptxsatnamsadguru491

183409-christina-rossetti.pdfdsfsdasggsagfardin123rahman07

Minions Want to eat presentacion muy lindaCarlaAndradesSoler1

Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management

By James Francis, CEO of Paradigm Asset Management In the landscape of urban safety innovation, Mt. Vernon is emerging as a compelling case study for neighboring Westchester County cities. The municipality’s recently launched Public Safety Camera Program not only represents a significant advancement in community protection but also offers valuable insights for New Rochelle and White Plains as they consider their own safety infrastructure enhancements.

Principles of information security Chapter 5.pptEstherBaguma

LLM finetuning for multiple choice google bertChadapornK

chapter3 Central Tendency statistics.pptjustinebandajbn

Adobe Analytics NOAM Central User Group April 2025 Agent AI: Uncovering the S...gmuir1066

Classification_in_Machinee_Learning.pptxwencyjorda88

md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004

FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d

GenAI for Quant Analytics: survey-analytics.aiInspirient

CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS

03 Daniel 2-notes.ppt seminario escatologiaAlexander Romero Arosquipa

Simple_AI_Explanation_English somplr.pptxssuser2aa19f

Stack_and_Queue_Presentation_Final (1).pptxbinduraniha86

Ch3MCT24.pptx measure of central tendencyayeleasefa2

How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345

VKS-Python-FIe Handling text CSV Binary.pptxVinod Srivastava

Medical Dataset including visualizationsvishrut8750588758

Digilocker under workingProcess Flow.pptxsatnamsadguru491

183409-christina-rossetti.pdfdsfsdasggsagfardin123rahman07

Minions Want to eat presentacion muy lindaCarlaAndradesSoler1

Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management

Principles of information security Chapter 5.pptEstherBaguma

LLM finetuning for multiple choice google bertChadapornK

chapter3 Central Tendency statistics.pptjustinebandajbn

Adobe Analytics NOAM Central User Group April 2025 Agent AI: Uncovering the S...gmuir1066

Classification_in_Machinee_Learning.pptxwencyjorda88

md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004

FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d

GenAI for Quant Analytics: survey-analytics.aiInspirient

CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS

03 Daniel 2-notes.ppt seminario escatologiaAlexander Romero Arosquipa

Simple_AI_Explanation_English somplr.pptxssuser2aa19f

Stack_and_Queue_Presentation_Final (1).pptxbinduraniha86

Spark for Recommender Systems

1. Using Spark’s Machine Learning Library to Make Product Recommendations Sorin Pește Technology Solutions Professional, Data & AI Microsoft source: xkcd.com

2. (demo)

3. A PA C H E S PA R K A unified, distributed, open source engine for large-scale data processing Spark Structured Streaming Stream processing Spark MLlib Machine Learning Spark Core Engine Spark SQL Interactive Queries Yarn Mesos Standalone Scheduler Spark MLlib Machine Learning Spark Streaming Stream processing GraphX Graph Computation

4. S PA R K : A B R I E F H I S T O R Y

5. S PA R K D ATA F R A M E S A distributed collection of data that’s conceptually equivalent to a table

6. S P A R K M A C H I N E L E A R N I N G ( M L L I B )  Offers a set of parallelized machine learning algorithms for ML  Supports Model Selection (hyperparameter tuning) using Cross Validation and Train-Validation Split.  Supports Java, Scala or Python apps using DataFrame-based API Enables Parallel, Distributed ML for large datasets on Spark Clusters

7. S P A R K M L L I B A L G O R I T H M S Spark MLlib Algorithms

8. S P A R K M L L I B P I P E L I N E S

9. C O L L A B O R A T I V E F I L T E R I N G

10. C O L L A B O R A T I V E F I L T E R I N G User Latent Factors Item Latent Factors

11. A L T E R N A T I N G L E A S T S Q U A R E S ( A L S )

12. ALS https://github.com/neaorin/databricks-demos/

13. A L S : E X P L I C I T V S I M P L I C I T F E E D B A C K  Explicit feedback — user rates items  Implicit feedback — system records user activity  Browses a product page  Watches a movie trailer  Plays a song  Shares on social media  etc Implicit feedback is generally used in real-world implementations

14. A L S : H Y P E R P A R A M E T E R T U N I N G  Hyperparameters which can be adjusted:  rank = the number of latent factors in the model  maxIter = the maximum number of iterations  regParam = the regularization parameter

15. A L S : H Y P E R P A R A M E T E R T U N I N G

16. A L S : W H A T A B O U T R E A L - T I M E ?  Near real-time computation of ALS algorithm may be unfeasible  Streaming variant of ALS, using Stochastic Gradient Descent https://github.com/brkyvz/streaming-matrix-factorization • Oryx Framework (http://oryx.io ) also offers streaming ALS

17. B E Y O N D A L S  ALS-learned latent factors can be useful as input for other algorithms

18. D E E P L E A R N I N G  A set of machine learning techniques that use multiple layers of non-linear processing units to learn useful data representations of input

19. D E E P L E A R N I N G W I T H S P A R K  Integrations with existing DL libraries • Microsoft CNTK (mmlspark) • TensorFlow (TensorFlowOnSpark) • DeepLearning4J • Caffe (CaffeOnSpark) • Keras (Elephas) • mxnet • Paddle • and more…  Implementations of DL on Spark • BigDL • DeepDist • SparkCL • SparkNet • Deep Learning Pipelines (Databricks) • and more… Distributed Hyperparameter Tuning

20. D E E P L E A R N I N G F O R R E C O M M E N D E R S • Neural Collaborative Filtering (He et al, 2017) https://arxiv.org/abs/1708.05031 https://github.com/hexiangnan/neural_collaborative_filtering Neural Collaborative Filtering

21. D E E P L E A R N I N G F O R R E C O M M E N D E R S • Predict the next item the user will want to interact with Recommendations as sequence prediction [a] -> b [a, b] -> c [a, b, c] -> d [0, 0, 0, a] -> b [0, 0, a, b] -> c [0, a, b, c] -> d

22. D E E P L E A R N I N G F O R R E C O M M E N D E R S • Predict the next item the user will want to interact with Recommendations as sequence prediction

23. D E E P L E A R N I N G F O R R E C O M M E N D E R S  Session-based Recommendations with Recurrent Neural Networks (Hidasi et al., 2015) https://arxiv.org/abs/1511.06939 https://github.com/hidasib/GRU4Rec Recommendations as sequence prediction

24. D E E P L E A R N I N G F O R R E C O M M E N D E R S  https://arxiv.org/pdf/1510.01784.pdf Featurize product images

Spark for Recommender Systems

Recommended

More Related Content

What's hot (20)

Similar to Spark for Recommender Systems (20)

More from Sorin Peste (8)

Recently uploaded (20)

Spark for Recommender Systems