Recommendations and Discovery at StumbleUpon
Download as PPTX, PDF•33 likes•8,591 views
StumbleUpon aims to enhance user discovery by shifting from traditional search to personalized content recommendations based on user interests and behavior. The platform uses an automated URL index and various techniques to separate quality content from poor content while continuously updating user interest graphs. Key challenges include understanding user expectations, optimizing recommendation algorithms, and managing user feedback for improved content discovery.
Recommendations and Discovery at StumbleUpon
- 1. Recommendations and Discovery at StumbleUpon Sumanth Kolar, Director, Engineering @_5K
- 2. StumbleUpon’s Mission Help users find content they did not expect to find Be the best way to discover new and interesting things from across the Web.
- 3. How StumbleUpon works 1. Register 2. Tell us your interests 3. Start Stumbling and rating web pages We use your interests and behavior to recommend new content for you!
- 4. Discovery is very different from search Discovery at StumbleUpon Search Serendipitous Intent driven One at a time List of articles Never repeats Always repeats Constantly adapting Fixed results Tailored for you Impersonal There is a ongoing shift from search to discovery
- 5. StumbleUpon
- 6. StumbleUpon Overview 1 Users Automated URL Index Discovery Crawled 3 Ingestion Pipeline Rec Engine Yes 2 Pass Sampling ?
- 7. What are the key challenges to good recommendations?
- 8. Pillars of good recommendations Understand who the user is and what he is interested in. Separate good content from the bad. Explore various techniques for matching users to content. Learn from your recommendations.
- 9. Pillars of good recommendations Understand who the user is and what he is interested in. Separate good content from the bad. Explore various techniques for matching users to content. Learn from your recommendations.
- 10. User self reports topics of interest Part of the sign up flow…
- 11. User’s Interest Graph Italian Food/ User Recipes Cooking Cars Vintage Cars
- 12. Continually Enhance a User’s Interest Graph Analyze user’s StumbleUpon history to expand on interest preferences: • Add/remove topics • Follow/block particular domains
- 13. Continually Enhance a User’s Interest Graph Leverage social network data: • Find friends & people to follow • Find content trending in your social circles • Find additional interests
- 14. Continually Enhance a User’s Interest Graph Mine internal StumbleUpon rating and sharing data to suggest other stumblers, topics.
- 15. Enhanced Interest Graph Friends News Italian Food/ Trending User Recipes Cooking Cars nasa.gov Vintage Cars 1x.com
- 16. Pillars of good recommendations Understand who the user is and what he is interested in. Separate good content from the bad. Explore various techniques for matching users to content. Learn from your recommendations.
- 17. Sampling On average hundreds of URLs are ingested into the StumbleUpon pipeline every minute. • Sampling key goals: 1. Determine which URLs to sample and which to skip completely 2. Examine sampling results to identify good URLs • URL features used when sampling: • Known domain performance(ratings, timespent) • Content related features (#images, #ads, url length etc) • User features of the discoverer (spammer vs trusted user)
- 18. Recommendations at StumbleUpon: Sampling Classifier based on User Feedback Random Forest Vote Recommend (Timespent, Ratings) Rating Timespent Yes Good 35sec Good 22sec Webpage Bad 15sec Yes No Yes Good 45sec Good 14sec Yes Good 28sec
- 19. Leveraging In-Network Experts • Users who thumb-up good content and thumb-down bad content • For example – Joe DiMaggio – Baseball – Julia Child- Food/Cooking – Da Vinci- Art and Architecture • Ratings from Experts are more trustworthy and earn more weight.
- 20. Non Expert Expert P(Thumb Up | Page Quality) P(Thumb Up | Page Quality) Page Quality Page Quality Recommendations at StumbleUpon: Experts
- 21. Pillars of good recommendations Understand who the user is and what he is interested in. Separate good content from the bad. Explore various techniques for matching users to content. Learn from your recommendations.
- 22. Challenge: User expectations are different “I LOVE cars!” “Me too!” -Anonymous Stumbler -Another Stumbler
- 23. Like-Minded Users • Find users who like content similar to the content you do • Signals can be ratings, time spent, interests, etc. • Use the content they’ve liked
- 24. PLSI based like-minded Vintage Cars Action movies Astronomy Astronomy Space Exploration Robotics Physics Classic Movies Movies Cars Space Neuroscience Astronomy Space Exploration Science Comedy Movies
- 25. Like-Minded Users: Challenges Scaling Total Pairwise Similarity Calculations = 50K users * 5 million users * 1K features = 250 Trillion Probabilistic Latent Semantic Index (PLSI) based similarity over 500 trillion calculations PLSI based similarity framework computes in less than an hour
- 26. Grow User’s Interest Graph: Implicit + Explicit Experts Friends Likeminded Users News User Food/ Trending Italian Recipes Cooking Cars nasa.gov Vintage 1x.com Cars
- 27. Different methods perform differently for different users at different times 100% 75% Trending Follow 50% Bias domains Experts News 25% Like-minded 0% User 1 User 2 User 3 User 4 User 5
- 29. Pillars of good recommendations Understand who the user is and what he is interested in. Separate good content from the bad. Explore various techniques for matching users to content. Learn from your recommendations.
- 30. Two Main Signals from Recommendation Rating Time Spent Both present numerous challenges . . .
- 31. Ratings: volume decay Users rate more during their initial experience # Ratings Time Why is this happening?
- 32. Time Spent ? ? Images Video Text Images Video T5 sec T3 sec T4 sec T2 sec T1 sec • Ratings are sparse • < 10% of recommendations have explicit ratings. • Using time spent decide whether the stumble was skipped • Timespent on videos is longer than images. • Solution: Estimate p(Like | Timespent) • Model based on user, content patterns
- 33. Challenges: Time spent on different devices Stumble Bar Median time spent per stumble Mobile / Tablets Installed plugin 5th percentile time spent per stumble
- 34. Pillars of good recommendations Understand who the user is and what he is interested in. Separate good content from the bad. Explore various techniques for matching users to content. Learn from your recommendations.
- 35. How do we know we are doing a good job?
- 36. Extensive A/B Testing AB Tests on metrics such as session length, retention, rating behavior etc
- 37. 0 2 6 8 10 14 16 4 12 Dec-08 Feb-09 Apr-09 Jun-09 Aug-09 Oct-09 +111% improvement! Dec-09 Feb-10 Apr-10 Jun-10 Aug-10 Oct-10 Dec-10 Recent Months Feb-11 Apr-11 Normalized Likes vs Dislikes Jun-11 Aug-11 Oct-11 Dec-11 Feb-12 Apr-12 Measurable Improvements In Rec Quality Jun-12 R² = 0.736 Aug-12
- 38. Many other interesting problems… • Dupe detection • Anti-spam • News • Topic classification • Metrics, quality analysis • Trending • Search We are HIRING !!! • User biases, mood • Many more…
Editor's Notes
- #2: At the end of this talk, you would have a good understanding of problems with discovery, some solutions, some data insights.
- #3: Our goal is to show content that you did not know you would likeTo surprise you, enlighten youBasically to enable exploration, discovery
- #4: -During signup, we ask interesting questions to learn more about you – solve the cold start problem
- #5: - Think of discovery as search without a term and add the complexities i.e, nothing repeats etcFor example, if you want to learn about astronomy or genetic algorithms its hard to do on search or any other services --- way more work
- #6: When I started a couple of years back, we were 6M users and 15 employeesGrowing rapidly, especially on mobileTalk about time spent and how users are super hooked.
- #11: Users are good at choosing topics that they like.. We have had repeated good success at increasing the topics they pickBut, the problem is more about having them pick the right topics for them.. Arts vs AI.. Its not simple to build a user experience that accounts for that and gets us that dataHuge area of research for StumbleUpon --- how do we get as much as possible from the user without losing them or setting completely different expectations than what the product is
- #12: Now we have a basic version of the interests graph.. Some topics you like
- #13: StumbleSenseBased on you likes/dislikes we build a SENSE for other things you may like. Hence suggest topics, domains, etc that we think you will like as you stumble alongMakes interest elicitation a part of the core productYou are learning about the user and the user understands the product a lot.. Dialgoue and back, forthNotice that we give the reason why it was recommended.. Transparency is very important.
- #14: Leverage other networks you are part of to get data about what you like and jumpstart interest graph
- #15: Also, show suggested stumblers, interests etc
- #16: More dense interest graph. Affinity, confidence to the interest varies and depending on that we can exploit, explore.
- #18: When new content is discovered /ingested how do we determine if its good or not.You will always have exceptions that need to be handled. For ex: - Domains such as youtube, basically UGC in which content is diverse.. You need to build models that account for thatUser features of the raters/discoverers .. Just because a spammer rated cnn.com you can’t ignore it.. Look at multiple sources of information and decide whether the url is worth sampling or not
- #19: Now, one way of doing this is to use a random forest with content features
- #20: And also we can sample to expertsThat’s one huge advantage SU has – the fact that we can decide which site to send and get data for that url.But, sometimes you could be recommending bad content to the expert – you get around by telling the expert that we think he is an expert and we need to get more data from him about the url. Again transparency for the win Transparency allows us to set the right expectations..
- #21: One way of defining experts is users who thumbup high quality pages and thumbdown low quality pages.. There are multiple ways you can find high quality pages-- Have a seed of experts pick urls and use them to find other experts-- Or looks at your current quality scores and see which user ratings are more predictive of that .. Use them as experts-- Social endorsement.. Have users rate others as experts, use external data sources similar to what klout is doing to do this – very hard problem.
- #23: How to you match right content to right user ? User expectations are very different. When you say you like cars and I like cars.. We are not talking about the same thingNeed to deeper understand the interest graph
- #24: One solution is find other users that are similar to you.. But then just because you are similar to me in Physics.. does not mean I would like the Music you listen to.
- #25: One solution.. Figure out latent topicsand then use them to cluster/find similar users
- #27: Now we have an interest graph that is both explicit and implicit
- #28: Different users have varying method mix.We learn the mix and balance it.. But this needs to account for mood – for example, we see that you like stumbling news in the morning and videos in the weekend. But there are always exceptions
- #29: Context i.e, showing why a recommendation was shown to a user is very important. There should be a back and forth. Recommendations should be very transparent. Context can that your friend on Facebook liked it or it can be that this is trending in Politics
- #32: Immediate conclusion is quality of recommendations is not good.. But this is both thumbups and downs Stumbling is cheap and so clicking the stumble button is better than rating. One could argue that we are doing a really good job and the marginal utility of rating is not highSolutions: Use other data such as time spent to figure out what you like. Make you rate more ;) work very closely with product on what we can do to remind the user that their ratings matter
- #33: Now, we know we need to use timespent..Last stumble, time spent Great we have a solution
- #34: Mobile – shorter attention spans,