Ranking the Web with Spark
3 likes968 views
This document discusses ranking web pages using Apache Spark. It begins by introducing the speaker and their background. It then provides an overview of how search engines traditionally work, including crawling, indexing, and ranking pages. It discusses using static features like URL depth and dynamic features like click-through rates to calculate page scores. The document proposes using Spark to perform learning to rank by training models on features and user data to improve results. It also demonstrates calculating PageRank on the Common Crawl dataset using GraphFrames in SparkSQL. Finally, it provides links to learn more about the Common Search open source project.
Ranking the Web with Spark
- 1. Ranking the Web with Spark Apache Big Data Europe 2016 [email protected] @sylvinus
- 2. /usr/bin/whoami • Jamendo (Founder & CTO, 2004-2011) • TEDxParis (Co-founder, 2009-2012) • dotConferences (Founder, 2012-) • Pricing Assistant (Co-founder & CTO, 2012-)
- 7. Ranking
- 8. Disclaimer: IANASRE (I Am Not A Search Relevance Engineer)
- 9. What's in a score score = fn( doc, query, language, user, time )
- 10. What's in a score score = fn( doc, query )
- 11. What's in a score score = fn( static_score, dynamic_score ( query ))
- 12. Static score
- 13. Static features • Scopes: • Page: URL depth, markup stats, ... • Domain: Age, page count, blacklists, ... • WebGraph: PageRank, ...
- 14. http://infolab.stanford.edu/~backrub/google.html The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998) Crawler Indexer Database SearcherRanker
- 15. Dynamic score
- 16. Dynamic features • Text match: TF-IDF, BM25, proximity, topic, ... • Query-level: number of words, popularity, ... • Usage: clicks, dwell time, reformulations, ... • Time
- 17. Scoring function
- 18. Users Database Elasticsearch Indexer Python, Spark Data sources Common Crawl, Alexa top 1M, ... words, static score query top 10 docs, final scores Offline Online Searcher Go
- 21. Issues with this architecture • Static & dynamic scoring are in different codebases • No control over result diversity • Hard to optimize • Very dependent on Elasticsearch
- 22. Rescoring
- 23. Users Database Indexer words, static score, features query Searcher top 1k docs, features Rescorer final 10 docs
- 24. Issues with rescoring • Latency • Pagination • Harder to explain
- 25. Learning to rank
- 26. LTR Model • Features • Training dataset • Evaluation: NDCG, ERR, ... • Algorithms: AdaRank, ListNet, LambdaMART, ... • Learning with Spark!
- 27. The right questions • What do users expect? • What features? • How to evaluate and fine-tune in the real world?
- 32. Common Search Pipeline Doc sources Common Crawl, WARC files, URLs ... Filter plugins Document parsing Output plugins Data output Database, file, HDFS, S3, ...
- 33. Most popular Wikipedia pages
- 34. Dumping the web graph
- 36. GraphFrames
- 42. Top 10
- 44. Spam
- 46. Spamdexing • Keyword stuffing, hidden text • Scraper sites, Mirrors • Link farms • Splogs, Comment spam • Domaining • Cloaking • Bombing