The Search for Better Search at Reddit - Nick Caldwell, Chris Slowe, and Luis Bitencourt-Emilio, Reddit

Editor's Notes

• #25: 2005 - Steve Huffman, cofounder and now CEO, implements postgres tsearch. 2006 - Chris Slowe, founding engineer and now CTO, implements pylucene. “we fixed a bug in the search results ordering” - Steve Huffman ‘06 “We updated the search system this morning to help alleviate some load problems” - Steve ‘06 “Jeremy is working on search! It’s not a complicated fix (basically, the sorting is whacky).” - Steve ‘06 “Search works much better, tagging and user-controlled subreddits are right around the corner” - Steve ‘07 “Search is better, but not quite where we’d like it.” - Steve ‘07 “Stats and search are temporarily disabled, but will be coming back as soon as we can get them repaired.” - Steve ‘07 “we were hoping to include an upgraded search, which, unlike the last version, was actually useful and helped you find what you were looking for. Unfortunately, the version we settled on didn’t quite load test as nicely” - Steve ‘ 07 “I made a quick fix to search that I hope helps until we get a chance to really fix it.” - Steve ‘07 2008 - David King, first employee and now search engineer, implements Solr. “[David]’s been fixing search and hacking mystery projects in Erlang.” - Alexis Ohanian ‘08 “I’ve totally replaced the reddit search function.” - David King ‘08 2010 - David King replaces Solr with IndexTank. “We launched a new search engine yesterday. Calm down. It’s okay. I know. You’ve been hurt before.” - David King ‘10 2012 - u/kemitche implements CloudSearch after LinkedIn shut down IndexTank “Q: Where do you see reddit in 10 years? A: Reddit search might work by then.” - Steve AMA ‘16 “Today we moved from the old Amazon CloudSearch domain to a new Amazon CloudSearch domain. The old search domain had significant performance issues: roughly 33% of queries took over 5 seconds to complete and would result in the search error page.” - u/bsimpson ‘17 NOW - Lucidworks Fusion! “As /u/bitofsalt mentioned a few months ago, we’ve been working on some improvements to search. We may even be ahead of spez’s 10 year plan.” u/starfishjenga ‘17
• #26: https://redditblog.com/2006/07/25/searching/ https://redditblog.com/2007/06/08/a-note-on-search-and-what-were-working-on/ https://redditblog.com/2008/04/21/new-search-2/ https://www.reddit.com/r/announcements/comments/cs4ll/new_search/ https://www.reddit.com/r/changelog/comments/694o34/reddit_search_performance_improvements/ https://www.reddit.com/r/changelog/comments/6pi0kk/improving_search/
• #27: 2005 - Steve Huffman, cofounder and now CEO, implements postgres tsearch. 2006 - Chris Slowe, founding engineer and now CTO, implements pylucene. “we fixed a bug in the search results ordering” - Steve Huffman ‘06 “We updated the search system this morning to help alleviate some load problems” - Steve ‘06 “Jeremy is working on search! It’s not a complicated fix (basically, the sorting is whacky).” - Steve ‘06 “Search works much better, tagging and user-controlled subreddits are right around the corner” - Steve ‘07 “Search is better, but not quite where we’d like it.” - Steve ‘07 “Stats and search are temporarily disabled, but will be coming back as soon as we can get them repaired.” - Steve ‘07 “we were hoping to include an upgraded search, which, unlike the last version, was actually useful and helped you find what you were looking for. Unfortunately, the version we settled on didn’t quite load test as nicely” - Steve ‘ 07 “I made a quick fix to search that I hope helps until we get a chance to really fix it.” - Steve ‘07 2008 - David King, first employee and now search engineer, implements Solr. “[David]’s been fixing search and hacking mystery projects in Erlang.” - Alexis Ohanian ‘08 “I’ve totally replaced the reddit search function.” - David King ‘08 2010 - David King replaces Solr with IndexTank. “We launched a new search engine yesterday. Calm down. It’s okay. I know. You’ve been hurt before.” - David King ‘10 2012 - u/kemitche implements CloudSearch after LinkedIn shut down IndexTank “Q: Where do you see reddit in 10 years? A: Reddit search might work by then.” - Steve AMA ‘16 “Today we moved from the old Amazon CloudSearch domain to a new Amazon CloudSearch domain. The old search domain had significant performance issues: roughly 33% of queries took over 5 seconds to complete and would result in the search error page.” - u/bsimpson ‘17 NOW - Lucidworks Fusion! “As /u/bitofsalt mentioned a few months ago, we’ve been working on some improvements to search. We may even be ahead of spez’s 10 year plan.” u/starfishjenga ‘17
• #28: ●It’s so great to be here, when I joined Reddit the first project Nick had me jump on was Search. I had no idea what I was in for at the time ●Search is a hard problem, you wouldn’t all be here today otherwise, it is also huge opportunity from both a business and product perspective ●Luckily for us we had the great support of Lucidworks in rising to this challenge and I’m proud to be here today to tell you a little about a thing we did
• #29: ●I’ve always liked to lead with the punchline, here’s what we’ve built in conjunction with the Lucidwork’s team ●We needed a scalable system that could index many billions of pieces of content and stand up to the traffic of the US’s 4th largest website. ●Our Search stack is composed of 3 main pieces: the search microservice, our data pipeline and our index cluster. ○We spun out search specific functionality from our monolith R2 to a separate autoscaling microservice, this greatly improved our ability to iterate quickly and closely monitor and scale this service as we rolled out search. Today, we generally run between 2-3 instances, but expect a significant increase as we begin to ingest comments and other content types and experiences. ○We then piped into our existing data pipeline to build canonical views of our datasets to be ingested by our index, as well as tapping into the live events for streaming updates ○Lastly, we stood up a 24 node Fusion Cluster to index and serve search results ●For comparison, our previous Cloudsearch iteration was running on around 200 machines (that’s a lot of Reddit Gold) and this new and significantly smaller stack is servicing just as much traffic at around 400QPS of first party app searches with 1/6th the resources. ●We can now build on top of this solid platform as we ingest orders of maginitude more content and move Search to the front and center of the Reddit user experience. ●Let’s dig in a bit on how this works…
• #30: ●A critical component of any search stack is the ingestion pipeline, and doing this at Reddit scale is a challenge on it’s own. ●Our first iteration here was the beast above, which pulls and aggregates data from both postgres and our events pipeline to create canonical views of Reddit content to be indexed. ●In doing so, we thought we could also tackle a broader challenge at Reddit and provide a canonical and easily consumable view of all our data not just for Search but also for ML model training and analytics. ●TLDR; we bit off more than we could chew, this pipeline was too complex and had way too many moving pieces, leading to numerous failures, data quality issues and low performance particularly around backfilling. ●So we went back to the drawing board….
• #31: ●…and this time we got it right… or at least less wrong. ●We retargetted ourselves at only solving the search ingestion challenge, which allowed us to remove a significant amount of complexity from the system and go from over 20 ETLs down to 7, and now pulling only from a single data source. ●This change took us from ingesting around 150M posts up to over 200M, and resolved numerous data quality issues we had been seeing in the previous pipeline. It also more than halved our overall time to run and significantly decreased hadoop resource utilization. ●But we’re not done yet… we also built out a streaming ingestion sytstem based on our kafka clusters to serve intra-day content updates. ●This live ingestion is scaling beyound our expectations and we’re currently looking at replacing all the ETLs above with this event based system, which could take an all-time backfill/reindex building from around 2 weeks down to 2 days. ●Enough about ingestion though… let’s look at some charts!
• #32: ● ●Dark Traffic ●Shunting API for backwards compatibility ●Staged rollout (1, 5, 10, 25, 50, 90) for scale and bugs ●THANK YOU ALL FOR FEEDBACK
• #33: ●Virtually eradicated errors, main source now happen to in-flight queries in the rare case when a node goes down. ●From 9 x 4’s to 4 x 9’s ●Some performance costs due to being on US-East/West, fixing this next.
• #34: ●Early results before we actually work through relevance tweak ●These are promissing especially considering cloudsearch uses solr at it’s core ●We can now build on top of this platform with personalization, query intent/synonyms/rewriting, and ML models to further tune
• #36: “learning to code” “Coming out” “Oil leaking” “Iron infusion”
• #37: “learning to code” “Coming out”
• #38: “Oil leaking” “Iron infusion”
• #39: Normalization Comment indexing for image, video, link, sentiment, etc….
• #41: Self harm search partnership in particular.