Scaling Pinterest
42 likes6,872 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1awkL99. Details on Pinterest's architeture, its systems -Pinball, Frontdoor-, and stack - MongoDB, Cassandra, Memcache, Redis, Flume, Kafka, EMR, Qubole, Redshift, Python, Java, Go, Nutcracker, Puppet, etc. Filmed at qconsf.com. Yash Nelapati is an infrastructure engineer at Pinterest where he focusses on scalability, capacity planning and architecture. Prior to Pinterest he was into web development and rapidly prototyping UI. Marty Weiner joined Pinterest in early 2011 as the 2nd engineer. Previously worked at Azul Systems as a VM engineer focused on building/improving the JIT compilers in HotSpot.
![쿠키런: 킹덤 대규모 인프라 및 서버 운영 사례 공유 [데브시스터즈 - 레벨 200] - 발표자: 용찬호, R&D 엔지니어, 데브시스터즈 ...](https://cdn.slidesharecdn.com/ss_thumbnails/t3s3-221108102039-c0f48289-thumbnail.jpg?width=560&fit=bounds)
![7. 게임 스트리밍 서비스를 위한 아키텍처 - 언리얼 엔진을 중심으로! [레벨 300] - 발표자: 하흥수, 솔루션즈 아키텍트, AWS :...](https://cdn.slidesharecdn.com/ss_thumbnails/t2s3-221108101206-4410a0ce-thumbnail.jpg?width=560&fit=bounds)
![[Partner TechForum] 고객을 360도로 이해하고 수익으로 연결하는 글로벌 선도 금융 기업들의 데이터 플랫폼 활용 사례](https://cdn.slidesharecdn.com/ss_thumbnails/treasuredatatechforrm2017-171122013630-thumbnail.jpg?width=560&fit=bounds)
![대용량 데이터레이크 마이그레이션 사례 공유 [카카오게임즈 - 레벨 200] - 조은희, 팀장, 카카오게임즈 ::: Games on AWS ...](https://cdn.slidesharecdn.com/ss_thumbnails/t4s2-221108115925-5b63bf11-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (17)
Ad
Similar to Scaling Pinterest (20)
![[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석](https://cdn.slidesharecdn.com/ss_thumbnails/ofa-and-redshift-pangyo-131203232750-phpapp01-thumbnail.jpg?width=560&fit=bounds)
Ad
More from C4Media (20)
Recently uploaded (20)
Scaling Pinterest
- 1. Scaling Pinterest Marty Weiner Cloud Ninja Yash Nelapati Ascii Artist Monday, November 11, 13
- 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /scaling-pinterest InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
- 3. Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 5. Growth March 2010 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 6. Growth March 2010 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 7. Growth March 2010 Page views per day · · · · RackSpace 1 small Web Engine 1 small MySQL DB 1 Engineer + 2 Founders Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 8. Growth March 2010 Scaling Pinterest Monday, November 11, 13
- 9. Growth March 2010 Scaling Pinterest Monday, November 11, 13
- 10. Growth January 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
- 11. Growth January 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
- 12. Growth January 2011 Page views per day · Amazon EC2 + S3 + CloudFront · · · 1 NGinX, 4 Web Engines 1 MySQL DB + 1 Read Slave 1 Task Queue + 2 Task Processors · · 1 MongoDB 2 Engineers + 2 Founders Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012
- 13. Scaling Pinterest Monday, November 11, 13
- 14. Growth September 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 15. Growth September 2011 Page views per day Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 16. Growth September 2011 Page views per day · · Amazon EC2 + S3 + CloudFront 2 NGinX, 16 Web Engines + 2 API Engines · 5 Functionally Sharded MySQL DB + 9 read slaves · · 4 Cassandra Nodes 15 Membase Nodes (3 separate clusters) · · · · · · 8 Memcache Nodes 10 Redis Nodes 3 Task Routers + 4 Task Processors 4 Elastic Search Nodes 3 Mongo Clusters 3 Engineers (8 Total) Scaling Pinterest Monday, November 11, 13 Mar 2010 Jan 2011 Jan 2012 May 2012
- 17. It will fail. Keep it simple. Scaling Pinterest Monday, November 11, 13
- 18. If you’re the biggest user of a technology, the challenges will be greatly amplified Scaling Pinterest Monday, November 11, 13
- 19. Growth January 2012 Scaling Pinterest Monday, November 11, 13
- 20. Growth April 2012 Page views per day Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 21. Growth April 2012 Page views per day Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 22. Growth April 2012 Page views per day · · · · Amazon EC2 + S3 + Edge Cast 135 Web Engines + 75 API Engines 10 Service Instances 80 MySQL DBs (m1.xlarge) + 1 slave each · · · 110 Redis Instances 60 Memcache Instances 2 Redis Task Manager + 60 Task Mar 2010 Processors · 3rd party sharded Solr Scaling Pinterest Monday, November 11, 13 Mar 2010 Jan 2011 Jan 2012 May 2012
- 23. Growth April 2012 Page views per day · 12 Engineers · · · · · 1 Data Infrastructure 1 Ops 2 Mobile 8 Generalists 10 Non-Engineers Mar 2010 Mar 2010 Scaling Pinterest Monday, November 11, 13 Jan 2011 Jan 2012 May 2012
- 24. Scaling Pinterest Scaling Pinterest Monday, November 11, 13
- 25. Growth April 2013 Page views per day April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
- 26. Growth April 2013 Page views per day April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
- 27. Growth April 2013 · · Page views per day Amazon EC2 + S3 + Edge Cast 400+ Web Engines + 400+ API Engines · 70+ MySQL DBs (hi.4xlarge on SSDs) + 1 slave each · · · 100+ Redis Instances 230+ Memcache Instances 10 Redis Task Manager + 500 Task Processors · 65+ Engineers (130+ total) April 2012 Scaling Pinterest Monday, November 11, 13 April 2013
- 28. Growth April 2013 · · Page views per day Amazon EC2 + S3 + Edge Cast 400+ Web Engines + 400+ API Engines · 70+ MySQL DBs (hi.4xlarge on SSDs) + 1 slave each · · · 100+ Redis Instances 230+ Memcache Instances 10 Redis Task Manager + 500 Task Processors · · · · · · · 65+ Engineers (130+ total) 8 services (80 instances) Sharded Solr 20 HBase 12 Kafka + Azkabhan 8 Zookeeper Instances 12 Varnish Scaling Pinterest Monday, November 11, 13 April 2012 April 2013
- 29. Growth April 2013 · 65+ Engineers · · · · · · · · · · Page views per day 7 Data Infrastructure + Science 7 Search and Discovery 9 Business and Platform 6 Spam, Abuse, Security 9 Web 9 Mobile 2 growth 10 Infrastructure 6 Ops 65+ Non-Engineers Scaling Pinterest Monday, November 11, 13 April 2012 April 2013
- 30. Scaling Pinterest Monday, November 11, 13
- 32. Arch Overview ELB Puppet StatsD Routing & Filtering (Varnish) Task Queue (Redis) Web App (Python) API App (Python / JS / HTML) Monit Sensu Task Processing (Python/Pyres) All connection pairings managed by ZooKeeper MySQL Service (Java/Finagle) Images (S3 + CDN) Scaling Pinterest Monday, November 11, 13 Memcache Mux (Nutcracker) Sharded MySQL Memcache Follower Service (Python/Thrift) Feed Service (Python/Thrift) Redis Search Service (Python/Thrift) HBase Spam Service (Python/Thrift)
- 33. Data Pipeline Web App (Python) API App (Python) Task Processing (Python/Pyres) Kafka S3 Copier Tripwire (Spam) S3 Qubole Pinball Scaling Pinterest Monday, November 11, 13 Redshift
- 34. Web App NGinX Website Rendering (x8) (Python / JS / HTML) API Scaling Pinterest Monday, November 11, 13
- 35. Our MySQL Sharding? http://www.infoq.com/presentations/Pinterest Scaling Pinterest Monday, November 11, 13
- 36. Choosing Your Tech Questions to ask • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Scaling Pinterest Monday, November 11, 13
- 37. Hosting Why Amazon Web Services (AWS)? • Variety of servers running Linux • Very good peripherals: load balancing, DNS, map reduce, basic security, and more • Good reliability • Very active dev community • Not cheap, but... Scaling Pinterest Monday, November 11, 13
- 38. Hosting Why Amazon Web Services (AWS)? • Variety of servers running Linux • Very good peripherals: load balancing, DNS, map reduce, basic security, and more • Good reliability • Very active dev community • Not cheap, but... • New instances ready in seconds Scaling Pinterest Monday, November 11, 13
- 39. Hosting AWS Usage • Route 53 for DNS • ELB for 1st tier load balance • EC2 Ubuntu Linux • Varnish layer • All web, API, background appliances • All services • All databases and caches • S3 for images, logs Scaling Pinterest Monday, November 11, 13
- 40. Code Why Python? • Extremely mature • Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Open Source Scaling Pinterest Monday, November 11, 13
- 41. Code Why Python? • Extremely mature • Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Open Source Some Java and Go... • Faster, lower variance response time Scaling Pinterest Monday, November 11, 13
- 42. Code Python Usage • All web backend, API, and related business logic • Most services Scaling Pinterest Monday, November 11, 13
- 43. Code Python Usage • All web backend, API, and related business logic • Most services Java and Go Usage • Varnish plugins • Search indexers • High frequency services (e.g., MySQL service) Scaling Pinterest Monday, November 11, 13
- 44. Production Data Why MySQL and Memcache? • Extremely mature • Well known and well liked • (MySQL) Rarely catastrophic loss of data • Response time to request rate increases linearly • Very good software support: XtraBackup, Innotop, Maatkit • Solid active community • Open Source Scaling Pinterest Monday, November 11, 13
- 45. Production Data MySQL and Memcache Usage • Storage / Caching of core data • Users, boards, pins, comments, domains • Mappings (e.g., users to boards, user likes, repin info) • Legal compliance data Scaling Pinterest Monday, November 11, 13
- 46. Production Data Why Redis? • Well known and well liked • Active community • Consistently good performance • Variety of convenient and efficient data structures • 3 Flavors of Persistence: Now, Snapshot, Never • Open Source Scaling Pinterest Monday, November 11, 13
- 47. Production Data Redis Usage • Follower data • Configurations • Public feed pin IDs • Caching of various core mappings (e.g., board to pins) Scaling Pinterest Monday, November 11, 13
- 48. Production Data Why HBase? • Small, but growing loyal community • Difficult to hire for, but... • Non-volatile, O(1), extremely fast and efficient storage • Strong Hadoop integration • Consistently good performance • Used by Facebook (bigger than us) • Seems to work well • Open Source Scaling Pinterest Monday, November 11, 13
- 49. Production Data HBase Usage • User feeds (pin IDs are pushed to feeds) • Rich pin details • Spam features • User relationships to pins Scaling Pinterest Monday, November 11, 13
- 50. Production Data What happened to Cassandra, Mongo, ES, and Membase? • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? Can you get help? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Scaling Pinterest Monday, November 11, 13
- 51. A 2nd chance... Scaling Pinterest Monday, November 11, 13
- 52. A 2nd Chance Stuff we could have done better • Logging on day 1 (StatsD, Kafka, Map Reduce) • Log every request, event, signup • Basic analytics • Recovery from data corruption or failure • Alerting on day 1 Scaling Pinterest Monday, November 11, 13
- 53. A 2nd Chance Stuff we could have done better • Shard our MySQL storage much earlier • Once you start relying on read slaves, start the timebomb countdown • We also fell into the NoSQL trap (Membase, Cassandra, Mongo, etc) • Pyres for background tasks day 1 • Hire technical operations eng earlier • Chef / Puppet earlier • Unit testing earlier (Jenkins for builds) Scaling Pinterest Monday, November 11, 13
- 54. A 2nd Chance Stuff we could have done better • A/B testing earlier • Decider on top of Zookeeper WATCH • Progressive roll out • Kill switches Scaling Pinterest Monday, November 11, 13
- 55. What’s next? Looking Forward • Continually improve Pinner experience • Help Pinners discover more of the things they love • Better uptime and lower latency • Faster development times • Reduce spam and abuse • Continually improve collaboration and build bigger, better, faster products • 180 Pinployees and beyond Scaling Pinterest Monday, November 11, 13
- 56. Have fun Scaling Pinterest Monday, November 11, 13
- 57. [email protected] pinterest.com/martaaay Monday, November 11, 13 [email protected] pinterest.com/yashh
- 58. [email protected] pinterest.com/martaaay Monday, November 11, 13 [email protected] pinterest.com/yashh
- 59. My 2nd Chance If I could do it all over again... • Stronger ACID transactional guarantees across multiple systems • Currently have: sometimes A, best effort C, I, D, no silent failure • Want: sometimes A, eventual C, I, D, no silent completion Scaling Pinterest Monday, November 11, 13
- 60. My 2nd Chance Transactional tasks • All tasks become a dependency tree of repeatable synchronous or asynchronous actions • All actions must be repeatable • Otherwise, must add repeatability • All tasks get a unique transaction number • Counters are tricky Scaling Pinterest Monday, November 11, 13
- 61. My 2nd Chance Transactional tasks • All tasks become a dependency tree of repeatable synchronous or asynchronous actions • Sync actions are executed in order • Async actions are executed in any order • Repeat until successful or too many failures • Too many failures -> put in per task failure queue • Gives eventual C, I, D • No silent completion and A require extra effort Scaling Pinterest Monday, November 11, 13
- 62. My 2nd Chance Transactional tasks example • Pin create sync • Write empty pin object • Write pin ID to board, likes, user’s pins, clear caches • Write pin object • Pin not shown until pin object created -> Atomicity! Scaling Pinterest Monday, November 11, 13
- 63. My 2nd Chance Transactional tasks example • Pin create async • Write pin to required user feeds and public feeds • Feeds are sorted sets. Reinsertion is okay. • Send emails, Facebook Likes, Twitter Tweets • Before send, check / record in temporary storage -> Gives (temporary) repeatability Scaling Pinterest Monday, November 11, 13
- 64. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/scalingpinterest