Stranger Things: The Forces that Disrupt Netflix
1 like1,450 views
The document discusses the challenges faced by Netflix in managing distributed systems and the strategies implemented to mitigate failures. It highlights the importance of verifying data consistency, managing resource constraints, and implementing circuit breakers to handle service disruptions effectively. Key takeaways include the need for light fallbacks and application sharding based on operational characteristics.
Stranger Things: The Forces that Disrupt Netflix
- 1. The Forces That Disrupt Netflix Nov. 7, 2016Haley Tucker
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ netflix-resilience
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 5. # A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. --Leslie Lamport
- 11. Notes on Distributed Systems for Young Bloods # Distributed systems are different because they fail often. --Jeff Hodges
- 12. TABLE OF CONTENTS CHAPTER 1: THE WEIRD DATA IN THE CATALOG • Metadata impacts on availability CHAPTER 2: THE VANISHING OF CRITICAL SERVICES • Crashing services and cascading failures CHAPTER 3: THE THROTTLE • Latency spikes and the impact of fallbacks FORCES AT WORK
- 14. Whoops, something went wrong… Netflix Streaming Error We’re having trouble playing this title right now. Please try again later or select a different title.
- 15. CHAPTER ONE THE WEIRD DATA IN THE CATALOG
- 17. 45 MINUTES!! Clock, by heyyobecky4lyfe, Tumblr
- 20. Amazon S3 Netflix Playback Service { String msg = “This should never happen!”; throw new IllegalStateException(msg); }
- 21. MITIGATION BLAST RADIUS Explosion, CC BY 2.0, Andrew Kuznetsov 2008, Flikr
- 22. Amazon WS Global Infrastructure
- 23. Amazon WS Global Infrastructure STAGGERED ROLLOUT
- 24. Pager Diagnosis?
- 26. Canary, CC BY 2.0, Steve P2008 2014, Flikr PREVENTION CANARIES
- 27. TRADITIONAL CANARY Canary (New Code) Baseline (Old Code) TrafficTraffic Video Metadata Service Amazon S3 Netflix Services Netflix Services Netflix Services Netflix ServicesNetflix Service Source System Source System Traffic
- 30. Australia with AAT, CC BY-SA 2.0, Ssolbergj 2010, Wikimedia SEEING RETURNS
- 31. Verify consistency prior to applying state changes. …one tool is a data canary.
- 32. CHAPTER TWO THE VANISHING OF CRITICAL SERVICES
- 35. # A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. --Leslie Lamport
- 36. Proxy/Routing Devices LOG DATA Log Data Service Traffic Cassandra Playback ServiceNetflix Playback Service Netflix Playback Service Edge Service Edge Service Edge Service Playback Service Traffic
- 38. Proxy/Routing Devices Proxy Log Data Service Traffic Cassandra Playback ServiceNetflix Playback Service Netflix Playback Service Edge Service Edge Service Edge Service Playback Service Traffic
- 40. Proxy/Routing Devices CASCADING FAILURE Log Data Service Traffic Cassandra Playback ServiceNetflix Playback Service Netflix Playback Service Edge Service Edge Service Edge Service Playback Service Traffic
- 41. { throw new OutOfMemoryError(); } Log Data Service Cassandra Playback Service
- 42. PREVENTION MANAGING RESOURCE CONSTRAINTS Whatever you ask, CC BY-SA 2.0, Kreg Steppe 2008, Flikr
- 43. Astronomical Clock, CC BY 2.0, Andrew Fleming 2011, Flikr REDUCE SURFACE AREA
- 44. 1 Keep Only Dependencies which are Necessary SO MANY JARS!!
- 45. 2 LIMIT “MAGIC” Magic, CC BY-ND 2.0, Daniel Lee 2013, Flikr
- 46. 3 Medusa Kill Switch, CC BY-NC-ND 2.0, Scott Hart 2013, Flikr ADD KILL SWITCHES
- 47. DEV FAVOR IMMUTABILITY Playback Service TEST Playback Service PROD Playback Service 4
- 48. try { remoteService.call(); } catch( Throwable t ){ //Oops! System.exit(1); } Log Data Service Cassandra Playback Service Proxy/Routing Traffic
- 49. It's Electric, CC BY ND 2.0, Alan Hochberg 2008, Flikr MITIGATION CIRCUIT BREAKERS
- 51. Wrecking Ball in Building, CC BY 2.0, Jason Eppink 2008, Flikr FAILURE TESTING
- 52. Proxy/Routing Devices FAILURE TESTING Log Data Service Traffic Cassandra Playback Service Automating Chaos Experiments in Production by Ali Basiri Applying Failure Testing Research @Netflix by Kolton Andrus and Peter Alvaro
- 53. Manage resource constraints by reducing surface area. Leverage circuit breakers and rigorously test failures.
- 56. Proxy/Routing Devices PLAYBACK ARCHITECTURE Edge Service Edge Service Edge Service Playback Service Traffic Traffic URL Service
- 57. NETFLIX CLIENT JARS Playback Service URL Service URL Client Circuit-breakers and Fallbacks Metrics Retries and Timeouts RPCService Discovery
- 59. Playback Service Traffic Concurrent Requests Throttled Requests (HTTP 503) THROTTLING
- 62. NETFLIX CLIENT JARS Playback Service URL Service URL Client Circuit-breakers Metrics Retries and Timeouts RPCService Discovery Heavy Fallback
- 63. FALLBACK TESTING With 100% Fallback, CPU held at 90% 15 RPSNo fallback, CPU held at 90% 58 RPS Siege: https://github.com/JoeDog/siege
- 65. URL Service Playback Service Edge Service Proxy/Routing Traffic } return Response .status(503) .build(); }
- 66. REQUEST BUCKETING NON-CRITICAL Experience or Performance Impact CRITICAL Customer Streaming Impact Fire Buckets at Oakworth Statione, CC BY 2.0, Tim Greene 2015, Flikr
- 67. APPLICATION SHARDING Non-Critical Playback Service Proxy/Routing Devices Edge Service Edge Service Edge Service Traffic Traffic URL Service Critical Playback Service Non-Critical URL Service
- 68. CRITICAL Country Road at Sunrisee, CC BY-SA 2.0, Susanne Nilssone 2014, Flikr
- 69. NON-CRITICAL Traffice, CC BY-NC 2.0, jonbgeme 2008, Flikr
- 70. APPLICATION SHARDING Non-Critical Playback Service Proxy/Routing Devices Edge Service Edge Service Edge Service Traffic Traffic Critical Playback Service URL Service Non-Critical URL Service
- 71. No heavy fallbacks!! Fallbacks should be light and fast. Shard your application based on operational characteristics.
- 73. KEY TAKEAWAYS CHAPTER 1: THE WEIRD DATA IN THE CATALOG • Verify consistency prior to applying state changes. • One tool is a data canary. CHAPTER 2: THE VANISHING OF CRITICAL SERVICES • Manage resource constraints by reducing surface area. • Leverage circuit breakers and rigorously test failures. CHAPTER 3: THE THROTTLE • No heavy fallbacks!! Fallbacks should be light and fast. • Shard your application based on operational characteristics.
- 74. The unexpected will happen. Plan to fail.
- 77. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ netflix-resilience