How Netflix Directs 1/3rd of Internet Traffic
517 likes•157,701 views
The document discusses Netflix's content delivery strategy presented at QCon San Francisco, detailing how Netflix manages its streaming infrastructure using a combination of a data plane and control plane to optimize user experience. It covers technical aspects such as content positioning, load distribution, and dynamic business rules that ensure efficient streaming across various devices and networks. Additionally, it highlights the evolution of Netflix’s architecture and the challenges faced in providing a seamless playback experience.
How Netflix Directs 1/3rd of Internet Traffic
- 1. How Netflix directs 1/3rd of Haley Tucker Mohit Vora QCon San Francisco Nov 16, 2015
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /netflix-streaming-arch
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 9. Project 366 #59; 280212 Days Gone By..., CC BY-SA, Pete 2012, Flickr
- 11. How do we build a streaming “tape”?
- 12. Determine the preferred experience DEVICETITLE CONNECTIONS COUNTRY NETWORK Broadband - wired or wifi Cellular - Edge, 3G, LTE, ... CUSTOMER
- 13. That’s exactly what I want ...now where can I get it?
- 14. Point the device to appropriate locations Steering
- 16. Uh-oh, the content is encrypted! Keymaster, CC BY-SA, Sean McGrath 2007, Flickr
- 17. LICENSE LICENSE
- 18. And...Action!
- 19. SESSION (START, STOP, PAUSE, RESUME, KEEPALIVE) SESSION EVENTS
- 20. LICENSE PLAYBACK MANIFEST GENERATE PLAYBACK MANIFEST SESSION (START, STOP, PAUSE, RESUME, KEEPALIVE) PLAYBACK LIFECYCLE
- 22. Data Plane (CDN)
- 23. What is a Content Delivery Network?
- 25. CONTENT RANK BYTES STREAMED PREDICTABLE VIEWING PATTERNS
- 26. FILLING WHEN YOU SLEEP Dreaming…,CCBY-SA,EleniBoulsaiki2009,Flickr
- 27. FILLING WHEN YOU SLEEP Open Connect A NETFLIX ORIGINAL READ XOR WRITE ONEWAY,CCBY-SA,KennyLouie2010,Flickr
- 40. STREAM ISP DATA CENTER NETFLIX DEVICE IXP DATA CENTER NFLX ROUTER ISP ROUTER ISP ROUTER IXP INTERCONNECTION NETFLIX
- 41. Control Plane
- 42. OPEN CONNECTSTREAM NETFLIX DEVICE CDN CONTROL PLANE DEVICE CONTROL PLANE DON’T KEEP SECRETS Network Proximity Content Positioning Load Distribution
- 43. Network Proximity Social Network in a Course, CC BY-SA, Hans Põldoja 2010, Flickr
- 46. Border Gateway Protocol TAKEAWAY BGP ROUTE 175.231.128.0/24 (+ proximity attributes) Use BGP
- 47. ISP2 DATA CENTER ISP2 BGP ROUTES CONTROL PLANE IXP DATA CENTER ISP1 BGP ROUTES ISP1 DATA CENTER ISP1 NFLX BGP ROUTE 175.231.128.0/24 (+ proximity attributes)
- 49. LOCALIZE TRAFFIC ISP DATA CENTER SERVE CACHE MISS
- 50. HOW DO WE DETERMINE WHAT CONTENT WILL BE POPULAR TOMORROW?
- 51. CHANGING CATALOG
- 53. MINIMIZE FILL CHURN ISP DATA CENTER OFF PEAK FILL
- 54. USE HISTORICAL DATA CONTENT RANKBYTES STREAMED bytesStreamed/bytesStored
- 55. IS ONE DAY OF HISTORY ENOUGH?
- 56. EXPONENTIALLY WEIGHTED MOVING AVERAGE WEIGHT DAYS AGO 0 10 20 30 40 … = 0.9 TAKEAWAY Weigh Recent Data Higher
- 57. HOW SHOULD CONTENT BE ALLOCATED?
- 58. MILLIONS OF FILES THOUSANDS OF SERVERS HOW SHOULD CONTENT BE ALLOCATED?
- 59. SVR4 SVR2 SVR1 SVR3 FILE1 FILE3 FILE1 TAKEAWAY ALLOCATE MULTIPLE REPLICAS RESILIENT TO CLUSTER CHANGES REPEATABLE Consistent Hashing
- 60. ISP2 DATA CENTER WHAT TO FILL? CONTROL PLANE IXP DATA CENTER WHERE TO FILL FROM? ISP1 DATA CENTER S3 FILL OVER HTTP
- 62. CONTENT RANKBYTES STREAMED LOTS OF THROUGHPUT LOTS OF STORAGE CONTENT WITH CONFLICTING CONSTRAINTS
- 63. SSD BASED SPINNING DISK BASED WITHIN CLUSTERS ON EACH SERVER MEMORY CONTENT RANK BYTES STREAMED SSD SPINNING DISK TAKEAWAY Tier Infrastructure
- 64. ACROSS SERVERS WITHIN CLUSTERS BALANCE BALANCE ACROSS EQUIDISTANT CLUSTERS HOW DO WE BALANCE LOAD?
- 66. USING CONTENT DISTRIBUTION HOW DO WE BALANCE LOAD?
- 67. FLIP A COIN AND WHEN WE HAVE EQUALLY ATTRACTIVE LOCATIONS TO SERVE FROM –
- 68. INCIDENT LOAD SYSTEM METRICS MAX INSANESANE HOW DO WE LOAD SERVERS OPTIMALLY?
- 69. … AMIDST EVER CHANGING INTERNET WEATHER
- 70. TRAFFIC t … AND DAILY TRAFFIC EBBS AND FLOWS
- 71. + SERVE STREAMS FEEDBACK - TRAFFIC EFFECT ON SYSTEM METRICS CONTROL WE INTRODUCE A FEEDBACK LOOP
- 73. TAKEAWAY PID CONTROLLER Process Variable Set Point Control Variable Current RPM Desired RPM Input Voltage System Metrics System Metrics Max Controlled Traffic DC MOTOR
- 74. TAKEAWAY PID CONTROLLER Process Variable Set Point Control Variable System Metrics System Metrics Max Controlled Traffic Current RPM Desired RPM Input Voltage LOADING SERVERS
- 75. ISP2 DATA CENTER CONTROL TO 80% CONTROL PLANE IXP DATA CENTER NO CONTROL ISP1 DATA CENTER 0.0 < CONTROL VAR < 1.0
- 76. TRAFFIC t NEXT HOP TRAFFIC SHIFTS TO NEXT HOP LOCATION
- 77. Steering
- 78. STREAM NETFLIX DEVICE CDN CONTROL PLANE PLAYBACK SERVICES STEERING Got URLs for f1, f2, …, fn? Yes, here’s the URLs PROXIMITY HEALTH CONTENT CASS KAFKA OPEN CONNECT
- 80. API STEERING SESSION MANIFEST DRM LICENSE How did we evolve from here...
- 81. API STEERING SESSION MANIFEST DRM LICENSE CLIENT SCRIPTS SERVICE LAYER RULES INSIGHTS ...to here. 5 SOLUTIONS CACHE
- 82. DEVICE CUSTOMER TITLE NETWORK Broadband - wired or wifi Cellular - Edge, 3G, LTE, ... CONNECTIONS COUNTRY High dimensionalityCHALLENGE
- 84. How can we quickly alter the playback experience in a targeted manner?
- 86. EXAMPLE RULES
- 91. Pinpoint what is brokenCHALLENGE Haystacks,CCBY-SA,JohnPavelka2008,Flickr
- 92. 3:00 AM : Pager goes off
- 94. OK...error code 105 is elevated. But why?
- 95. Indexed Logging
- 97. Large amount of stateCHALLENGE
- 98. How can we enable faster UIs and low-end devices?
- 99. We introduced a server-side caching tier MANIFESTSCUSTOMERA CUSTOMERA CUSTOMERB
- 100. Watch out for resiliency issues!! Ping Pong project, CC BY-SA, Michael Knowles 2008, Flickr
- 102. Managing device protocolsCHALLENGE Square peg, round hole, CC BY-SA, Simon Law 2006, Flickr
- 103. Can we allow devices to define their own protocols?
- 106. Enabling high-velocity innovationCHALLENGE CC BY-SA, Nathan E Photography 2008, Flickr
- 107. How can we expose new data with the least amount of churn?
- 108. API MANIFEST Stream ● Bitrate ● Framerate ● Dynamic Data Stream’ ● Bitrate ● Dynamic Data This works from API: ● stream.getBitrate() ● stream.getDynamicData().get(“FRAME_RATE”) Works both ways!
- 109. This works from CLIENT SCRIPT! ● stream.getDynamicData().get(“BIT_RATE”) ● stream.getDynamicData().get(“FRAME_RATE”) CLIENT SCRIPT Stream’’ ● Dynamic Data Works both ways! API MANIFEST Stream ● Bitrate ● Framerate ● Dynamic Data Stream’ ● Bitrate ● Dynamic Data Works both ways!
- 111. TAKEAWAYS ● BGP based proximity ● Tiered Infrastructure ● PID Controller ● EWMA for historical data ● Consistent Hashing ● Dynamic business rules ● Detailed domain insights ● Reduce client state ● Client-driven protocols ● Data pass-thru
- 112. TAKEAWAYS ● BGP based proximity ● Tiered Infrastructure ● PID Controller ● EWMA for historical data ● Consistent Hashing ● Dynamic business rules ● Detailed domain insights ● Reduce client state ● Client-driven protocols ● Data pass-thru Questions? Haley Tucker @hwilson1204 Mohit Vora @mohitvora
- 113. STREAM NETFLIX DEVICE NETFLIX DEVICE STREAM SPINNING DISK SERVERS SSD SERVERS WHAT TO FILL? WHERE TO FILL FROM? API CLIENT SCRIPTS SERVICE LAYER CACHE CONTROL DON’T KEEP SECRETS STEERING SESSION MANIFEST DRMLICENSE RULES CACHE INSIGHTS IXP DATA CENTER ISP1 ISP2 ISP2 BGP ROUTES ISP1 BGP ROUTES CONTROL TO 80%
- 114. ● Background image from https://www.flickr.com/photos/centralasian/4099515384, Image was cropped and red lines and dots were drawn on top, https://creativecommons.org/licenses/by/2.0/. ● Image from https://www.flickr.com/photos/28705377@N04/4142872268, No modifications made, https://creativecommons.org/licenses/by/2.0/. ● Image of cassette is from https://www.flickr.com/photos/comedynose/6939206771, Image was cropped, https://creativecommons.org/licenses/by/2.0/. ● Image of speaker is from https://www.flickr.com/photos/av_hire_london/5578975575, No changes made, https://creativecommons.org/licenses/by/2.0/. ● Image of television is from https://www.flickr.com/photos/jvcamerica/3660897684/, No changes made, https://creativecommons.org/licenses/by/2.0/. ● Image of text is from https://www.flickr.com/photos/dno1967b/5754743006, No changes made, https://creativecommons.org/licenses/by/2.0/. ● Background image from https://www.flickr.com/photos/mcgraths/866572532, Image was cropped, https://creativecommons.org/licenses/by/2.0/. ● Image from https://www.flickr.com/photos/thatguyfromcchs08/2300190277, Image is dimmed, https://creativecommons.org/licenses/by/2.0/. ● Image from https://www.flickr.com/photos/mknowles/3134373590, Image was cropped, https: //creativecommons.org/licenses/by-sa/2.0/. Image Attributions
- 115. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/netflix- streaming-arch