Living on the edge
3 likes•2,839 views
The document presents an overview of how Netflix manages streaming services, focusing on geo DNS, personalization, and content delivery infrastructure. It discusses the architecture involving AWS cloud services, user data processing, and advanced DNS configurations for traffic management and failover strategies. Additionally, it introduces 'denominator,' a multi-cloud DNS abstraction tool developed by Netflix, emphasizing its utility in enhancing traffic flow based on geographical locations.
![Basic list
$ denominator -n ultradns-prod zone
[UltraDNS#accountId] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#accountId] <--- HTTP/1.1 200 OK (2062ms)
[UltraDNS#zonesOfAccount] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#zonesOfAccount] <--- HTTP/1.1 200 OK (2169ms)](https://image.slidesharecdn.com/livingontheedge-130419000308-phpapp01/85/Living-on-the-edge-33-320.jpg)
![Add Record
$ denominator -n ultradns-test record -z ultradnstest.denominator.io.
replace -n www.ultradnstest.denominator.io. -t A -d 192.0.2.1
[UltraDNS#recordsInZoneByNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#recordsInZoneByNameAndType] <--- HTTP/1.1 200 OK (1663ms)
[UltraDNS#createRRPoolInZoneForNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#createRRPoolInZoneForNameAndType] <--- HTTP/1.1 200 OK
(2108ms)
[UltraDNS#createRecordInRRPoolInZone] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#createRecordInRRPoolInZone] <--- HTTP/1.1 200 OK (3263ms)](https://image.slidesharecdn.com/livingontheedge-130419000308-phpapp01/85/Living-on-the-edge-34-320.jpg)
Living on the edge
- 1. Living on the Edge Adrian Cole @adrianfcole #netflixoss @denominatorOSS http://www.linkedin.com/in/adrianforrestcole
- 2. adrian • engineer at Square • founded apache jclouds • focus on (small) libraries * Worked exclusively on what’s in this deck while at Netflix!
- 3. How Netflix Streaming Works Geo DNS at Netflix Denominator
- 4. Netflix Member Web Site Home Page Personalization Driven – What goes on to make this?
- 5. How Netflix Streaming Works Consumer Electronics User Data Web Site or Discovery API AWS Cloud Services Personalization CDN Edge Locations DRM Customer Device (PC, PS3, TV…) Streaming API QoS Logging OpenConnect CDN Boxes CDN Management and Steering Content Encoding
- 6. Content Delivery Service Open Source Hardware Design + FreeBSD, bird, nginx
- 8. Real Web Server Dependencies Flow (Netflix Home page business transaction as seen by AppDynamics) Each icon is three to a few hundred instances across three AWS zones Cassandra memcached Start Here Three Personalization movie group choosers (for US, Canada and Latam) Web service S3 bucket
- 9. Netflix entrypoints are Geo DNS Geo == Directional
- 10. DNS Things • Nameserver – Server that listens on port 53 for queries • Resolver – Client that makes queries • API – Creates and controls configuration and data on the nameservers.
- 11. DNS Lingo • Zone – Name (ex. denominator.io.) – Id (if ambiguous) • Records – Name (ex. www.denominator.io.) – Type (ex. CNAME) – TTL (ex. 300) – RData (ex. myLB-1234567890.us-east1.elb.amazonaws.com)
- 12. Record Set • Records visible to the resolver that have the same name and type (also TTL). • Ex. If www.denominator.io has 4 ip addresses, they can be in the same recordset. • Concept is helpful for DNSSEC
- 13. Geo (Directional) Record Set Visible to clients in specific territories Group or Qualifier: maybe “US-WEST” Territories: subset directed to this rrset
- 14. EU-WEST-1 cbp.nccp.netflix.com Zone A Western US or Canada? Zone C Cassandra Replicas Europe? Zone B Cassandra Replicas Cassandra Replicas Everybody Else US-EAST-1 US-WEST-2 Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas
- 15. Changing territories reshapes traffic
- 16. DNS for Region Failover aka: why this deck is labeled advanced
- 18. Route53 Alias Reference to an AWS resource such as a load balancer (ELB). Appears to the client as an ddress record set. if IPv4 or for IPv6
- 19. Example setup for failover • Point Geo record set to a normal, site-specific . • These s point to a Route53 ELB lias.
- 21. (at least) 2 failover options • When Route53 API is available • When Route53 API isn’t available, but your normal DNS is
- 22. On Failover When Route53 API is available… Update the route53 alias to point to the healthy region.
- 24. On Failover When Route53 API isn’t available, but your normal DNS is… Update normal CNAME for each for each host and geo group pointing to a healthy ELB.
- 26. Implications • You are pointing to a healthy region, not a specific ELB – No read-lookups needed – Can increase ELBs in healthy region • More setup needed – Must create Geo + region * (normal CNAME + A, AAAA Aliases in Route53)
- 27. PORTABLE CONTROL OF DNS CLOUDS
- 28. A Cloud Native Open Source Platform
- 30. Feature Set • • • • Do stuff in batches Cleanly handle advanced records Play nice with persistence Don’t do too much • … Use cool things like Dagger
- 31. Model ResourceRecordSet is the central class Record types (A, CNAME, etc) extend Map<String, Object> mxData.preference() mxData.get("preference”)
- 32. Hello Denominator get denominator from bintray or homebrew create ~/.denominatorconfig name: ultradns-prod provider: ultradns credentials: username: your_user password: your_password denominator -n ultradns-prod zone list
- 33. Basic list $ denominator -n ultradns-prod zone [UltraDNS#accountId] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#accountId] <--- HTTP/1.1 200 OK (2062ms) [UltraDNS#zonesOfAccount] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#zonesOfAccount] <--- HTTP/1.1 200 OK (2169ms)
- 34. Add Record $ denominator -n ultradns-test record -z ultradnstest.denominator.io. replace -n www.ultradnstest.denominator.io. -t A -d 192.0.2.1 [UltraDNS#recordsInZoneByNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#recordsInZoneByNameAndType] <--- HTTP/1.1 200 OK (1663ms) [UltraDNS#createRRPoolInZoneForNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#createRRPoolInZoneForNameAndType] <--- HTTP/1.1 200 OK (2108ms) [UltraDNS#createRecordInRRPoolInZone] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#createRecordInRRPoolInZone] <--- HTTP/1.1 200 OK (3263ms)
- 35. From Java mgr = Denominator.create(”ultradns”, (username, password)) for (Zone zone : mgr.api(). processZone(zone); } mgr. . ()) { (“denominator.io.”) (a("www.denominator.io.", 300, "192.0.2.1"));
- 36. Thanks! adrianco jdamick colmmacc everett-toews digitalsanctum quidryan cfieber davidmc24 (github ids)
- 37. Takeaway Geo (Directional) DNS helps you manage the flow of traffic based on location. Vendors engagement in OSS >> better place for availability Denominator is a multi-cloud DNS abstraction built as a library and a cli. https://github.com/Netflix/denominator https://groups.google.com/forum/#!forum/denominator-user http://www.linkedin.com/in/adrianforrestcole @adrianfcole #netflixoss @denominatorOSS
- 38. Denominator Sidebar Dagger A fast dependency injector for Android and Java.
- 39. Dagger • Guice for libraries, particularly android • Speed and Simplicity over features • Extension averse, feature conservative • Friendly forks
- 40. Declare Dependencies class DNSAPIManager { } @Inject ZoneApi zoneApi; ... class Route53ZoneApi implements ZoneApi { } @Inject Route53 route53; ...
- 41. Satisfy Dependencies @Module(injects = DNSApiManager.class … class Route53Module { @Provides ZoneApi zoneApi(Route53ZoneApi zone) { return zones; } @Provides @Singleton Route53 route53(Feign feign, Route53Target target) { return feign.newInstance(target); } ...
- 42. Create the Graph manager = Denominator.create(new Route53Module()); class Denominator { public static DNSApiManager create(Object module) { ObjectGraph objectGraph = ObjectGraph.create(module); return objectGraph.get(DNSApiManager.class); } ... }
- 43. javac with compiler dependencies { compile "com.squareup.dagger:dagger” // 52k! provided "com.squareup.dagger:dagger-compiler” ... } Dagger’s compiler writes binding classes instead of reflection binding at runtime.
- 44. Takeaway Dagger is a leaner version of Guice, great for android and libraries. http://square.github.io/dagger/ https://groups.google.com/forum/#!forum/dagger-discuss
Editor's Notes
- #37: Adrian Cockroft championed the project in Netflix, also acting as a design mentor.Jeff Damick was the first pre-oss developer, also helping adjust UltraDNS.Colm helped design data structures and vetted concepts against Amazon Route53. Everett helped implement OpenStack and Rackspace DNS services.Shane helped shine up the commandline tool, particularly yaml config.Justin helped with single-jar syntax for gradleCameron wrote gradle scripts to setup Eclipse and Idea for Dagger processingDavid maintains Feign, the http interface layer used by denominator.