CMP376 - Another Week, Another Million Containers on Amazon EC2
1 like882 views
Netflix’s container management platform, Titus, powers critical aspects of the Netflix business, including video streaming, recommendations, machine learning, big data, content encoding, studio technology, internal engineering tools, and other Netflix workloads. Titus offers a convenient model for managing compute resources, enables developers to maintain just their application artifacts, and provides a consistent developer experience from a developer’s laptop to production by leveraging Netflix container-focused engineering tools.
![[AWS Container Service] Getting Started with Kubernetes on AWS](https://cdn.slidesharecdn.com/ss_thumbnails/200gettingstartedwithkubernetesonaws-190326085832-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to CMP376 - Another Week, Another Million Containers on Amazon EC2 (9)
![[AWS Container Service] Introducing AWS Fargate](https://cdn.slidesharecdn.com/ss_thumbnails/200introducingawsfargate-190326085914-thumbnail.jpg?width=560&fit=bounds)
Ad
More from aspyker (13)
Ad
Recently uploaded (20)
CMP376 - Another Week, Another Million Containers on Amazon EC2
- 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Another Week, Another Million Containers on Amazon EC2 Andrew Spyker Software Engineering Manager Netflix C M P 3 7 6 Joe Hsieh Principal Technical Account Manager Amazon Web Services
- 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why containers? Given our VM architecture comprised of … Amazingly resilient Microservice driven Cloud native CI/CD DevOps enabled Elastically scalable Do we really need containers?
- 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What was missing from our VM environment? Packaging • Simple to customize application focused artifacts • Especially for growth of polyglot environments • Notably for platforms with OS level dependencies Local development • Ability to run applications locally on developer laptops Simple way to manage compute resources • Especially for ad hoc batch processing
- 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus, Netflix’s container management platform Scheduling • Service & batch job lifecycle • Resource management Container execution • AWS Integration • Netflix Ecosystem Support Job and Fleet Management Batch Resource Management & Optimization Container Execution Service
- 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The Titus team • Design • Develop • Operate • Support
- 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus and containers product strategy • Ordered priority focus on • Developer velocity • Reliability • Cost efficiency Easy migration from VMs to containers Easy container integration with VMs and Amazon Services Focus on just what Netflix needs
- 8. “Our focus is to leverage EC2 deeply in Titus, not abstract it away or implement similar features. We see this as a differentiator of Titus versus other container management solutions.”
- 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Mesos High level architecture Titus Control Plane • API • Scheduling • Job Lifecycle Control Fenzo Titus Agents User Containers Docker Mesos Agent Netflix System Services AWS Virtual Machines Docker Registry Cassandra AWS Auto Scaling
- 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EC2 virtual machine portability Early on we decided a container MUST … • Natively integrate with VPC for networking • Natively integrate with security groups for firewalling • Work with IAM based Amazon Web Services
- 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Key leverage points
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EC2 GPUs - 10’s of p2.8xlarges Memory optimized - 100’s of r4.16xlarges General purpose - 1000’s of m4.16xlarges
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. VPC and security groups EC2 VM ENI0 (to control plane) ENI1 SG = w ENI2 SG = x ENIn SG = z Container 1 SG = w ENI1 IP1 Container 2 SG = w ENI1 IP2 Container 3 SG = y ENI3 IP1 Titus Container Mgmt
- 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. IAM based services EC2 VM ENI0 Container 1 eth0 ethMD ENI1 Titus Metadata Proxy Normal networking 169.254.169.254 Amazon Metadata Service and Security Token Service (STS)
- 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus Host Instance cryptographic identity Metatron Service User Container
- 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. All I really needed to know about containers, I learned from Titus …
- 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choices for Auto Scaling Titus applications Use the two existing Netflix autoscaling engines we already had • Pro: Code existed • Con: Lacking features, we’d have to operate Write a new one Look for one from Amazon Web Services
- 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choices for Auto Scaling Titus applications Use the two existing Netflix autoscaling engines we already had Write a new one • Pro: Would be specific to our needs • Con: Would be lacking features, we’d have to operate Look for one from Amazon Web Services
- 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choices for Auto Scaling Titus applications Use the two existing Netflix autoscaling engines we already had Write a new one Look for one from Amazon Web Services • Pro: Already well understood for VMs, feature-rich • Con: Only works for VMs
- 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A true story
- 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A product manager introduction, development team interchanges, and multiple iterations later …
- 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Application Auto Scaling with custom resources
- 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuring Auto Scaling in Spinnaker
- 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus and Application Auto Scaling integration User Containers Control Plane
- 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus API call pattern CreateNetworkInterface Total CreateNetworkInterface Throttled AttachNetworkInterface Total AttachNetworkInterface Throttled ModifyNetworkInterfaceAttribute Total ModifyNetworkInterfaceAttribute Throttled AssignPrivateIpAddresses Total AssignPrivateIpAddresses Throttled
- 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling group Auto Scaling group Auto Scaling group An infrastructure view of applications
- 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. An infrastructure view of applications Auto Scaling group VPC
- 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. API calls RunInstances CreateNetworkInterface AttachNetworkInterface AssignPrivateIpAddress ModifyNetworkInterface
- 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Netflix regional failover Kong evacuation of us-east-1 Traffic diverted to other regions Fail back to us-east-1 Traffic moved back to us-east-1 us-east-1 eu-west-1
- 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure challenge • Increase capacity during scale up of savior region • Launch 1000s of containers in seven minutes
- 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Easy right? “we reduced time to schedule 30,000 pods onto 1,000 nodes from 8,780 seconds to 587 seconds”
- 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Easy right? “we reduced time to schedule 30,000 pods onto 1,000 nodes from 8,780 seconds to 587 seconds”
- 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus can do this by … • Dynamically changeable scheduling behavior • Fleet wide networking optimizations
- 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Normal scheduling VM1 App 1 App 2 ENI 1 App 2 IP1 IP1 VM2 App 1 ENI 1 IP1 VMn App 1 App 2 ENI 1 App 2 IP1 IP1 Trade-off for reliability
- 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failover scheduling VM1 App 1 App 2 ENI 1 App 2 IP1 IP1 VM2 App 1 ENI 1 IP1 VMn App 1 App 2 ENI 1 App 2 IP1 IP1 App 1 App 1 App 1 App 1 App 1 App 2 App 2 IP2, IP3 IP2, IP3, IP4 IP2, IP3 Trade-off for speed
- 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. On each host Change when create and attach ENIs is performed • Moved this to instance start time • No longer needed on-demand Need to burst allocate IP addresses • Opportunistically batch allocate at container launch time • Likely if one container was launched more are coming • Garbage collect unused later
- 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus API pattern ModifyNetworkInterfaceAttribute Total ModifyNetworkInterfaceAttribute Throttled AssignPrivateIpAddresses Total AssignPrivateIpAddresses Throttled
- 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Results us-east-1 / prod containers started per minute } 7500 Launched in 5 minutes
- 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Netflix load balancing
- 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. IP based Application Load Balancing
- 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuring EC2 load balancers in Spinnaker
- 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Titus and Load Balancing integration User Containers Control Plane IP Target Group
- 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use cases on Titus • Netflix API, Node.js Backend UI Scripts • Machine Learning (GPUs) for personalization • Encoding and Content use cases • Netflix Studio use cases • CDN tracking and planning • Massively parallel CI system • Data Pipeline routing and SPaaS • Big Data platform use cases Batch Q4 15 Basic Services 1Q 16 Production Services 4Q 16 Customer Facing Services 2Q 17
- 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Q4 2018 container usage Common Jobs launched 255K jobs / day Different applications 1K+ different images Isolated Titus deployments 7 stacks Services Single app cluster size 5K (real), 12K containers (benchmark) Hosts managed 7K VMs (435,000 CPUs) Batch Containers launched 450K / day (750K / day peak) Hosts managed (autoscaled) 55K VMs / month
- 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Open Source Open sourced April 2018 Help other communities by sharing our approach
- 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Current and future work Advanced CPU Isolation Opportunistic Workloads Nitro and Bare Metal Instances Next Amazon and Netflix Partnership
- 54. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Andrew Spyker @aspyker Joe Hsieh
- 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.