Builders' Day- Mastering Kubernetes on AWS
0 likes293 views
The document discusses running Kubernetes on AWS. It begins with an agenda on Kubernetes, AWS, and mastering Kubernetes. It then provides examples of running containers locally and considerations for moving workloads to production. It discusses Kubernetes networking concepts and how the AWS VPC CNI plugin handles IP allocation. It also covers different Kubernetes service types like NodePort, LoadBalancer, ExternalName and Ingress resources. The document shares an example of implementing logging with EFK and discusses Snap's use of Amazon EKS to move to a microservices architecture.
![[AWS Container Service] Introducing AWS Fargate](https://cdn.slidesharecdn.com/ss_thumbnails/200introducingawsfargate-190326085914-thumbnail.jpg?width=560&fit=bounds)
![[AWS Container Service] Getting Started with Kubernetes on AWS](https://cdn.slidesharecdn.com/ss_thumbnails/200gettingstartedwithkubernetesonaws-190326085832-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
Similar to Builders' Day- Mastering Kubernetes on AWS (8)
More from Amazon Web Services LATAM (20)
Ad
Recently uploaded (20)
Ad
Builders' Day- Mastering Kubernetes on AWS
- 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 2. Martin Dominguez Solutions Architect AWS Mastering Kubernetes on AWS
- 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • Kubernetes • AWS • Mastering
- 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. $ vi Dockerfile $ docker build -t mykillerapp:0.0.1 . $ docker run -it mykillerapp:0.0.1 Running containers in development is easy…
- 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Moving to production: data plane Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS Server Guest OS AZ 1 AZ 2 AZ 3
- 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Moving to production: control plane etcd etcdetcd Master Master Master Availability zone 1 Availability zone 2 Availability zone 1
- 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Yup. This is hard. - Lucas Käldström, volunteer ambassador for the Cloud Native Computing Foundation
- 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Run Kubernetes for me.” 51% of Kubernetes workloads run on AWS today — Cloud Native Computing Foundation
- 9. “Give us an upstream experience.” “Please don’t fork.” “Make sure it’s compatible”
- 10. Amazon EKS is Kubernetes Certified
- 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EKS mycluster.eks.amazonaws.com EKS Workers Kubectl AZ 1 AZ 2 AZ 3 VPC
- 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cross account Kubernetes EKS VPCCustomer VPC Worker Nodes EKS-Owned ENI Kubernetes API calls Exec, Logs, Proxy Internet
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. • All the pods can communicate with each other directly without NAT • All the nodes can communicate with all pods (and vice versa) without NAT • The IP that a pod sees itself as is the same IP that others see it as The three rules of Kubernetes networking…
- 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS VPC CNI plugin ENI Secondary IPs: 10.0.0.1 10.0.0.2 10.0.0.1 10.0.0.2 ENI 10.0.0.20 10.0.0.22 Secondary IPs: 10.0.0.20 10.0.0.22 ec2.associateaddress() VPC Subnet – 10.0.0.0/24 Instance 1 Instance 2 VPC
- 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS VPC CNI plugin – understanding IP allocation • primary CIDR range è RFC 1918 addresses è 10/8, 172.16/12, 192.168/16 Used in EKS for: • Pods • Cross-account ENIs for (masters à workers) communication (exec, logs, proxy etc.) • Internal Kubernetes services network (10.100/16 or 172.20/16 – chosen based on your VPC range) Setup: • EKS cluster creation è provide list of subnets (in at least 2 AZs!) è tagging
- 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS VPC CNI plugin – understanding IP allocation • secondary CIDR ranges (new!) è non-RFC 1918 address blocks (100.64.0.0/10 and 198.19.0.0/16) Used in Amazon EKS for: • Pods only How? • Amazon EKS custom network config è enable è create ENIConfig CRD è annotate nodes CNI 1.2.1+
- 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Exposes the service on a cluster-internal IP • Only reachable from within the cluster • Access possible via kube-proxy • Useful for debugging services, connecting from your laptop or displaying internal dashboards Kubernetes ServiceType: ClusterIP
- 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes ServiceType: NodePort • Exposes the service on each Node’s IP at a static port. • Routes to a ClusterIP service, which is automatically created. • from outside the cluster: <NodeIP>:<NodePort> • 1 service per port • Uses ports 30000-32767
- 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Exposes the service externally using a cloud provider’s load balancer. • NodePort and ClusterIP services (to which LB will route) automatically created. • Each service exposed with a LoadBalancer (ELB or NLB) will get its own IP address • Exposes L4 (TCP) or L7 (HTTP) services Kubernetes ServiceType: LoadBalancer
- 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service load balancer: NLB apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" spec: externalTrafficPolicy: Local ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer
- 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service load balancer: NLB • NLB supports forwarding the client’s IP through to the node • .spec.externalTrafficPolicy = Local è client ip passed to pod • Nodes with no matching pods will be removed by specified NLB’s health check .spec.healthCheckNodePort • Use DaemonSet or pod anti-affinity to verify even traffic split
- 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Maps: service => CNAME(externalName field) • No proxying • Accessing my-service works in the same way as other Services • redirection happens at the DNS level (rather than via proxying or forwarding) kind: Service apiVersion: v1 metadata: name: my-service namespace: prod spec: type: ExternalName externalName: my.database.example.com Kubernetes ServiceType: ExternalName
- 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. • exposes HTTP/HTTPS routes to services within the cluster • Many implementations: ALB, Nginx, F5, HAProxy etc • Default Service Type: ClusterIP Kubernetes Ingress Object
- 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ALB Ingress Controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTP ListenerHTTPS Listener Rule: /cheesesRule: /charcuterie TargetGroup: Green (IP Mode) TargetGroup: Blue (Instance Mode) NodePort NodePort
- 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Implementing logging with EFK fluentd is an open source data collector providing a unified logging layer elasticsearch is a distributed, RESTful search and analytics engine kibana lets you visualize your Elasticsearch data
- 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Implementing logging with EFK - DIY EKS Worker pod fluentd daemonset
- 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Implementing logging with EFK - DIY EKS Worker pod fluentd daemonset
- 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer Story: Snap
- 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Snap’s Approach to Infrastructure Goals Flexibility Security Availability / Performance Cost Reduction Minimize operational work
- 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Technical State of Snap 2016 Small number of large monolithic applications Projects slowed due to inflexibility Infrastructure started to be the long pole
- 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Technical State of Snap 2016 (cont.) Organizational boundaries also got in the way Work was single threaded through central teams New product teams were not happy with the constraints
- 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Technical State of Snap 2016 (cont.) Regionalization was impossible in our old architecture Performance matters a lot Stuck with “the way things have always been done” Teams couldn’t “spin up their service in a new region”
- 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service-oriented architectures We had seen the value of microservices in other organizations Solve a smaller problem in the best way possible Separates data, responsibilities (security) Organizational division Scaling tied to usage But what is the best approach to SOA?
- 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Portability Our strategy has always been to use the best of breed Containers were obvious. Orchestration is half the battle Let a vendor do that for us.
- 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Amazon EKS Still highly portable Less operations – managed control plane (and more management coming) Run it ourselves? Most portability But a lot of complexity And a lot of operational work Solves a lot of the problems of managing a large set of services + =
- 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EKS at Snap 2018 6 services in production today 2019 30-50 services in production by end of 2019 End State Several hundred services on EKS Multi-region Different policies on redundancy based on service
- 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EKS at Snap Production services at scale on EKS 7,500 cores 250,000 transactions per second High density pod to node ratio in a secure service mesh 2019 – Global regionalization
- 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EKS Cluster Standard Architecture Kubernetes Pod Envoy Proxy Application Service AuthN/Z Logs Metrics
- 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Results Saved a lot of money Containers and Amazon EKS give us a lot of flexibility to adopt new technologies Envoy is one example, but we expect to continue to reap this benefit Performance improvements Amazon EKS is already widely adopted at Snap
- 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Next Steps Continue the march Service by service API by API Optimize regionalization
- 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Recap Kubernetes and AWS • 51% of Kubernetes workloads run on AWS • Amazon EKS is Kubernetes Certified Kubernetes Networking • The three rules of Kubernetes networking • CNI plugin • Kubernetes ServiceTypes Kubernetes Security • Pod permissions to an AWS service Kubernetes Logging • Implementing logging with EFK
- 45. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Martin Dominguez [email protected] @mp_dominguez