Network services on Kubernetes on premise
Download as ODP, PDF1 like905 views
Deep dive into Kubernetes Networking and presentation of a usecase of running network services like DNS on a bare metal Kubernetes cluster for a major Danish e-sport event.
Ad
More Related Content
What's hot (20)
Similar to Network services on Kubernetes on premise (20)
![[KubeCon NA 2020] containerd: Rootless Containers 2020](https://cdn.slidesharecdn.com/ss_thumbnails/kubeconna2020rootlesscontainers-201117203340-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
Ad
Network services on Kubernetes on premise
- 1. Network Services on Kubernetes on-premise Hans Duedal Network Services Engineer @ NPF CTO @ Visma e-conomic
- 2. Networking in Kubernetes Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. Every pod gets its own IP address. Fundamental requirements 1. all containers can communicate with all other containers without NAT 2. all nodes can communicate with all containers (and vice-versa) without NAT 3. the IP that a container sees itself as is the same IP that others see it as
- 3. Container Network Interface ● Originated at CoreOS as part of rkt ● Cloud Native project ● Many container runtimes, ie. kubernetes, rkt, mesos ● If your network solution adheres to the CNI Spec, the runtimes can use it ● Sometimes the choice of CNI plugins are (almost) made for you, ie. on public clouds ● For an on-premise project, choosing the right CNI solution is critical to success
- 5. Project Calico ● IP-in-IP with option for BGP Based ● Independent of Kubernetes (ships with it’s own etcd cluster) ● Backbone of Github.com’s metal cloud https://thenewstack.io/github-goes-kubernetes-tells/ ● Policy based access control ● Fully featured for Datacenter Level Kubernetes Networking ● Doesn’t make much sense to use without BGP
- 6. Kube-router Three components 1. IPVS/LVS (Part of Linux kernel since 1999) 2. GoBGP 3. Kuberouter itself ● BGP Based https://cloudnativelabs.github.io/post/2017-05-22-kube-pod-networking/ ● Heavy lifting by IPVS/LVS (in-kernel load balancer) and the routing tables ● Kubernetes specific ● Small and lightweight, with excellent community on slack ● Network Policies (but not as advanced as Calico)
- 7. flannel ● Full userspace software based solution, no hardware or kernel involved ● IP-in-IP based ● Often the goto/default choice for on-premise deployments ● Only network, no policy ● Networking: “Developer Edition” ● CoreOS project, but plays extremely well with Kubernetes ● The reference plugin for CNI
- 8. How Rancher does it ● Magically network nodes in AWS with GCE or on-premise ● It’s all an illusion, it sets up a full mesh of IPSec (VPN) tunnels between each node ● High overhead of IPSec stack and encryption ● Easy to use, and most of the time “just works”
- 9. That’s all fine, but you said something about e-sports?
- 11. NPF in Numbers ● 3600 Amps of mains, 3 phase power ● Almost 14 kilometers of network cables (2.4 km fiber, 11.5 km copper) ● 400+ volunteers ● 300+ network switches ● Built up in five days, 47 hour event, and then torn down in two days ● Almost 6000 gamers with their computers ● Internet consumption the size of Aalborg, so internet wise we become the fourth largest “city” in Denmark for a weekend
- 12. Network at NPF
- 13. Distribution Network ● Layer 3 Distribution ● Approx. 1100 Participants pr ring ● OSPF Routing
- 14. Rings
- 15. Network “Services”? ● DHCP ● DNS ● Prometheus Monitoring ● Grafana ● SNMP Polling (Blackbox exporter) ● SNMP Traps (ELK Cluster)
- 18. Kube-router in NPF Key learnings ● Think in terms of ToR (top-of-rack) design ● Often it’s not the new fancy kubernetes network that fails ● Kube-router toolbox is essential ○ You just kubectl exec -it into the kube-router box Why kube-router ● We tried calico and got overwhelmed ● Very low overhead, must of it is done in-kernel ● Minimal code base, easily understood ● We run full kube-router, so no kubenet or kubeproxy on the nodes
- 19. DHCP Server Troubles Modern DHCP server Kea failed, so we had to go with isc-dhcpd ISC DHCPd needs to know the IP of all availability peers on startup - ISC DHCPd can take a hostname rather than the IP - Headless Services expose the pod ip to kube-dns - Kubernetes DNS resolution to the rescue! - Stateful Sets could be a possible non-hacky solution, but not sure ISC DHCPd will play nice with stateful sets.
- 20. So how did it go?
Editor's Notes
- #19: Kubeproxy disadvantages arcane implementation of load balancer, hard to troubleshoot anything with out understanding the how Kube-proxy uses iptables to implement load balancer not a true load balancer but a simple round robin forwarder no load balancing mechanisms iptable performance degrades as number of services increases. More number of services means long list of iptable rules in a chain to match a packet against in the data path, and latency in insert/delete rules in control path