Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter)
Download as PPTX, PDF1 like895 views
- CORD aims to virtualize telecom central offices using open source software and commodity hardware. Kubernetes could help integrate NFV apps but challenges remain. - Issues include converting existing VM-based NFVs to containers, supporting both OpenStack and Kubernetes, and ensuring the SDN controller ONOS can communicate with Kubernetes network components. - The presenter's team addressed these by designing a multi-interface CNI plugin and centralized IPAM using Etcd to integrate ONOS and provide pod networking. Further work is needed to fully integrate ONOS control and test the solution.
![[20200720]cloud native develoment - Nelson Lin](https://cdn.slidesharecdn.com/ss_thumbnails/20200720cloudnativedeveloment-200824031228-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter) (20)
Ad
More from inwin stack (20)
Ad
Recently uploaded (20)
Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter)
- 1. KUBERNETES INTEGRATE WITH CORD HUNG-WEI CHIU LINKER NETWORKS.INC
- 2. WHO AM I • Hung-Wei Chiu (邱宏瑋) • [email protected] • hwchiu.com • Experience • Software Engineer at Linker Netowrks • Software Engineer at Synology (2014~2017) • Co-Found of SDNDS-TW • Open Source experience • SDN related projects (mininet, ONOS, Floodlight, awesome-sdn)
- 3. OUTLINE • What is CORD • Challenge of kubernetes with CORD. • What have we done now • Next steps
- 4. What is CORD ?
- 5. CENTRAL OFFICE RE-ARCHITECTED AS A DATACENTER 5 SDN + NFV + Cloud Open Source Software Commodity Hardware (Servers, White-Box Switches, I/O Blades) Large number of COs Evolved over 40-50 years 300+ Types of equipment Huge source of CAPEX/OPEX
- 6. DATA CENTER LEAF-SPINE FABRIC 6 Reliability 😃 Flexibility 😃 Bandwidth 😃 Spine Leaf
- 7. Metro Router White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box Open Source SDN-based Bare-metal White Box White Box R,E,M- Access 7 ONOS Controller Cluster vRouter Control XOS (Orchestrator) vSG VNF VNF VNFVNFvSG VNF VNF VNFVNF vSG VNF VNF VNF VNF OVS OVS OVS OVS OVS Residential Mobile Enterprise Underlay Control Underlay Overlay Control Overlay vOLT Control Final CORD Architecture
- 8. Metro Router White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box Open Source SDN-based Bare-metal White Box White Box R,E,M- Access 8 ONOS Controller Cluster vRouter Control XOS (Orchestrator) vSG VNF VNF VNFVNFvSG VNF VNF VNFVNF vSG VNF VNF VNF VNF OVS OVS OVS OVS OVS Residential Mobile Enterprise Underlay Control Underlay Overlay Control Overlay vOLT Control Final CORD Architecture
- 9. Metro Router White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box Open Source SDN-based Bare-metal White Box White Box R,E,M- Access 9 ONOS Controller Cluster vRouter Control XOS (Orchestrator) vSG VNF VNF VNFVNFvSG VNF VNF VNFVNF vSG VNF VNF VNF VNF OVS OVS OVS OVS OVS Residential Mobile Enterprise Underlay Control Underlay Overlay Control Overlay vOLT Control Final CORD Architecture
- 10. SUMMARY • VM-based NFV (Network Function Virtualization) • Use ONOS (SDN controller) + Open VSwitch to control packets. • Use XOS (Service Orchestration) to control all services (VNF) • ONOS/XOS need to communicate with OpenStack component.
- 11. Why kubernetes CORD ?
- 12. WHO? • Ciena • Hate OpenStack • Intel China • Some Telecom service providers • China unicom
- 13. CHANGE TO KUBERNETES • VM-based NFV (Network Function Virtualization) • vSG, vPGW, vSGW, etc • Who owns the NFV? • Vendors • We can’t force them to convert all NFVs to container. • It’s impossible to have a kubernetes solution for CORD now.
- 14. CHANGE TO KUBERNETES • How about hybrid architecture? • Support OpenStack + kubernetes
- 16. CHANGE TO KUBERNETES • Use ONOS (SDN controller) + Open vSwitch to control packets. • There’re so many CNI for kubernetes now. • None of them are Open vSwitch based solution. • Linen-CNI is Open vSwitch + Linux Bridge solution • Same subnet traffics between POD are controlled by Bridge NODE POD POD OVS br0 ens0p3
- 17. CHENLAGE TO KUBERNETES • Use ONOS (SDN controller) + Open vSwitch to control packets. • Create our own CNI to support pure Open vSwitch version. • Still some problem need to conquer future. NODE POD POD OVS ens0p3 ？
- 18. CHANGE TO KUBERNETES NODE POD POD Bridge ens0p3 many iptable rules. many iptable rules.
- 19. CHANGE TO KUBERNETES • Use XOS (Service Orchestration) to control all services (VNF) • Kubernetes can handle most thing. • XOS should communicate with Kubernetes via its API service.
- 20. CHANGE TO KUBERNETES • ONOS/XOS need to communicate with OpenStack component. • ONOS need to know the IP information of each host (VMs) from neutron component. • In kubernetes, we need to provide the IP information of each POD. • Since we implement our CNI, we can send the IP information after CNI assigns IP to POD. • Send the information via Restful API/gRPC
- 21. PROBLEMS WE MET • Deploy ONOS controller as container • Multiple network interface for POD • Centralized IP management
- 22. DEPLOY PROBLEM • We need to deploy ONOS as container • The chicken-and-egg conundrum! • Hard to solve, we need to work-around now. • We decide to move the ONOS out of POD. • For each node, it should have multiple network interface including data network and control network. • Out of band.
- 23. DEPLOY PROBLEM NODE POD POD OVS ens0p3 ens0p4 NODE POD POD OVS ens0p3 ens0p4 NODE POD POD OVS ens0p3 ens0p4 Data network Control network
- 24. DEPLOY PROBLEM NODE POD POD OVS ens0p3 ens0p4 NODE POD POD OVS ens0p3 ens0p4 NODE POD POD OVS ens0p3 ens0p4 Data network Control network
- 25. MULTIPLE NETWORK INTERFACE • For some NFV,(vSG) they need to multiple interface in its POD. • For example. NODE POD POD OVS ens0p3
- 26. MULTIPLE NETWORK INTERFACE • We found a open source project (multus-CNI) • Provides the multi interface support in a pod • We don’t figure out how to use it first. • We thought it’s a global setting. • We try to implement by ourself.
- 27. MULTIPLE NETWORK INTERFACE • Multiple network interface means call CNI multiple times. • For CNI, we need to know the namespace location for each POD. • We want to provide a interface to dynamically call CNI for any existing POD. • Input • Pod Name, Network configuration (CNI name, configuration) • Output • Success (add another interface for existing POD) • Error
- 28. MULTIPLE NETWORK INTERFACE • For each CNI, it should know the namespace location of each POD. • We also need to know the POD name • In the CNI plugin, we can get above information via • Args.Args (many information, separate by semi-colon) • Args.Netns • We store those information in ETCD.
- 29. MULTIPLE NETWORK INTERFACE • After I have finished almost tasks. • One intel guy tell us multus-cni support pod configuration. • …..….. • Ok, we use multus-cni.
- 30. CENTRALIZED IP MANAGEMENT • Our CNI plugin use IPAM to handle the IP management. • Official IPAM support two types. • Host-local • DHCP
- 31. IPAM DHCP • Requirement • You should run a IPAM DHCP daemon on each node. • You should setup a DHCP server on your network. • How it works. • Start a DHCP client when a POD is be created. • Forward the DHCP packet to DHCP server(it depends on your CNI forward L2 broadcast) • Official recommend to use MacVLan as CNI. • Limitation • All node in same subnet. • Simple configuration. You only specify “type=dhcp” in CNI configuration.
- 32. IPAM HOST-LOCAL • Requirement • None • How it works • Use a local file to record how many IP address has been used. • Lookup the file and choose an available IP address for CNI. • Limitation. • You should prepare a configuration for each Node with different setting. • Complex configuration. You need to specify which subnet the Node will use. • Make sure no duplicate for each Node.
- 33. IPAM • What we want ? • Simple configuration • Support multiple subnet • We decided to create new IPAM
- 34. DHCP • Refer to trellis (CORD network infrastructure) • Requirement • DHCP server • How it works • We set the Gateway Address for each dhcp request to support multiple subnet via L3 unicast. • It looks like
- 35. DHCP NODE 1 POD POD OVS ens0p3 POD POD OVS ens0p3 NODE 2 network Master DHCP Server DHCP relay DHCP relay192.168.1.1 192.168.2.1 Subnet 192.168.1.0/24 { } Subnet 192.168.2.0/24 { }
- 36. DHCP – PROBLEM • The Problem is.. • How to decide the IP address of each Open vSwitch? • Gateway address of each subnet. • Use ETCD • Maybe we can use ETCD to replace the DHCP
- 37. ETCD • Implement a new IPAM which use etcd to record the subnet of each node. • Simple configuration • Network: 10.12.0.0/16 • Subnet length: 24 • etcd address. • The subnet will from 10.12.1.0/24 to 10.12.255.0/24 • Simple and easy to implement.
- 38. NEXT STEP • Integrate the ONOS (SDN controller) with our CNI. • Make sure the ONOS can control the network. • For the kubernetes internal communication, the ONOS should implement all of them by Openflow, not iptables.NODE POD POD OVS ens0p3 Use openflow rules many iptable rules.
- 40. Q&A