Proto kubernetes onswitc_hengines_tue100418
1 likeâ˘867 views
The document outlines the offerings of SWITCH, focusing on cloud services for Swiss universities and research institutions, including tailored virtual private clouds and support for academic use cases. It details the implementation of Kubernetes on OpenStack for container management and highlights features such as Keystone authentication and persistent storage management. Additionally, it discusses networking strategies and the use of Ingress for handling external access to services within Kubernetes clusters.
![Š 2018 SWITCH | 12
The playbook returns an admin.conf that looks like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: [..CUT..]
server: https://<ipaddress>:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: [..CUT..]
client-key-data: [..CUT..]
Client config](https://image.slidesharecdn.com/protokubernetesonswitchenginestue100418-180508145428/85/Proto-kubernetes-onswitc_hengines_tue100418-12-320.jpg)
![Š 2018 SWITCH | 19
/etc/kubernetes/cloud-config
[Global]
auth-url = https://keystone.cloud.switch.ch:5000/v2.0
username = saverio.proto@switch.ch
password = ******
tenant-name = saverio.proto@switch.ch
region = ZH
[BlockStorage]
trust-device-path = false
bs-version = v2
[Route]
router-id = 3a6cd142-91cd-4a04-9a86-a73455a0155d
[LoadBalancer]
lb-version = v2
floating-network-id = 3cc83f7d-9119-475b-ba17-f3510c7902e8
subnet-id = 5e18c72c-1902-4846-bd84-ec54cf028375
K8s Master interaction with Openstack](https://image.slidesharecdn.com/protokubernetesonswitchenginestue100418-180508145428/85/Proto-kubernetes-onswitc_hengines_tue100418-19-320.jpg)
Proto kubernetes onswitc_hengines_tue100418
- 1. Š 2018 SWITCH | 1 Saverio Proto [email protected] Lugano Kubernetes on Openstack
- 2. Š 2018 SWITCH | 2 Infrastructure & Data Services Our offer Our customers Your added value
- 3. Š 2018 SWITCH | 3 Your added value SWITCH made â Swiss made ⢠Swiss law and data location ⢠Scalable storage and computing power on demand with immediate availability in accordance to the need of â and controlled by â the institutions ⢠Flexible usage and charging model, no up-front investment ⢠Simple administration; integrated into the academic network of SWITCH; security and identity services included ⢠Support for academic use cases ⢠Created together with you
- 4. Š 2018 SWITCH | 4 Our customers Higher education ⢠Cantonal universities ⢠ETH domain with research institutions ⢠Universities of applied sciences ⢠Universities of teacher education University-related organizations ⢠Spin-Offs ⢠Research institutions ⢠eLearning Center ⢠University hospitals
- 5. Š 2018 SWITCH | 5 Our offer ⢠SWITCHengines ⢠Virtual Private Cloud (VPC) ⢠SCALE-UP (Project)* * For developing academic services with 9 universities, as part of the âScientific Informationâ projects mandated by swissuniversities.
- 6. Š 2018 SWITCH | 6 SWITCHengines Customer tailored computing and storage performance for universities, research and teaching âfurther developed in the SCALE-UP project mandated by swissuniversities. Your benefits ⢠Your data in Switzerland ⢠Integrated network and security ⢠Support for academic use cases ⢠Simple administration and billing ⢠Created together with you Customers ⢠Universities ⢠Research institutions ⢠eLearning Center ⢠University hospitals ⢠Spin-Offs Services ⢠SWITCHengines (IaaS) ⢠Virtual Private Cloud (VPC) ⢠SCALE-UP (academic project)
- 7. Š 2018 SWITCH | 7 Openstack Cloud Engineer at SWITCH. He works in NRENs since 2011, first in Italy and then in Switzerland. He has significant experience in running critical infrastructures using Open Source software. He works together with the Infrastructure & Data team, to deliver an Openstack based cloud to the Swiss Universities. Saverio Proto
- 8. Š 2018 SWITCH | 8 Container orchestration. Open Source system to deploy containerized applications. ⢠What you should already know: ⢠Docker, build docker containers ⢠Run your application in a container ⢠What Kubernentes will do for me ? ⢠Manage the lifecycle of containers ⢠Schedule the containers to the hosts ⢠Attach storage volumes to the containers ⢠Setup networking Kubernetes (K8s): what is it ?
- 9. Š 2018 SWITCH | 9 ⢠K8s can interact with the Openstack API ⢠Use keystone for authentication ⢠Create volumes ⢠Create load balancers ⢠Set routes in Neutron routers Kubernetes on Openstack
- 10. Š 2018 SWITCH | 10 ⢠https://github.com/switch-ch/k8s-on-openstack/ ⢠Forked from https://github.com/infraly/k8s-on-openstack This ansible playbook will create Openstack instances, and will install Kubernetes on them. export KEY=keyname export IMAGE="Ubuntu Xenial 16.04 (SWITCHengines)" export NETWORK=k8s ( read REAME for complete variables list) ansible-playbook site.yaml Deploy K8s on Openstack
- 11. Š 2018 SWITCH | 11 ubuntu@k8s-master:~$ kubectl get nodes NAME STATUS AGE VERSION k8s-1 Ready 3m v1.9.4 k8s-2 Ready 3m v1.9.4 k8s-3 Ready 3m v1.9.4 k8s-master Ready 4m v1.9.4 Start !
- 12. Š 2018 SWITCH | 12 The playbook returns an admin.conf that looks like this: apiVersion: v1 clusters: - cluster: certificate-authority-data: [..CUT..] server: https://<ipaddress>:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: [..CUT..] client-key-data: [..CUT..] Client config
- 13. Š 2018 SWITCH | 13 ⢠kubectl is the command line tool ⢠~/.kube/config is the config file ⢠Supports Openstack Keystone authentication ⢠kubectl is able to read usual Openstack OS_ env vars ⢠kubectl config set-credentials osuser --auth-provider=openstack ⢠source ~/openrc Client config
- 14. Š 2018 SWITCH | 14 Create a context to use the Openstack User kubectl config set-context --cluster=kubernetes --user=osuser osuser@kubernetes Lets use the context kubectl config use-context osuser@kubernetes Client config
- 15. Š 2018 SWITCH | 15 Keystone authentication for k8s Kubectl client Openstack Keystone KubeApi server WebHook 2. Request with bearer token 1. Token issue 3. Calling WebHook for token validation 4. Token validation with keystone
- 16. Š 2018 SWITCH | 16 kube-apiserver --authentication-token-webhook-config-file=webhook.kubeconfig.yaml --- apiVersion: v1 clusters: - cluster: insecure-skip-tls-verify: true server: https://localhost:8443/webhook name: webhook contexts: - context: cluster: webhook user: webhook name: webhook current-context: webhook kind: Config preferences: {} users: - name: webhook Configure the webhook
- 17. Š 2018 SWITCH | 17 It is just a container: https://hub.docker.com/r/zioproto/k8s-keystone-auth/ More recent source code can be found here: https://github.com/kubernetes/cloud-provider-openstack make image-k8s-keystone-auth Start the webhook container
- 18. Š 2018 SWITCH | 18 ⢠Keystone tested for Authentication ⢠Default RBAC for Kubernetes implements Authorization ⢠Keystone projects are mapped to Kubernetes Groups Authorization via RBAC
- 19. Š 2018 SWITCH | 19 /etc/kubernetes/cloud-config [Global] auth-url = https://keystone.cloud.switch.ch:5000/v2.0 username = [email protected] password = ****** tenant-name = [email protected] region = ZH [BlockStorage] trust-device-path = false bs-version = v2 [Route] router-id = 3a6cd142-91cd-4a04-9a86-a73455a0155d [LoadBalancer] lb-version = v2 floating-network-id = 3cc83f7d-9119-475b-ba17-f3510c7902e8 subnet-id = 5e18c72c-1902-4846-bd84-ec54cf028375 K8s Master interaction with Openstack
- 20. Š 2018 SWITCH | 20 /etc/kubernetes/manifests/kube-controller-manager.yaml spec: containers: - command: - kube-controller-manager - --cloud-provider=openstack - --cloud-config=/etc/kubernetes/cloud-config Interaction with Openstack
- 21. Š 2018 SWITCH | 21 I changed my the way of doing Networking after the Openstack summit in Sydney Use --network-plugin=kubenet ⢠https://github.com/zioproto/k8s-on- openstack/commit/f4506ed202ecc6fc4ff5ac603fd28f3664c b2871 ⢠https://www.openstack.org/videos/sydney- 2017/kubernetes-on-openstack-the-technical-details Kubernetes Neutron Networking
- 22. Š 2018 SWITCH | 22 Kubernetes Neutron Networking neutron router VM aka k8s node VM aka k8s node Default gateway is the neutron router Has static routes to reach Pod Networks Default gateway is the neutron router Pod Pod Pod network assigned by the k8s master Pod Pod Pod network assigned by the k8s master
- 23. Š 2018 SWITCH | 23 ⢠Docker containers run into Openstack instances ⢠Persistent storage is cinder volumes ⢠K8s will attach the Cinder Volume to the right Openstack instance were the docker container is scheduled volumes: - name: mysql-persistent-storage cinder: volumeID: <uuid> fsType: ext4 Persistent Storage
- 24. Š 2018 SWITCH | 24 ⢠This method is not practical ⢠we have to manage Cinder volumes on the side ⢠Export UUID ⢠Solution ⢠Define a Storage Class Persistent storage
- 25. Š 2018 SWITCH | 25 kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: default annotations: storageclass.beta.kubernetes.io/is- default-class: "true" provisioner: kubernetes.io/cinder parameters: type: fast availability: nova Storage Class
- 26. Š 2018 SWITCH | 26 kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mysql-pvc annotations: volume.beta.kubernetes.io/storage-class: default spec: accessModes: - ReadWriteOnce resources: requests: storage: 30Gi Persistent Storage pvc
- 27. Š 2018 SWITCH | 27 volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc Persistent Storage with pvc
- 28. Š 2018 SWITCH | 28 I suggest not to use it, at least in Openstack Newton ⢠Both DNAT and SNAT are necessary ⢠you lose the client IP information ⢠IPv6 support status is unknown to me ⢠It is a pure L4 loadbalancer, cannot terminate TLS connections or rewrite HTTP headers Openstack LBaaSv2 with K8s
- 29. Š 2018 SWITCH | 29 Openstack LBaaSv2 neutron router VM aka k8s node VM aka k8s node Default gateway is the neutron router Default gateway is the neutron router Pod Pod Pod network assigned by the k8s master Pod Pod Pod network assigned by the k8s master neutron LBaaS Svc FloatingIP configured here Openstack project network Both DNAT and SNAT are necessary here.
- 30. Š 2018 SWITCH | 30 ⢠An API object that manages external access to the services in a cluster, typically HTTP. ⢠Ingress can provide load balancing, TLS termination and name-based virtual hosting. ⢠https://kubernetes.io/docs/concepts/services- networking/ingress/ ⢠At least two known implementations ⢠Nginx based ⢠HAProxy Kubernetes NGINX Ingress
- 31. Š 2018 SWITCH | 31 Kubernetes NGINX Ingress VM aka k8s node VM aka k8s node Pod Pod Pod network assigned by the k8s master Pod Pod Pod network assigned by the k8s master VM k8s master Kubernetes-nginx-ingress Default network namespace (docker ânet=host) ⢠It is a special privileged Docker container running NGINX ⢠On the k8s master (s) or dedicated VM ⢠Has access to external IP addresses and cluster IP addresses Kubernetes cluster network
- 32. Š 2018 SWITCH | 32 How I use the Ingress ?
- 33. Š 2018 SWITCH | 33 Slack channel #sig-openstack ⢠Join via http://slack.k8s.io/ Follow our cloud blog: ⢠https://cloudblog.switch.ch/ Get in touch with me ⢠[email protected] Getting involved
- 34. Š 2018 SWITCH | 34 Thank you ! Questions ?