网易云K8S应用实践 | practices for kubernetes cluster provisioning, management and application deployment
Download as PPTX, PDF5 likes645 views
This document summarizes best practices for managing Kubernetes clusters and deploying containerized applications on Netease Cloud. It discusses implementing container networking in VPC, using cloud disks for storage, automating cluster creation, managing nodes, exposing services via ingress, extending ingress controllers, developing deployment pipelines with Helm, and managing application environments with operators. Overall principles and resources are also provided.
![认证与授权
• kubernetes 认证和授权
– Service account
– 客户端证书
– OpenID
– RBAC
• nginx-ingress-controller + OpenID Connect
– lua-resty-openidc
– id_token 刷新
kind: Ingress
metadata:
name: kubernetes-dashboard
annotations:
ext.ingress.kubernetes.io/oidc-access: >
{
"id_token_refresh": true,
"enc_id_token" : true,
"claim_headers": { "Authorization": "bearer_enc_id_token" },
"no_auth_locations": ["/api"],
"pass_locations": ["/assets","/static","/favicon.ico"]
}
spec:
rules:
- host: k8s.example.com
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 80
cloudndp/kube-oidc-ingress-controller](https://image.slidesharecdn.com/2018-190501000222/85/K8S-practices-for-kubernetes-cluster-provisioning-management-and-application-deployment-11-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to 网易云K8S应用实践 | practices for kubernetes cluster provisioning, management and application deployment (20)
![[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus](https://cdn.slidesharecdn.com/ss_thumbnails/e61520monitoringopeninfrawithprometheusv1-180704062709-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Ad
网易云K8S应用实践 | practices for kubernetes cluster provisioning, management and application deployment
- 3. 物理机 vs 云主机
- 4. 选择容器网络 • 网络类型 – overlay，underlay, L3, L2, … • 插件方案 – calico, contiv, flannel, openshift sdn, … • 实现技术 – hostnet, routing, vxlan, macvlan, ipvlan, gre, bgp, … • 问题和限制 – 拓扑, 隔离, 性能, MTU, … • … • WHY？ Kubernetes network 1. All Pods should get their own IP Address, and the container should be able to see itself as the same IP that others see it 2. All containers can communicate with each other without NAT 3. All nodes can communicate with all containers (and vice-versa) without NAT
- 5. 在VPC中实现容器网络 node 1 pod/container eth0 10.177.5.1 docker0 10.176.100.0/26 veth1 10.176.100.1 pod/container veth2 10.176.100.2 node 2 pod/container eth0 10.177.5.2 docker0 10.176.100.64/26 veth1 10.176.100.65 pod/container veth2 10.176.100.66 node 3 pod/container eth0 10.177.5.3 docker0 10.176.100.128/26 veth1 10.176.100.129 pod/container veth2 10.176.100.130 VPC Network # cat /etc/docker/daemon.json { "bip": "10.176.100.0/26", "ip-masq": false, … } # iptables -P FORWARD ACCEPT kubelet --network-plugin=kubenet --pod-cidr=10.176.100.0/26 --non-masquerade-cidr=10.176.0.0/15 cbr0 cbr0 cbr0 Open API
- 6. 容器存储 • Kubernetes的容器存储 – emptyDir – hostPath – ConfigMap & Secret – PersistentVolume/PVClaim • 支持云硬盘 master node 1 node 2 pod pvc(bound) kubelet ctrl mgr Open API 1. create/find 2. attach/detach 3. mkfs 4. mount/umount kubelet watch disk1 可用区B SSD云盘 100GB disk2 可用区B SSD云盘 100GB disk3 可用区B SSD云盘 100GB disk4 可用区B SSD云盘 100GB cloudndp/flexvolume-npc-disk npc-disk driver
- 7. 启动集群 • 配置集群 – 内网DNS发现：apiserver，etcd – 集群服务：kubelet 清单 vs 集群 addons – 使用蜂巢镜像仓库 • 自动化 – ansible – 云计算 Open API cloudndp/ansible-role-npc_setupcloudndp/npc-shell https://............/ndp/ndp-k8s-launcher
- 9. 管理节点 kind: NpcInstance metadata: name: builder-node spec: instance_count: 3 pod_cidr_prefix: 26 instance_type: { cpu: 8, memory: '16G' } cloud_volume: capacity: 100G cloud_volume_binds: - /var/lib/builder-cache node_labels: node-role.kubernetes.io/builder: '' ingress_enabled: no nagent_enabled: yes cmdb_clusters: - ndp-k8s/ndp-k8s-nodes@ndp-k8s ... k8s-node-operator $ kubectl patch NpcInstance builder-node -p '{"spec":{"instance_count":5}}' NpcInstance builder-node3 kubelet builder-node2 kubelet builder-node1 kubelet Open API builder-node1 builder-node2 builder-node3 watch 云服务器 初始化(ssh) 节点管理控制器 builder-node5 kubelet builder-node4 kubelet
- 10. 发布集群中的服务 • NLB + nginx-ingress-controller • External Services – lua-resty-upstream-healthcheck nodePort 30080 ingress-nginx ingress service kind: Service metadata: name: external-svc annotations: ext.ingress.kubernetes.io/check-http-uri: /status ext.ingress.kubernetes.io/check-http-extras: > { "interval": 2000, "timeout": 1000, "fall": 3, "rise": 2 } spec: ports: - port: 80 cloudndp/kube-oidc-ingress-controller
- 11. 认证与授权 • kubernetes 认证和授权 – Service account – 客户端证书 – OpenID – RBAC • nginx-ingress-controller + OpenID Connect – lua-resty-openidc – id_token 刷新 kind: Ingress metadata: name: kubernetes-dashboard annotations: ext.ingress.kubernetes.io/oidc-access: > { "id_token_refresh": true, "enc_id_token" : true, "claim_headers": { "Authorization": "bearer_enc_id_token" }, "no_auth_locations": ["/api"], "pass_locations": ["/assets","/static","/favicon.ico"] } spec: rules: - host: k8s.example.com http: paths: - path: / backend: serviceName: kubernetes-dashboard servicePort: 80 cloudndp/kube-oidc-ingress-controller
- 13. $ mvn versions:set -DnewVersion=v1.2.0 && mvn package $ docker build –t …/example-app:v1.2.0 && docker push …/example-app:v1.2.0 $ helm package --version=1.2.0 --app-version=v1.2.0 && npc nos PUT /charts/example-app-1.2.0.tgz $ kubectl annotate ndpapps example-app-beta ndpapps/chart='example-app-1.2.0.tgz' 应用交付实践：部署流水线
- 14. 使用Helm example-app/ ├── Chart.yaml ├── charts/ ├── templates/ │ ├── _helpers.tpl │ ├── deployment.yaml │ ├── service.yaml │ ├── ingress.yaml │ └── NOTES.txt └── values.yaml appVersion=v1.2.0 Chart image: repository: hub.c.163.com/cloudndp/example-app service: port: 80 ingress: annotations: ext.ingress.kubernetes.io/oidc-access-extras: > { … } hosts: - ndp.example.org Values NAME: example-app-beta LAST DEPLOYED: Tue Dec 25 19:41:56 2018 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-app-beta ClusterIP 10.0.0.23 <none> 80/TCP 0s ==> v1beta2/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-app-beta 1 0 0 0 0s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE example-app-beta-6895d7487-cb9kx 0/1 ContainerCreating 0 0s NOTES: ... Release $ helm upgrade –install example-app-beta ... spec: containers: - name: example image: "{{ .Values.image.repository }}:{{ default .Chart.AppVersion .Values.image.tag }}" ...
- 15. Helm部署控制器 kind: NdpApp metadata: name: example-app-beta annotations: ndpapps/chart=example-app-1.2.0 spec: image: repository: … ndpapps-operator 1. watch 3. Install, upgrade uninstall 2. fetch $ kubectl create -f ndpapps/example-feature-wip.yaml kind: NdpApp … cloudndp/helm-app-operator
- 16. 小结 • 在网易云创建K8S集群 – 通过VPC实现容器网络 – 使用云盘作为容器存储 – 自动化集群创建 • 管理K8S集群 – 自动化节点管理 – 通过 ingress 发布服务 – 扩展 nginx-ingress-controller • 开发和部署容器应用 – 部署流水线 – 使用Helm组织应用 – 管理应用环境
- 17. 一般性原则…
- 19. 主办方： 谢谢观看