K8s cluster autoscaler
0 likes238 views
The document discusses Kubernetes cluster autoscaler, including how it works, deployment steps, configuration, and limitations. It describes setting up the cluster autoscaler manager, preparing extra nodes, protecting nodes from scale down, and installing the autoscaler using Helm. Some key points are that it can automatically scale a cluster from 0 to 1000 nodes handling 30 pods each, but has restrictions like not supporting regional instance groups and taking 10 minutes to scale nodes down.
![Step 1 Steup CA manager
● Setup IG/ASG ACL for GCE / EC2
● Setup /etc/gce.conf
[global]
project-id = <Your project name>
network-project-id =<Your project name>
network-name = ayla-kube-gcp
subnetwork-name = ayla-app
node-instance-prefix = ayla-minion-pert-extra](https://image.slidesharecdn.com/k8sclusterautoscaler-190711150732/85/K8s-cluster-autoscaler-8-320.jpg)
![--set
"autoscalingGroups[0].name=https://www.googleapis.com/
compute/v1/projects/xxx/zones/us-east1-c/instanceGroup
s/k8s-minion-perf-scaler,autoscalingGroups[0].maxSize=
3,autoscalingGroups[0].minSize=0"](https://image.slidesharecdn.com/k8sclusterautoscaler-190711150732/85/K8s-cluster-autoscaler-16-320.jpg)
![Bug ?
● After draining node pod cann’t remove form k8s metadata
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/scale
_down.go
● How to fix it ?
*/10 * * * * /opt/extra/clean-CA-NotReadyNode.sh >>
/var/log/cleanCANode.log 2>&1
#!/bin/bash
/usr/local/bin/kubectl delete node $(/usr/local/bin/kubectl get nodes | grep
NotReady | grep "[0-9][h|m]" | awk '{print $1}' | xargs)](https://image.slidesharecdn.com/k8sclusterautoscaler-190711150732/85/K8s-cluster-autoscaler-19-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to K8s cluster autoscaler (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
Ad
K8s cluster autoscaler
- 1. K8s Cluster Autoscaler Docs didn’t tell you … Sr. Devops of Ayla - Sunny Hu.
- 3. Official Docs for deployment
- 4. In the real world ● Deploy by Kubespray ● Not set cloud-provider ● Pod with local-storage ● Deploy with node selector
- 5. Restriction & Limit ● Not support regional Instance Group ● Handle up to 1000 nodes running 30 pods each ● Scale time - Scale Up : 30 seconds - Down : 10 mins ● Scale UP - Not scale up/down when pod has node selector ● Scale Down - Bulk delete up to 10 nodes - Pods with restrictive PodDisruptionBudget. - Pods with local storage. * - CPU usage > 20%
- 6. - Kube-system pods that: ○ are not run on the node by default, * ○ don't have PDB or their PDB is too restrictive (since CA 0.6). - Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc). * - Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, matching anti-affinity, etc) - Pods that have the following annotation set: "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
- 7. STEPS
- 8. Step 1 Steup CA manager ● Setup IG/ASG ACL for GCE / EC2 ● Setup /etc/gce.conf [global] project-id = <Your project name> network-project-id =<Your project name> network-name = ayla-kube-gcp subnetwork-name = ayla-app node-instance-prefix = ayla-minion-pert-extra
- 9. Step 2 Prepare Extra Minion ● Create Base Image : - Setup system config - Install kubelet, kubeadm, kubectl ● Create Instance Templete: - Startup script for kubelet config (set Provider ID) - kubeadm join ( Never expire token ) ● Create Instance Group - Multi zone, *ACL - Do not enable auto scaling of instance group
- 10. Provider ID Source code have already told you kubectl describe node Every nodes include master, ingresses, minions need Provider ID.
- 12. GCE gce://<project_id>/<zone>/<hostname> curl http://metadata.google.internal/computeMetadata/v1/project/project-id -H "Metadata-Flavor: Google" curl http://metadata.google.internal/computeMetadata/v1/instance/zone -H "Metadata-Flavor: Google" | cut -d "/" -f4 curl http://metadata.google.internal/computeMetadata/v1/instance/name -H "Metadata-Flavor: Google"
- 14. STEP 3 Protect your node from Scale Down
- 15. Step 4 HELM Install helm install stable/cluster-autoscaler --name gcp-scaler --namespace kube-system --set cloudProvider=gce --set rbac.create=true --set nodeSelector.aylanetwork="camanager" --set regional=true --set skip-nodes-with-local-storage=false --set balance-similar-node-groups=false
- 17. CA Deployment
- 18. Advance Setup ● Control more than 1 Instance Grouop ● Debug level : 1 Less -10 More ● Expendar : ○ random ○ most-pods ○ Least-waste ○ price
- 19. Bug ? ● After draining node pod cann’t remove form k8s metadata https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/scale _down.go ● How to fix it ? */10 * * * * /opt/extra/clean-CA-NotReadyNode.sh >> /var/log/cleanCANode.log 2>&1 #!/bin/bash /usr/local/bin/kubectl delete node $(/usr/local/bin/kubectl get nodes | grep NotReady | grep "[0-9][h|m]" | awk '{print $1}' | xargs)
- 20. All Proerties ● Helm chart https://github.com/helm/charts/tree/master/stable/cluster-auto scaler#configuration ● Cluster Autoscaler container https://github.com/kubernetes/autoscaler/blob/cluster-autosca ler-1.13.5/cluster-autoscaler/FAQ.md#what-are-the-paramete rs-to-ca
- 21. Reference ● Cluster Autoscaler FAQ ● Cluster Autoscaler Helm ● K8s Autoscaler 101 ● Understand K8s Autoscaler ● Official K8s Cluster Autoscaling Document ● Pod Disruption Budget