Exploring Magnum and Senlin integration for autoscaling containers
Download as PPTX, PDF1 like728 views
1. The document discusses integrating Magnum and Senlin for autoscaling containers across multiple levels. Magnum provisions and manages container clusters while Senlin provides clustering and autoscaling capabilities. 2. A key goal is to coordinate scaling at both the application and cluster level based on policies. This includes scaling containers within clusters managed by COEs like Kubernetes, as well as adding or removing nodes to clusters. 3. The demo shows autoscaling a Kubernetes cluster managed by Magnum using Senlin profiles, policies, and triggers to scale pods and nodes based on metrics from Ceilometer. This provides an integrated OpenStack solution for multi-level autoscaling of containers.
Ad
More Related Content
What's hot (20)
Similar to Exploring Magnum and Senlin integration for autoscaling containers (20)
![[RightScale Webinar] Architecting Databases in the cloud: How RightScale Doe...](https://cdn.slidesharecdn.com/ss_thumbnails/rightscalewebinar-architectingdatabasesinthecloud-howrsdoesitfinal1-141106125853-conversion-gate01-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Ad
Exploring Magnum and Senlin integration for autoscaling containers
- 1. 1 Exploring Magnum and Senlin integration for autoscaling containers Hongbin Lu (Huawei) Jay Lau, Ton Ngo (IBM) Qiming Teng, Julio Ruano (IBM) Exploring Magnum and Senlin integration
- 2. Contributors Exploring Magnum and Senlin integration2 Zilian Ji, Linjuan Xia, Shijia E, Peiyu Lin, Tianyuan Zhuang, Ji Jiang, Yongkui Wang
- 3. Talk Outline 1. Use cases 2. Magnum 3. Senlin 4. How it works 5. Demo Exploring Magnum and Senlin integration3
- 4. Use cases Exploring Magnum and Senlin integration4
- 5. Use case • Shared COE cluster – Autoscale on some policies (utilization, etc.) – Scale number of nodes • Multiple apps on cluster – Autoscale on some policies (request rate, etc.) – Scale number of containers • Example: – 2, 3-tiered web apps: handle requests – SLA determines policy Exploring Magnum and Senlin integration5
- 6. Scaling scenarios (1) Exploring Magnum and Senlin integration6 node c c c node c cc node c scale app scale cluster Single app on cluster • scale out containers as needed • scale cluster by utilization • coordination = utilization
- 7. Scaling scenarios (2) Exploring Magnum and Senlin integration7 node c node c c node c c c c scale 2 apps Two apps on cluster • scale out containers as needed • scale cluster by utilization • coordination = utilization scale cluster unlimited scale cluster limited
- 8. Scaling scenarios (3) Exploring Magnum and Senlin integration8 node c c node scale 2 apps scale cluster limited c node c c c c c c Two apps on cluster, limited cluster size • blue scales first, retains containers • red scales later, fails • maybe acceptable • what if red is critical and blue is not • inverted priority: blue < red
- 9. Scaling scenarios (4) Exploring Magnum and Senlin integration9 node c c scale 2 apps Limit on apps • Place max on blue • blue scales, red does not scale • cannot use full resources scale cluster limited c nodenode c limited
- 10. Scaling scenarios (5) Exploring Magnum and Senlin integration10 node c c node scale 2 apps scale cluster limited c node c c c c c cc c c c Priority on apps • blue scales first, red scales later • blue scales back to free up resource for red • 2 ways linkage: container cluster host cluster
- 11. Magnum Exploring Magnum and Senlin integration11
- 12. Instance Magnum Overview Exploring Magnum and Senlin integration12 • Provision • Kubernetes • Docker Swarm • Mesos • Scale • Add instances • Remove instances • Manage • Pod • Service • Replication Controller • Container Nova Instance container container Magnum CLI Magnum Heat
- 13. Magnum Architecture Exploring Magnum and Senlin integration13
- 14. Magnum: Autoscale Bay 1. Raw data: Magnum pulls raw data from Bays. 2. Metrics: Metrics, like memory utilization, are computed and sent them to Ceilometer. 3. Alarm: Ceilometer triggers alarms based on user-specified scenario and metrics. 4. Scale: Heat scales the bays. Exploring Magnum and Senlin integration14 Ceilometer Magnum Heat Bay 1) Raw data 2) Metrics 3) Alarm 4) Scale
- 15. Kubernetes: Autoscale Pod 1. Metrics: The autoscaler periodically queries pods’ metrics. 2. Scale: Based on collected metrics, the autoscaler uses a built-in algorithm to trigger the scaling. 3. Resize: Replication Controller resize the pods. Exploring Magnum and Senlin integration15 Horizontal Pod Autoscaler Pods 1) Metrics 3) Resize Replication Controller 2) Scale
- 16. In discussion • Magnum AutoScaler – A new component in Magnum • Scaling Bays • Scaling Applications – Interacts with AutoScaler from COEs • Kubernetes AutoScaler as a driver for container-level scaling • Magnum responsible for scaling RCs without Kubernetes autoscaler – Customizability • data sources • metrics and thresholds • standby bay nodes (i.e. resource pool) Exploring Magnum and Senlin integration16
- 17. Two-Levels of AutoScaling Exploring Magnum and Senlin integration17 Working K8S Bay NodeNode K8S Master Heapster Kublet cAdvisor Kublet cAdvisor Storage Backend RC AutoScaler Pod Magnum AutoScaler Service K8S Driver Collector Analyzer Operator Senlin Magnum Conductor Standby Swarm Driver Mesos Driver
- 18. Senlin Exploring Magnum and Senlin integration18
- 19. Clustering - A Missing Service in OpenStack Exploring Magnum and Senlin integration19 Scalable Load-Balanced Highly-Available Manageable ...... of any (OpenStack) objects
- 20. Senlin Architecture Exploring Magnum and Senlin integration20 Senlin API Senlin Client REST RPC Senlin Engine Senlin Database Policies Profiles
- 21. Clustering/AutoScaling/LoadBalancing Exploring Magnum and Senlin integration21 Nova Kubernetes Heat Ironic BareMetal VMs Stacks Containers Senlin placement deletion scaling health load-balance batching
- 22. Scaling A Cluster using Senlin Exploring Magnum and Senlin integration22 type: os.nova.server version: 1.0 properties: image: fedora-21 flavor: 3 networks: - network: private userdata: ... Profile Cluster type: senlin.policy.scaling version: 1.0 properties: event: CLUSTER_SCALE_IN adjustment: type: CHANGE_IN_CAPACITY number: 1 min_step: 1 best_effort: True attach Policy type: senlin.policy.scaling version: 1.0 properties: event: CLUSTER_SCALE_OUT adjustment: . . . attach Policy webhook RESIZE
- 23. Policies Exploring Magnum and Senlin integration23 type: senlin.policy.scaling version: 1.0 properties: adjustment: min_step: 1 best_effort: true number: 1 type: CHANGE_IN_CAPACITY event: CLUSTER_SCALE_OUT • Declarative • Can be mapped to from a standard • Example: SUR_Cluster (Add k8s Minion / Add Spark worker node)
- 24. Trigger Exploring Magnum and Senlin integration24 type: CeilometerThresholdAlarm version: 1.0 rule: meter_name: memory_util comparison_operator: gt threshold: 70 period: 60 evaluation_periods: 1 statistic: avg query: - field: resource_metadata.cluster op: eq value: SUR_Cluster • Generic abstraction • alarms in OpenStack monitoring services • e.g. Ceilometer Alarm
- 25. How it works Exploring Magnum and Senlin integration25
- 26. Design goals 1. Coordinate scaling at apps and cluster level • Control flows in both direction • Apps -> Cluster: utilization, other methods • Cluster -> Apps: priority, other methods 2. Integrate scaling support from COE if available • Kubernetes autoscaler 3. Policy driven • Easy to express, manage complex cases Exploring Magnum and Senlin integration26
- 27. template Outer Stack Autoscaling: OpenStack-Only Solution (Today) 27 Exploring Magnum and Senlin integration AutoScalingGroup Stack Heat Ceilometer AH PUp PDn AL ASGS1 S2 S3 update ASG
- 28. Senlin profile Autoscaling: OpenStack-Only Solution (New) 28 Exploring Magnum and Senlin integration Cluster Ceilometer AH PUp PDn AL SCALE_IN SCALE_OUT Node1 Node2 Node3 webhookspolicies triggers
- 29. Demo Exploring Magnum and Senlin integration29
- 30. Autoscaling with Kubernetes 30 Exploring Magnum and Senlin integration VMs K8S Bay NodeNode Heapster Kublet cAdvisor Kublet cAdvisor Pod Senlin Magnum profile CeilometerWorkload policy
- 31. Questions Exploring Magnum and Senlin integration31
- 32. Thank You Exploring Magnum and Senlin integration32
Editor's Notes
- #18: Collector will help get some Pod metrics k8s via REST API of Storagebackend. Analyzer will help check if the RC need scaling and k8s bay node need scaling. Operator can interact with both k8s and Magnum Conductor Analyzer logic: Scale up/down based on Periodic task in Analyzer: Check if current standby k8s nodes greater than desired standby nodes, if not, scale up k8s nodes, otherwise, scale down the k8s nodes. NOTE: The scale up and scale down can only be triggered if a specified stabilization reach. For example: We set a stabilization as 3, periodic task interval is 5min, if the periodic task detect need scale up/down nodes in a continually 3 times, then scale will be triggered. (This stabilization should be applied to both scale pods and nodes) Scale up k8s pods without k8s autoscaler and k8s nodes Analyzer will check all of the RCs in k8s, if the RC has a native k8s autoscaler, the Operator will ignore this RC; If the RC do not have an autoscaler, the Operator will help scale up/scale down the RC. If the Pods in RC exceeds the configured threshold, trigger scale up; Get node difference for k8s nodes minus max number of pods in rc without k8s autoscaler (Current standby k8s nodes) If the standby k8s node is less than desired standby k8s nodes, Operator call Magnum Conductor scale up k8s nodes. Scale up k8s nodes (For RC with k8s autoscaler): Get max required nodes for rc with k8s autoscaler: Max number of Pods in RC – current pods Check if current standby k8s nodes greater than number from 1), if not, scale up to the desired standby k8s nodes
- #30: magnum baymodel-create magnum bay-create senlin node-list senlin cluster-list heat stack-list senlin profile-list nova list magnum bay-list magnum bay-show SURbay -> 2 minion senlin policy-list -> scale out, scale in senlin policy-show SUR_Cluster_so_policy -> scale out by 1 senlin cluster-policy-list SUR_Cluster -> scale out, scale in senlin webhook-list -> for scale out ceilometer alarm-list -> memory > 70% senlin node-delete -> delete one minion from the bay senlin cluster-list senlin node-create <minion_profile> -> add back one minion called test_minion senlin node-list -> see new minion added back log into minion node, run docker run -it gcr.io/google_containers/spark-base memory utilization on both minions rises: before = 34% + 71%, after = 67% + 87% ceilometer alarm-list ceilometer alarm-history senlin node-list -> show new node being created kubectl get pods -> another spark-worker pod created